We provide
Solution
for your UI designs

Get Started Our Tool

About UIED

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation.
We provide a UIED, a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection.

We provide support on

  • GUI Element Detection

    3 Computer Vision & 3 Deep Learning approaches

  • Interactive Dashborad

    Editable dashbord to adjust the detection result, including position and tag

  • Software Engineering Output

    Output reusable precise detection result for further application (Testing, UI2CODE)

Our Detection Approach

In addition to several classic detection techniques, we proposed our own top-down divide-and-conquer approach to detect GUI text and GUI elements separately

  • Text Detection

    We use a state-of-the-art scene text detector EAST to detect the text element on GUI.

  • Graphic Element Detection

    We adopt a top-down strategy that first segments layout block and then locates atomic elements.

    • Block Detection - Segment the UI by layout blocks

    • Component Extraction Locate components from binary map of each block

    • Element Classification Classify categories of extracted element

  • Merge

    The approach incorporates results from the two branches to produce the final result.

Demonstration

UIED detects GUI elements from the input GUI image and provides a editable dashboard to adjust the detection.
It produces an easy to use json file for any further developement.

Home Page

  • 1

    Go to Tool page to use UIED

  • 2

    Select given example or upload your own GUI image

    UIED is able to handle various GUI, including mobile app, web page and even hand-drawn sketch

  • 3

    Select given example or upload your own GUI image

    We provide totally 6 approaches to perform element detection, but due to hardware limitation, only 3 of them are available now

    For our own approach, the best set of key parameters may vary with GUI size and type, feel free to adjust them if you are unsatisfied of the detection

Dashboard

  • 4

    Go to Tool page to use UIED

    We provide user a M A G I C Dashboard for editing the detection result

    You can Drag & Drop UI elements to adjust their position

    You can Change the size and tag of UI elements

    You can Delete any unwanted UI elements

    All the elements are Collected and Categorized

    You can Add more collected elements from the storage

    You can apply More Approach to detect UI elements

    All the new detected elements are also collected

    You can Compound the result to get the best one

  • 5

    Export the result as reusable file for further application

    The adjusted precise element detection is useful for various application and can be easily expanded

Tool

GUI Element Detection

Convert Your GUI Design Image Or Screenshot To High-Precision Element

Quick Start

Using examples to see how the tool works

Upload Custom UI

Try the tool with your upload UI image

Select a processing method

Web Code Generation

Generate Web Code (HTML + CSS) from the Input GUI Image

Quick Start

Using examples to see how the tool works

Test Version

Meet The Team

Developers

Mulong Xie

Frontend, Backend and Detection Algorithm

Sidong Feng

Frontend

Supervisors

Jieshan Chen

Ph.D

Zhenchang Xing

Associate Professor

Chunyang Chen

Lecturer

Our Collaborators

Contact Us

Office N223, Building 108
Research School of Computer Science
Australian National University
Canberra ACT 2600 Australia

mulong.xie@anu.edu.au

+61 0450674929

Your message has been sent. Thank you!