Thesis Details

Automated Identification of Graphical UI Elements for Robotic Quality Assurance

Master's Thesis Student: Válek Lukáš Academic Year: 2021/2022 Supervisor: Španěl Michal, Ing., Ph.D.
Czech title
Identifikace elementů grafického uživatelského rozhraní pro robotický testovací systém
Language
English
Abstract

This thesis explores the issue of graphical user interface (GUI) screen analysis using convolutional neural networks (CNN) and computer vision methods. The thesis aims to create a system which automatically identifies GUI elements based on pictogram and text information for detected components in an input image. A combination of EfficientNetB1 CNN, OCR, and traditional computer vision methods was used to develop the system. A custom dataset which contains 120k pictograms was used to train the CNN. A UI element semantic dictionary was created, which further utilises the text detected by OCR. Finally, a GUI hierarchy analysis subsystem was created to detect and semantically categorise sections in GUI. The resulting system automatically classifies detected pictograms, suggests additional text classes, and separates the GUI screen into hierarchical sections. The system achieves 81.1% UI element identification accuracy and, on average, analyses a single screen in 0.6 seconds. This system automates repetitive processes, thus decreasing needed person-hours. In the future, the system can be further developed to function as a foundation for automated exploratory testing.

Keywords

UI element identification, object identification, EfficientNetB1, neural networks, image classification, semantic analysis of graphical user interface, computer vision, machine learning, semantic dictionary of UI elements, OCR, Python, Keras, TensorFlow, OpenCV

Department
Degree Programme
Information Technology and Artificial Intelligence, Specialization Computer Vision
Files
Status
defended, grade C
Date
20 June 2022
Reviewer
Committee
Zemčík Pavel, prof. Dr. Ing. (DCGM FIT BUT), předseda
Beran Vítězslav, doc. Ing., Ph.D. (DCGM FIT BUT), člen
Čadík Martin, doc. Ing., Ph.D. (DCGM FIT BUT), člen
Juránek Roman, Ing., Ph.D. (DCGM FIT BUT), člen
Křivka Zbyněk, Ing., Ph.D. (DIFS FIT BUT), člen
Milet Tomáš, Ing., Ph.D. (DCGM FIT BUT), člen
Citation
VÁLEK, Lukáš. Automated Identification of Graphical UI Elements for Robotic Quality Assurance. Brno, 2022. Master's Thesis. Brno University of Technology, Faculty of Information Technology. 2022-06-20. Supervised by Španěl Michal. Available from: https://www-dev.fit.vutbr.cz/study/thesis/25178/
BibTeX
@mastersthesis{FITMT25178,
    author = "Luk\'{a}\v{s} V\'{a}lek",
    type = "Master's thesis",
    title = "Automated Identification of Graphical UI Elements for Robotic Quality Assurance",
    school = "Brno University of Technology, Faculty of Information Technology",
    year = 2022,
    location = "Brno, CZ",
    language = "english",
    url = "https://www.fit.vut.cz/study/thesis/25178/"
}
Back to top