Automated Identification of Graphical UI Elements for Robotic Quality Assurance

This thesis explores the issue of graphical user interface (GUI) screen analysis using convolutional neural networks (CNN) and computer vision methods. The thesis aims to create a system which automatically identifies GUI elements based on pictogram and text information for detected components in an input image. A combination of EfficientNetB1 CNN, OCR, and traditional computer vision methods was used to develop the system. A custom dataset which contains 120k pictograms was used to train the CNN. A UI element semantic dictionary was created, which further utilises the text detected by OCR. Finally, a GUI hierarchy analysis subsystem was created to detect and semantically categorise sections in GUI. The resulting system automatically classifies detected pictograms, suggests additional text classes, and separates the GUI screen into hierarchical sections. The system achieves 81.1% UI element identification accuracy and, on average, analyses a single screen in 0.6 seconds. This system automates repetitive processes, thus decreasing needed person-hours. In the future, the system can be further developed to function as a foundation for automated exploratory testing.

UI element identification, object identification, EfficientNetB1, neural networks, image classification, semantic analysis of graphical user interface, computer vision, machine learning, semantic dictionary of UI elements, OCR, Python, Keras, TensorFlow, OpenCV