Graphical User Interface auto-completion with element constraints

Abstract

Graphical User Interfaces (GUIs) are often the main touchpoint between a digital product and its users. Consistency is a key usability factor of GUIs that dictates reusing layout patterns throughout the designs of a single system. While visual styles can be easily specified across designs, spatial patterns of related elements often emerge in the creation process. Although these patterns are encoded in existing layouts, a large number is dicult to consider manually. This thesis investigates methods for a design assistance tool that suggests completions for a partial layout. Instead of generating arbitrary completions, we allow the designer to control the next element type and dimensions. The goal of the methods is then to predict the placement of this new element according to the layout patterns of reference designs. Two recently proposed methods for layout completion were evaluated, a Graph Neural Network (GNN) and a Transformer model, as well as a novel approach that leverages a sequence alignment algorithm and a nearest neighbor search (kNN). The methods were tested on handcrafted data sets with explicit layout patterns, as well as larger sets of diverse mobile layouts that lack consistent patterns. The implementation of the GNN mostly fails to predict high-quality results. The transformer model captures general layout structures and works reasonably well for spatial compositions that are close to the training data. The kNN approach achieves the best scores overall. The results suggest that leveraging data sets explicitly via instance-based learning algorithms can outperform neural network approaches for layout design problems. As such, this thesis contributes to establishing smarter design tools for professional designers that increase consistency in the design process.