Recently, transforming graphical user interface (GUI) mockups into code becomes a common challenging practice for current software developers. However, this transformation usually takes time especially when GUI changes keep pace with evolutionary features. There are many studies admitted this challenge and presented solutions in terms of computer-based GUI mockups. However, there is a research gap in this kind of research as very few of them adopted hand-drawn mockups as an input. In this study, the authors employed YOLOv5 is a fast and accurate deep learning framework to automate the process of converting hand-drawn GUI mockups into Android-based GUI prototype. The process starts with detecting all GUI mockups in an input image and determining their bounding boxes, classifying these mockups into their corresponding GUI objects, then finally aligning these objects together to form the output prototype based on the layout presented in the input image. Experimental results show the effectiveness of the proposed approach in generating a visually appealing Android GUI from hand-drawn mockups with a recognition accuracy of 98.54% when tested on various hand-drawn GUI structures designed by five developers.

References

1. 1)
  - 5. The digital design toolkit. Available at https://www.sketchapp.com/.
2. 2)
  - 38. FluidUI.com – Create Web and Mobile Prototypes in Minutes. Available at https://www.fluidui.com/.
3. 3)
  - 33. Seifert, J., Pfleging, B., Valderrama Bahamóndez, E.d.C., et al: ‘Mobidev: a tool for creating apps on Mobile phones’. Proc. of the 13th Int. Conf. on Human Computer Interaction with Mobile Devices and Services (MobileHCI ’11), New York, NY, USA, 2011, pp. 109–112.
4. 4)
  - 47. Free Mobile App Prototyping Tool for iPhone and Android. Available at https://www.appypie.com/app-prototype-builder.
5. 5)
  - 35. Lasecki, W.S., Kim, J., Rafter, N., et al: ‘Apparition: crowdsourced user interfaces that Come to life as you sketch them’. Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI ’15), New York, NY, USA, 2015, pp. 1925–1934.
6. 6)
  - 20. Robinson, A.J.: ‘Sketch2code: generating a website from a paper mockup’. ArXiv, abs/1905.13750, 2019.
7. 7)
  - 46. Mobile Prototyping. Available at https://www.irise.com/mobile-prototyping.
8. 8)
  - 1. Moran, K.P., Bernal-Cardenas, C., Curcio, M., et al: ‘Machine learning-based prototyping of graphical user interfaces for mobile apps’, IEEE Trans. Softw. Eng., 2020, 46, (2), pp. 196–221, doi: 10.1109/TSE.2018.2844788.
9. 9)
  - 7. Myers, B.: ‘Challenges of HCI design and implementation’, Interactions, 1994, 1, (1), pp. 73–83.
10. 10)
  - 10. Moran, K., Li, B., Bernal-Cárdenas, C., et al: ‘Automated reporting of GUI design violations for mobile apps’. Proc. of the 40th Int. Conf. on Software Engineering (ICSE ’18), Gothenburg, Sweden, 2018, pp. 165–175,.
11. 11)
  - 32. Chatty, S., Sire, S., Vinot, J.-L., et al: ‘Revisiting visual interface programming: creating GUI tools for designers and programmers’. Proc. of the 17th annual ACM Symp. on User interface software and technology (UIST ’04), Santa Fe, NM, USA, 2004, p. 267.
12. 12)
  - 13. Box bounding tool. Available at https://github.com/puzzledqs/BBox-Label-Tool.
13. 13)
  - 9. Lelli, V., Blouin, A., Baudry, B.: ‘Classifying and qualifying GUI defects’. 2015 IEEE 8th Int. Conf. on Software Testing, Verification and Validation (ICST), Graz, Austria, April 2015, pp. 1–10.
14. 14)
  - 49. Girshick, R.B.: ‘Fast r-cnn’. 2015 IEEE Int. Conf. on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 1440–1448.
15. 15)
  - 50. YOLOv5. Available at https://github.com/ultralytics/yolov5.
16. 16)
  - 2. Google Play. Available at https://play.google.com/store?hl=en.
17. 17)
  - 37. Proto.io - Prototypes that feel real. Available at https://proto.io.
18. 18)
  - 41. Xiffe | Mobile app prototyping in a jiffy. Available at https://xiffe.com.
19. 19)
  - 48. Supernova Studio | The World's First Design to Code Platform. Available at https://supernova.io/.
20. 20)
  - 29. Samir, H., Kamel, A.: ‘Automated reverse engineering of Java graphical user interfaces for web migration’, (:unav), 2007.
21. 21)
  - 44. Free prototyping tool for web & mobile apps – Justinmind. Available at https://www.justinmind.com/.
22. 22)
  - 23. Balog, M., Gaunt, A.L., Brockschmidt, M., et al: ‘Deepcoder: learning to write programs’. ArXiv, abs/1611.01989, 2017.
23. 23)
  - 40. Pixate. Available at http://www.pixate.com/.
24. 24)
  - 36. Mockup.io – Present and Manage iOS Mockups. Available at https://mockup.io/about/.
25. 25)
  - 25. Chang, T.-H., Yeh, T., Miller, R.: ‘Associating the visual representation of user interfaces with their internal structures and metadata’. Proc. of the 24th Annual ACM Symp. on User Interface Software and Technology (UIST ‘11), New York, NY, USA, 2011, pp. 245–256.
26. 26)
  - 42. MockingBot – Mobile prototyping without headaches. Available at https://mockingbot.com/.
27. 27)
  - 31. Landay, J.A., Myers, B.A.: ‘Sketching interfaces: toward more human interface design’, Computer, 2001, 34, (3), pp. 56–64.
28. 28)
  - 30. Coyette, A., Kieffer, S., Vanderdonckt, J.: ‘Multi-fidelity prototyping of user interfaces’. In Proc. of the 11th IFIP TC 13 Int. Conf. on Human-computer Interaction (INTERACT'07), Berlin, Heidelberg, 2007, pp. 150–164.
29. 29)
  - 34. Meng, X., Zhao, S., Huang, Y., et al: ‘WADE: simplified GUI add-on development for third-party software’. Proc. of the 32Nd Annual ACM Conf. on Human Factors in Computing Systems (CHI ’14), New York, NY, USA, 2014, pp. 2221–2230.
30. 30)
  - 6. Tucker, A.B.: ‘Computer science handbook’ (Chapman & Hall/CRC, Boca Raton, Fla, 2004, 2nd edn.).
31. 31)
  - 15. Wang, C.-Y., Liao, H.-Y.M., Yeh, I.-H., et al: ‘CSPNet: A new backbone that can enhance learning capability of CNN’. ArXiv, abs/1911.11929, 2019.
32. 32)
  - 27. Dixon, M., Fogarty, J.: ‘Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure’. Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI ’10), New York, NY, USA, 2010, pp. 1525–1534.
33. 33)
  - 24. Visual Studio IDE, Code Editor, Azure DevOps, & App Center. Available at https://visualstudio.microsoft.com/.
34. 34)
  - 12. Myers, B., Park, S.Y., Nakano, Y., et al: ‘How designers design and program interactive behaviors. (:unav)’, September 2008.
35. 35)
  - 16. Huang, G., Liu, Z., Weinberger, K.Q.: ‘Densely connected convolutional networks’. 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 2261–2269.
36. 36)
  - 18. Beltramelli, T.: ‘Pix2code: generating code from a graphical user interface screenshot’. arXiv:1705.07962 [cs], May 2017, arXiv: 1705.07962.
37. 37)
  - 21. Jain, V., Agrawal, P., Banga, S., et al: ‘Sketch2code: transformation of sketches to UI in real-time using deep neural network’. ArXiv, abs/1910.08930, 2019.
38. 38)
  - 14. Joseph, R., Ali, F.: ‘YOLOv3: an Incremental improvement’, 2018.
39. 39)
  - 22. Eyiokur, F.I., Yaman, D., Ekenel, H.K.: ‘Sketch classification with deep learning models’. 2018 26th Signal Processing and Communications Applications Conf. (SIU), Izmir, Turkey, 2018, pp. 1–4.
40. 40)
  - 45. Prott – Prototyping tool. Features. Available at https://prottapp.com/.
41. 41)
  - 19. Mohian, S., Csallner, C.: ‘Doodle2app: native app code by freehand ui sketching’. Proc. of the IEEE/ACM 7th Int. Conf. on Mobile Software Engineering and Systems, MOBILESoft ‘20, New York, NY, USA, 2020, pp. 81–84.
42. 42)
  - 28. Hinze, A., Bowen, J., Wang, Y., et al: ‘Model-driven GUI & interaction design using emulation’. Proc. of the 2Nd ACM SIGCHI Symp. on Engineering Interactive Computing Systems (EICS ’10), New York, NY, USA, 2010, pp. 273–278.
43. 43)
  - 17. Deka, B., Huang, Z., Franzen, C., et al: ‘Rico: a Mobile app dataset for building data-driven design applications’. Proc. of the 30th Annual ACM Symp. on User Interface Software and Technology (UIST ’17), Qubec City, QC, Canada, 2017, pp. 845–854.
44. 44)
  - 39. Prototype – Prototyping tool for designing website and app prototypes. Available at https://marvelapp.com/features/prototyping/.
45. 45)
  - 4. Photoshop Inspiration, Photoshop Information | Photoshop.com. Available at https://www.photoshop.com/.
46. 46)
  - 11. Landay, J.A., Myers, B.A.: ‘Interactive sketching for the early stages of user interface design’. Proc. of the SIGCHI Conf. on Human factors in computing systems (CHI ’95), Denver, Colorado, United States, 1995, pp. 43–50.
47. 47)
  - 43. Flinto. Available at https://www.flinto.com/.
48. 48)
  - 8. Nguyen, T.A., Csallner, C.: ‘Reverse engineering Mobile application user interfaces with REMAUI (T)’. 2015 30th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE), Lincoln, NE, USA, November 2015, pp. 248–259.
49. 49)
  - 26. Dixon, M., Leventhal, D., Fogarty, J.: ‘Content and hierarchy in pixel-based methods for reverse engineering interface structure’. Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI ’11), New York, NY, USA, 2011, pp. 969–978.
50. 50)
  - 3. Why Your App's UX is More Important than You Think. Available at https://www.codemag.com/Article/1401041.

Deep learning-based prototyping of android GUI from hand-drawn mockups

References

Related content