access icon free Cloud testing automation: industrial needs and ElasTest response

© The Institution of Engineering and Technology
Received 31/05/2019, Accepted 03/04/2020, Revised 12/02/2020, Published 23/04/2020

While great emphasis is given in the current literature about the potential of leveraging the cloud for testing purposes, the authors have scarce factual evidence from real-world industrial contexts about the motivations, drawbacks and benefits related to the adoption of automated cloud testing technology. In this study, the authors present an empirical study undertaken within the ongoing European Project ElasTest, which has developed an open source platform for end-to-end testing of large distributed systems. This study aims at validating the ElasTest solution, and consists of the assessment of four demonstrators belonging to different application domains, namely e-commerce, 5G networking, WebRTC and Internet of Things. For each demonstrator, they collected differing requirements, and achieved varying results, both positive and negative, showing that cloud testing needs careful assessment before adoption.

Inspec keywords: electronic commerce; cloud computing; public domain software; automatic testing; program testing; Internet of Things; home automation

Other keywords: Internet of Things; scarce factual evidence; WebRTC; real-world industrial contexts; application domains; cloud testing automation; e-commerce; testing purposes; demonstrator; open source platform; ElasTest solution; end-to-end testing; 5G networking; ElasTest response; European Project ElasTest; automated cloud testing technology; distributed systems

Subjects: Information networks; Diagnostic, testing, debugging and evaluating systems; Mobile, ubiquitous and pervasive computing; Internet software

References

    1. 1)
      • 14. Wang, W., Tian, N., Huang, S., et al: ‘Testing cloud applications under cloud-uncertainty performance effects’. 2018 IEEE 11th Int. Conf. on Software Testing, Verification and Validation (ICST), Västerås, Sweden, April 9–13 2018, pp. 81–92.
    2. 2)
      • 26. Gribbons, B., Herman, J.: ‘True and quasi-experimental designs. ERIC/AE digest’ (ERIC Clearinghouse on Assessment and Evaluation, Washington, DC, 1997).
    3. 3)
      • 21. NaevaTec: ‘Full teaching’, 2020. Available at https://github.com/OpenVidu/full-teaching.
    4. 4)
      • 3. Bertolino, A., De Angelis, G., Gallego, M., et al: ‘A systematic review on cloud testing’, ACM Comput. Surv., 2019, 52, (5), pp. 1–42. Available at https://doi.org/10.1145/3331447.
    5. 5)
      • 19. FOKUS: ‘Open5GCore website’, 2020. Available at https://www.open5gcore.org/.
    6. 6)
      • 2. Mell, P., Grance, T.: ‘The NIST definition of cloud computing’, National Institute of Standards and Technology (NIST), 2011. 800-145. Available at http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf.
    7. 7)
      • 15. Ahmad, A.A.S., Brereton, P., Andras, P.: ‘A systematic mapping study of empirical studies on software cloud testing methods’. IEEE Int. Conf. on Software Quality, Reliability and Security Companion (QRS-C), Prague, Czech Republic, 2017, pp. 555–562.
    8. 8)
      • 18. FOKUS: ‘Fraunhofer website’, 2020. Available at https://www.fokus.fraunhofer.de/.
    9. 9)
      • 7. Elastest: ‘Elastest’, 2018. Available at https://elastest.eu/.
    10. 10)
      • 22. Google: ‘WebRTC’, 2020. Available at https://webrtc.org/.
    11. 11)
      • 30. Silva, D.G.e., de Abreu, B.T., Jino, M.: ‘A simple approach for estimation of execution effort of functional test cases’. Proc. of the 2009 Int. Conf. on Software Testing Verification and Validation. ICST'09, USA, 2009, pp. 289–298. Available at https://doi.org/10.1109/ICST.2009.47.
    12. 12)
      • 8. Bertolino, A., Calabrò, A., Marchetti, E., et al: ‘Perceived needs and gains from an industrial study in cloud testing automation’. 2018 IEEE/ACM Int. Conf. on Utility and Cloud Computing Companion, UCC Companion 2018, Zurich, Switzerland, 17–20 December 2018, pp. 238–244. Available at https://doi.org/10.1109/UCC-Companion.2018.00062.
    13. 13)
      • 9. Gao, J., Bai, X., Tsai, W.T.: ‘Cloud testing-issues, challenges, needs and practice’, Softw. Eng., Int. J., 2011, 1, (1), pp. 9–23.
    14. 14)
      • 10. Bai, X., Li, M., Chen, B., et al: ‘Cloud testing tools’. 2011 IEEE 6th Int. Symp. on Service Oriented System Engineering (SOSE), Irvine, CA, USA, December 12–14 2011, pp. 1–12.
    15. 15)
      • 20. NaevaTec: ‘Naeva Tec website’, 2020. Available at https://www.naevatec.com/.
    16. 16)
      • 31. Meyer, A.N., Fritz, T., Murphy, G.C., et al: ‘Software developers’ perceptions of productivity’. Proc. of the 22nd ACM SIGSOFT Int. Symp. on Foundations of Software Engineering. FSE 2014, New York, NY, USA, 2014, pp. 19–29. Available at https://doi.org/10.1145/2635868.2635892.
    17. 17)
      • 16. Riungu-Kalliosaari, L., Taipale, O., Smolander, K., et al: ‘Adoption and use of cloud-based testing in practice’, Softw. Qual. J., 2016, 24, (2), pp. 337–364.
    18. 18)
      • 1. CapGemini, Sogeti, and Micro Focus: ‘World quality report’ (Micro Focus, Issy-les-Moulineaux, France, 2017-18, 9th edn.).
    19. 19)
      • 28. TestLink: ‘Testlink open source test management’, 2020. Available at http://testlink.org/.
    20. 20)
      • 12. ‘D2.2 SotA revision document v1’, Elastest Consortium, 2018. Available at https://elastest.eu//resources/deliverables/D2.2_SotA_revision_document_v1_FINAL.pdf.
    21. 21)
      • 27. Docker: ‘Elastest dockerhub’, 2020. Available at https://hub.docker.com/r/elastest/.
    22. 22)
      • 6. Gortazár, F., Gallego, M., Donato, M., et al: ‘The ElasTest platform: supporting automation of end-to-end testing of large complex applications’. Elastest Consortium, 2018. Available at https://elastest.io/resources/ElasTest_white_paper.pdf.
    23. 23)
      • 29. ElasTest Consortium: ‘Multi-configuration’, ElasTest Consortium, 2019. Available at https://elastest.io/docs/demos/multi-configuration/.
    24. 24)
      • 13. Liu, C., Chen, S.: ‘Evaluation of cloud testing strategies based on task decomposition and allocation for improving test efficiency’. 2016 Int. Conf. on Applied System Innovation (ICASI), Okinawa, Japan, May 28–June 1 2016, pp. 1–4.
    25. 25)
      • 23. TUB: ‘Technische Universität Berlin website’, 2020. Available at https://www.av.tu-berlin.de/next_generation_networks/.
    26. 26)
      • 17. Worldline: ‘Worldline website’, 2020. Available at https://worldline.com/.
    27. 27)
      • 24. FOKUS: ‘Open IoT Fog website’, 2020. Available at https://openiotfog.org/en/.
    28. 28)
      • 11. Vilkomir, S.: ‘Cloud testing: a state-of-the-art review’, Inf. Secur., Int. J., 2012, 28, (17), pp. 213–222.
    29. 29)
      • 25. ‘D7.1 ElasTest Validation methodology and its results v1’, Elastest Consortium, 2018. Available at https://elastest.eu//resources/deliverables/D7.1_ElasTest_validation_methodology_and_its_results_v1_FINAL.pdf.
    30. 30)
      • 4. Inçki, K., Ari, I., Sözer, H.: ‘A survey of software testing in the cloud’. Sixth Int. Conf. on Software Security and Reliability, SERE 2012, Gaithersburg, Maryland, USA, 20–22 June 2012, pp. 18–23. Available at https://doi.org/10.1109/SERE-C.2012.32.
    31. 31)
      • 5. Tilley, S.R., Parveen, T.: ‘Software testing in the cloud – migration and execution’, Springer Briefs in Computer Science (Springer-Verlag, Berlin, Heidelberg, Germany, 2012). Available at https://doi.org/10.1007/978-3-642-32122-1.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen.2019.0140
Loading

Related content

content/journals/10.1049/iet-sen.2019.0140
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading