Cloud testing automation: industrial needs and ElasTest response
- Author(s): Antonia Bertolino 1 ; Antonello Calabrò 1 ; Eda Marchetti 1 ; Anton Cervantes Sala 2 ; Guiomar Tuñón de Hita 3 ; Ilie Daniel Gheorghe Pop 4 ; Varun Gowtham 5
-
-
View affiliations
-
Affiliations:
1:
Istituto di Scienza e Tecnologie dell'Informazione “A.Faedo”, Consiglio Nazionale delle Ricerche , Pisa , Italy ;
2: Worldline Iberia SAU, Mobile Competence Center , Barcelona , Spain ;
3: Naeva Tec , Las Rozas (Madrid) , Spain ;
4: Fraunhofer FOKUS Institute , Berlin , Germany ;
5: Technische Universität Berlin , Berlin , Germany
-
Affiliations:
1:
Istituto di Scienza e Tecnologie dell'Informazione “A.Faedo”, Consiglio Nazionale delle Ricerche , Pisa , Italy ;
- Source:
Volume 14, Issue 5,
October
2020,
p.
553 – 562
DOI: 10.1049/iet-sen.2019.0140 , Print ISSN 1751-8806, Online ISSN 1751-8814
- « Previous Article
- Table of contents
- Next Article »
While great emphasis is given in the current literature about the potential of leveraging the cloud for testing purposes, the authors have scarce factual evidence from real-world industrial contexts about the motivations, drawbacks and benefits related to the adoption of automated cloud testing technology. In this study, the authors present an empirical study undertaken within the ongoing European Project ElasTest, which has developed an open source platform for end-to-end testing of large distributed systems. This study aims at validating the ElasTest solution, and consists of the assessment of four demonstrators belonging to different application domains, namely e-commerce, 5G networking, WebRTC and Internet of Things. For each demonstrator, they collected differing requirements, and achieved varying results, both positive and negative, showing that cloud testing needs careful assessment before adoption.
Inspec keywords: electronic commerce; cloud computing; public domain software; automatic testing; program testing; Internet of Things; home automation
Other keywords: Internet of Things; scarce factual evidence; WebRTC; real-world industrial contexts; application domains; cloud testing automation; e-commerce; testing purposes; demonstrator; open source platform; ElasTest solution; end-to-end testing; 5G networking; ElasTest response; European Project ElasTest; automated cloud testing technology; distributed systems
Subjects: Information networks; Diagnostic, testing, debugging and evaluating systems; Mobile, ubiquitous and pervasive computing; Internet software
References
-
-
1)
-
14. Wang, W., Tian, N., Huang, S., et al: ‘Testing cloud applications under cloud-uncertainty performance effects’. 2018 IEEE 11th Int. Conf. on Software Testing, Verification and Validation (ICST), Västerås, Sweden, April 9–13 2018, pp. 81–92.
-
-
2)
-
26. Gribbons, B., Herman, J.: ‘True and quasi-experimental designs. ERIC/AE digest’ (ERIC Clearinghouse on Assessment and Evaluation, Washington, DC, 1997).
-
-
3)
-
21. NaevaTec: ‘Full teaching’, 2020. Available at https://github.com/OpenVidu/full-teaching.
-
-
4)
-
3. Bertolino, A., De Angelis, G., Gallego, M., et al: ‘A systematic review on cloud testing’, ACM Comput. Surv., 2019, 52, (5), pp. 1–42. Available at https://doi.org/10.1145/3331447.
-
-
5)
-
19. FOKUS: ‘Open5GCore website’, 2020. Available at https://www.open5gcore.org/.
-
-
6)
-
2. Mell, P., Grance, T.: ‘The NIST definition of cloud computing’, National Institute of Standards and Technology (NIST), 2011. 800-145. Available at http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf.
-
-
7)
-
15. Ahmad, A.A.S., Brereton, P., Andras, P.: ‘A systematic mapping study of empirical studies on software cloud testing methods’. IEEE Int. Conf. on Software Quality, Reliability and Security Companion (QRS-C), Prague, Czech Republic, 2017, pp. 555–562.
-
-
8)
-
18. FOKUS: ‘Fraunhofer website’, 2020. Available at https://www.fokus.fraunhofer.de/.
-
-
9)
-
7. Elastest: ‘Elastest’, 2018. Available at https://elastest.eu/.
-
-
10)
-
22. Google: ‘WebRTC’, 2020. Available at https://webrtc.org/.
-
-
11)
-
30. Silva, D.G.e., de Abreu, B.T., Jino, M.: ‘A simple approach for estimation of execution effort of functional test cases’. Proc. of the 2009 Int. Conf. on Software Testing Verification and Validation. ICST'09, USA, 2009, pp. 289–298. Available at https://doi.org/10.1109/ICST.2009.47.
-
-
12)
-
8. Bertolino, A., Calabrò, A., Marchetti, E., et al: ‘Perceived needs and gains from an industrial study in cloud testing automation’. 2018 IEEE/ACM Int. Conf. on Utility and Cloud Computing Companion, UCC Companion 2018, Zurich, Switzerland, 17–20 December 2018, pp. 238–244. Available at https://doi.org/10.1109/UCC-Companion.2018.00062.
-
-
13)
-
9. Gao, J., Bai, X., Tsai, W.T.: ‘Cloud testing-issues, challenges, needs and practice’, Softw. Eng., Int. J., 2011, 1, (1), pp. 9–23.
-
-
14)
-
10. Bai, X., Li, M., Chen, B., et al: ‘Cloud testing tools’. 2011 IEEE 6th Int. Symp. on Service Oriented System Engineering (SOSE), Irvine, CA, USA, December 12–14 2011, pp. 1–12.
-
-
15)
-
20. NaevaTec: ‘Naeva Tec website’, 2020. Available at https://www.naevatec.com/.
-
-
16)
-
31. Meyer, A.N., Fritz, T., Murphy, G.C., et al: ‘Software developers’ perceptions of productivity’. Proc. of the 22nd ACM SIGSOFT Int. Symp. on Foundations of Software Engineering. FSE 2014, New York, NY, USA, 2014, pp. 19–29. Available at https://doi.org/10.1145/2635868.2635892.
-
-
17)
-
16. Riungu-Kalliosaari, L., Taipale, O., Smolander, K., et al: ‘Adoption and use of cloud-based testing in practice’, Softw. Qual. J., 2016, 24, (2), pp. 337–364.
-
-
18)
-
1. CapGemini, Sogeti, and Micro Focus: ‘World quality report’ (Micro Focus, Issy-les-Moulineaux, France, 2017-18, 9th edn.).
-
-
19)
-
28. TestLink: ‘Testlink open source test management’, 2020. Available at http://testlink.org/.
-
-
20)
-
12. ‘D2.2 SotA revision document v1’, Elastest Consortium, 2018. Available at https://elastest.eu//resources/deliverables/D2.2_SotA_revision_document_v1_FINAL.pdf.
-
-
21)
-
27. Docker: ‘Elastest dockerhub’, 2020. Available at https://hub.docker.com/r/elastest/.
-
-
22)
-
6. Gortazár, F., Gallego, M., Donato, M., et al: ‘The ElasTest platform: supporting automation of end-to-end testing of large complex applications’. Elastest Consortium, 2018. Available at https://elastest.io/resources/ElasTest_white_paper.pdf.
-
-
23)
-
29. ElasTest Consortium: ‘Multi-configuration’, ElasTest Consortium, 2019. Available at https://elastest.io/docs/demos/multi-configuration/.
-
-
24)
-
13. Liu, C., Chen, S.: ‘Evaluation of cloud testing strategies based on task decomposition and allocation for improving test efficiency’. 2016 Int. Conf. on Applied System Innovation (ICASI), Okinawa, Japan, May 28–June 1 2016, pp. 1–4.
-
-
25)
-
23. TUB: ‘Technische Universität Berlin website’, 2020. Available at https://www.av.tu-berlin.de/next_generation_networks/.
-
-
26)
-
17. Worldline: ‘Worldline website’, 2020. Available at https://worldline.com/.
-
-
27)
-
24. FOKUS: ‘Open IoT Fog website’, 2020. Available at https://openiotfog.org/en/.
-
-
28)
-
11. Vilkomir, S.: ‘Cloud testing: a state-of-the-art review’, Inf. Secur., Int. J., 2012, 28, (17), pp. 213–222.
-
-
29)
-
25. ‘D7.1 ElasTest Validation methodology and its results v1’, Elastest Consortium, 2018. Available at https://elastest.eu//resources/deliverables/D7.1_ElasTest_validation_methodology_and_its_results_v1_FINAL.pdf.
-
-
30)
-
4. Inçki, K., Ari, I., Sözer, H.: ‘A survey of software testing in the cloud’. Sixth Int. Conf. on Software Security and Reliability, SERE 2012, Gaithersburg, Maryland, USA, 20–22 June 2012, pp. 18–23. Available at https://doi.org/10.1109/SERE-C.2012.32.
-
-
31)
-
5. Tilley, S.R., Parveen, T.: ‘Software testing in the cloud – migration and execution’, Springer Briefs in Computer Science (Springer-Verlag, Berlin, Heidelberg, Germany, 2012). Available at https://doi.org/10.1007/978-3-642-32122-1.
-
-
1)