Judy – a mutation testing tool for Java

L. Madeyski; N. Radyk

Judy – a mutation testing tool for Java

Access Full Text

Judy – a mutation testing tool for Java

Author(s): L. Madeyski and N. Radyk
DOI: 10.1049/iet-sen.2008.0038

For access to this article, please select a purchase option:

Buy article PDF

Buy Knowledge Pack

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership

Recommend Title Publication to library

IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Author(s): L. Madeyski ¹ and N. Radyk ¹
- Affiliations: 1: Institute of Informatics, Wrocław University of Technology, Wrocław, Poland
Source: Volume 4, Issue 1, February 2010, p. 32 – 42
DOI: 10.1049/iet-sen.2008.0038 , Print ISSN 1751-8806, Online ISSN 1751-8814

Published

Popular code coverage measures, such as branch coverage, are indicators of the thoroughness rather than the fault detection capability of test suites. Mutation testing is a fault-based technique that measures the effectiveness of test suites for fault localisation. Unfortunately, use of mutation testing in the software industry is rare because generating and running vast numbers of mutants against the test cases is time-consuming and difficult to do without an automated, fast and reliable tool. Our objective is to present an innovative approach to mutation testing that takes advantage of a novel aspect-oriented programming mechanism, called ‘pointcut and advice’, to avoid multiple compilation of mutants and, therefore, to speed up mutation testing. An empirical comparison of the performance of the developed tool, called Judy, with the MuJava mutation testing tool on 24 open-source projects demonstrates the value of the presented approach. The results show that there is a statistically significant (t(23)=−12.28, p < 0.0005, effect size d=3.43) difference in the number of mutants generated per second between MuJava (M=4.15, SD=1.42) and Judy (M=52.05, SD=19.69). Apart from being statistically significant, this effect is considered very large and, therefore, represents a substantive finding. This therefore allows us to estimate the fault detection effectiveness of test suites of much larger systems.

References

1. 1)
  - Madeyski, L.: `On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests', Proc. Int. Conf. Product Focused Software Process, July 2007, p. 207–221, http://dx.doi.org/10.1007/978-3-540-73460-4_20.
2. 2)
  - Rothfuss, G.J.: `A framework for open source', 2002, MSc, University of Zurich.
3. 3)
  - L. Madeyski . The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Softw. Process Improve. Pract. , 3 , 281 - 295
4. 4)
  - P. Chevalley , P. Thévenod-Fosse . A mutation analysis tool for Java programs. Int. J. Softw. Tools Tech. Transfer , 1 , 90 - 103
5. 5)
  - Irvine, S.A., Tin, P., Trigg, L., Cleary, J.G., Inglis, S., Utting, M.: `Jumble Java byte code to measure the effectiveness of unit tests', Proc. Testing: Academic and Industrial Conf. Practice and Research Techniques, September 2007, p. 169–175.
6. 6)
  - Kim, S., Clark, J.A., McDermid, J.A.: `Class mutation: mutation testing for object-oriented programs', Proc. Net.ObjectDays, October 2000, http://www-users.cs.york.ac.uk/~jac/papers/ClassMutation.pdf, accessed March 2008.
7. 7)
  - Y. Ma , J. Offutt , Y.R. Kwon . MuJava: an automated class mutation system. Softw. Test Verif Rel. , 2 , 97 - 133
8. 8)
  - P. Ammann , J. Offutt . (2008) Introduction to software testing.
9. 9)
  - Rompaey, B.V., Bois, B.D., Demeyer, S.: `Characterizing the relative significance of a test smell', Proc. Int. Conf. Software Maintenance, September 2006, p. 391–400.
10. 10)
  - R.G. Hamlet . Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. , 4 , 279 - 290
11. 11)
  - Untch, R.H., Offutt, A.J., Harrold, M.J.: `Mutation analysis using mutant schemata', Proc. Int. Symp. Software Testing and Analysis, June 1993, p. 139–148.
12. 12)
  - Y.S. Ma , J. Offutt . (2005) Description of class mutation operators for Java.
13. 13)
  - Kiczales, G., Lamping, J., Mendhekar, A.: `Aspect-oriented programming', Proc. European Conf. Object-Oriented Programming, June 1997, p. 220–242.
14. 14)
  - Offutt, J., Ma, Y.S., Kwon, Y.R.: `The class-level mutants of MuJava', Proc. Int. Workshop Automation of Software Test, May 2006, p. 78–84.
15. 15)
  - A.J. Offutt , A. Lee , G. Rothermel , R.H. Untch , C. Zapf . An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Meth. , 2 , 99 - 118
16. 16)
  - Judy http://www.e-informatyka.pl/sens/Wiki.jsp?page=Projects.Judy, accessed June 2007.
17. 17)
  - R. Laddad . (2003) AspectJ in action: practical aspect-oriented programming.
18. 18)
  - Moore, I.: `Jester – a JUnit test tester', Proc. Second Int. Conf. Extreme Programming and Flexible Processes in Software Engineering, May 2001, p. 84–87.
19. 19)
  - W.E. Wong , A.P. Mathur . Reducing the cost of mutation testing: an empirical study. J. Syst. Softw. , 3 , 185 - 196
20. 20)
  - Y.S. Ma , J. Offutt . (2005) Description of method-level mutation operators for Java.
21. 21)
  - S. Loughran , E. Hatcher . (2007) Ant in action.
22. 22)
  - Offutt, A.J., Rothermel, G., Zapf, C.: `An experimental evaluation of selective mutation', Proc. 15th Int. Conf. Software Engineering, May 1993, p. 100–107.
23. 23)
  - B. Venners . (2007) Test-driven development. A conversation with Martin Fowler, Part V.
24. 24)
  - M. Gaelli , R. Wamper , O. Nierstrasz . Composing tests from examples. J. Object Tech. , 9 , 71 - 86
25. 25)
  - A.J. Offutt , J. Pan , K. Tewary , T. Zhang . An experimental evaluation of data flow and mutation testing. Softw. Pract. Exper. , 2 , 165 - 176
26. 26)
  - Ma, Y.S., Harrold, M.J., Kwon, Y.R.: `Evaluation of mutation testing for object-oriented programs', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 869–872.
27. 27)
  - Andrews, J.H., Briand, L.C., Labiche, Y.: `Is mutation an appropriate tool for testing experiments?', Proc. 27th Int. Conf. Software Engineering, May 2005, p. 402–411.
28. 28)
  - Walsh, P.J.: `A measure of test case completeness', 1985, PhD, University New York.
29. 29)
  - J. Offutt , Y.S. Ma , Y.R. Kwon . An experimental mutation system for Java. SIGSOFT Softw. Eng. Notes , 5 , 1 - 4
30. 30)
  - Smith, B.H., Williams, L.: `An empirical evaluation of the MuJava mutation operators', Proc. Testing: Academic and Industrial Conf., September 2007, p. 193–202.
31. 31)
  - K.N. King , A.J. Offutt . A Fortran language system for mutation-based software testing. Softw. Pract. Exper. , 7 , 685 - 718
32. 32)
  - Janzen, D.S.: `An empirical evaluation of the impact of test-driven development on software quality', 2006, PhD, University of Kansas.
33. 33)
  - R.A. DeMillo , R.J. Lipton , F.G. Sayward . Hints on test data selection: help for the practicing programmer. IEEE Comput. , 4 , 34 - 41
34. 34)
  - Chevalley, P.: `Applying mutation analysis for object-oriented programs using a reflective approach', Proc. Eighth Asia-Pacific Software Engineering Conf., December 2001, p. 267–270.
35. 35)
  - P.G. Frankl , S.N. Weiss , C. Hu . All-uses vs mutation testing: an experimental comparison of effectiveness. J. Syst. Softw. , 3 , 235 - 253
36. 36)
  - Ma, Y.S., Kwon, Y.R., Offutt, J.: `Inter-class mutation operators for java', Proc. 13th Int. Symp. Software Reliability Engineering, November 2002, p. 352–363.
37. 37)
  - A.J. Offutt , R.H. Untch , W.E. Wong . (2001) Mutation 2000: uniting the orthogonal, Mutation testing for the new century.
38. 38)
  - J.B. Rainsberger . (2004) JUnit recipes.
39. 39)
  - C. Wohlin , P. Runeson , M. Höst , M.C. Ohlsson , B. Regnell , A. Wesslén . (2000) Experimentation in software engineering: an introduction.
40. 40)
  - Huang, C.H., Huo, Y.C.: `A semi-automatic generator for unit testing code files based on JUnit', Proc. Int. Conf. Systems, Man and Cybernetics, October 2005, p. 140–145.
41. 41)
  - Ma, Y.S., Offutt, J., Kwon, Y.R.: `MuJava: a mutation system for Java', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 827–830.
42. 42)
  - H. Zhu , P.A.V. Hall , J.H.R. May . Software unit test coverage and adequacy. ACM Comput. Surv. , 4 , 366 - 427
43. 43)
  - Bogacki, B., Walter, B.: `Aspect-oriented response injection: an alternative to classical mutation testing', Proc. IFIP Work. Conf. Software Engineering Techniques, October 2006, p. 273–282.
44. 44)
  - DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, M.W., Offutt, A.J.: `An extended overview of the Mothra software testing environment', Proc. Second Workshop on Software Testing, Verification, and Analysis, July 1988, p. 142–151.
45. 45)
  - Delamaro, M.E., Maldonado, J.C.: `Proteum – a tool for the assessment of test adequacy for C programs', Proc. Conf. Performability in Computing Systems, July 1996, p. 75–95.
46. 46)
  - J. Offutt . (2006) An analysis of Jester based on published papers.
47. 47)
  - T. Husted , V. Massol . (2003) JUnit in action.
48. 48)
  - J.H. Andrews , L.C. Briand , Y. Labiche , A.S. Namin . Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. , 8 , 608 - 624

Login

Not registered yet?

Share

Tools

Login to add to favourites

Key

Judy – a mutation testing tool for Java

Judy – a mutation testing tool for Java

Buy article PDF

Buy Knowledge Pack

Thank you

References

Related content