Judy – a mutation testing tool for Java

Judy – a mutation testing tool for Java

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Popular code coverage measures, such as branch coverage, are indicators of the thoroughness rather than the fault detection capability of test suites. Mutation testing is a fault-based technique that measures the effectiveness of test suites for fault localisation. Unfortunately, use of mutation testing in the software industry is rare because generating and running vast numbers of mutants against the test cases is time-consuming and difficult to do without an automated, fast and reliable tool. Our objective is to present an innovative approach to mutation testing that takes advantage of a novel aspect-oriented programming mechanism, called ‘pointcut and advice’, to avoid multiple compilation of mutants and, therefore, to speed up mutation testing. An empirical comparison of the performance of the developed tool, called Judy, with the MuJava mutation testing tool on 24 open-source projects demonstrates the value of the presented approach. The results show that there is a statistically significant (t(23)=−12.28, p < 0.0005, effect size d=3.43) difference in the number of mutants generated per second between MuJava (M=4.15, SD=1.42) and Judy (M=52.05, SD=19.69). Apart from being statistically significant, this effect is considered very large and, therefore, represents a substantive finding. This therefore allows us to estimate the fault detection effectiveness of test suites of much larger systems.


    1. 1)
      • Hints on test data selection: help for the practicing programmer
    2. 2)
      • Testing programs with the aid of a compiler
    3. 3)
      • Ma, Y.S., Kwon, Y.R., Offutt, J.: `Inter-class mutation operators for java', Proc. 13th Int. Symp. Software Reliability Engineering, November 2002, p. 352–363
    4. 4)
      • Software unit test coverage and adequacy
    5. 5)
      • Mutation 2000: uniting the orthogonal, Mutation testing for the new century
    6. 6)
      • Madeyski, L.: `On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests', Proc. Int. Conf. Product Focused Software Process, July 2007, p. 207–221,
    7. 7)
      • The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites
    8. 8)
      • MuJava: an automated class mutation system
    9. 9)
      • Judy, accessed June 2007
    10. 10)
      • AspectJ in action: practical aspect-oriented programming
    11. 11)
      • JUnit in action
    12. 12)
      • JUnit recipes
    13. 13)
      • Composing tests from examples
    14. 14)
      • Huang, C.H., Huo, Y.C.: `A semi-automatic generator for unit testing code files based on JUnit', Proc. Int. Conf. Systems, Man and Cybernetics, October 2005, p. 140–145
    15. 15)
      • Janzen, D.S.: `An empirical evaluation of the impact of test-driven development on software quality', 2006, PhD, University of Kansas
    16. 16)
      • Rompaey, B.V., Bois, B.D., Demeyer, S.: `Characterizing the relative significance of a test smell', Proc. Int. Conf. Software Maintenance, September 2006, p. 391–400
    17. 17)
      • Ant in action
    18. 18)
      • Offutt, J., Ma, Y.S., Kwon, Y.R.: `The class-level mutants of MuJava', Proc. Int. Workshop Automation of Software Test, May 2006, p. 78–84
    19. 19)
      • Kim, S., Clark, J.A., McDermid, J.A.: `Class mutation: mutation testing for object-oriented programs', Proc. Net.ObjectDays, October 2000,, accessed March 2008
    20. 20)
      • Chevalley, P.: `Applying mutation analysis for object-oriented programs using a reflective approach', Proc. Eighth Asia-Pacific Software Engineering Conf., December 2001, p. 267–270
    21. 21)
      • A Fortran language system for mutation-based software testing
    22. 22)
      • An experimental determination of sufficient mutant operators
    23. 23)
      • Andrews, J.H., Briand, L.C., Labiche, Y.: `Is mutation an appropriate tool for testing experiments?', Proc. 27th Int. Conf. Software Engineering, May 2005, p. 402–411
    24. 24)
    25. 25)
      • Walsh, P.J.: `A measure of test case completeness', 1985, PhD, University New York
    26. 26)
      • All-uses vs mutation testing: an experimental comparison of effectiveness
    27. 27)
      • An experimental evaluation of data flow and mutation testing
    28. 28)
      • Offutt, A.J., Rothermel, G., Zapf, C.: `An experimental evaluation of selective mutation', Proc. 15th Int. Conf. Software Engineering, May 1993, p. 100–107
    29. 29)
      • Reducing the cost of mutation testing: an empirical study
    30. 30)
      • DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, M.W., Offutt, A.J.: `An extended overview of the Mothra software testing environment', Proc. Second Workshop on Software Testing, Verification, and Analysis, July 1988, p. 142–151
    31. 31)
      • Delamaro, M.E., Maldonado, J.C.: `Proteum – a tool for the assessment of test adequacy for C programs', Proc. Conf. Performability in Computing Systems, July 1996, p. 75–95
    32. 32)
      • Moore, I.: `Jester – a JUnit test tester', Proc. Second Int. Conf. Extreme Programming and Flexible Processes in Software Engineering, May 2001, p. 84–87
    33. 33)
      • Irvine, S.A., Tin, P., Trigg, L., Cleary, J.G., Inglis, S., Utting, M.: `Jumble Java byte code to measure the effectiveness of unit tests', Proc. Testing: Academic and Industrial Conf. Practice and Research Techniques, September 2007, p. 169–175
    34. 34)
      • An experimental mutation system for Java
    35. 35)
      • Ma, Y.S., Offutt, J., Kwon, Y.R.: `MuJava: a mutation system for Java', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 827–830
    36. 36)
      • A mutation analysis tool for Java programs
    37. 37)
      • Bogacki, B., Walter, B.: `Aspect-oriented response injection: an alternative to classical mutation testing', Proc. IFIP Work. Conf. Software Engineering Techniques, October 2006, p. 273–282
    38. 38)
      • Untch, R.H., Offutt, A.J., Harrold, M.J.: `Mutation analysis using mutant schemata', Proc. Int. Symp. Software Testing and Analysis, June 1993, p. 139–148
    39. 39)
      • Ma, Y.S., Harrold, M.J., Kwon, Y.R.: `Evaluation of mutation testing for object-oriented programs', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 869–872
    40. 40)
      • Description of class mutation operators for Java
    41. 41)
      • Description of method-level mutation operators for Java
    42. 42)
      • Smith, B.H., Williams, L.: `An empirical evaluation of the MuJava mutation operators', Proc. Testing: Academic and Industrial Conf., September 2007, p. 193–202
    43. 43)
      • Test-driven development. A conversation with Martin Fowler, Part V
    44. 44)
      • An analysis of Jester based on published papers
    45. 45)
      • Introduction to software testing
    46. 46)
      • Kiczales, G., Lamping, J., Mendhekar, A.: `Aspect-oriented programming', Proc. European Conf. Object-Oriented Programming, June 1997, p. 220–242
    47. 47)
      • Experimentation in software engineering: an introduction
    48. 48)
      • Rothfuss, G.J.: `A framework for open source', 2002, MSc, University of Zurich

Related content

This is a required field
Please enter a valid email address