http://iet.metastore.ingenta.com
1887

Judy – a mutation testing tool for Java

Judy – a mutation testing tool for Java

For access to this article, please select a purchase option:

Buy article PDF
£12.50
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Popular code coverage measures, such as branch coverage, are indicators of the thoroughness rather than the fault detection capability of test suites. Mutation testing is a fault-based technique that measures the effectiveness of test suites for fault localisation. Unfortunately, use of mutation testing in the software industry is rare because generating and running vast numbers of mutants against the test cases is time-consuming and difficult to do without an automated, fast and reliable tool. Our objective is to present an innovative approach to mutation testing that takes advantage of a novel aspect-oriented programming mechanism, called ‘pointcut and advice’, to avoid multiple compilation of mutants and, therefore, to speed up mutation testing. An empirical comparison of the performance of the developed tool, called Judy, with the MuJava mutation testing tool on 24 open-source projects demonstrates the value of the presented approach. The results show that there is a statistically significant (t(23)=−12.28, p < 0.0005, effect size d=3.43) difference in the number of mutants generated per second between MuJava (M=4.15, SD=1.42) and Judy (M=52.05, SD=19.69). Apart from being statistically significant, this effect is considered very large and, therefore, represents a substantive finding. This therefore allows us to estimate the fault detection effectiveness of test suites of much larger systems.

References

    1. 1)
      • R.A. DeMillo , R.J. Lipton , F.G. Sayward . Hints on test data selection: help for the practicing programmer. IEEE Comput. , 4 , 34 - 41
    2. 2)
      • R.G. Hamlet . Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. , 4 , 279 - 290
    3. 3)
      • Ma, Y.S., Kwon, Y.R., Offutt, J.: `Inter-class mutation operators for java', Proc. 13th Int. Symp. Software Reliability Engineering, November 2002, p. 352–363.
    4. 4)
      • H. Zhu , P.A.V. Hall , J.H.R. May . Software unit test coverage and adequacy. ACM Comput. Surv. , 4 , 366 - 427
    5. 5)
      • A.J. Offutt , R.H. Untch , W.E. Wong . (2001) Mutation 2000: uniting the orthogonal, Mutation testing for the new century.
    6. 6)
      • Madeyski, L.: `On the effects of pair programming on thoroughness and fault-finding effectiveness of unit tests', Proc. Int. Conf. Product Focused Software Process, July 2007, p. 207–221, http://dx.doi.org/10.1007/978-3-540-73460-4_20.
    7. 7)
      • L. Madeyski . The impact of pair programming on thoroughness and fault detection effectiveness of unit tests suites. Softw. Process Improve. Pract. , 3 , 281 - 295
    8. 8)
      • Y. Ma , J. Offutt , Y.R. Kwon . MuJava: an automated class mutation system. Softw. Test Verif Rel. , 2 , 97 - 133
    9. 9)
      • Judy http://www.e-informatyka.pl/sens/Wiki.jsp?page=Projects.Judy, accessed June 2007.
    10. 10)
      • R. Laddad . (2003) AspectJ in action: practical aspect-oriented programming.
    11. 11)
      • T. Husted , V. Massol . (2003) JUnit in action.
    12. 12)
      • J.B. Rainsberger . (2004) JUnit recipes.
    13. 13)
      • M. Gaelli , R. Wamper , O. Nierstrasz . Composing tests from examples. J. Object Tech. , 9 , 71 - 86
    14. 14)
      • Huang, C.H., Huo, Y.C.: `A semi-automatic generator for unit testing code files based on JUnit', Proc. Int. Conf. Systems, Man and Cybernetics, October 2005, p. 140–145.
    15. 15)
      • Janzen, D.S.: `An empirical evaluation of the impact of test-driven development on software quality', 2006, PhD, University of Kansas.
    16. 16)
      • Rompaey, B.V., Bois, B.D., Demeyer, S.: `Characterizing the relative significance of a test smell', Proc. Int. Conf. Software Maintenance, September 2006, p. 391–400.
    17. 17)
      • S. Loughran , E. Hatcher . (2007) Ant in action.
    18. 18)
      • Offutt, J., Ma, Y.S., Kwon, Y.R.: `The class-level mutants of MuJava', Proc. Int. Workshop Automation of Software Test, May 2006, p. 78–84.
    19. 19)
      • Kim, S., Clark, J.A., McDermid, J.A.: `Class mutation: mutation testing for object-oriented programs', Proc. Net.ObjectDays, October 2000, http://www-users.cs.york.ac.uk/~jac/papers/ClassMutation.pdf, accessed March 2008.
    20. 20)
      • Chevalley, P.: `Applying mutation analysis for object-oriented programs using a reflective approach', Proc. Eighth Asia-Pacific Software Engineering Conf., December 2001, p. 267–270.
    21. 21)
      • K.N. King , A.J. Offutt . A Fortran language system for mutation-based software testing. Softw. Pract. Exper. , 7 , 685 - 718
    22. 22)
      • A.J. Offutt , A. Lee , G. Rothermel , R.H. Untch , C. Zapf . An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Meth. , 2 , 99 - 118
    23. 23)
      • Andrews, J.H., Briand, L.C., Labiche, Y.: `Is mutation an appropriate tool for testing experiments?', Proc. 27th Int. Conf. Software Engineering, May 2005, p. 402–411.
    24. 24)
    25. 25)
      • Walsh, P.J.: `A measure of test case completeness', 1985, PhD, University New York.
    26. 26)
      • P.G. Frankl , S.N. Weiss , C. Hu . All-uses vs mutation testing: an experimental comparison of effectiveness. J. Syst. Softw. , 3 , 235 - 253
    27. 27)
      • A.J. Offutt , J. Pan , K. Tewary , T. Zhang . An experimental evaluation of data flow and mutation testing. Softw. Pract. Exper. , 2 , 165 - 176
    28. 28)
      • Offutt, A.J., Rothermel, G., Zapf, C.: `An experimental evaluation of selective mutation', Proc. 15th Int. Conf. Software Engineering, May 1993, p. 100–107.
    29. 29)
      • W.E. Wong , A.P. Mathur . Reducing the cost of mutation testing: an empirical study. J. Syst. Softw. , 3 , 185 - 196
    30. 30)
      • DeMillo, R.A., Guindi, D.S., King, K.N., McCracken, M.W., Offutt, A.J.: `An extended overview of the Mothra software testing environment', Proc. Second Workshop on Software Testing, Verification, and Analysis, July 1988, p. 142–151.
    31. 31)
      • Delamaro, M.E., Maldonado, J.C.: `Proteum – a tool for the assessment of test adequacy for C programs', Proc. Conf. Performability in Computing Systems, July 1996, p. 75–95.
    32. 32)
      • Moore, I.: `Jester – a JUnit test tester', Proc. Second Int. Conf. Extreme Programming and Flexible Processes in Software Engineering, May 2001, p. 84–87.
    33. 33)
      • Irvine, S.A., Tin, P., Trigg, L., Cleary, J.G., Inglis, S., Utting, M.: `Jumble Java byte code to measure the effectiveness of unit tests', Proc. Testing: Academic and Industrial Conf. Practice and Research Techniques, September 2007, p. 169–175.
    34. 34)
      • J. Offutt , Y.S. Ma , Y.R. Kwon . An experimental mutation system for Java. SIGSOFT Softw. Eng. Notes , 5 , 1 - 4
    35. 35)
      • Ma, Y.S., Offutt, J., Kwon, Y.R.: `MuJava: a mutation system for Java', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 827–830.
    36. 36)
      • P. Chevalley , P. Thévenod-Fosse . A mutation analysis tool for Java programs. Int. J. Softw. Tools Tech. Transfer , 1 , 90 - 103
    37. 37)
      • Bogacki, B., Walter, B.: `Aspect-oriented response injection: an alternative to classical mutation testing', Proc. IFIP Work. Conf. Software Engineering Techniques, October 2006, p. 273–282.
    38. 38)
      • Untch, R.H., Offutt, A.J., Harrold, M.J.: `Mutation analysis using mutant schemata', Proc. Int. Symp. Software Testing and Analysis, June 1993, p. 139–148.
    39. 39)
      • Ma, Y.S., Harrold, M.J., Kwon, Y.R.: `Evaluation of mutation testing for object-oriented programs', Proc. 28th Int. Conf. Software Engineering, May 2006, p. 869–872.
    40. 40)
      • Y.S. Ma , J. Offutt . (2005) Description of class mutation operators for Java.
    41. 41)
      • Y.S. Ma , J. Offutt . (2005) Description of method-level mutation operators for Java.
    42. 42)
      • Smith, B.H., Williams, L.: `An empirical evaluation of the MuJava mutation operators', Proc. Testing: Academic and Industrial Conf., September 2007, p. 193–202.
    43. 43)
      • B. Venners . (2007) Test-driven development. A conversation with Martin Fowler, Part V.
    44. 44)
      • J. Offutt . (2006) An analysis of Jester based on published papers.
    45. 45)
      • P. Ammann , J. Offutt . (2008) Introduction to software testing.
    46. 46)
      • Kiczales, G., Lamping, J., Mendhekar, A.: `Aspect-oriented programming', Proc. European Conf. Object-Oriented Programming, June 1997, p. 220–242.
    47. 47)
      • C. Wohlin , P. Runeson , M. Höst , M.C. Ohlsson , B. Regnell , A. Wesslén . (2000) Experimentation in software engineering: an introduction.
    48. 48)
      • Rothfuss, G.J.: `A framework for open source', 2002, MSc, University of Zurich.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen.2008.0038
Loading

Related content

content/journals/10.1049/iet-sen.2008.0038
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address