access icon free How test suites impact fault localisation starting from the size

© The Institution of Engineering and Technology
Received 05/02/2017, Accepted 27/01/2018, Revised 16/11/2017, Published 30/01/2018

Although a test suite is indispensable for conducting effective fault localisation, not much work has been done to study how the test suite impacts fault localisation. This study presents a systematic study for a deeper understanding of their relation. Specifically, the authors' study reveals an interesting fact that there is no strong correlation between localisation effectiveness and the size of the test suite. Furthermore, they show that, in a test suite, (i) the passing test cases that do not execute the faulty statements and the failing test cases have a positive effect on the fault localisation effectiveness, while (ii) the passing test cases that exercise the faulty statements have a negative impact on localisation performance. Their result is drawn from a large-scale empirical analysis on the localisation effectiveness with respect to randomly sampled test suites. This study presents the details of the study and their follow-up investigation on the findings. Their work provides a new perspective on fault localisation and suggests fresh directions of research on an extensively studied topic.

Inspec keywords: program debugging; program testing

Other keywords: fault localisation; software test suites; large-scale empirical analysis; debugging

Subjects: Systems analysis and programming; Diagnostic, testing, debugging and evaluating systems

References

    1. 1)
      • 41. Zhang, Y., Harman, M., Jia, Y., et al: ‘Inferring test models from Kate's bug reports using multi-objective search’. Proc. the 7th Int. Conf. Search Based Software Engineering (SSBSE 2015), 2015, pp. 301–307.
    2. 2)
      • 45. Fraser, G., Arcuri, A.: ‘1600 faults in 100 projects: automatically finding faults while achieving high coverage with evosuite’, Empir. Softw. Eng., 2015, 20, (3), pp. 611–639.
    3. 3)
      • 6. Xie, X., Chen, T. Y., Kuo, F.-C., et al: ‘A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization’, Trans. Soft. Eng. Methodol. (TOSEM), 2013, 22, (4), p. 31.
    4. 4)
      • 2. Jin, W., Orso, A.: ‘F3: fault localization for field failures’. Proc. the 2013 Int. Symp. on Software Testing and Analysis (ISSTA 2013), 2013, pp. 213–223.
    5. 5)
      • 4. Naish, L., Lee, H., Ramamohanarao, K.: ‘A model for spectra-based software diagnosis’, Trans. Softw. Eng. Methodol. (TOSEM), 2011, 20, (3), p. 11.
    6. 6)
      • 28. Lei, Y., Mao, X., Dai, Z., et al: ‘Effective fault localization approach using feedback’, IEICE Trans. Inf. Syst., 2012, 95, (9), pp. 2247–2257.
    7. 7)
      • 17. McNaught, A.D., Wilkinson, A.: ‘Compendium of chemical terminology’ (Blackwell Science, Oxford, 1997).
    8. 8)
      • 50. Gong, L., Lo, D., Jiang, L., et al: ‘Diversity maximization speedup for fault localization’. Proc. the 27th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2012), 2012, pp. 30–39.
    9. 9)
      • 48. Hao, D., Xie, T., Zhang, L., et al: ‘Test input reduction for result inspection to facilitate fault localization’, Autom. Softw. Eng., 2010, 17, (1), pp. 5–31.
    10. 10)
      • 42. Geronimo, L.D., Ferrucci, F., Murolo, A., et al: ‘A parallel genetic algorithm based on Hadoop MapReduce for the automatic generation of JUnit test suites’. Proc. the 5th Int. Conf. Software Testing, Verification and Validation (ICST 2012), 2012, pp. 785–793.
    11. 11)
      • 36. Perez, A., Abreu, R., Deursen, A.V.: ‘A test-suite diagnosability metric for spectrum-based fault localization approaches’. Proc. the 39th Int. Conf. Software Engineering (ICSE 2017), 2017, pp. 654–664.
    12. 12)
      • 59. Masri, W., Assi, R.A.: ‘Cleansing test suites from coincidental correctness to enhance fault-localization’. Proc. the 3rd Int. Conf. Software Testing, Verification and Validation (ICST 2010), 2010, pp. 165–174.
    13. 13)
      • 3. Wong, W.E., Debroy, V., Gao, R., et al: ‘The dstar method for effective software fault localization’, IEEE Trans. Reliab., 2014, 63, (1), pp. 290–308.
    14. 14)
      • 47. Yu, Y., Jones, J.A., Harrold, M.J.: ‘An empirical study of the effects of test-suite reduction on fault localization’. Proc. the 30th Int. Conf. Software Engineering (ICSE 2008), 2008, pp. 201–210.
    15. 15)
      • 43. Zhang, X., Tallam, S., Gupta, N., et al: ‘Towards locating execution omission errors’, ACM SIGPLAN Notices, 2007, 42, (6), pp. 415–424.
    16. 16)
      • 22. DiGiuseppe, N., Jones, J.: ‘On the influence of multiple faults on coverage-based fault localization’. Proc. the 2011 Int. Symp. on Software Testing and Analysis (ISSTA 2011), 2011, pp. 210–220.
    17. 17)
      • 44. Fraser, G., Staats, M., McMinn, P., et al: ‘Does automated unit test generation really help software testers? A controlled empirical study’, Trans. Softw. Eng. Methodol. (TOSEM), 2015, 24, (4), p. 23.
    18. 18)
      • 27. Wang, T., Roychoudhury, A.: ‘Automated path generation for software fault localization’. Proc. the 20th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2005), 2005, pp. 347–351.
    19. 19)
      • 25. Artzi, S., Dolby, J., Tip, F., et al: ‘Directed test generation for effective fault localization’. Proc. the 19th Int. Symp. on Software Testing and Analysis (ISSTA 2010), 2010, pp. 49–60.
    20. 20)
      • 14. Wong, W., Qi, Y., Zhao, L., et al: ‘Effective fault localization using code coverage’. Proc. the 31st Annual Int. Computer Software and Applications Conf. (COMPSAC 2007), 2007, pp. 449–456.
    21. 21)
      • 19. Wang, X., Cheung, S., Chan, W., et al: ‘Taming coincidental correctness: coverage refinement with context patterns to improve fault localization’. Proc. the 31st Int. Conf. Software Engineering (ICSE 2009), 2009, pp. 45–55.
    22. 22)
      • 26. Rößler, J., Fraser, G., Zeller, A., et al: ‘Isolating failure causes through test case generation’. Proc. the 2012 Int. Symp. on Software Testing and Analysis (ISSTA 2012), 2012, pp. 309–319.
    23. 23)
      • 58. Wang, R., Qu, B., Lu, Y.: ‘Empirical study of the effects of different profiles on regression test case reduction’, IET Softw., 2015, 9, (2), pp. 29–38.
    24. 24)
      • 37. Jost, L.: ‘Entropy and diversity’, Oikos, 2006, 113, (2), pp. 363–375.
    25. 25)
      • 52. Masri, W., Assi, R.A.: ‘Prevalence of coincidental correctness and mitigation of its impact on fault localization’, ACM Trans. Softw. Eng. Methodol., 2014, 23, (1), p. 8.
    26. 26)
      • 54. Dandan, G., Tiantian, W., Xiaohong, S., et al: ‘A test-suite reduction approach to improving fault-localization effectiveness’, Comput. Lang. Syst. Struct., 2013, 39, (3), pp. 95–108.
    27. 27)
      • 33. Zhang, L., Yan, L., Zhang, Z., et al: ‘A theoretical analysis on cloning the failed test cases to improve spectrum-based fault localization’, J. Syst. Softw., 2017, 129, pp. 35–57.
    28. 28)
      • 9. Moon, S., Kim, Y., Kim, M., et al: ‘Ask the mutants: mutating faulty programs for fault localization’. Proc. the 7th IEEE Int. Conf. Software Testing, Verification and Validation (ICST 2014), 2014, pp. 153–162.
    29. 29)
      • 12. Abreu, R., Zoeteweij, P., Van Gemund, A.: ‘On the accuracy of spectrum-based fault localization’. Proc. the Testing: Academic and Industrial Conf. Practice and Research Techniques – MUTATION, 2007, pp. 89–98.
    30. 30)
      • 1. Cleve, H., Zeller, A.: ‘Locating causes of program failures’. Proc. the 27th Int. Conf. Software Engineering (ICSE 2005), 2005, pp. 342–351.
    31. 31)
      • 18. Wohlinv, C., Runeson, P., Höst, M., et al: ‘Experimentation in software engineering’ (Springer, Berlin, 2012).
    32. 32)
      • 46. Arcuri, A., Fraser, G.: ‘On the effectiveness of whole test suite generation’. Proc. the 6th Int. Conf. Search Based Software Engineering (SSBSE 2014), 2014, pp. 1–15.
    33. 33)
      • 38. Inozemtseva, L., Holmes, R.: ‘Coverage is not strongly correlated with test suite effectiveness’. Proc. the 36th Int. Conf. Software Engineering (ICSE 2014), 2014, pp. 435–445.
    34. 34)
      • 32. Artzi, S., Dolby, J., Tip, F., et al: ‘Fault localization for dynamic web applications’, IEEE Trans. Softw. Eng., 2012, 38, (2), pp. 314–335.
    35. 35)
      • 51. Bandyopadhyay, A.: ‘Mitigating the effect of coincidental correctness in spectrum based fault localization’. Proc. the 5th Int. Conf. Software Testing, Verification and Validation (ICST 2012), 2012, pp. 479–482.
    36. 36)
      • 31. Abreu, R., Zoeteweij, P., Golsteijn, R., et al: ‘A practical evaluation of spectrum-based fault localization’, J. Syst. Softw., 2009, 82, (11), pp. 1780–1792.
    37. 37)
      • 30. Xuan, J., Monperrus, M.: ‘Test case purification for improving fault localization’. Proc. the 22nd ACM SIGSOFT Int. Symp. on Foundations of Software Engineering (FSE 2014), 2014, pp. 52–63.
    38. 38)
      • 11. Wong, W.E., Gao, R., Li, Y., et al: ‘A survey on software fault localization’, IEEE Trans. Softw. Eng., 2016, 42, (8), p. 1.
    39. 39)
      • 24. Baudry, B., Fleurey, F., Le Traon, Y.: ‘Improving test suites for efficient fault localization’. Proc. the 28th Int. Conf. Software Engineering (ICSE 2006), 2006, pp. 82–91.
    40. 40)
      • 40. McMinn, P.: ‘Search-based software test data generation: a survey’, Softw.Test. Verif. Reliab., 2004, 14, (2), pp. 105–156.
    41. 41)
      • 20. Corder, G.W., Foreman, D.I.: ‘Nonparametric statistics for non-statisticians: a step-by-step approach’ (John Wiley & Sons, Hoboken, 2009).
    42. 42)
      • 15. Parnin, C., Orso, A.: ‘Are automated debugging techniques actually helping programmers?’. Proc. the 2011 Int. Symp. on Software Testing and Analysis (ISSTA 2011), 2011, pp. 199–209.
    43. 43)
      • 16. Xie, X., Wong, W.E., Chen, T.Y., et al: ‘Metamorphic slice: an application in spectrum-based fault localization’, Inf. Softw. Technol., 2013, 55, (5), pp. 866–879.
    44. 44)
      • 56. Alipour, M.A., Shi, A., Gopinath, R., et al: ‘Evaluating non-adequate test-case reduction’. Proc. the 31st IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2016), 2016, pp. 16–26.
    45. 45)
      • 34. González-Sanchez, A., Gross, H., Van Gemund, A.J.C.: ‘Modeling the diagnostic efficiency of regression test suites’. Proc. the 4th IEEE Int. Conf. on Software Testing, Verification and Validation (ICST 2011), Workshop Proc., 2011, pp. 634–643.
    46. 46)
      • 8. Zhang, L., Zhang, L., Khurshid, S.: ‘Injecting mechanical faults to localize developer faults for evolving software’. Proc. the 2013 ACM SIGPLAN Int. Conf. Object Oriented Programming Systems Languages and Applications (OPPSLA 2013), 2013, pp. 765–784.
    47. 47)
      • 53. Masri, W., Abouassi, R., Elghali, M., et al: ‘An empirical study of the factors that reduce the effectiveness of coverage-based fault localization’. Proc. the 2009 Int. Workshop on Defects in Large Software Systems, 2009, pp. 1–5.
    48. 48)
      • 55. Yoo, S., Harman, M.: ‘Using hybrid algorithm for pareto efficient multi-objective test suite minimisation’, J. Syst. Softw., 2010, 83, (4), pp. 689–701.
    49. 49)
      • 7. Sun, C., Khoo, S.-C.: ‘Mining succinct predicated bug signatures’. Proc. the 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013), 2013, pp. 576–586.
    50. 50)
      • 35. González-Sanchez, A., Abreu, R., Gross, H., et al: ‘Prioritizing tests for fault localization through ambiguity group reduction’. Proc. the 26th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2011), 2011, pp. 83–92.
    51. 51)
      • 49. Zhao, L., Zhang, Z., Wang, L., et al: ‘PAFL: fault localization via noise reduction on coverage vector’. Proc. the 23rd Int. Conf. Software Engineering and Knowledge Engineering (SEKE 2011), 2011, pp. 203–206.
    52. 52)
      • 39. Harman, M.: ‘Automated test data generation using search based software engineering’. Proc. the 2nd Int. Workshop on Automation of Software Test (AST 2007), 2007, pp. 1–2.
    53. 53)
      • 5. Lei, Y., Mao, X., Dai, Z., et al: ‘Effective statistical fault localization using program slices’. Proc. the 36th Annual Int. Computer Software and Applications Conf. (COMPSAC 2012), 2012, pp. 1–10.
    54. 54)
      • 29. Campos, J., Abreu, R., Fraser, G., et al: ‘Entropy-based test generation for improved fault localization’. The 28th Int. Conf. Automated Software Engineering (ASE 2013), 2013, pp. 257–267.
    55. 55)
      • 21. Jones, J.A., Bowring, J.F., Harrold, J.F.: ‘Debugging in parallel’. Proc. the 2007 Int. Symp. on Software Testing and Analysis (ISSTA 2007), 2017, pp. 16–26.
    56. 56)
      • 13. Xie, X., Kuo, F.-C., Chen, T.Y., et al: ‘Provably optimal and human-competitive results in SBSE for spectrum based fault localisation’. Proc. the 5th Symp. on Search-Based Software Engineering (SSBSE 2013), 2013, pp. 224–238.
    57. 57)
      • 23. Steinder, M., Sethi, A.S.: ‘A survey of fault localization techniques in computer networks’, Sci. Comput. Program., 2004, 53, (2), pp. 165–194.
    58. 58)
      • 10. Papadakis, M., Traon, Y.L.: ‘Metallaxis-fl: mutation-based fault localization’, Softw. Test. Verif. Reliab., 2015, 25, pp. 605–628.
    59. 59)
      • 57. Lin, C.T., Tang, K.W., Wang, J.S., et al: ‘Empirically evaluating greedy-based test suite reduction methods at different levels of test suite complexity’, Sci. Comput. Program., 2017, 150, pp. 1–25.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen.2017.0026
Loading

Related content

content/journals/10.1049/iet-sen.2017.0026
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading