http://iet.metastore.ingenta.com
1887

How test suites impact fault localisation starting from the size

How test suites impact fault localisation starting from the size

For access to this article, please select a purchase option:

Buy article PDF
£12.50
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Although a test suite is indispensable for conducting effective fault localisation, not much work has been done to study how the test suite impacts fault localisation. This study presents a systematic study for a deeper understanding of their relation. Specifically, the authors' study reveals an interesting fact that there is no strong correlation between localisation effectiveness and the size of the test suite. Furthermore, they show that, in a test suite, (i) the passing test cases that do not execute the faulty statements and the failing test cases have a positive effect on the fault localisation effectiveness, while (ii) the passing test cases that exercise the faulty statements have a negative impact on localisation performance. Their result is drawn from a large-scale empirical analysis on the localisation effectiveness with respect to randomly sampled test suites. This study presents the details of the study and their follow-up investigation on the findings. Their work provides a new perspective on fault localisation and suggests fresh directions of research on an extensively studied topic.

References

    1. 1)
      • 1. Cleve, H., Zeller, A.: ‘Locating causes of program failures’. Proc. the 27th Int. Conf. Software Engineering (ICSE 2005), 2005, pp. 342351.
    2. 2)
      • 2. Jin, W., Orso, A.: ‘F3: fault localization for field failures’. Proc. the 2013 Int. Symp. on Software Testing and Analysis (ISSTA 2013), 2013, pp. 213223.
    3. 3)
      • 3. Wong, W.E., Debroy, V., Gao, R., et al: ‘The dstar method for effective software fault localization’, IEEE Trans. Reliab., 2014, 63, (1), pp. 290308.
    4. 4)
      • 4. Naish, L., Lee, H., Ramamohanarao, K.: ‘A model for spectra-based software diagnosis’, Trans. Softw. Eng. Methodol. (TOSEM), 2011, 20, (3), p. 11.
    5. 5)
      • 5. Lei, Y., Mao, X., Dai, Z., et al: ‘Effective statistical fault localization using program slices’. Proc. the 36th Annual Int. Computer Software and Applications Conf. (COMPSAC 2012), 2012, pp. 110.
    6. 6)
      • 6. Xie, X., Chen, T. Y., Kuo, F.-C., et al: ‘A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization’, Trans. Soft. Eng. Methodol. (TOSEM), 2013, 22, (4), p. 31.
    7. 7)
      • 7. Sun, C., Khoo, S.-C.: ‘Mining succinct predicated bug signatures’. Proc. the 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2013), 2013, pp. 576586.
    8. 8)
      • 8. Zhang, L., Zhang, L., Khurshid, S.: ‘Injecting mechanical faults to localize developer faults for evolving software’. Proc. the 2013 ACM SIGPLAN Int. Conf. Object Oriented Programming Systems Languages and Applications (OPPSLA 2013), 2013, pp. 765784.
    9. 9)
      • 9. Moon, S., Kim, Y., Kim, M., et al: ‘Ask the mutants: mutating faulty programs for fault localization’. Proc. the 7th IEEE Int. Conf. Software Testing, Verification and Validation (ICST 2014), 2014, pp. 153162.
    10. 10)
      • 10. Papadakis, M., Traon, Y.L.: ‘Metallaxis-fl: mutation-based fault localization’, Softw. Test. Verif. Reliab., 2015, 25, pp. 605628.
    11. 11)
      • 11. Wong, W.E., Gao, R., Li, Y., et al: ‘A survey on software fault localization’, IEEE Trans. Softw. Eng., 2016, 42, (8), p. 1.
    12. 12)
      • 12. Abreu, R., Zoeteweij, P., Van Gemund, A.: ‘On the accuracy of spectrum-based fault localization’. Proc. the Testing: Academic and Industrial Conf. Practice and Research Techniques – MUTATION, 2007, pp. 8998.
    13. 13)
      • 13. Xie, X., Kuo, F.-C., Chen, T.Y., et al: ‘Provably optimal and human-competitive results in SBSE for spectrum based fault localisation’. Proc. the 5th Symp. on Search-Based Software Engineering (SSBSE 2013), 2013, pp. 224238.
    14. 14)
      • 14. Wong, W., Qi, Y., Zhao, L., et al: ‘Effective fault localization using code coverage’. Proc. the 31st Annual Int. Computer Software and Applications Conf. (COMPSAC 2007), 2007, pp. 449456.
    15. 15)
      • 15. Parnin, C., Orso, A.: ‘Are automated debugging techniques actually helping programmers?’. Proc. the 2011 Int. Symp. on Software Testing and Analysis (ISSTA 2011), 2011, pp. 199209.
    16. 16)
      • 16. Xie, X., Wong, W.E., Chen, T.Y., et al: ‘Metamorphic slice: an application in spectrum-based fault localization’, Inf. Softw. Technol., 2013, 55, (5), pp. 866879.
    17. 17)
      • 17. McNaught, A.D., Wilkinson, A.: ‘Compendium of chemical terminology’ (Blackwell Science, Oxford, 1997).
    18. 18)
      • 18. Wohlinv, C., Runeson, P., Höst, M., et al: ‘Experimentation in software engineering’ (Springer, Berlin, 2012).
    19. 19)
      • 19. Wang, X., Cheung, S., Chan, W., et al: ‘Taming coincidental correctness: coverage refinement with context patterns to improve fault localization’. Proc. the 31st Int. Conf. Software Engineering (ICSE 2009), 2009, pp. 4555.
    20. 20)
      • 20. Corder, G.W., Foreman, D.I.: ‘Nonparametric statistics for non-statisticians: a step-by-step approach’ (John Wiley & Sons, Hoboken, 2009).
    21. 21)
      • 21. Jones, J.A., Bowring, J.F., Harrold, J.F.: ‘Debugging in parallel’. Proc. the 2007 Int. Symp. on Software Testing and Analysis (ISSTA 2007), 2017, pp. 1626.
    22. 22)
      • 22. DiGiuseppe, N., Jones, J.: ‘On the influence of multiple faults on coverage-based fault localization’. Proc. the 2011 Int. Symp. on Software Testing and Analysis (ISSTA 2011), 2011, pp. 210220.
    23. 23)
      • 23. Steinder, M., Sethi, A.S.: ‘A survey of fault localization techniques in computer networks’, Sci. Comput. Program., 2004, 53, (2), pp. 165194.
    24. 24)
      • 24. Baudry, B., Fleurey, F., Le Traon, Y.: ‘Improving test suites for efficient fault localization’. Proc. the 28th Int. Conf. Software Engineering (ICSE 2006), 2006, pp. 8291.
    25. 25)
      • 25. Artzi, S., Dolby, J., Tip, F., et al: ‘Directed test generation for effective fault localization’. Proc. the 19th Int. Symp. on Software Testing and Analysis (ISSTA 2010), 2010, pp. 4960.
    26. 26)
      • 26. Rößler, J., Fraser, G., Zeller, A., et al: ‘Isolating failure causes through test case generation’. Proc. the 2012 Int. Symp. on Software Testing and Analysis (ISSTA 2012), 2012, pp. 309319.
    27. 27)
      • 27. Wang, T., Roychoudhury, A.: ‘Automated path generation for software fault localization’. Proc. the 20th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2005), 2005, pp. 347351.
    28. 28)
      • 28. Lei, Y., Mao, X., Dai, Z., et al: ‘Effective fault localization approach using feedback’, IEICE Trans. Inf. Syst., 2012, 95, (9), pp. 22472257.
    29. 29)
      • 29. Campos, J., Abreu, R., Fraser, G., et al: ‘Entropy-based test generation for improved fault localization’. The 28th Int. Conf. Automated Software Engineering (ASE 2013), 2013, pp. 257267.
    30. 30)
      • 30. Xuan, J., Monperrus, M.: ‘Test case purification for improving fault localization’. Proc. the 22nd ACM SIGSOFT Int. Symp. on Foundations of Software Engineering (FSE 2014), 2014, pp. 5263.
    31. 31)
      • 31. Abreu, R., Zoeteweij, P., Golsteijn, R., et al: ‘A practical evaluation of spectrum-based fault localization’, J. Syst. Softw., 2009, 82, (11), pp. 17801792.
    32. 32)
      • 32. Artzi, S., Dolby, J., Tip, F., et al: ‘Fault localization for dynamic web applications’, IEEE Trans. Softw. Eng., 2012, 38, (2), pp. 314335.
    33. 33)
      • 33. Zhang, L., Yan, L., Zhang, Z., et al: ‘A theoretical analysis on cloning the failed test cases to improve spectrum-based fault localization’, J. Syst. Softw., 2017, 129, pp. 3557.
    34. 34)
      • 34. González-Sanchez, A., Gross, H., Van Gemund, A.J.C.: ‘Modeling the diagnostic efficiency of regression test suites’. Proc. the 4th IEEE Int. Conf. on Software Testing, Verification and Validation (ICST 2011), Workshop Proc., 2011, pp. 634643.
    35. 35)
      • 35. González-Sanchez, A., Abreu, R., Gross, H., et al: ‘Prioritizing tests for fault localization through ambiguity group reduction’. Proc. the 26th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2011), 2011, pp. 8392.
    36. 36)
      • 36. Perez, A., Abreu, R., Deursen, A.V.: ‘A test-suite diagnosability metric for spectrum-based fault localization approaches’. Proc. the 39th Int. Conf. Software Engineering (ICSE 2017), 2017, pp. 654664.
    37. 37)
      • 37. Jost, L.: ‘Entropy and diversity’, Oikos, 2006, 113, (2), pp. 363375.
    38. 38)
      • 38. Inozemtseva, L., Holmes, R.: ‘Coverage is not strongly correlated with test suite effectiveness’. Proc. the 36th Int. Conf. Software Engineering (ICSE 2014), 2014, pp. 435445.
    39. 39)
      • 39. Harman, M.: ‘Automated test data generation using search based software engineering’. Proc. the 2nd Int. Workshop on Automation of Software Test (AST 2007), 2007, pp. 12.
    40. 40)
      • 40. McMinn, P.: ‘Search-based software test data generation: a survey’, Softw.Test. Verif. Reliab., 2004, 14, (2), pp. 105156.
    41. 41)
      • 41. Zhang, Y., Harman, M., Jia, Y., et al: ‘Inferring test models from Kate's bug reports using multi-objective search’. Proc. the 7th Int. Conf. Search Based Software Engineering (SSBSE 2015), 2015, pp. 301307.
    42. 42)
      • 42. Geronimo, L.D., Ferrucci, F., Murolo, A., et al: ‘A parallel genetic algorithm based on Hadoop MapReduce for the automatic generation of JUnit test suites’. Proc. the 5th Int. Conf. Software Testing, Verification and Validation (ICST 2012), 2012, pp. 785793.
    43. 43)
      • 43. Zhang, X., Tallam, S., Gupta, N., et al: ‘Towards locating execution omission errors’, ACM SIGPLAN Notices, 2007, 42, (6), pp. 415424.
    44. 44)
      • 44. Fraser, G., Staats, M., McMinn, P., et al: ‘Does automated unit test generation really help software testers? A controlled empirical study’, Trans. Softw. Eng. Methodol. (TOSEM), 2015, 24, (4), p. 23.
    45. 45)
      • 45. Fraser, G., Arcuri, A.: ‘1600 faults in 100 projects: automatically finding faults while achieving high coverage with evosuite’, Empir. Softw. Eng., 2015, 20, (3), pp. 611639.
    46. 46)
      • 46. Arcuri, A., Fraser, G.: ‘On the effectiveness of whole test suite generation’. Proc. the 6th Int. Conf. Search Based Software Engineering (SSBSE 2014), 2014, pp. 115.
    47. 47)
      • 47. Yu, Y., Jones, J.A., Harrold, M.J.: ‘An empirical study of the effects of test-suite reduction on fault localization’. Proc. the 30th Int. Conf. Software Engineering (ICSE 2008), 2008, pp. 201210.
    48. 48)
      • 48. Hao, D., Xie, T., Zhang, L., et al: ‘Test input reduction for result inspection to facilitate fault localization’, Autom. Softw. Eng., 2010, 17, (1), pp. 531.
    49. 49)
      • 49. Zhao, L., Zhang, Z., Wang, L., et al: ‘PAFL: fault localization via noise reduction on coverage vector’. Proc. the 23rd Int. Conf. Software Engineering and Knowledge Engineering (SEKE 2011), 2011, pp. 203206.
    50. 50)
      • 50. Gong, L., Lo, D., Jiang, L., et al: ‘Diversity maximization speedup for fault localization’. Proc. the 27th IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2012), 2012, pp. 3039.
    51. 51)
      • 51. Bandyopadhyay, A.: ‘Mitigating the effect of coincidental correctness in spectrum based fault localization’. Proc. the 5th Int. Conf. Software Testing, Verification and Validation (ICST 2012), 2012, pp. 479482.
    52. 52)
      • 52. Masri, W., Assi, R.A.: ‘Prevalence of coincidental correctness and mitigation of its impact on fault localization’, ACM Trans. Softw. Eng. Methodol., 2014, 23, (1), p. 8.
    53. 53)
      • 53. Masri, W., Abouassi, R., Elghali, M., et al: ‘An empirical study of the factors that reduce the effectiveness of coverage-based fault localization’. Proc. the 2009 Int. Workshop on Defects in Large Software Systems, 2009, pp. 15.
    54. 54)
      • 54. Dandan, G., Tiantian, W., Xiaohong, S., et al: ‘A test-suite reduction approach to improving fault-localization effectiveness’, Comput. Lang. Syst. Struct., 2013, 39, (3), pp. 95108.
    55. 55)
      • 55. Yoo, S., Harman, M.: ‘Using hybrid algorithm for pareto efficient multi-objective test suite minimisation’, J. Syst. Softw., 2010, 83, (4), pp. 689701.
    56. 56)
      • 56. Alipour, M.A., Shi, A., Gopinath, R., et al: ‘Evaluating non-adequate test-case reduction’. Proc. the 31st IEEE/ACM Int. Conf. Automated Software Engineering (ASE 2016), 2016, pp. 1626.
    57. 57)
      • 57. Lin, C.T., Tang, K.W., Wang, J.S., et al: ‘Empirically evaluating greedy-based test suite reduction methods at different levels of test suite complexity’, Sci. Comput. Program., 2017, 150, pp. 125.
    58. 58)
      • 58. Wang, R., Qu, B., Lu, Y.: ‘Empirical study of the effects of different profiles on regression test case reduction’, IET Softw., 2015, 9, (2), pp. 2938.
    59. 59)
      • 59. Masri, W., Assi, R.A.: ‘Cleansing test suites from coincidental correctness to enhance fault-localization’. Proc. the 3rd Int. Conf. Software Testing, Verification and Validation (ICST 2010), 2010, pp. 165174.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen.2017.0026
Loading

Related content

content/journals/10.1049/iet-sen.2017.0026
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address