http://iet.metastore.ingenta.com
1887

Improved pairwise test suites for non-prime-power orders

Improved pairwise test suites for non-prime-power orders

For access to this article, please select a purchase option:

Buy article PDF
£12.50
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Software — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Software testing has become a critical component of the modern software development process. Therefore, a lot of research has been done in this area in recent years, and as a result new algorithms, methodologies, and tools have been created. One of the most used testing strategies is pairwise testing; this technique ensures that all possible combinations of values between any two input parameters are covered by at least one test. In this work, a new algorithm called add factor and stochastic optimisation (AFSO) is used to build small pairwise test suites for non-prime-power orders. Starting from an orthogonal array of order , AFSO iteratively adds a factor and then reduces to zero the number of uncovered combinations by means of a simulated annealing algorithm. The results of the AFSO algorithm improved the size of 92 pairwise test suites with non-prime-power orders. One of these improved test suites is used in a real-word application to show the usefulness of the new results.

References

    1. 1)
      • D.R. Kuhn , R.N. Kacker , Y. Lei . (2010)
        1. Kuhn, D.R., Kacker, R.N., Lei, Y.: ‘Practical combinatorial testing’ (National Institute of Standards & Technology, Gaithersburg, MD, USA, 2010).
        .
    2. 2)
      • P. Yang , X. Tan , H. Sun .
        2. Yang, P., Tan, X., Sun, H., et al: ‘Fire accident reconstruction based on LES field model by using orthogonal experimental design method’, Adv. Eng. Softw., 2011, 42, (11), pp. 954962.
        . Adv. Eng. Softw. , 11 , 954 - 962
    3. 3)
      • C.H. Chen , F.J. Shiou .
        3. Chen, C.H., Shiou, F.J.: ‘Determination of optimal ball-burnishing parameters for plastic injection moulding steel’, Int. J. Adv. Manuf. Technol., 2003, 21, (3), pp. 177185.
        . Int. J. Adv. Manuf. Technol. , 3 , 177 - 185
    4. 4)
      • D.E. Shasha , A.Y. Kouranov , L.V. Lejay .
        4. Shasha, D.E., Kouranov, A.Y., Lejay, L.V., et al: ‘Using combinatorial design to study regulation by multiple input signals: a tool for parsimony in the post-genomics era’, Plant Physiol., 2001, 127, (4), pp. 15901594.
        . Plant Physiol. , 4 , 1590 - 1594
    5. 5)
      • K.K. Vadde , V.R. Syrotiuk .
        5. Vadde, K.K., Syrotiuk, V.R.: ‘Factor interaction on service delivery in mobile ad hoc networks’, IEEE J. Sel. Areas Commun., 2004, 22, (7), pp. 13351346.
        . IEEE J. Sel. Areas Commun. , 7 , 1335 - 1346
    6. 6)
      • S.R. Dalal , A. Jain , N. Karunanithi .
        6. Dalal, S.R., Jain, A., Karunanithi, N., et al: ‘Model-based testing in practice’. Proc. 21st Int. Conf. Software Engineering, Los Angeles, California, USA, May 1999, pp. 285294.
        . Proc. 21st Int. Conf. Software Engineering , 285 - 294
    7. 7)
      • D.R. Wallace , D.R. Kuhn .
        7. Wallace, D.R., Kuhn, D.R.: ‘Failure modes in medical device software: an analysis of 15 years of recall data’, Int. J. Reliab. Qual. Safety Eng., 2001, 8, (4), pp. 351371.
        . Int. J. Reliab. Qual. Safety Eng. , 4 , 351 - 371
    8. 8)
      • A.P. Godbole , D.E. Skipper , R.A. Sunley .
        8. Godbole, A.P., Skipper, D.E., Sunley, R.A.: ‘t-covering arrays: upper bounds and Poisson approximations’, Comb. Probab. Comput., 1996, 5, (2), pp. 105117.
        . Comb. Probab. Comput. , 2 , 105 - 117
    9. 9)
      • K. Sarkar , C.J. Colbourn , A. de Bonis .
        9. Sarkar, K., Colbourn, C.J., de Bonis, A., et al: ‘Partial covering arrays: algorithms and asymptotics’. Int. Workshop on Comb. Algorithms, Helsinki, Finland, August 2016, pp. 437448.
        . Int. Workshop on Comb. Algorithms , 437 - 448
    10. 10)
      • N. Francetić , B. Stevens .
        10. Francetić, N., Stevens, B.: ‘Asymptotic size of covering arrays: an application of entropy compression’, J. Comb. Des., 2017, 25, (6), pp. 243257.
        . J. Comb. Des. , 6 , 243 - 257
    11. 11)
      • K.A. Bush .
        11. Bush, K.A.: ‘Orthogonal arrays of index unity’, Ann. Math. Stat., 1952, 23, (3), pp. 426434.
        . Ann. Math. Stat. , 3 , 426 - 434
    12. 12)
      • C.J. Colbourn , G. Kéri , P.P. Rivas Soriano .
        12. Colbourn, C.J., Kéri, G., Rivas Soriano, P.P., et al: ‘Covering and radius-covering arrays: constructions and classification’, Discrete Appl. Math., 2010, 158, (11), pp. 11581180.
        . Discrete Appl. Math. , 11 , 1158 - 1180
    13. 13)
      • C.J. Colbourn .
        13. Colbourn, C.J.: ‘Strength two covering arrays: existence tables and projection’, Discrete Math., 2008, 308, (5), pp. 772786.
        . Discrete Math. , 5 , 772 - 786
    14. 14)
      • A.H. Ronneseth , C.J. Colbourn .
        14. Ronneseth, A.H., Colbourn, C.J.: ‘Merging covering arrays and compressing multiple sequence alignments’, Discrete Appl. Math., 2009, 157, (9), pp. 21772190.
        . Discrete Appl. Math. , 9 , 2177 - 2190
    15. 15)
      • M.I. Younis , K.Z. Zamli , M.F.J. Klaib .
        15. Younis, M.I., Zamli, K.Z., Klaib, M.F.J., et al: ‘Assessing IRPS as an efficient pairwise test data generation strategy’, Int. J. Adv. Intell. Paradigms, 2010, 2, (1), pp. 90104.
        . Int. J. Adv. Intell. Paradigms , 1 , 90 - 104
    16. 16)
      • A. Calvagna , A. Gargantini .
        16. Calvagna, A., Gargantini, A.: ‘T-wise combinatorial interaction test suites construction based on coverage inheritance’, Softw. Test. Verif. Reliab., 2012, 22, pp. 507526.
        . Softw. Test. Verif. Reliab. , 507 - 526
    17. 17)
      • Y. Lei , K.C. Tai .
        17. Lei, Y., Tai, K.C.: ‘In-parameter-order: a test generation strategy for pairwise testing’. Proc. 3rd IEEE Int. Symp. High-Assurance Systems Engineering, Washington, DC, USA, August 1998, pp. 254261.
        . Proc. 3rd IEEE Int. Symp. High-Assurance Systems Engineering , 254 - 261
    18. 18)
      • R.C. Bryce , C.J. Colbourn .
        18. Bryce, R.C., Colbourn, C.J.: ‘The density algorithm for pairwise interaction testing’, Softw. Test. Verif. Reliab., 2007, 17, (3), pp. 159182.
        . Softw. Test. Verif. Reliab. , 3 , 159 - 182
    19. 19)
      • M.B. Cohen , C.J. Colbourn , A.C.H. Ling .
        19. Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: ‘Augmenting simulated annealing to build interaction test suites’. Proc. 14th Int. Symp. Software Reliability Engineering, Denver, CO, USA, November 2003, pp. 394405.
        . Proc. 14th Int. Symp. Software Reliability Engineering , 394 - 405
    20. 20)
      • J. Torres-Jimenez , E. Rodriguez-Tello .
        20. Torres-Jimenez, J., Rodriguez-Tello, E.: ‘New bounds for binary covering arrays using simulated annealing’, Inf. Sci., 2012, 185, (1), pp. 137152.
        . Inf. Sci. , 1 , 137 - 152
    21. 21)
      • K.J. Nurmela .
        21. Nurmela, K.J.: ‘Upper bounds for covering arrays by tabu search’, Discrete Appl. Math., 2004, 138, pp. 143152.
        . Discrete Appl. Math. , 143 - 152
    22. 22)
      • G.B. Sherwood .
        22. Sherwood, G.B.: ‘Getting the most from pairwise testing: a guide for practicing software engineers’, Testcover, 2011.
        .
    23. 23)
      • G. Tarry .
        23. Tarry, G.: ‘Le problème des 36 officiers’, Compte Rendu de l'Association Française pour l'Avancement de Science Naturel, 1901, 29, (2), pp. 170203.
        . Compte Rendu de l'Association Française pour l'Avancement de Science Naturel , 2 , 170 - 203
    24. 24)
      • H. Avila-George , J. Torres-Jimenez , N. Rangel-Valdez .
        24. Avila-George, H., Torres-Jimenez, J., Rangel-Valdez, N., et al: ‘Supercomputing and grid computing on the verification of covering arrays’, J. Supercomputing, 2012, 62, (2), pp. 916945.
        . J. Supercomputing , 2 , 916 - 945
    25. 25)
      • H. Avila-George , J. Torres-Jimenez , L. Gonzalez-Hernandez .
        25. Avila-George, H., Torres-Jimenez, J., Gonzalez-Hernandez, L., et al: ‘Metaheuristic approach for constructing functional test-suites’, IET Softw., 2013, 7, (2), pp. 104117.
        . IET Softw. , 2 , 104 - 117
    26. 26)
      • E.T. Parker .
        26. Parker, E.T.: ‘Orthogonal Latin squares’, Proc. Natl. Acad. Sci., 1959, 45, (6), pp. 859862.
        . Proc. Natl. Acad. Sci. , 6 , 859 - 862
    27. 27)
      • D.M. Johnson , A.L. Dulmage , N.S. Mendelsohn .
        27. Johnson, D.M., Dulmage, A.L., Mendelsohn, N.S.: ‘Orthomorphisms of groups and orthogonal Latin squares’, Can. J. Math., 1961, 13, pp. 356372.
        . Can. J. Math. , 356 - 372
    28. 28)
      • D.T. Todorov .
        28. Todorov, D.T.: ‘Four mutually orthogonal Latin squares of order 14’, J. Comb. Des., 2012, 20, (8), pp. 363367.
        . J. Comb. Des. , 8 , 363 - 367
    29. 29)
      • P.J. Schellenberg , G.H.J. Van Rees , S.A. Vanstone .
        29. Schellenberg, P.J., Van Rees, G.H.J., Vanstone, S.A.: ‘Four pairwise orthogonal Latin squares of order 15’, Ars Combinatoria, 1978, 6, pp. 141150.
        . Ars Combinatoria , 141 - 150
    30. 30)
      • R.J.R. Abel .
        30. Abel, R.J.R.: ‘Existence of five MOLS of orders 18 and 60’, J. Comb. Des., 2015, 23, (4), pp. 135139.
        . J. Comb. Des. , 4 , 135 - 139
    31. 31)
      • D.T. Todorov .
        31. Todorov, D.T.: ‘Four mutually orthogonal Latin squares of order 20’, Ars Combinatoria, 1989, 27, pp. 6365.
        . Ars Combinatoria , 63 - 65
    32. 32)
      • R.J.R. Abel , D.T. Todorov .
        32. Abel, R.J.R., Todorov, D.T.: ‘Four MOLS of orders 20, 30, 38, and 44’, J. Comb. Theory A, 1993, 64, (1), pp. 144148.
        . J. Comb. Theory A , 1 , 144 - 148
    33. 33)
      • A.V. Nazarok .
        33. Nazarok, A.V.: ‘Five pairwise orthogonal Latin squares of order 21’, Issled. oper. i ASU, 1991, 1, pp. 5456.
        . Issled. oper. i ASU , 54 - 56
    34. 34)
      • R.J.R. Abel , X. Zhang , H. Zhang .
        34. Abel, R.J.R., Zhang, X., Zhang, H.: ‘Three mutually orthogonal idempotent Latin squares of orders 22 and 26’, J. Stat. Plan. Inference, 1996, 51, (2), pp. 101106.
        . J. Stat. Plan. Inference , 2 , 101 - 106
    35. 35)
      • R.J.R. Abel , C.J. Colbourn , M. Wojtas .
        35. Abel, R.J.R., Colbourn, C.J., Wojtas, M.: ‘Concerning seven and eight mutually orthogonal Latin squares’, J. Comb. Des., 2004, 12, (2), pp. 123131.
        . J. Comb. Des. , 2 , 123 - 131
    36. 36)
      • T. Developers . (2016)
        36. Developers, T.: ‘The SAGE mathematics software system (version 7.1)’ (SageMath, 2016).
        .
    37. 37)
      • C.J. Colbourn , S.S. Martirosyan , G.L. Mullen .
        37. Colbourn, C.J., Martirosyan, S.S., Mullen, G.L., et al: ‘Products of mixed covering arrays of strength two’, J. Comb. Des., 2006, 12, (2), pp. 124138.
        . J. Comb. Des. , 2 , 124 - 138
    38. 38)
      • C.J. Colbourn , J. Torres-Jimenez .
        38. Colbourn, C.J., Torres-Jimenez, J.: ‘Heterogeneous hash families and covering arrays’, Contemp. Math., 2010, 523, pp. 315.
        . Contemp. Math. , 3 - 15
    39. 39)
      • P. Quiz-Ramos , J. Torres-Jimenez , N. Rangel-Valdez .
        39. Quiz-Ramos, P., Torres-Jimenez, J., Rangel-Valdez, N.: ‘Constant Row maximizing problem for covering arrays’. Proc. Eighth Mexican Int. Conf. Artificial Intelligence, Guanajuato, Mexico, February 2009, pp. 159164.
        . Proc. Eighth Mexican Int. Conf. Artificial Intelligence , 159 - 164
    40. 40)
      • C.J. Colbourn .
        40. Colbourn, C.J.: ‘Augmentation of covering arrays of strength two’, Graphs Comb., 2015, 31, (6), pp. 21372147.
        . Graphs Comb. , 6 , 2137 - 2147
    41. 41)
      • B. Smith , W. Millar , J. Dunphy .
        41. Smith, B., Millar, W., Dunphy, J., et al: ‘Validation and verification of the remote agent for spacecraft autonomy’. IEEE Aerospace Conf., Snowmass at Aspen, CO, USA, August 1999, pp. 449468.
        . IEEE Aerospace Conf., Snowmass at Aspen, CO , 449 - 468
    42. 42)
      • D.R. Kuhn , V. Okun .
        42. Kuhn, D.R., Okun, V.: ‘Pseudo-exhaustive testing for software’. Proc. 30th Annual IEEE/NASA Software Engineering Workshop, Columbia, MD, USA, April 2006, pp. 153158.
        . Proc. 30th Annual IEEE/NASA Software Engineering Workshop , 153 - 158
    43. 43)
      • M.B. Cohen , M.B. Dwyer , J. Shi .
        43. Cohen, M.B., Dwyer, M.B., Shi, J.: ‘Interaction testing of highly-configurable systems in the presence of constraints’. Proc. 2007 Int. Symp. Software Testing and Analysis, London, UK, July 2007, pp. 129139.
        . Proc. 2007 Int. Symp. Software Testing and Analysis , 129 - 139
    44. 44)
      • M.B. Cohen , M.B. Dwyer , J. Shi .
        44. Cohen, M.B., Dwyer, M.B., Shi, J.: ‘Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach’, IEEE Trans. Softw. Eng., 2008, 34, (5), pp. 633650.
        . IEEE Trans. Softw. Eng. , 5 , 633 - 650
    45. 45)
      • J. Yan , J. Zhang .
        45. Yan, J., Zhang, J.: ‘A backtracking search tool for constructing combinatorial test suites’, J. Syst. Softw., 2008, 81, (10), pp. 16811693.
        . J. Syst. Softw. , 10 , 1681 - 1693
    46. 46)
      • I. Segall , R. Tzoref-Brill , E. Farchi .
        46. Segall, I., Tzoref-Brill, R., Farchi, E.: ‘Using binary decision diagrams for combinatorial test design’. Proc. 2011 Int. Symp. Software Testing and Analysis, Toronto, ON, Canada, July 2011, pp. 254264.
        . Proc. 2011 Int. Symp. Software Testing and Analysis , 254 - 264
    47. 47)
      • M.I. Younis , K.Z. Zamli .
        47. Younis, M.I., Zamli, K.Z.: ‘MIPOG - an efficient t-way minimization strategy for combinatorial testing’, Int. J. Comput. Theory Eng., 2011, 3, (3), pp. 388397.
        . Int. J. Comput. Theory Eng. , 3 , 388 - 397
    48. 48)
      • B.S. Ahmed , K.Z. Zamli , C.P. Lim .
        48. Ahmed, B.S., Zamli, K.Z., Lim, C.P.: ‘Application of particle swarm optimization to uniform and variable strength covering array construction’, Appl. Soft Comput., 2012, 12, (4), pp. 13301347.
        . Appl. Soft Comput. , 4 , 1330 - 1347
    49. 49)
      • D. Hillmer .
        49. Hillmer, D.: ‘Introducing combinatorial testing in the organization a report on a first attempt’. IEEE Eighth Int. Conf. Software Testing, Verification and Validation Workshops, Graz, Austria, May 2015, pp. 19.
        . IEEE Eighth Int. Conf. Software Testing, Verification and Validation Workshops , 1 - 9
    50. 50)
      • J. Torres-Jimenez , A. Rodriguez-Cristerna .
        50. Torres-Jimenez, J., Rodriguez-Cristerna, A.: ‘Metaheuristic post-optimization of the NIST repository of covering arrays’, CAAI Trans. Intell. Technol., 2017, 2, (1), pp. 3138.
        . CAAI Trans. Intell. Technol. , 1 , 31 - 38
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-sen.2017.0107
Loading

Related content

content/journals/10.1049/iet-sen.2017.0107
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address