Combinatorial Testing

Mandl, 1985, Orthogonal Latin squares: an application of experiment design to compiler testing, Commun. ACM, 28, 1054, 10.1145/4372.4375 Tatsumi, 1987, Test-case design support system, 615 Tatsumi, 1987, Conceptual support for test case design, 285 Dalal, 1998, Factor-covering designs for testing software, Technometrics, 40, 234, 10.1080/00401706.1998.10485524 Wallace, 2001, Failure modes in medical device software: an analysis of 15 years of recall data, Int. J. Reliab. Qual. Saf. Eng., 8, 351, 10.1142/S021853930100058X Kuhn, 2002, An investigation of the applicability of design of experiments to software testing Kuhn, 2004, Software fault interactions and implications for software testing, IEEE Trans. Softw. Eng., 30, 418, 10.1109/TSE.2004.24 Bell, 2005, On effectiveness of pairwise methodology for testing network-centric software, 221 Bell, 2006 Zhang, 2011, Characterizing failure-causing parameter interactions by adaptive testing, 331 Chilenski, 2001 Kacker, 2013, Combinatorial testing for software: an adaptation of design of experiments, Measurement, 46, 3745, 10.1016/j.measurement.2013.02.021 Cochran, 1950 Kempthorne, 1952 Snedecor, 1967 Box, 1978 Montgomery, 2004 Fisher, 1925 Fisher, 1935 C. McQueary, Using design of experiments for operational test and evaluation, Memo, Office of the Secretary of Defense. www.dote.osd.mil/pub/policies/2009/200905UsingDoEforOTE_MOA.pdf, May 2009. Taguchi, 1986 Taguchi, 1987, vols. 1 and 2 Taguchi, 1993 Kackar, 1985, Off-line quality control, parameter design and the Taguchi method, J. Qual. Technol., 17, 176, 10.1080/00224065.1985.11978964 Phadke, 1989 Rao, 1947, Factorial experiments derivable from combinatorial arrangements of arrays, J. R. Stat. Soc. (Suppl.), 9, 128, 10.2307/2983576 Raghavarao, 1971 Hedayat, 1999 N.J.A. Sloane, Webpage. http://www2.research.att.com/~njas/oadir/. Sato, 1984, Methods for setting software test parameters using the design of experiments method (in Japanese), 1 Shimokawa, 1985 Sloane, 1993, Covering arrays and intersecting codes, J. Comb. Des., 1, 51, 10.1002/jcd.3180010106 Lawrence, 2011, A survey of binary covering arrays, Electron. J. Comb., 18, P84, 10.37236/571 Torres-Jimenez, 2012, New bounds for binary covering arrays using simulated annealing, Inform. Sci., 185, 137, 10.1016/j.ins.2011.09.020 D. Richard Kuhn, R.N. Kacker, Y. Lei, Practical combinatorial testing, NIST Special Publication 800-142. http://csrc.nist.gov/groups/SNS/acts/documents/SP800-142-101006.pdf, 2010. Kitsos, 2015 Britton, 2013 D.R. Kuhn, R.N. Kacker, Y. Lei, Combinatorial measurement tool user guide, Available online at http://csrc.nist.gov/groups/SNS/acts/documents/ComCoverage110130.pdf, 2011 (accessed 14.05.12). Kuhn, 2013, Combinatorial coverage measurement concepts and applications, 352 Ammann, 1994, Using formal methods to derive test frames in category-partition testing, 69 Maximoff, 2010, A method for analyzing system state-space coverage within a t-wise testing framework, 10.1109/SYSTEMS.2010.5482481 NIST Covering Array Tables, Available online at http://math.nist.gov/coveringarrays/ipof/ipof-results.html (accessed 23.03.15). C. Colbourn, Covering array tables for t=2,3,4,5,6. Available online at http://www.public.asu.edu/~ccolbou/src/tabby/catable.html (accessed 23.03.15). J. Torres-Jimenez, Covering arrays, Available at http://www.tamps.cinvestav.mx/~jtj/# (accessed 23.03.15). Bryce, 2006 Cohen, 2003, Constructing test suites for interaction testing, 28 Cohen, 2008, Constructing strength three covering arrays with augmented annealing, Disc. Math., 308, 2709, 10.1016/j.disc.2006.06.036 J. Czerwonka, Pairwise testing in the real world. http://msdn.microsoft.com/en-us/library/cc150619.aspx. Czerwonka, 2013, On use of coverage metrics in assessing effectiveness of combinatorial test designs, 257 Combinatorial methods in software testing, National Institute of Standards and Technology. http://csrc.nist.gov/acts. Satish, 2013, Deriving combinatorial test design model from UML activity diagram, 331 Qu, 2013, A study in prioritization for higher strength combinatorial testing, 285 Arcaini, 2014, Validation of models and tests for constrained combinatorial interaction testing, 98 Wu, 2014, Test suite prioritization by switching cost, 133 Farchi, 2014, Combinatorial testing with order requirements, 118 Gargantini, 2012, Citlab: a laboratory for combinatorial interaction testing, 559 Kuhn, 2013, Test parameter analysis Segall, 2012, Simplified modeling of combinatorial test spaces, 573 Segall, 2012, Common patterns in combinatorial models, 624 Arcuri, 2011, Formal analysis of the probability of interaction fault detection using random testing Grindal, 2006 Yu, 2013, An efficient algorithm for constraint handling in combinatorial test generation, 242 Cohen, 2007, Exploiting constraint solving history to construct interaction test suites, 121 Yu, 2014, Combinatorial test generation for software product lines using minimum invalid tuples, 65 IEEE, International workshop on combinatorial testing. http://ieeexplore.ieee.org/xpl/conhome.jsp?reload=true&punumber=1001832. World Wide Web Consortium, Document object model. http://www.w3.org/DOM/ (accessed 28.03.15). World Wide Web Consortium, DOM level 3 events specification. http://www.w3.org/TR/DOM-Level-3-Events/ (accessed 08.09.09). Montanez-Rivera, 2012, Evaluation of fault detection effectiveness for combinatorial and exhaustive selection of discretized test inputs, Softw. Qual. Prof., 14, 12 Hagar, 2014, Introducing combinatorial testing in a large organization: pilot project experience report [poster], 153 Cunningham, 2012, A system analysis study comparing reverse engineered combinatorial testing to expert judgment, 630 Hagar, 2015, Introducing combinatorial testing in a large organization, IEEE Comput., 48, 64, 10.1109/MC.2015.114 Bryce, 2011, Developing a single model and test prioritization strategies for event-driven software, Trans. Softw. Eng., 37, 48, 10.1109/TSE.2010.12 Rothermel, 2001, Prioritizing test cases for regression testing, IEEE Trans. Softw. Eng., 27, 929, 10.1109/32.962562 Sampath, 2011, A tool for combinatorial-based prioritization and reduction of user-session-based test suites, 574 Parnas, 1969, On the use of transition diagrams in the design of user interface for an interactive computer system, 379 Howden, 1996, Linear and structural event sequence analysis, International Symposium on Software Testing and Analysis, 98 Chow, 1978, Testing software design modeled by finite-state machines, IEEE Trans. Softw. Eng., 4, 178, 10.1109/TSE.1978.231496 Offutt, 2003, Generating test data from state-based specifications, J. Softw. Test. Verif. Reliab., 13, 25, 10.1002/stvr.264 Sarikaya, 1989, Conformance testing: architectures and test sequences, Comput. Netw. ISDN Syst., 17, 111, 10.1016/0169-7552(89)90004-4 Yuan, 2007, Covering array sampling of input event sequences for automated GUI testing, 405 Kuhn, 2012, Combinatorial methods for event sequence testing, 601 Kuhn, 2012, Combinatorial methods for event sequence testing, CrossTalk J. Defense Software Eng., 25, 15 Dushnik, 1950, Concerning a certain set of arrangements, Proc. Am. Math. Soc, 1, 788, 10.1090/S0002-9939-1950-0038922-4 Spencer, 1971, Minimal scrambling sets of simple orders, Acta Math. Acad. Sci. Hungary, 22, 349, 10.1007/BF01896428 Margalit, 2013, Better bounds for event sequencing testing, 281 Erdem, 2011, Answer-set programming as a new approach to event-sequence testing, 25 Chee, 2013, Sequence covering arrays, SIAM J. Discret. Math., 27, 1844, 10.1137/120894099 Yilmaz, 2004, Covering arrays for efficient fault characterization in complex configuration spaces, 45 Shi, 2005, A software debugging method based on pairwise testing, 1088 Wang, 2010, Adaptive interaction fault location based on combinatorial testing, 495 Ghandehari, 2012, Identifying failure-inducing combinations in a combinatorial test set, 370 Jones, 2002, Visualization of test information to assist fault localization, 467 Renieris, 2003, Fault localization with nearest neighbor queries, 30 Ghandehari, 2013, Fault localization based on failure-inducing combinations, 168 Bartholomew, 2013, An industry proof-of-concept demonstration of automated combinatorial test, 118 Bartholomew, 2014, Using combinatorial testing to reduce software rework, CrossTalk, 23, 23 Moy, 2013, Testing or formal verification: DO-178C alternatives and industrial experience, IEEE Softw., 30, 50, 10.1109/MS.2013.43 Cohen, 2007, Interaction testing of highly-configurable systems in the presence of constraints, 129 C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, Y.L. Traon, 2012, Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test suites for large software product lines. http://arxiv.org/pdf/1211.5451v1.pdf White, 2009, Selecting highly optimal architectural feature sets with filtered cartesian flattening, J. Syst. Softw., 82, 1268, 10.1016/j.jss.2009.02.011 Lee, 2012, A survey on software product line testing, 31 Xu, 2013, Continuous test suite augmentation in software product lines, 52 Garvin, 2013, Failure avoidance in configurable systems through feature locality, 266 Henard, 2015, Flattening or not the combinatorial interaction testing models?, 10.1109/ICSTW.2015.7107443 Li, 2014, Refining a randomized post-optimization method for covering arrays, 143 Segall, 2015, Combining minimization and generation for combinatorial testing, 10.1109/ICSTW.2015.7107434 Petke, 2015, Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection, IEEE Transactions on Software Engineering, 10.1109/TSE.2015.2421279 Yu, 2015, Constraint handling in combinatorial test generation using forbidden tuples, 10.1109/ICSTW.2015.7107441 Mayo, 2014, Test suite reduction by combinatorial-based coverage of event sequences, 128 Garn, 2014, Eris: a tool for combinatorial testing of the Linux system call interface, 58 Erick Lopez-Herrejon, 2015, A first systematic mapping study on combinatorial interaction testing for software product lines, 10.1109/ICSTW.2015.7107435 Satish, 2014, Extracting the combinatorial test parameters and values from UML sequence diagrams, 88 Zeppetzauer, 2014, Automating test case design within the classification tree editor, 1585