Higher order mutation testing: A Systematic Literature Review

Bezier, 1990 Myers, 2004 Copeland, 2003 Binder, 1999 Kapfhammer, 2004 Schach, 1996, Testing: Principles and practice, ACM Comput. Surv., 28, 277, 10.1145/234313.234422 Zhu, 1997, Software unit test coverage and adequacy, ACM Comput. Surv., 29, 366, 10.1145/267580.267590 DeMillo, 1978, Hints on test data selection: Help for the practicing programmer, Computer, 11, 34, 10.1109/C-M.1978.218136 Hamlet, 1977, Testing programs with the aid of a compiler, IEEE Trans. Softw. Eng., 3, 279, 10.1109/TSE.1977.231145 M. Harman, Y. Jia, W.B. Langdon, A manifesto for higher order mutation testing, in: Third International Conference on Software Testing, Verification, and Validation Workshops, ICSTW, 2010, pp. 80–89. Howden, 1982, Weak mutation testing and completeness of test sets, IEEE Trans. Softw. Eng., 8, 371, 10.1109/TSE.1982.235571 D. Schuler, A. Zeller, Javalanche: Efficient mutation testing for java, in: Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering, ESEC/FSE’09, New York, NY, USA, 2009, pp. 297–298. Offutt, 2001, 34 R. DeMillo, E. Krauser, A. Mathur, Compiler-integrated program mutation, in: Computer Software and Applications Conference, COMPSAC’91, Proceedings of the Fifteenth Annual International, 1991, pp. 351–356. A.P. Mathur, W.E. Wong, An empirical comparison of mutation and data low based test adequacy criteria, 1993. M.A. Cachia, M. Micallef, C. Colombo, Towards incremental mutation testing, in: Electronic Notes in Theoretical Computer Science, Proceedings of the 2013 Validation Strategies for Software Evolution, VSSE Workshop, 2013, pp. 2–11. Jia, 2011, An analysis and survey of the development of mutation testing, IEEE Trans. Softw. Eng., 37, 649, 10.1109/TSE.2010.62 Jia, 2009, Higher order mutation testing, J. Inf. Softw. Technol., 51, 1379, 10.1016/j.infsof.2009.04.016 Langdon, 2010, Efficient multi-objective higher order mutation testing with genetic programming, J. Syst. Softw., 83, 2416, 10.1016/j.jss.2010.07.027 Papadakis, 2010, An empirical evaluation of the first and second order mutation testing strategies, 90 Madeyski, 2014, Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation, IEEE Trans. Softw. Eng.., 40, 23, 10.1109/TSE.2013.44 M. Kintis, M. Papadakis, N. Malevris, Evaluating mutation testing alternatives: A collateral experiment, in: Proc. 17th Asia Pacific Soft. Eng. Conf., APSEC, 2010. Kitchenham, 2007, Guidelines for performing systematic literature reviews in software engineering Offutt, 1992, Investigations of the software testing coupling effect, ACM Trans. Softw. Eng. Methodol., 1, 5, 10.1145/125489.125473 Akinde, 2012, Using higher order mutation for reducing equivalent mutants in mutation testing, Asian J. Comput. Sci. Inf. Technol., 2, 13 M. Kintis, M. Papadakis, N. Malevris, Isolating First Order Equivalent Mutants via Second Order Mutation, in: IEEE Fifth International Conference on Software Testing, Verification and Validation, 2012, pp. 701–710. Langdon, 2009, Multi objective mutation testing with genetic programming, 21 Y. Jia, M. Harman, Constructing Subtle Faults Using Higher Order Mutation Testing, in: Proceedings of the 8th International Working Conference on Source Code Analysis and Manipulation, SCAM’08, Beijing, China, 2008, pp. 249–258. E. Omar, S. Ghosh, D. Whitely, Constructing subtle higher order mutants for Java and AspectJ programs, in: Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering, ISSRE, 2013, , pp. 340–349. Nguyen, 2015, Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm, 391 Omar, 2017, Subtle higher order mutants, Inf. Softw. Technol., 81, 3, 10.1016/j.infsof.2016.01.016 E. Omar, S. Ghosh, D. Whitley, Comparing search techniques for finding subtle higher order mutants, in: GECCO’14, Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, 2014, pp. 1271–1278. Harman, 2014, Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation, 397 Acree, 1980 Polo, 2008, Decreasing the cost of mutation testing with second-order mutants, Softw. Test. Verif. Reliab., 19, 111, 10.1002/stvr.392 Jia, 2008, MILU: A customizable, runtime-optimized higher order mutation testing tool for the full c language, 94 Kapoor, 2011, Test case effectiveness of higher order mutation testing, Int. J. Comput. Technol. Appl., 2, 1206 M. Harman, Y. Jia, W.B. Langdon, Strong higher order mutation-based test data generation, in: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, ESEC/FSE’11, 2011, pp. 212–222. E. Blanco-Muñoz, A. García-Domínguez, J.J. Domínguez-Jiménez, I. Medina-Bulo, Towards higher-order mutant generation for WS-BPEL, in: Proceedings of the International Conference on e-Business, 2011, pp. 1–6. E. Omar, S. Ghosh, An Exploratory Study of Higher Order Mutation Testing in Aspect-Oriented Programming, in: Proceedings of the 23rd IEEE International Symposium on Software Reliability Engineering, ISSRE 2012, 2012, pp. 1–10. Mateo, 2013, Validating 2nd-order mutation at system level, IEEE Trans. Softw. Eng., 39, 570, 10.1109/TSE.2012.39 Jia, 2013 Ghiduk, 2014, Using evolutionary algorithms for higher-order mutation testing, IJCSI Int. J. Comput. Sci., 11, 93 Derezińska, 2014, Experimental evaluation of mutation testing approaches to python programs, 156 E. Omar, S. Ghosh, D. Whitley, HOMAJ: A Tool for Higher Order Mutation Testing in AspectJ and Java, Software Testing, in: Verification and Validation Workshops, ICSTW, IEEE Seventh International Conference on, 2014, pp. 165–170. Nguyen, 2014, Problems of mutation testing and higher order mutation testing, 157 Y. Jia, F. Wu, M. Harman, J. Krinke, Genetic Improvement using Higher Order Mutation, in: GECCO Companion’15: Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, 2015, pp. 803–804. Jia, 2015, Introduction to the special issue on mutation testing, Softw. Test. Verif. Reliab., 25, 461, 10.1002/stvr.1582 Nguyen, 2016, Empirical evaluation of multi-objective optimization algorithms searching for higher order mutants, vol. 47, 48 Vu Nguyen, 2016, On the relationship between the order of mutation testing and the properties of generated higher order mutants, vol. 9621 Ghiduk, 2016, Reducing the number of higher-order mutants with the aid of data flow, e-Inform. Softw. Eng. J., 10, 31 Nguyen, 2016, Higher order mutation testing to drive development of new test cases: An empirical comparison of three strategies, 235 S. Tokumoto, H. Yoshida, K. Sakamoto, S. Honiden, MuVM: Higher Order Mutation Analysis Virtual Machine for C, in: 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST, Chicago, IL, 2016, pp. 320–329. J.A.P. Lima, G. Guizzo, S.R. Vergilio, A.P.C. Silva, H.L. Jakubovski Filho, H.V. Ehrenfried, Evaluating Different Strategies for Reduction of Mutation Testing Costs, in: Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing, Article No. 4, 2016. F. Wu, M. Harman, Y. Jia, J. Krinke, HOMI: Searching Higher Order Mutants For Software Improvement, in: Symposium on Search-Based Software Engineering, Raleigh, NC, USA, October 2016. Nguyen, 2017, Addressing mutation testing problems by applying multi-objective optimization algorithms and higher order mutation, J. Intell. Fuzzy Systems, 32, 1173, 10.3233/JIFS-169117 B. Kitchenham, S. Charters, Guidelines for performing systematic literature reviews in software engineering, Technical Report EBSE, 2007. K.S. Khan, G.Ter. Riet, J. Glanville, A.J. Sowden, J. Kleijnen, Undertaking systematic reviews of research on effectiveness CRDs guidance for those carrying out or commissioning reviews, Technical Report, 2001. D. Schuler, A. Zeller, (Un-) Covering Equivalent Mutants, in Software Testing, Verification and Validation, in: ICST, 2010 Third International Conference on, 2010, pp. 45–54. W.B. Langdon, Genetic Programming and Data Structures: Genetic Programming Data Structures = Automatic Programming!, Boston. 1998. DeMillo, 1991, Constraint-based automatic test data generation, IEEE Trans. Softw. Eng., 17, 900, 10.1109/32.92910 Morell, 1990, A theory of fault-based testing, IEEE Trans. Softw. Eng., 16, 844, 10.1109/32.57623 Hanh, 2014, Mutation-based test data generation for simulink models using genetic algorithm and simulated annealing, Int. J. Comput. Inf. Technol., 3, 763 Papadakis, 2013, Searching and generating test inputs for mutation testing, J. Springer Plus, 2, 1 Louzada, 2012, An elitist evolutionary algorithm for automatically generating test data, 1 Malhotra, 2011, An adequacy based test data generation technique using genetic algorithms, J. Inf. Process. Syst., 7, 363, 10.3745/JIPS.2011.7.2.363 M. Papadakis, N. Malevris, M. Kallia, Towards automating the generation of mutation tests, in: 5th Workshop on Automation of Software Test, 2010, pp. 111–118. M. Rad, F. Akbari, A. Bakht, Implementation of common genetic and bacteriological algorithms in optimizing testing data in mutation testing, in: Computational Intelligence and Software Engineering, CiSE, 2010 International Conference on, 2010, pp. 1–6. G. Fraser, A. Zeller, Mutation-driven generation of unit tests and oracles, in: International Symposium on Software Testing and Analysis, ISSTA’10, 2010, pp. 147–157. M. Papadakis, N. Malevris, Automatic mutation test case generation via dynamic symbolic execution, in: 21st International Symposium on Software Reliability Engineering, ISSRE’10, 2010, pp. 121–130. Mishra, 2010, An approach for mutation testing using elitist genetic algorithm, 426 L. Zhang, T. Xie, N. Tillmann, J. Halleux, H. Mei, Test generation via dynamic symbolic execution for mutation testing, in: International Conference on Software Maintenance, ICSM’10, 2010. May, 2007, Immune and evolutionary approaches to software mutation testing, 336 Ayari, 2007, Automatic mutation test input data generation via ant colony, 1074 Liu, 2006, An approach to test data generation for killing multiple mutants, 113 Russell, 2003 Dorigo, 1999, The ant colony optimization meta-heuristic, 11 Back, 1996 Whitley, 2001, An overview of evolutionary algorithms: Practical issues and common pitfalls, Inf. Softw. Technol., 43, 817831, 10.1016/S0950-5849(01)00188-4 Goldberg, 1989 Gopal, 1983 DeMillo, 1983 Centre for Reviews and Dissemination, What are the criteria for the inclusion of reviews on DARE? 2007. Available at http://www.york.ac.uk/inst/crd/faq4.htm. Budd, 1978, The design of a prototype mutation system for program testing, 623 R.J. Lipton, F.G. Sayward, The Status of Research on Program Mutation, in: Proceedings of the Workshop on Software Testing and Test Documentation, 1978, pp. 355–373. Offutt, 1996 Bowser, 1988 Agrawal, 1989 Delamaro, 2001, Interface mutation: An approach for integration testing, IEEE Trans. Softw. Eng., 27, 228, 10.1109/32.910859 H. Shahriar, M. Zulkernine, Mutation-Based Testing of Buffer Overflow Vulnerabilities, in: Proceedings of the 2nd Annual IEEE International Workshop on Security in Software Engineering, Turku, Finland, 2008, pp. 979–984. M.E. Delamaro, J.C. Maldonado, Interface Mutation: Assessing Testing Quality at Inter procedural Level, in: Proceedings of the 19thInternational Conference of the Chilean Computer Science Society, SCCC’99, Talca, Chile,1999, pp. 78–86. M.E. Delamaro, J.C. Maldonado, A.P. Mathur, Integration Testing Using Interface Mutation, in: Proceedings of the seventh International Symposium on Software Reliability Engineering, ISSRE’96, White Plains, New York, 1996, pp. 112–121. S. Kim, J.A. Clark, J. A. McDermid, Class Mutation: Mutation Testing for Object-oriented Programs, in: Proceedings of the Net. Object Days Conference on Object-Oriented Software Systems. 2000. A. Derezinska, Object-oriented Mutation to Assess the Quality of Tests, in: Proceedings of the 29th Euro micro Conference, Belek, Turkey, 2003, pp. 417–420. Derezinska, 2005 A. Derezinska, Quality Assessment of Mutation Operators Dedicated for C# Programs, in: Proceedings of the 6th International Conference on Quality Software, QSIC’06, Beijing, China, 2006. A.S. Namin, J.H. Andrews, D. Murdoch, Sufficient mutation operators for measuring test effectiveness. in: Software Engineering, ICSE’08 ACM/IEEE 30th International Conference on, 2008, pp. 351–360. Offutt, 1996, An experimental determination of sufficient mutant operators, ACM Trans. Softw. Eng. Methodol., 5, 99, 10.1145/227607.227610 Y.-S. Ma, A.J. Offutt, Y.-R. Kwon, Inter-class mutation operators for java, in: Software Reliability Engineering, Proceedings. 13th International Symposium on, 2002, pp. 352–363. Chen, 2006 Derezinska, 2012, Quality evaluation of object-oriented and standard mutation operators applied to c# programs, 42 King, 1991, A fortran language system for mutation-based software testing, Softw. - Pract. Exp., 21, 685, 10.1002/spe.4380210704 Ahmed, 2010 Shahriar, 2008, Mutation-Based testing of format string bugs, 229 Bradbury, 2006, Mutation operators for concurrent java, j2se 50 Budd, 1980 Tanaka, 1981 DeMillo, 1988, An extended overview of the mothra software testing environment, 142 Delamaro, 1993 Parasoft, Parasoft Insure++, 2006, http://www.parasoft.com/jsp/products/home.jsp?product=Insure. Offutt, 1994, An empirical evaluation of weak mutation, IEEE Trans. Softw. Eng., 20, 337, 10.1109/32.286422 I. Moore, Jester and Pester, 2001, http://jester.sourceforge.net/. Ma, 2005, MuJava: An automated class mutation system, Softw. Test. Verif. Reliab., 15, 97, 10.1002/stvr.308 SourceForge, Nester, 2002, http://nester.sourceforge.net/. C. Zhou, P. Frankl, Mutation Testing for Java Database Applications, in: Proceedings of the 2nd International Conference on Software Testing Verification and Validation, ICST’09, Davor Colorado, 2009, pp. 396–405. H. Shahriar, M. Zulkernine, MUSIC: Mutation-based SQL Injection Vulnerability Checking, in: Proceedings of the 8th International Conference on Quality Software, QSIC’08, Oxford, UK, 2008, pp. 77–86. Offutt, 1994, Using compiler optimization techniques to detect equivalent mutants, Softw. Test. Verif. Reliab., 4, 131, 10.1002/stvr.4370040303 Hierons, 1999, Using program slicing to assist in the detection of equivalent mutants, Softw. Test. Verif. Reliab.., 10.1002/(SICI)1099-1689(199912)9:4<233::AID-STVR191>3.0.CO;2-3 Arcaini, 2017, A novel use of equivalent mutants for static anomaly detection in software artifacts, Inf. Softw. Technol., 81, 52, 10.1016/j.infsof.2016.01.019 Umar, 2006 T. Ueshiba, H. Haga, Detecting equivalent mutants using symbolic computation, in: Proceedings of the International Conference on Electrical, Electronics, Computer Engineering and their Applications, Kuala Lumpur, Malaysia, 2014, pp. 6–11. Kintis, 2016 Adamopoulos, 2004, How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution, 1338 Kintis, 2015, MEDIC: A static analysis framework for equivalent mutant identification, Inf. Softw. Technol., 68, 1, 10.1016/j.infsof.2015.07.009 Papadakis, 2014, Mitigating the effects of equivalent mutants with mutant classification strategies, J. Sci. Comput. Program., 95, 298, 10.1016/j.scico.2014.05.012 Offutt, 1997, Automatically detecting equivalent mutants and infeasible paths, Softw. Test. Verif. Reliab., 7, 165, 10.1002/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-U A.J. Offutt, J. Pan, Detecting equivalent mutants and the feasible path problem, in: Computer Assurance. 1996 COMPASS’96, Systems Integrity. Software Safety. Process Security. Proceedings of the Eleventh Annual Conference on, Gaithersburg, MD, 1996, pp. 224–236. P. Arcaini, A. Gargantini, E. Riccobene, P. Vavassori, Rehabilitating equivalent mutants as static anomaly detectors in software artifacts, in: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops, ICSTW, Graz, 2015, pp. 1–6. Orzeszyna, 2011 B.J.M. Grün, D. Schuler, A. Zeller, The Impact of Equivalent Mutants, in: International Conference on Software Testing, Verification, and Validation Workshops, Denver, CO, 2009, pp. 192–199. M. Papadakis, Y. Jia, M. Harman, Y. Le Traon, Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique, in: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, 2015, pp. 936–946. Pan, 1994 S. Nica, F. Wotawa, Using Constraints for Equivalent Mutant Detection. WS-FMDS, 2012, pp. 1–8. M. Kintis, N. Malevris, Using data flow patterns for equivalent mutant detection, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, Cleveland, OH, 2014, pp. 196–205. Patrick, 2012, MESSI: Mutant evaluation by static semantic interpretation, 711 R. Just, M.D. Ernst, G. Fraser, Using state infection conditions to detect equivalent mutants and speed up mutation analysis, in: Proceedings of the Dagstuhl Seminar 13021: Symbolic Methods in Testing. 2013. S. Nica, M. Nica, F. Wotawa, Detecting equivalent mutants by means of constraint systems, in: VALID 2011: The Third International Conference on Advances in System Testing and Validation Lifecycle, 2011, pp. 21–24. Baldwin, 1979, 276 Ellims, 2007, The Csaw C mutation tool: Initial results, 185 Martin, 2007, A fault model and mutation testing of access control policies, 667 du Bousquet, 2008, Towards mutation analysis for Lustre programs, Electron. Notes Theor. Comput. Sci., 203, 35, 10.1016/j.entcs.2008.05.009 Mresa, 1999, Efficiency of mutation operators and selective mutation strategies: An empirical study, Softw. Test. Verif. Reliab., 9, 205, 10.1002/(SICI)1099-1689(199912)9:4<205::AID-STVR186>3.0.CO;2-X Harman, 2001, The relationship between program dependence and mutation analysis, 5 Offutt, 2006, The class-level mutants of mujava, 78 G. Kaminski, P. Ammann, Using a fault hierarchy to improve the efficiency of DNF logic mutation testing, in: Proc. Int. Conf. Software Testing Verification and Validation ICST’09, 2009, pp. 386–395. C. Ji, Z. Chen, B. Xu, Z. Wang, A new mutation analysis method for testing Java exception handling, in: Proc. 33rd Annual IEEE Int. Computer Software and Applications Conf. COMPSAC’09, vol. 2, 2009, pp. 556–561. Vincenzi, 2002, Bayesian-learning based guidelines to determine equivalent mutants, Int. J. Softw. Eng. Knowl. Eng., 12, 675, 10.1142/S021819400200113X Schuler, 2009, Efficient mutation testing by checking invariant violations, 69 Schuler, 2013, Covering and uncovering equivalent mutants, Softw. Test. Verif. Reliab., 23, 353, 10.1002/stvr.1473 Kintis, 2015, Employing second-order mutation for isolating first-order equivalent mutants, Softw. Test. Verif. Reliab., 25, 508, 10.1002/stvr.1529 Kintis, 2013, Identifying more equivalent mutants via code similarity, 1, 180 Just, 2014, Are mutants a valid substitute for real faults in software testing?, 654 Andrews, 2005, Is mutation an appropriate tool for testing experiments?, 402 Gopinath, 2014, Mutations: How close are they to real faults?, 189 Daran, 1996, Software error analysis: A real case study involving real faults and mutations, 158 Namin, 2011, The use of mutation in testing experiments and its sensitivity to external threats, 342 Kitchenham, 2010, Systematic literature reviews in software engineering: A tertiary study, Inf. Softw. Technol., 52, 792, 10.1016/j.infsof.2010.03.006 Kitchenham, 2008, Systematic literature reviews in software engineering - a systematic literature review, Inf. Softw. Technol., 51, 7, 10.1016/j.infsof.2008.09.009