Understanding replication of experiments in software engineering: A classification

Adams, 2005, Service-learning outcomes research: the role and scarcity of replication studies, J Appl Sociol, 22, 55, 10.1177/19367244052200204

J.P.F. Almqvist, Replication of controlled experiments in empirical software engineering – a survey. Master’s thesis, Department of Computer Science, Faculty of Science, Lund University, 2006. Supervisors: Amela Karahasanovic, Simula RL and Goran Fries, Lund University.

Bahr, 1983, Middletown iii: problems of replication, longitudinal measurement, and triangulation, Annu. Rev. Sociol., 9, 243, 10.1146/annurev.so.09.080183.001331

Barker, 1972, Replication versus tests of equivalence, Percept. Motor Skills, 35, 807, 10.2466/pms.1972.35.3.807

Basili, 1999, Building knowledge through families of experiments, IEEE Trans. Softw. Eng., 25, 456, 10.1109/32.799939

Beck, 1994, Replication strategies for nursing research, J. Nurs. Scholarship, 26, 191, 10.1111/j.1547-5069.1994.tb00312.x

Beloff, 1985, Research strategies for dealing with unstable phenomena

Bisell, 2013, The risks of the replication drive, Nature, 503, 333, 10.1038/503333a

Blomquist, 1986, Replication of research, Res. Nurs. Health, 9, 193, 10.1002/nur.4770090302

Brinberg, 1985

A. Brooks, J. Daly, J. Miller, M. Roper, M. Wood, Replication of experimental results in software engineering, Technical Report ISERN-96-10, Univ. of Strathclyde, Scotland, 1996.

Brown, 1976, Building a replication tradition in marketing

Campbell, 1963

Carver, 2014, Replications of software engineering experiments. Guest’s editor introduction to special issue on replication, Empirical Softw. Eng., 19, 267, 10.1007/s10664-013-9290-8

Christensen, 2001

Collins, 1985

Cook, 1976

G. de Investigación en Ingeniería del Software Empírica (GrISE). Descriptions of the replications typologies, 2013.

Dieste, 2009, Developing search strategies for detecting relevant experiments, Empirical Softw. Eng., 14, 513, 10.1007/s10664-008-9091-7

Finifter, 1972, The generation of confidence: evaluating research findings by random subsample replication, Sociol. Methodol., 4, 112, 10.2307/270731

Finifter, 1975, Replication and Extension of Social Research through Secondary Analysis, Soc. Sci. Inform., 14, 119, 10.1177/053901847501400205

Fuess, 1996, On replications in business and economics research: the qjbe case, Quart. J. Business Econ.

O.S. Gómez, N. Juristo, S. Vegas, Replication types in experimental disciplines, in: 4th International Symposium on Empirical Software Engineering and Measurement (ESEM’2010), Bolzan, Italy, 2010, pp. 71–75.

Halas, 1961, Types of responses elicited in planaria by light, J. Comp. Physiol. Psychol., 54, 302, 10.1037/h0045739

Hannay, 2009, The effectiveness of pair programming: a meta-analysis, Inf. Softw. Technol., 51, 1110, 10.1016/j.infsof.2009.02.001

W. Hayes, Research synthesis in software engineering: a case for meta-analysis, in: METRICS ’99: Proceedings of the 6th International Symposium on Software Metrics, IEEE Computer Society, Washington, DC, USA, 1999, p. 143.

Hendrick, 1990

Hunt, 1975, Do we really need more replications?, Psychol. Rep., 36, 587, 10.2466/pr0.1975.36.2.587

Hunter, 2001, The desperate need for replications, J. Consumer Res., 28, 149, 10.1086/321953

Hyman, 1967, Evaluating social action programs, 769

N. Juristo, Towards Understanding Replication of Software Engineering Experiments, Keynote, in: International Symposium on Empirical Software Engineering and Measurement (ESEM’13), Baltimore, USA, September 2013.

Juristo, 2004, Reviewing 25 years of testing technique experiments, Empirical Softw. Eng., 9, 7, 10.1023/B:EMSE.0000013513.48963.1b

Juristo, 2011, The role of non-exact replications in software engineering experiments, Empirical Softw. Eng., 16, 295, 10.1007/s10664-010-9141-9

Juristo, 2013, A process for managing interaction between experimenters to get useful similar replications, Inf. Softw. Technol., 55, 215, 10.1016/j.infsof.2012.07.016

Jørgensen, 2004, A review of studies on expert estimation of software development effort, J. Syst. Softw., 70, 37, 10.1016/S0164-1212(02)00156-5

Kelly, 1979, Replication in experimental communication research: an analysis, Human Commun. Res., 5, 338, 10.1111/j.1468-2958.1979.tb00646.x

Keppel, 1982

Kitchenham, 2008, The role of replications in empirical software engineering – a word of warning, Empirical Softw. Eng., 13, 219, 10.1007/s10664-008-9061-0

J.L. Krein, C.D. Knutson, A case for replication: Synthesizing research methodologies in software engineering, in: 1st International Workshop on Replication in Empirical Software Engineering Research (RESER’2010), Cape Town, South, Africa, 2010.

La Sorte, 1972, Replication as a verification technique in survey research: a paradigm, Sociol. Quart., 13, 218, 10.1111/j.1533-8525.1972.tb00805.x

Lindsay, 1993, The design of replicated studies, Am. Stat., 47, 217, 10.1080/00031305.1993.10475983

Lung, 2008, On the difficulty of replicating human subjects studies in software engineering, 191

Lykken, 1968, Statistical significance in psychological research, Psychol. Bull., 70, 151, 10.1037/h0026141

Lüdtke, 2008, Do that to me one more time! – what kind of trial replications do we need?, Complementary Therapies Med., 16, 181, 10.1016/j.ctim.2008.06.003

V. Mandić, J. Markkula, M. Oivo. Towards multi-method research approach in empirical software engineering, vol. 32, 2009, pp. 96–110.

McConnell, 1967, Specific factors influencing planarian behavior, 217

McConnell, 1961, The effects of ingestion of conditioned planaria on the response level of native planaria: a pilot study, Worm Runner’s Digest., 3, 41

M. Mendonça, J. Maldonado, M. de Oliveira, J. Carver, S. Fabbri, F. Shull, G. Travassos, E. Höhn, V. Basili. A framework for software engineering experimental replications, in: ICECCS ’08: Proceedings of the 13th IEEE International Conference on Engineering of Complex Computer Systems (iceccs 2008), IEEE Computer Society, Washington, DC, USA, 2008, pp. 203–212.

J. Miller,. Can results from software engineering experiments be safely combined? in: METRICS ’99: Proceedings of the 6th International Symposium on Software Metrics, IEEE Computer Society, Washington, DC, USA, 1999, p. 152.

Miller, 2000, Applying meta-analytical procedures to software engineering experiments, J. Syst. Softw., 54, 29, 10.1016/S0164-1212(00)00024-8

Miller, 2005, Replicating software engineering experiments: a poisoned chalice or the holy grail, Inf. Softw. Technol., 47, 233, 10.1016/j.infsof.2004.08.005

Mittelstaedt, 1984, Econometric replication: lessons from the experimental sciences, Quart. J. Business Econ., 23

Moonesinghe, 2007, Most published research findings are false – but a little replication goes a long way, PLoS Med., 4, 0218, 10.1371/journal.pmed.0040028

Moyer, 2012, Fish oil supplement research remains murky, Sci. Am., 24

Pickard, 1998, Combining empirical results in software engineering, Inf. Softw. Technol., 40, 811, 10.1016/S0950-5849(98)00101-3

H. Radder. Experimental reproducibility and the experimenters’ regress, in: PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, vol. 1, 1992, pp. 63–73.

Rao, 1981, On the question of replication, J. Parapsychol., 45, 311

Reid, 1981, Replication in advertising research: 1977, 1978, 1979, J. Advert., 10, 3, 10.1080/00913367.1981.10672750

Rosenthal, 1990, Replication in behavioral research, J. Social Behav. Personality, 4, 1

Sargent, 1981, The repeatability of significance and the significance of repeatability, Eur. J. Parapsychol., 3, 423

Schmidt, 2009, Shall we really do it again? The powerful concept of replication is neglected in the social sciences, Rev. Gener. Psychol., 13, 90, 10.1037/a0015108

F. Shull, V. Basili, J. Carver, J. Maldonado, G. Travassos, M. Mendonça, S. Fabbri, Replicating software engineering experiments: Addressing the tacit knowledge problem, in: ISESE ’02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, IEEE Computer Society, Washington, DC, USA, 2002, p. 7.

Shull, 2008, The role of replications in empirical software engineering, Empirical Softw. Eng., 13, 211, 10.1007/s10664-008-9060-1

Sidman, 1960

Silva, 2012, Replication of empirical studies in software engineering research: a systematic mapping study, Empirical Softw. Eng., 1

Sjøberg, 2005, A survey of controlled experiments in software engineering, IEEE Trans. Softw. Eng., 31, 733, 10.1109/TSE.2005.97

Thompson, 1955, Classical conditioning in the planarian, Dugesia dorotocephala, J. Comp. Physiol. Psychol., 48, 65, 10.1037/h0041147

Tichy, 1998, Should computer scientists experiment more?, Computer, 31, 32, 10.1109/2.675631

Tsang, 1999, Replication and theory development in organizational science: a critical realist perspective, Acad. Manage. Rev., 24, 759, 10.5465/amr.1999.2553252