Is the replication crisis a base-rate fallacy?
Tóm tắt
Is science in the midst of a crisis of replicability and false discoveries? In a recent article, Alexander Bird offers an explanation for the apparent lack of replicability in the biomedical sciences. Bird argues that the surprise at the failure to replicate biomedical research is a result of the fallacy of neglecting the base rate. The base-rate fallacy arises in situations in which one ignores the base rate—or prior probability—of an event when assessing the probability of this event in the light of some observed evidence. By extension, the replication crisis would result from ignoring the low prior probability of biomedical hypotheses. In this paper, my response to Bird’s claim is twofold. First, I show that the argument according to which the replication crisis is due to the low prior of biomedical hypotheses is incomplete. Second, I claim that a simple base-rate fallacy model does not account for some important methodological insights that have emerged in discussions of the replication crisis.
Tài liệu tham khảo
Ioannidis, John P.A. 2005. Why most published research findings are false. PLoS Medicine 2: e124. https://doi.org/10.1371/journal.pmed.0020124.
Begley, C. Glenn, and Lee M. Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature 483: 531–533.
Prinz, Florian, Thomas Schlange, and Khusru Asadullah. 2011. Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery 10: 712.
Bird, Alexander. 2021. Understanding the replication crisis as a base rate fallacy. British Journal for the Philosophy of Science 72: 965–993. https://doi.org/10.1093/bjps/axy051.
Fisher, R.A. 1925. Statistical methods for research workers. Edinburgh: Oliver and Boyd.
Neyman, J., and E.S. Pearson. 1933. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London A 231: 289–337.
Gigerenzer, Gerd. 2004. Mindless statistics. Journal of Socio-Economics 33: 587–606.
Baum, Mark L., David S. Anish, Thomas C. Chalmers, Henry S. Sacks, Harry Smith, and Richard M. Fagerstrom. 1981. A survey of clinical trials of antibiotic prophylaxis in colon surgery: Evidence against further use of no-treatment controls. New England Journal of Medicine 305: 795–799.
Ioannidis, John P.A., and Joseph Lau. 1999. State of the evidence: Current status and prospects of meta-analysis in infectious diseases. Clinical Infectious Diseases 29: 1178–1185.
Greenland, Sander, Manuela Gago-Dominguez, and Jose Esteban Castelao. 2004. The value of risk-factor (“black box”) epidemiology. Epidemiology 15: 529–535.
Feinstein, Alvan R. 1988. Scientific standards in epidemiologic studies of the menace of daily life. Science 242: 1257–1263.
Skrabanek, Petr. 1994. The emptiness of the black box. Epidemiology 5: 553–555.
Kirwan, Peter D., Cuong Chau, Alison Brown, O. Noel Gill, Valerie Delpech, and contributors. 2016. HIV in the UK: 2016 report. London: Public Health England. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/602942/HIV_in_the_UK_report.pdf.
Goodman, Steven N. 2014. Discussion: An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15: 23–27.
Page, Matthew J., Joanne E. McKenzie, and Andrew Forbes. 2013. Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. Journal of Clinical Epidemiology 66: 524–537.
Pfeiffer, Thomas, Lars Bertram, and John P.A. Ioannidis. 2011. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6: e18362. https://doi.org/10.1371/journal.pone.0018362.
Jager, Leah R., and Jeffrey T. Leek. 2014. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 15: 1–12.
Wagenmakers, Eric-Jan, Ruud Wetzels, Denny Borsboom, Han L.J. van der Maas, and Rogier A. Kievit. 2012. An agenda for purely confirmatory research. Perspectives on Psychological Science 7: 632–638.
Hahn, S., P.R. Williamson, and J.L. Hutton. 2002. Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee. Journal of Evaluation in Clinical Practice 8: 353–359.
Chan, Ann-Wen, and Douglas G. Altman. 2005. Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ 330: 753–759.
Chan, Ann-Wen, Asbjørn Hróbjartsson, Mette T. Haahr, Peter C. Gøtzsche, and Douglas G. Altman. 2004. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. Journal of the American Medical Association 291: 2457–2465.
Dwan, Kerry, Carrol Gamble, Paula R. Williamson, and Jamie J. Kirkham. 2013. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS ONE 8: e66844. https://doi.org/10.1371/journal.pone.0066844.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. The preregistration revolution. PNAS 115: 2600–2606.
Kaplan, Robert M., and Veronica L. Irvin. 2015. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS ONE 10: e0132382. https://doi.org/10.1371/journal.pone.0132382.