Preregistration of Modeling Exercises May Not Be Useful
Tóm tắt
This is a commentary on Lee et al.’s (2019) article encouraging preregistration of model development, fitting, and evaluation. While we are in general agreement with Lee et al.’s characterization of the modeling process, we disagree on whether preregistration of this process will move the scientific enterprise forward. We emphasize the subjective and exploratory nature of model development, and point out that “under-modeling” of data (relying on black-box approaches applied to data without data exploration) is as big a problem as “over-modeling” (fitting noise, resulting in models that generalize poorly). We also note the potential long-run negative impact of preregistration on future generations of cognitive scientists. It is our opinion that preregistration of model development will lead to less, and to less creative, exploratory analysis (i.e., to more under-modeling), and that Lee at al.’s primary goals can be achieved by requiring publication of raw data and code. We conclude our commentary with suggestions on how to move forward.
Tài liệu tham khảo
Breiman, L. (2001). Statistical modeling: the two cultures. Statistical Science, 16, 199–231.
Cavagnaro, D.R., Aranovich, G.J., McClure, S.M., Pitt, M.A., Myung, J.I. (2016). On the functional form of temporal discounting: an optimized adaptive test. Journal of Risk and Uncertainty, 52, 233–254.
Lee, M.D., Criss, A.H., Devezer, B., et al. (2019). Computational Brain & Behavior. https://doi.org/10.1007/s42113-019-00029-y.
Lewis, J., Lee, Y., MacEachern, S. (2012). Robust inference via the blended paradigm. In Proceedings of the Joint Statistical Meetings (pp. 1773–1786).
Ratcliff, R., & Smith, P. (2004). A comparison of sequential sampling model for two-choice reaction time. Psychological Review, 111, 333–367.
Turner, B., Dennis, S., Van Zandt, T. (2014). Likelihood free Bayesian analysis of memory models. Psychological Review, 120, 667–678.
Van Zandt, T., Colonius, H., Proctor, R. (2000). A comparison of two response time models applied to perceptual matching. Psychonomic Bulletin Review, 7, 208–256.
Xu, X., Lu, P., MacEachern, S.N., Xu, R. (2019). Calibrated Bayes factors for model comparison. Journal of Statistical Computation and Simulation, 89(4), 591–614.
Yu, Q., MacEachern, S., Peruggia, M. (2011). Bayesian synthesis: combining subjective analyses, with an application to ozone data. Annals of Applied Statistics, 5, 1678–1698.