Multilevel mediation analysis in R: A comparison of bootstrap and Bayesian approaches

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001 Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01 Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006). Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations. Psychological Methods, 11(2), 142–163. https://doi.org/10.1037/1082-989X.11.2.142 Biesanz, J. C., Falk, C. F., & Savalei, V. (2010). Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research, 45(4), 661–701. https://doi.org/10.1080/00273171.2010.498292 Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31(2), 144–152. https://doi.org/10.1111/j.2044-8317.1978.tb00581.x Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1), 1–20. https://doi.org/10.5334/joc.10 Buhle, J., & Wager, T. D. (2010). Performance-dependent inhibition of pain by an executive working memory task. PAIN®, 149(1), 19–26. https://doi.org/10.1016/j.pain.2009.10.027 Bürkner, P. C. (2017). Brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80, 1–28. https://doi.org/10.18637/jss.v080.i01 Cain, M. K., Zhang, Z., & Bergeman, C. S. (2018). Time and other considerations in mediation design. Educational and Psychological Measurement, 78(6), 952–972. https://doi.org/10.1177/0013164417743003 Carpenter, J. R., Goldstein, H., & Rasbash, J. (2003). A novel bootstrap procedure for assessing the relationship between class size and achievement. Applied Statistics, 52(4), 431–443. Chib, S., & Greenberg, E. (1995). Understanding the Metropolis–Hastings algorithm. The American Statistician, 49(4), 327–335. https://doi.org/10.1080/00031305.1995.10476177 Chu, Y. (2013). Bayesian mediation analysis for partially clustered designs[Unpublished Master’s thesis]. The University of Texas at Austin. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155 Dohle, S., & Siegrist, M. (2014). Fluency of pharmaceutical drug names predicts perceived hazardousness, assumed side effects and willingness to buy. Journal of Health Psychology, 19(10), 1241–1249. https://doi.org/10.1177/1359105313488974 Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall. Falk, C. F. (2018). Are robust standard errors the best approach for interval estimation with nonnormal data in structural equation modeling? Structural Equation Modeling: A Multidisciplinary Journal, 25(2), 244–266. https://doi.org/10.1080/10705511.2017.1367254 Fang, J., Wen, Z., & Hau, K.-T. (2019). Mediation effects in 2-1-1 multilevel model: Evaluation of alternative estimation methods. Structural Equation Modeling: A Multidisciplinary Journal, 26(4). https://doi.org/10.1080/10705511.2018.1547967 Fenerci, C., & Sheldon, S. (2022). The role of episodic memory in imagining autobiographical events: The influence of event expectancy and context familiarity. Memory, 30(5), 573–590. https://doi.org/10.1080/09658211.2022.2032178 Fritz, M. S., Taylor, A. B., & MacKinnon, D. P. (2012). Explanation of two anomalous results in statistical mediation analysis. Multivariate Behavioral Research, 47(1), 61–87. https://doi.org/10.1080/00273171.2012.640596 Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis. CRC Press. Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel / hierarchical models. Cambridge University Press. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472. Goldstein, H. (2011). Bootstrapping in multilevel models. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 163–171). Routledge. Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1), 97–109. https://doi.org/10.1093/biomet/57.1.97 Hoffman, M. D., & Gelman, A. (2014). The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623. Hox, J., & van de Schoot, R. (2013). Robust methods for multilevel analysis. In M. A. Scott, J. S. Simonoff, & B. D. Marx (Eds.), The SAGE handbook of multilevel modeling (pp. 387–402). SAGE Publications Ltd.. https://doi.org/10.4135/9781446247600.n22 Jackman, S. (2009). Bayesian analysis for the social sciences. John Wiley & Sons. Kackar, R. N., & Harville, D. A. (1984). Approximations for standard errors of estimators of fixed and random effects in mixed linear models. Journal of the American Statistical Association, 79(388), 853–862. Kelcey, B., Dong, N., Spybrook, J., & Cox, K. (2017). Statistical power for causally defined indirect effects in group-randomized trials with individual-level mediators. Journal of Educational and Behavioral Statistics, 42(5), 499–530. https://doi.org/10.3102/1076998617695506 Kenny, D. A., Korchmaros, J. D., & Bolger, N. (2003). Lower-level mediation in multilevel models. Psychological Methods, 8(2), 115–128. https://doi.org/10.1037/1082-989X.8.2.115 Krull, J. L., & MacKinnon, D. P. (2001). Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36(2), 249–277. https://doi.org/10.1207/S15327906MBR3602_06 Lachowicz, M. J., Sterba, S. K., & Preacher, K. J. (2015). Investigating multilevel mediation with fully or partially nested data. Group Processes & Intergroup Relations, 18(3), 274–289. https://doi.org/10.1177/1368430214550343 Lai, M. (2020). Bootmlm: Bootstrap resampling for multilevel models. R package version 0.0.1.1000. https://github.com/marklhc/bootmlm Lai, M. (2021). Bootstrap confidence intervals for multilevel standardized effect size. Multivariate Behavioral Research, 56(4), 558–578. https://doi.org/10.1080/00273171.2020.1746902 Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9), 1989–2001. https://doi.org/10.1016/j.jmva.2009.04.008 MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7(1), 83–104. https://doi.org/10.1037/1082-989X.7.1.83 MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39(1), 99–128. MacKinnon, D. P., & Pirlott, A. G. (2015). Statistical approaches for enhancing causal interpretation of the M to Y relation in mediation analysis. Personality and Social Psychology Review, 19(1), 30–43. https://doi.org/10.1177/1088868314542878 Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing type I error and power in linear mixed models. Journal of Memory and Language, 94, 305–315. https://doi.org/10.1016/j.jml.2017.01.001 Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953). Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6), 1087–1092. Miočević, M., MacKinnon, D. P., & Levy, R. (2017). Power in Bayesian mediation analysis for small sample research. Structural Equation Modeling: A Multidisciplinary Journal, 24(5), 666–683. https://doi.org/10.1080/10705511.2017.1312407 Neal, R. (2011). MCMC using Hamiltonian dynamics. In S. Brooks, A. Gelman, G. L. Jones, & X.-L. Meng (Eds.), Handbook of Markov chain Monte Carlo (pp. 116–162). Chapman and Hall/CRC. Page-Gould, E., & Sharples, A. (2016). Bootstrapping accurate indirect effects in multilevel models. Retrieved from http://www.page-gould.com/r/indirectmlm/. Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & R Core Team. (2021). Nlme: Linear and nonlinear mixed effects models. R package version, 3, 1–152 https://CRAN.R-project.org/package=nlme Pituch, K. A., & Stapleton, L. M. (2008). The performance of methods to test upper-level mediation in the presence of nonnormal data. Multivariate Behavioral Research, 43(2). https://doi.org/10.1080/00273170802034844 Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41(3), 367–400. https://doi.org/10.1207/s15327906mbr4103_5 Pituch, K. A., Whittaker, T. A., & Stapleton, L. M. (2005). A comparison of methods to test for mediation in multisite experiments. Multivariate Behavioral Research, 40(1), 1–23. https://doi.org/10.1207/s15327906mbr4001_1 Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40(3), 879–891. https://doi.org/10.3758/BRM.40.3.879 Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2011). Alternative methods for assessing mediation in multilevel data: The advantages of multilevel SEM. Structural Equation Modeling: A Multidisciplinary Journal, 18(2), 161–182. https://doi.org/10.1080/10705511.2011.557329 Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. SAGE. R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing https://www.R-project.org/ Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309–338. https://doi.org/10.1037/met0000184 Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02 Sherman, M., & le Cessie, S. (1997). A comparison between bootstrap methods and generalized estimating equations for correlated outcomes in generalized linear models. Communications in Statistics – Simulation and Computation, 26(3), 901–925. https://doi.org/10.1080/03610919708813417 Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7(4), 422–445. https://doi.org/10.1037/1082-989X.7.4.422 Snijders, T. A. B., & Bosker, R. J. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling. SAGE. Stan Development Team. (2020a). Stan modeling language users guide and reference manual, Version 2.21.0. Stan Development Team. (2020b). RStan: The R interface to Stan. R Package Version, 2(21), 2. http://mc-stan.org/ Tabry, V., Vogel, T. A., Lussier, M., Brouillard, P., Buhle, J., Rainville, P., Bherer, L., & Roy, M. (2020). Inter-individual predictors of pain inhibition during performance of a competing cognitive task. Scientific Reports, 10, 21785. https://doi.org/10.1038/s41598-020-78653-z Vallejo, G., Tuero-Herrero, E., Nunez, J. C., & Rosario, P. (2014). Performance evaluation of recent information criteria for selecting multilevel models in behavioral and social sciences. International Journal of Clinical and Health Psychology, 14(1), 48–57. https://doi.org/10.1016/S1697-2600(14)70036-5 Vallejo Seco, G., Ato García, M., Fernández García, P. M., & Livacic Rojas, P. E. (2013). Multilevel bootstrap analysis with assumptions violated. Psicothema, 25(4), 520–528. van der Leeden, R., Meijer, E., & Busing, F. M. T. A. (2008). Resampling multilevel models. In J. de Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 401–433). Springer. van de Schoot, R., & Depaoli, S. (2014). Bayesian analyses: Where to start and what to report. The European Health Psychologist, 16(2), 75–84. van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & Aken, M. A. (2014). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85, 842–860. https://doi.org/10.1111/cdev.12169 Wang, L., & Preacher, K. J. (2015). Moderated mediation analysis using Bayesian methods. Structural Equation Modeling: A Multidisciplinary Journal, 22(2), 249–263. https://doi.org/10.1080/10705511.2014.935256 Wu, W., Carroll, I. A., & Chen, P.-Y. (2018). A single-level random-effects cross-lagged panel model for longitudinal mediation analysis. Behavior Research Methods, 50, 2111–2124. https://doi.org/10.3758/s13428-017-0979-2 Yuan, Y., & MacKinnon, D. P. (2009). Bayesian mediation analysis. Psychological Methods, 14(4), 301–322. https://doi.org/10.1037/a0016972 Zhang, Q., Wang, L., & Bergeman, C. S. (2018). Multilevel autoregressive mediation models: Specification, estimation, and applications. Psychological Methods, 23(2), 278–297. https://doi.org/10.1037/met0000161 Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12(4), 695–719. https://doi.org/10.1177/1094428108327450