Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Diagnostic and Prognostic Research - Tập 5 Số 1 - 2021
David Jenkins1, Glen P. Martin1, Matthew Sperrin1, Richard D Riley2, Thomas P A Debray3, Gary S. Collins4, Niels Peek5
1Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
2Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Staffordshire, UK
3Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
4Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
5NIHR Greater Manchester Patient Safety Translational Research Centre, The University of Manchester, Manchester, UK

Tóm tắt

AbstractClinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, “living” (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.

Từ khóa


Tài liệu tham khảo

Hemingway H, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346:1–11.

Steyerberg E, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10:e1001381.

Hippisley-Cox J, Coupland C, Brindle P. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ. 2017;357:1–21.

Gray LJ, et al. The Leicester Risk Assessment score for detecting undiagnosed type 2 diabetes and impaired glucose regulation for use in a multiethnic UK setting. Diabet. Med. 2010;27:887–95.

Martin, G. P., Sperrin, M. & Sotgiu, G. Performance of prediction models for covid-19: the caudine forks of the external validation. Eur. Respir. J. 2020; 2003728. doi:https://doi.org/10.1183/13993003.03728-2020.

Wynants L, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369.

Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am. J. Epidemiol. 2010;172:971–80.

Hickey GL, et al. Dynamic trends in cardiac surgery: why the logistic euroscore is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur. J. Cardio-thoracic Surg. 2013;43:1146–52.

Debray TPA, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356.

Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. Calibration drift in regression and machine learning models for acute kidney injury. J. Am. Med. Informatics Assoc. 2017;24:1052–61.

Nashef SAM, et al. European system for cardiac operative risk evaluation (EuroSCORE). Eur. J. Cardio-thoracic Surg. 1999;16:9–13.

Luijken K, et al. Changing predictor measurement procedures affected the performance of prediction models in clinical examples. J. Clin. Epidemiol. 2020;119:7–18.

van Houwelingen HC, Thorogood J. Construction, validation and updating of a prognostic model for kidney graft survival. Stat. Med. 1995;14:1999–2008.

Janssen KJM, Moons KGM, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J. Clin. Epidemiol. 2008;61:76–86.

Debray TPA, Koffijberg H, Vergouwe Y, Moons KGM, Steyerberg EW. Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat. Med. 2012;31:2697–712.

Martin GP, Mamas MA, Peek N, Buchan I, Sperrin M. A multiple-model generalisation of updating clinical prediction models. Stat. Med. 2018;37:1343–58.

Riley RD, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016;353:27–30.

Debray TPA, Riley RD, Rovers MM, Reitsma JB, Moons KGM. Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med. 2015;12:1–12.

Martin GP, Mamas MA, Peek N, Buchan I, Sperrin M. Clinical prediction in defined populations: a simulation study investigating when and how to aggregate existing models. BMC Med. Res. Methodol. 2017;17:1.

Hippisley-Cox J, et al. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. Br. Med. J. 2007;335:136–41.

Hippisley-Cox J, et al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ. 2008;336:1475–82.

Nashef SAM, et al. Euroscore II. Eur. J. Cardio-thoracic Surg. 2012;41:734–745.

Raftery AE, Kárný M, Ettler P. Online Prediction Under Model Uncertainty via Dynamic Model Averaging: Application to a Cold Rolling Mill. Technometrics. 2010;52(1):52–66. https://doi.org/10.1198/TECH.2009.08104.

Jenkins DA, Sperrin M, Martin GP, Peek N. Dynamic models to predict health outcomes: current status and methodological challenges. Diagnostic Progn. Res. 2018;2:23.

Fan J, Zhang W. Statistical methods with varying coefficient models. Stat Interface. 2008;1:179–95.

Hoover DR, Rice JA, Wu CO, Yang L-P. Nonparametric smoothing estimates of time-varying coefficient models with longitudinal data. Biometrika. 1998;85:809–22.

Mccormick TH, Raftery AE, Madigan D, Burd RS. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics. 2012;68:23–30.

Su TL, Jaki T, Hickey GL, Buchan I, Sperrin M. A review of statistical updating methods for clinical prediction models. Stat Methods Med Res. 2016:1–16. https://doi.org/10.1177/0962280215626466.

Hickey GL, et al. Dynamic prediction modeling approaches for cardiac surgery. Circ. Cardiovasc. Qual. Outcomes. 2013;6:649–58.

Halabi S, et al. Updated prognostic model for predicting overall survival in first-line chemotherapy for patients with metastatic castration-resistant prostate cancer. J. Clin. Oncol. 2014;32:671–7.

Dawid, A. Present position and potential developments: some personal views: statistical theory: the prequential approach Author ( s ): A . P . Dawid Source: Journal of the Royal Statistical Society . Series A ( General ), Vol . 147 , No . 2 , The 150th Annivers. J. R. Stat. Soc. Ser. A 147, 1984 278–292.

Lenert, M. C., Matheny, M. E. & Walsh, C. G. Prognostic models will be victims of their own success, unless…. J. Am. Med. Inform. Assoc. 2019;26:1645–1650.

Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastructure for global health improvement. Yearb. Med. Inform. 2017;26:16–23.

Adibi, A., Sadatsafavi, M. & Ioannidis, J. P. A. Validation and utility testing of clinical prediction models. JAMA 2020; 2004.

Booth, S., Riley, R. D., Ensor, J., Lambert, P. C. & Rutherford, M. J. Temporal recalibration for improving prognostic model development and risk predictions in settings where survival is improving over time. Int. J. Epidemiol. 2020; 1–10. doi:https://doi.org/10.1093/ije/dyaa030.

Vergouwe Y, et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat. Med. 2017;36:4529–39.

Altman DG, Royston P. What do we mean by validating a prognistic model? Stat. Med. 2000;19:453–73.

Thông tin
Thông tin xuất bản

Diagnostic and Prognostic Research

Tập 5 Số 1

Thông tin tác giả