Cross validation for the classical model of structured expert judgment

Aspinall, 2010, A route to more tractable expert advice, Nature, 463, 294, 10.1038/463294a Aspinall, 2016, Evaluation of a performance-based expert elicitation: WHO global attribution of foodborne diseases, PLoS ONE, 11, e0149817, 10.1371/journal.pone.0149817 Aspinall, 2002, The montserrat volcano observatory: its evolution, organization, role and activities, 21, 71 Bamber, 2016, A commentary on ‘how to interpret expert judgment assessments of twenty-first century sea-level rise’ by Hylke de Vries and Roderik SW van de Wal, Clim Change, 137, 321, 10.1007/s10584-016-1672-7 Clemen, 2008, Comment on Cooke's classical method, Reliab Eng Syst Saf, Expert Judgement, 93, 760, 10.1016/j.ress.2008.02.003 Cooke, 1991 Cooke, 2008, Discussion: response to discussants, Reliab Eng Syst Saf, Expert Judgement, 93, 775, 10.1016/j.ress.2008.02.006 Cooke, 2012, Pitfalls of ROAT cross-validation: comment on effects of overconfidence and dependence on aggregated probability judgments, J Model Manag, 7, 20 Cooke, 2012, Uncertainty analysis comes to integrated assessment models for climate change…and conversely, Clim Change, 117, 467, 10.1007/s10584-012-0634-y Cooke, 2014, Validating expert judgment with the classical model, 191–212 Cooke, 2015, Messaging climate change uncertainty, Nat Clim Change, 5, 8, 10.1038/nclimate2466 Cooke, 2008, TU Delft expert judgment data base, Reliab Eng Syst Saf, Expert Judgement, 93, 657, 10.1016/j.ress.2007.03.005 Cooke, 1988, Calibration and information in expert resolution; a classical approach, Automatica, 24, 87, 10.1016/0005-1098(88)90011-8 Cooke, 1992 Cooke, 2014, Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake Erie, Integr Environ Assess Manag, 10, 522, 10.1002/ieam.1559 Eggstaff, 2014, The effect of the number of seed variables on the performance of Cooke's classical model, Reliab Eng Syst Saf, 121, 72, 10.1016/j.ress.2013.07.015 Flandoli, 2011, Comparison of a new expert elicitation model with the classical model, equal weights and single experts, using a cross-validation technique, Reliab Eng Syst Saf, 96, 1292, 10.1016/j.ress.2011.05.012 Genest, 1986, Combining probability distributions: a critique and an annotated bibliography, Stat Sci, 1, 114, 10.1214/ss/1177013825 Gillingham, 2015, Modeling uncertainty in climate change: a multi-model comparison, Cowles Found Discuss Pap New Haven, CT: Cowles Found Res Econ Yale Univ, 2022 Hald, 2016, World health organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: a structured expert elicitation, PLoS ONE, 11, e0145839, 10.1371/journal.pone.0145839 Harper, 1994, Probabilistic Accid Conséq Uncertain Study: Dispers Depos Uncertain Assess, Volume I Laddaga, 1977, Lehrer and the consensus proposal, Synthese, 36, 473, 10.1007/BF00486109 Lichtendahl, 2013, Is It better to average probabilities or quantiles?, Manag Sci, 59, 1594, 10.1287/mnsc.1120.1667 Lin, 2008, Can Cooke's model sift out better experts and produce well-calibrated aggregated probabilities?, IEEE Int Conf Ind Eng Eng Manag, 2008 IEEM, 2008, 425 Lin, 2009, The reliability of aggregated probability judgments obtained through Cooke's classical model, J Model Manag, 4, 149, 10.1108/17465660910973961 Lin, 2012, Effects of overconfidence and dependence on aggregated probability judgments, J Model Manag, 7, 6, 10.1108/17465661211208785 NAS National Academies of Sciences, Engineering, and Medicine, 2017 Vries, 2015, How to interpret expert judgment assessments of 21st century sea-level rise, Clim Change, 130, 87, 10.1007/s10584-015-1346-x Wadge, 2014, A review of volcanic hazard and risk-assessment Praxis at the Soufrière Hills Volcano, Montserrat from 1997 to 2011, 39, 439 Winkler, 1969, Scoring rules and the evaluation of probability assessors, J Am Stat Assoc, 64, 1073, 10.1080/01621459.1969.10501037 Ungar L, Mellers B, Satopää V, Tetlock P, Baron J. The good judgment project: a large scale test of different methods of combining expert predictions. 2012 AAAI Fall Symposium Series; 2012.