Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Monthly Weather Review - Tập 143 Số 4 - Trang 1321-1334 - 2015
Michael Scheuerer1, Thomas M. Hamill1
1NOAA/Earth System Research Laboratory, Boulder, Colorado

Tóm tắt

Abstract Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

Từ khóa


Tài liệu tham khảo

Anderson, 1996, A method for producing and evaluating probabilistic forecasts from ensemble model integrations, J. Climate, 9, 1518, 10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2

Berrocal, 2007, Combining spatial statistical and ensemble information in probabilistic weather forecasts, Mon. Wea. Rev., 135, 1386, 10.1175/MWR3341.1

Bruno, 1989

Buizza, 2005, A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems, Mon. Wea. Rev., 133, 1076, 10.1175/MWR2905.1

Cressie, 1980, Robust estimation of the variogram I, Math. Geol., 12, 115, 10.1007/BF01035243

Cressie, 1998, The variance-based cross-variogram: You can add apples and oranges, Math. Geol., 30, 789, 10.1023/A:1021770324434

Dawid, 1999, Coherent dispersion criteria for optimal experimental design, Ann. Stat., 27, 65, 10.1214/aos/1018031101

Emery, 2005, Variograms of order ω: A tool to validate a bivariate distribution model, Math. Geol., 37, 163, 10.1007/s11004-005-1307-4

Feldmann, 2015, Spatial postprocessing of ensemble forecasts for temperature using nonhomogeneous Gaussian regression, Mon. Wea. Rev., 143, 955, 10.1175/MWR-D-14-00210.1

Fricker, 2013, Three recommendations for evaluating climate predictions, Meteor. Appl., 20, 246, 10.1002/met.1409

Gneiting, 2011, Making and evaluating point forecasts, J. Amer. Stat. Assoc., 106, 746, 10.1198/jasa.2011.r10138

Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, J. Amer. Stat. Assoc., 102, 359, 10.1198/016214506000001437

Gneiting, 2014, Probabilistic forecasting, Ann. Rev. Stat. Appl., 1, 125, 10.1146/annurev-statistics-062713-085831

Gneiting, 2007, Probabilistic forecasts, calibration and sharpness, J. Roy. Stat. Soc., 69B, 243, 10.1111/j.1467-9868.2007.00587.x

Gneiting, 2008, Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds, TEST, 17, 211, 10.1007/s11749-008-0114-x

Hamill, 1999, Hypothesis tests for evaluating numerical precipitation forecasts, Wea. Forecasting, 14, 155, 10.1175/1520-0434(1999)014<0155:HTFENP>2.0.CO;2

Hamill, 2001, Interpretation of rank histograms for verifying ensemble forecasts, Mon. Wea. Rev., 129, 550, 10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2

Hamill, 1997, Verification of Eta-RSM short-range ensemble forecasts, Mon. Wea. Rev., 125, 1312, 10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2

Hamill, 2001, Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter, Mon. Wea. Rev., 129, 2776, 10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2

Hamill, 2003, Ensemble forecasts and the properties of flow-dependent analysis-error covariance, Mon. Wea. Rev., 131, 1741, 10.1175/2559.1

Hamill, 2013, NOAA’s second-generation global medium-range ensemble reforecast dataset, Bull. Amer. Meteor. Soc., 94, 1553, 10.1175/BAMS-D-12-00014.1

Hersbach, 2000, Decomposition of the continuous ranked probability score for ensemble prediction systems, Wea. Forecasting, 15, 559, 10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2

Houtekamer, 2001, A sequential ensemble Kalman filter for atmospheric data assimilation, Mon. Wea. Rev., 129, 123, 10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2

Jung, 2008, Scale-dependent verification of ensemble forecasts, Quart. J. Roy. Meteor. Soc., 134, 973, 10.1002/qj.255

Leutbecher, 2008, Ensemble forecasting, J. Comput. Phys., 227, 3515, 10.1016/j.jcp.2007.02.014

Lewis, 2005, Roots of ensemble forecasting, Mon. Wea. Rev., 133, 1865, 10.1175/MWR2949.1

Matérn, 1986

Pinson, 2012, Evaluating the quality of scenarios of short-term wind power generation, Appl. Energy, 96, 12, 10.1016/j.apenergy.2011.11.004

Pinson, 2013

Roulston, 2002, Evaluating probabilistic forecasts using information theory, Mon. Wea. Rev., 130, 1653, 10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2

Schefzik, 2013, Uncertainty quantification in complex simulation models using ensemble copula coupling, Stat. Sci., 28, 616, 10.1214/13-STS443

Smith, 2004, Extending the limits of ensemble forecast verification with the minimum spanning tree, Mon. Wea. Rev., 132, 1522, 10.1175/1520-0493(2004)132<1522:ETLOEF>2.0.CO;2

Thorarinsdottir, 2010, Probabilistic forecasts of wind speed: Ensemble model output statistics using heteroskedastic censored regression, J. Roy. Stat. Soc., 173A, 371, 10.1111/j.1467-985X.2009.00616.x

Thorarinsdottir, 2015, 10.1080/10618600.2014.977447

Wilks, 2004, The minimum spanning tree histogram as verification tool for multidimensional ensemble forecasts, Mon. Wea. Rev., 132, 1329, 10.1175/1520-0493(2004)132<1329:TMSTHA>2.0.CO;2

Wilks, 2014, Multivariate ensemble Model Output Statistics using empirical copulas, Quart. J. Roy. Meteor. Soc., 10.1002/qj.2414

Ziegel, 2014, 10.1214/14-EJS964

Thông tin
Thông tin xuất bản

Monthly Weather Review

Tập 143 Số 4

1321-1334

Thông tin tác giả