Using Bayesian Model Averaging to Calibrate Forecast Ensembles

Monthly Weather Review - Tập 133 Số 5 - Trang 1155-1174 - 2005
Adrian E. Raftery1,2, Tilmann Gneiting2, Leonhard Held2, Michael Polakowski2
1Department of Statistics, University of Washington, Box 354320, Seattle, WA 98195-4320.
2Department of Statistics, University of Washington, Seattle, Washington

Tóm tắt

Abstract Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models' relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution. The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive. The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in January–June 2000 using the University of Washington fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed.

Từ khóa


Tài liệu tham khảo

Anderson, 1996, A method for producing and evaluating probabilistic forecasts from ensemble model integrations., J. Climate, 9, 1518, 10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2

Buizza, 1997, Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF ensemble prediction system., Mon. Wea. Rev, 125, 99, 10.1175/1520-0493(1997)125<0099:PFSOEP>2.0.CO;2

Buizza, 1999, Stochastic representation of model uncertainties in the ECMWF ensemble prediction system., Quart. J. Roy. Meteor. Soc, 125, 2887, 10.1002/qj.49712556006

Buizza, 2005, A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems., Mon. Wea. Rev, 133, 1076, 10.1175/MWR2905.1

Carter, 1989, Statistical forecasts based on the National Meteorological Center's numerical weather prediction system., Wea. Forecasting, 4, 401, 10.1175/1520-0434(1989)004<0401:SFBOTN>2.0.CO;2

Casella, 2001, Statistical Inference.

Dempster, 1977, Maximum likelihood from incomplete data via the EM algorithm., J. Roy. Stat. Soc, 39B, 1

Eckel, 1998, Calibrated probabilistic quantitative precipitation forecasts based on the MRF ensemble., Wea. Forecasting, 13, 1132, 10.1175/1520-0434(1998)013<1132:CPQPFB>2.0.CO;2

Eckel, 2003, Towards an effective short-range ensemble forecast system.

Epstein, 1969, Stochastic dynamic prediction., Tellus, 21, 739, 10.3402/tellusa.v21i6.10143

Fisher, 1922, On the mathematical foundations of theoretical statistics., Philos. Trans. Roy. Soc. London, 222A, 309

Gel, 2004, Calibrated probabilistic mesoscale weather field forecasting: The geostatistical output perturbation (GOP) method (with discussion)., J. Amer. Stat. Assoc, 99, 575, 10.1198/016214504000000872

Glahn, 1972, The use of model output statistics (MOS) in objective weather forecasting., J. Appl. Meteor, 11, 1202, 10.1175/1520-0450(1972)011<1203:TUOMOS>2.0.CO;2

Gneiting, 2003, Verifying probabilistic forecasts: Calibration and sharpness.

Good, 1952, Rational decisions., J. Roy. Stat. Soc, 14, 107

Grimit, 2002, Initial results of a mesoscale short-range ensemble forecasting system over the Pacific Northwest., Wea. Forecasting, 17, 192, 10.1175/1520-0434(2002)017<0192:IROAMS>2.0.CO;2

Hamill, 2001, Interpretation of rank histograms for verifying ensemble forecasts., Mon. Wea. Rev, 129, 550, 10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2

Hamill, 1997, Verification of Eta–RSM short-range ensemble forecasts., Mon. Wea. Rev, 125, 1312, 10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2

Hamill, 1998, Evaluation of Eta–RSM ensemble probabilistic precipitation forecasts., Mon. Wea. Rev, 126, 711, 10.1175/1520-0493(1998)126<0711:EOEREP>2.0.CO;2

Hamill, 2000, A comparison of probabilistic forecasts from bred, singular-vector, and perturbed observation ensembles., Mon. Wea. Rev, 128, 1835, 10.1175/1520-0493(2000)128<1835:ACOPFF>2.0.CO;2

Hamill, 2004, Ensemble re-forecasting: Improving medium-range forecast skill using retrospective forecasts., Mon. Wea. Rev, 132, 1434, 10.1175/1520-0493(2004)132<1434:ERIMFS>2.0.CO;2

Hersbach, 2000, Decomposition of the continuous ranked probability score for ensemble prediction systems., Wea. Forecasting, 15, 559, 10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2

Hersbach, 2000, A short-range to early-medium-range ensemble prediction system for the European area., Mon. Wea. Rev, 128, 3501, 10.1175/1520-0493(2000)128<3501:ASRTEM>2.0.CO;2

Hoeting, 1999, Bayesian model averaging: A tutorial (with discussion)., Stat. Sci, 14, 382

Hou, 2001, Objective verification of the SAMEX'98 ensemble forecast., Mon. Wea. Rev, 129, 73, 10.1175/1520-0493(2001)129<0073:OVOTSE>2.0.CO;2

Houtekamer, 1993, Global and local skill forecasts., Mon. Wea. Rev, 121, 1834, 10.1175/1520-0493(1993)121<1834:GALSF>2.0.CO;2

Houtekamer, 1995, Methods for ensemble prediction., Mon. Wea. Rev, 123, 2181, 10.1175/1520-0493(1995)123<2181:MFEP>2.0.CO;2

Houtekamer, 1998, Data assimilation using an ensemble Kalman filter technique., Mon. Wea. Rev, 126, 796, 10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2

Houtekamer, 2001, A sequential ensemble Kalman filter for atmospheric data assimilation., Mon. Wea. Rev, 129, 123, 10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2

Kass, 1995, Bayes factors., J. Amer. Stat. Assoc, 90, 773, 10.1080/01621459.1995.10476572

Kharin, 2002, Climate predictions with multimodel ensembles., J. Climate, 15, 793, 10.1175/1520-0442(2002)015<0793:CPWME>2.0.CO;2

Krishnamurti, 1999, Improved weather and seasonal climate forecasts from multimodel superensembles., Science, 258, 1548, 10.1126/science.285.5433.1548

Leamer, 1978, Specification Searches.

Leith, 1974, Theoretical skill of Monte-Carlo forecasts., Mon. Wea. Rev, 102, 409, 10.1175/1520-0493(1974)102<0409:TSOMCF>2.0.CO;2

McCullagh, 1989, Generalized Linear Models., 10.1007/978-1-4899-3242-6

McLachlan, 1997, The EM Algorithm and Extensions.

McLachlan, 2000, Finite Mixture Models., 10.1002/0471721182

Molteni, 1996, The ECMWF ensemble system: Methodology and validation., Quart. J. Roy. Meteor. Soc, 122, 73, 10.1002/qj.49712252905

Pellerin, 2003, Increasing the horizontal resolution of ensemble forecasts at CMC., Nonlinear Processes Geophys, 10, 463, 10.5194/npg-10-463-2003

Raftery, 1993, Bayesian model selection in structural equation models.

Raftery, 2003, Long-run performance of Bayesian model averaging., J. Amer. Stat. Assoc, 98, 931, 10.1198/016214503000000891

Roulston, 2002, Evaluating probabilistic forecasts using information theory., Mon. Wea. Rev, 130, 1653, 10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2

Roulston, 2003, Combining dynamical and statistical ensembles., Tellus, 55A, 16, 10.3402/tellusa.v55i1.12082

Scherrer, 2004, Analysis of the spread-skill relations using the ECMWF ensemble prediction system over Europe., Wea. Forecasting, 19, 552, 10.1175/1520-0434(2004)019<0552:AOTSRU>2.0.CO;2

Stefanova, 2002, Interpretation of seasonal climate forecast using Brier skill score, the Florida State University superensemble, and the AMIP-I dataset., J. Climate, 15, 537, 10.1175/1520-0442(2002)015<0537:IOSCFU>2.0.CO;2

Stensrud, 1999, Using ensembles for short-range forecasting., Mon. Wea. Rev, 127, 433, 10.1175/1520-0493(1999)127<0433:UEFSRF>2.0.CO;2

Stern, 1984, A model fitting analysis of daily rainfall data (with discussion)., J. Roy. Stat. Soc, 147A, 1

Talagrand, O., R.Vautard, and B.Strauss, 1997: Evaluation of probabilistic prediction systems. Proc. ECMWF Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–25. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

Toth, 1993, Ensemble forecasting at the NMC: The generation of perturbations., Bull. Amer. Meteor. Soc, 74, 2317, 10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2

Toth, 2001, The use of ensembles to identify forecasts with small and large uncertainty., Wea. Forecasting, 16, 463, 10.1175/1520-0434(2001)016<0463:TUOETI>2.0.CO;2

Van den Dool, 1994, On the weights for an ensemble-averaged 6–10-day forecast., Wea. Forecasting, 9, 457, 10.1175/1520-0434(1994)009<0457:OTWFAE>2.0.CO;2

Wilks, 1995, Statistical Methods in the Atmospheric Sciences.

Wilks, 2002, Smoothing forecast ensembles with fitted probability distributions., Quart. J. Roy. Meteor. Soc, 128, 2821, 10.1256/qj.01.215

Wu, 1983, On the convergence properties of the EM algorithm., Ann. Stat, 11, 95, 10.1214/aos/1176346060

Thông tin
Thông tin xuất bản

Monthly Weather Review

Tập 133 Số 5

1155-1174

Thông tin tác giả