Intercomparison of Spatial Forecast Verification Methods

Weather and Forecasting - Tập 24 Số 5 - Trang 1416-1430 - 2009
Eric Gilleland1, David Ahijevych1, Barbara G. Brown1, Barbara Casati2, Elizabeth E. Ebert3
1National Center for Atmospheric Research Boulder, Colorado
2Ouranos, Montreal, Quebec, Canada
3Center for Australian Weather and Climate Research, Melbourne, Victoria, Australia

Tóm tắt

Abstract Advancements in weather forecast models and their enhanced resolution have led to substantially improved and more realistic-appearing forecasts for some variables. However, traditional verification scores often indicate poor performance because of the increased small-scale variability so that the true quality of the forecasts is not always characterized well. As a result, numerous new methods for verifying these forecasts have been proposed. These new methods can mostly be classified into two overall categories: filtering methods and displacement methods. The filtering methods can be further delineated into neighborhood and scale separation, and the displacement methods can be divided into features based and field deformation. Each method gives considerably more information than the traditional scores, but it is not clear which method(s) should be used for which purpose. A verification methods intercomparison project has been established in order to glean a better understanding of the proposed methods in terms of their various characteristics and to determine what verification questions each method addresses. The study is ongoing, and preliminary qualitative results for the different approaches applied to different situations are described here. In particular, the various methods and their basic characteristics, similarities, and differences are described. In addition, several questions are addressed regarding the application of the methods and the information that they provide. These questions include (i) how the method(s) inform performance at different scales; (ii) how the methods provide information on location errors; (iii) whether the methods provide information on intensity errors and distributions; (iv) whether the methods provide information on structure errors; (v) whether the approaches have the ability to provide information about hits, misses, and false alarms; (vi) whether the methods do anything that is counterintuitive; (vii) whether the methods have selectable parameters and how sensitive the results are to parameter selection; (viii) whether the results can be easily aggregated across multiple cases; (ix) whether the methods can identify timing errors; and (x) whether confidence intervals and hypothesis tests can be readily computed.

Từ khóa


Tài liệu tham khảo

Ahijevych, 2009, Application of spatial verification methods to idealized and NWP gridded precipitation forecasts., Wea. Forecasting, 10.1175/2009WAF2222298.1

Alexander, 1999, The effect of assimilating rain rates derived from satellites and lightning on forecasts of the 1993 Superstorm., Mon. Wea. Rev., 127, 1433, 10.1175/1520-0493(1999)127<1433:TEOARR>2.0.CO;2

Atger, 2001, Verification of intense precipitation forecasts from single models and ensemble prediction systems., Nonlinear Processes Geophys., 8, 401, 10.5194/npg-8-401-2001

Baldwin, 2003, Development of an events-oriented verification system using data mining and image processing algorithms.

Baldwin, 2006, Sensitivity of several performance measures to displacement error, bias, and event frequency., Wea. Forecasting, 21, 636, 10.1175/WAF933.1

Briggs, 1997, Wavelets and field forecast verification., Mon. Wea. Rev., 125, 1329, 10.1175/1520-0493(1997)125<1329:WAFFV>2.0.CO;2

Brooks, 1998, Objective limits on forecasting skill of rare events.

Casati, 2009, New developments of the intensity-scale technique within the Spatial Verification Methods Inter-Comparison Project., Wea. Forecasting

Casati, 2004, A new intensity-scale approach for the verification of spatial precipitation forecasts., Meteor. Appl., 11, 141, 10.1017/S1350482704001239

Casati, 2008, Forecast verification: Current status and future directions., Meteor. Appl., 15, 3, 10.1002/met.52

Damrath, 2004, Verification against precipitation observations of a high density network—What did we learn?

Davis, 2006, Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas., Mon. Wea. Rev., 134, 1772, 10.1175/MWR3145.1

Davis, 2009, The Method for Object-based Diagnostic Evaluation (MODE) applied to WRF forecasts from the 2005 Spring Program., Wea. Forecasting, 24, 1252, 10.1175/2009WAF2222241.1

Dickinson, 1996, A study of near-surface winds in marine cyclones using multiple satellite sensors., J. Appl. Meteor., 35, 769, 10.1175/1520-0450(1996)035<0769:ASONSW>2.0.CO;2

Ebert, 2008, Fuzzy verification of high resolution gridded forecasts: A review and proposed framework., Meteor. Appl., 15, 51, 10.1002/met.25

Ebert, 2009, Neighborhood verification: A strategy for rewarding close forecasts., Wea. Forecasting, 10.1175/2009WAF2222251.1

Ebert, 2000, Verification of precipitation in weather systems: Determination of systematic errors., J. Hydrol., 239, 179, 10.1016/S0022-1694(00)00343-7

Ebert, 2009, Toward better understanding of the contiguous rain area (CRA) method for spatial forecast verification., Wea. Forecasting, 24, 1401, 10.1175/2009WAF2222252.1

Ebert, 2004, Verification of nowcasts from the WWRP Sydney 2000 Forecast Demonstration Project., Wea. Forecasting, 19, 73, 10.1175/1520-0434(2004)019<0073:VONFTW>2.0.CO;2

Germann, 2004, Scale dependence of the predictability of precipitation from continental radar images. Part II: Probability forecasts., J. Appl. Meteor., 43, 74, 10.1175/1520-0450(2004)043<0074:SDOTPO>2.0.CO;2

Gilleland, 2008, Computationally efficient spatial forecast verification using Baddeley’s delta image metric., Mon. Wea. Rev., 136, 1747, 10.1175/2007MWR2274.1

Grams, 2006, The use of a modified Ebert–McBride technique to evaluate mesoscale model QPF as a function of convective system morphology during IHOP 2002., Wea. Forecasting, 21, 288, 10.1175/WAF918.1

Harris, 2001, Multiscale statistical properties of a high-resolution precipitation forecast., J. Hydrometeor., 2, 406, 10.1175/1525-7541(2001)002<0406:MSPOAH>2.0.CO;2

Hoffman, 1995, Distortion representation of forecast errors., Mon. Wea. Rev., 123, 2758, 10.1175/1520-0493(1995)123<2758:DROFE>2.0.CO;2

Huttenlocher, 1999, Object recognition using subspace methods., IEEE Trans. Pattern Anal. Mach. Intell., 21, 951, 10.1109/34.790437

Jolliffe, 2003, Forecast Verification. A Practitioner’s Guide in Atmospheric Science.

Keil, 2007, A displacement-based error measure applied in a regional ensemble forecasting system., Mon. Wea. Rev., 135, 3248, 10.1175/MWR3457.1

Keil, 2009, A displacement and amplitude score employing an optical flow technique., Wea. Forecasting, 24, 1297, 10.1175/2009WAF2222247.1

Lack, 2009, An object-oriented multiscale verification scheme., Wea. Forecasting

Marchok, 2002, How the NCEP tropical cyclone tracker works.

Marsigli, 2006, Verification of the COSMOLEPS new suite in terms of precipitation distribution.

Marzban, 2006, Cluster analysis for verification of precipitation fields., Wea. Forecasting, 21, 824, 10.1175/WAF948.1

Marzban, 2008, Cluster analysis for object-oriented verification of fields: A variation., Mon. Wea. Rev., 136, 1013, 10.1175/2007MWR1994.1

Marzban, 2009, Verification with variograms., Wea. Forecasting, 24, 1102, 10.1175/2009WAF2222122.1

Marzban, 2009, Three spatial verification techniques: Cluster analysis, variogram, and optical flow., Wea. Forecasting, 10.1175/2009WAF2222261.1

Mesinger, 2008, Bias-adjusted precipitation threat scores., Adv. Geosci., 16, 137, 10.5194/adgeo-16-137-2008

Micheas, 2007, Cell identification and verification of QPF ensembles using shape analysis techniques., J. Hydrol., 343, 105, 10.1016/j.jhydrol.2007.05.036

Mittermaier, 2006, Using an intensity-scale technique to assess the added benefit of high-resolution model precipitation forecasts., Atmos. Sci. Lett., 7, 35, 10.1002/asl.127

Mittermaier, 2009, Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score., Wea. Forecasting

Nachamkin, 2004, Mesoscale verification using meteorological composites., Mon. Wea. Rev., 132, 941, 10.1175/1520-0493(2004)132<0941:MVUMC>2.0.CO;2

Nachamkin, 2009, Application of the composite method to the Spatial Forecast Verification Methods Intercomparison Dataset., Wea. Forecasting, 24, 1390, 10.1175/2009WAF2222225.1

Nachamkin, 2005, Evaluation of heavy precipitation forecasts using composite-based methods: A distributions-oriented approach., Mon. Wea. Rev., 133, 2163, 10.1175/MWR2975.1

Nehrkorn, 2003, Feature calibration and alignment to represent model forecast errors: Empirical regularization., Quart. J. Roy. Meteor. Soc., 129, 195, 10.1256/qj.02.18

Nychka, 1998, Spatial process estimates as smoothers.

Ogden, 1996, Essential Wavelets for Statistical Applications and Data Analysis.

Rezacova, 2007, A radar-based verification of precipitation forecast for local convective storms., Atmos. Res., 83, 221, 10.1016/j.atmosres.2005.08.011

Roberts, 2005, An investigation of the ability of a storm-scale configuration of the Met Office NWP model to predict flood-producing rainfall.

Roberts, 2008, Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events., Mon. Wea. Rev., 136, 78, 10.1175/2007MWR2123.1

Templeton, 1982, Tropical cyclone strike probability forecasting in the Australian region.

Theis, 2005, Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach., Meteor. Appl., 12, 257, 10.1017/S1350482705001763

Tustison, 2003, Scale-recursive estimation for multisensory quantitative forecast verification: A preliminary assessment., J. Geophys. Res., 108, 8377, 10.1029/2001JD001073

Venugopal, 2005, A new metric for comparing precipitation patterns with an application to ensemble forecasts., J. Geophys. Res., 110, D08111, 10.1029/2004JD005395

Wernli, 2008, SAL—A novel quality measure for the verification of quantitative precipitation forecasts., Mon. Wea. Rev., 136, 4470, 10.1175/2008MWR2415.1

Wernli, 2009, Spatial Forecast Verification Methods Intercomparison project: Application of the SAL technique., Wea. Forecasting, 10.1175/2009WAF2222271.1

Weygandt, 2004, Scale sensitivities in model precipitation skill scores during IHOP.

Wilks, 2005, Statistical Methods in the Atmospheric Sciences. An Introduction.

Yates, 2006, Point and areal validation of forecast precipitation fields., Meteor. Appl., 13, 1, 10.1017/S1350482705001921

Zepeda-Arce, 2000, Space–time rainfall organization and its role in validating quantitative precipitation forecasts., J. Geophys. Res., 105, 10129, 10.1029/1999JD901087

Thông tin
Thông tin xuất bản

Weather and Forecasting

Tập 24 Số 5

1416-1430

Thông tin tác giả