Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)

Journal of Applied Ecology - Tập 43 Số 6 - Trang 1223-1232 - 2006
Omri Allouche1, Asaf Tsoar1, Ronen Kadmon1
1Department of Evolution, Systematics and Ecology, Institute of Life Sciences, The Hebrew University, Givat-Ram, Jerusalem 91904, Israel

Tóm tắt

Summary In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence. Synthesis and applications. Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.

Từ khóa


Tài liệu tham khảo

10.1175/WAF854.1

10.1111/j.1365-2486.2004.00828.x

10.1111/j.1365-2486.2005.01000.x

10.1111/j.1365-2664.2006.01136.x

10.1016/j.ecolmodel.2005.01.030

10.1111/j.0906-7590.2004.03553.x

10.1111/j.1365-2486.2005.00997.x

10.1111/j.0906-7590.2004.03764.x

10.1016/0895-4356(93)90018-V

10.1016/0006-3207(95)00102-6

10.1016/0895-4356(90)90159-M

10.1177/001316446002000104

10.1046/j.1365-2699.2000.00408.x

10.1046/j.1365-2699.2000.00419.x

10.1175/1520-0434(1990)005<0576:OSMOSI>2.0.CO;2

10.1175/1520-0434(2003)018<0953:OECMFS>2.0.CO;2

10.1111/j.0021-8901.2004.00881.x

10.1016/S0304-3800(02)00327-7

10.1017/S0376892997000088

10.1046/j.1365-2699.2003.00914.x

10.1111/j.1461-0248.2005.00792.x

10.1111/j.1365-2664.2006.01164.x

10.1148/radiology.143.1.7063747

10.1177/000456329303000601

10.1016/S0895-4356(99)00174-2

10.1038/28843

10.1111/j.1461-0248.2004.00598.x

10.1890/1051-0761(2003)013[0853:ASAOFA]2.0.CO;2

10.1016/0895-4356(95)00571-4

10.1111/j.0906-7590.2005.03957.x

10.1111/j.1523-1739.2003.00233.x

10.1175/1520-0434(2000)015<0103:VOQPFF>2.0.CO;2

10.1111/j.0021-8901.2004.00943.x

10.1016/S0304-3800(99)00113-1

10.1046/j.1365-2664.2001.00647.x

10.1038/35012251

10.1037/1040-3590.7.3.404

Nix H.A., 1986, Atlas of Elapid Snakes of Australia, 4

10.1111/j.0021-8901.2004.00910.x

10.1577/1548-8659(2002)131<0329:PMOFSD>2.0.CO;2

10.1111/j.1472-4642.2004.00051.x

10.1111/j.0906-7590.2004.03822.x

10.1016/S0304-3800(00)00322-7

10.1111/j.0906-7590.2004.03740.x

Pearson R.G., 2006, Model‐based uncertainty in species’ range prediction, Journal of Biogeography, 10.1111/j.1365-2699.2006.01460.x

10.1046/j.1523-1739.2003.02206.x

10.1034/j.1600-0587.2003.03545.x

Reese G.C., 2005, Factors affecting species distribution predictions. A simulation modelling experiment, Ecological Applications, 15, 556, 10.1890/03-5374

10.1111/j.1366-9516.2004.00118.x

10.1111/j.0021-8901.2004.00903.x

10.17161/bi.v2i0.9

10.1175/1520-0434(2002)017<0832:COWCRF>2.0.CO;2

10.1111/j.1366-9516.2005.00185.x

10.1111/j.1365-2699.2004.01076.x

10.1016/j.ecolmodel.2004.12.012

10.2307/2845837

10.1111/j.0906-7590.2004.03823.x

10.1080/136588199241391

10.1016/S0304-3800(01)00388-X

Thuiller W., 2003, BIOMOD: optimising predictions of species distributions and projecting potential future shifts under global change, Global Change Biology, 9, 1353, 10.1046/j.1365-2486.2003.00666.x

10.1111/j.1466-822X.2005.00162.x

10.1073/pnas.0409902102

10.1111/j.1365-2486.2005.001018.x

10.1016/j.apmr.2003.12.002

10.1016/j.biocon.2004.01.009

10.1111/j.1523-1739.2003.00359.x

10.1111/j.1365-2664.2005.01052.x

Zweig M.H., 1993, Receiver‐operating characteristics (ROC) plots: a fundamental evaluation tool in clinical medicine, Clinical Chemistry, 39, 561, 10.1093/clinchem/39.4.561

Thông tin
Thông tin xuất bản

Journal of Applied Ecology

Tập 43 Số 6

1223-1232

Thông tin tác giả