Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)
Tóm tắt
In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence.
Từ khóa
Tài liệu tham khảo
Nix H.A., 1986, Atlas of Elapid Snakes of Australia, 4
Pearson R.G., 2006, Model‐based uncertainty in species’ range prediction, Journal of Biogeography, 10.1111/j.1365-2699.2006.01460.x
Reese G.C., 2005, Factors affecting species distribution predictions. A simulation modelling experiment, Ecological Applications, 15, 556, 10.1890/03-5374
Thuiller W., 2003, BIOMOD: optimising predictions of species distributions and projecting potential future shifts under global change, Global Change Biology, 9, 1353, 10.1046/j.1365-2486.2003.00666.x