Appendix D. Model evaluation.
Sensitivity and specificity are products of a confusion matrix used to evaluate binomial distribution models (See Table D1). A confusion matrix compares the ability of a habitat model to accurately predict observed presences and absences by tabulating true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) predictions. Sensitivity is a measure of commission error (TP/(TP+FN)) and specificity is a measure of omission error (TN/(TN+FP)). Sensitivity and specificity are calculated independent of each other and also independent of prevalence, the proportion of presence locations. Sensitivity and specificity values range from 0, indicating a high error rate, to 1 which describes perfect agreement between observed and predicted values.
Sensitivity and specificity have long been used to generate other measures of model performance (Fielding and Bell 1997, Pearce and Ferrier 2000, Manel et al. 2001, Segurado and Araujo 2004) such as receiver operator curves (ROC). The ROC technique generates probabilities, which are inappropriate for models based on binomial data. The Cohen’s kappa statistic (Cohen 1960) is a popular accuracy measure of binomial predictions, but many recent studies criticize the kappa statistic for its inherent dependence on prevalence and subsequent generation of bias (Fielding and Bell 1997, Manel et al. 2001, McPherson et al. 2004, Vaughan and Ormerod 2005). TSS is a variant of a ROC that is applicable as an accuracy measure of binomial models.
TABLE D1. A confusion matrix used to tabulate the predictive capacity of presence/absence models. TP = presence observed and predicted by model; FP = absence observed but predicted as a presence location; FN = presence observed but location predicted as absence; TN = absence observed and predicted by model.
LITERATURE CITED
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37–46.
Fielding, A. H., and J. F. Bell. 1997. A review of methods for the assessment of prediction errors in conservation presence-absence models. Environmental Conservation 24:38–49.
Manel, S., H. C. Williams, and S. J. Ormerod. 2001. Evaluating presence-absence models in ecology: the need to account for prevalence. Journal of Applied Ecology 38:921–931.
McPherson, J. M., W. Jetz, and D. J. Rogers. 2004. The effects of species' range sizes on the accuracy of distribution models: ecological phenomenon or statistical artefact? Journal of Applied Ecology 41:811–823.
Pearce, J., and S. Ferrier. 2000. Evaluating the predictive performance of habitat models developed using logistic regression. Ecological Modelling 133:225–245.
Segurado, P., and M. B. Araujo. 2004. An evaluation of methods for modelling species distributions. Journal of Biogeography 31:1555–1568.
Vaughan, I. P., and S. J. Ormerod. 2005. The continuing challenges of testing species distribution models. Journal of Applied Ecology 42:720–730.