An Improved Model Selection Heuristic for AUC

Wu, Shaomin and Flach, Peter A. and Ferri, Cesar (2007) An Improved Model Selection Heuristic for AUC. Lecture Notes in Computer Science, 4701 (NA). pp. 478-489. ISSN 978-3-540-74975-2. (Full text available)

PDF
Download (336kB) Preview
[img]
Preview

Abstract

The area under the ROC curve (AUC) has been widely used to measure ranking performance for binary classification tasks. AUC only employs the classifier’s scores to rank the test instances; thus, it ignores other valuable information conveyed by the scores, such as sensitivity to small differences in the score values. However, as such differences are inevitable across samples, ignoring them may lead to overfitting the validation set when selecting models with high AUC. This problem is tackled in this paper. On the basis of ranks as well as scores, we introduce a new metric called scored AUC (sAUC), which is the area under the sROC curve. The latter measures how quickly AUC deteriorates if positive scores are decreased. We study the interpretation and statistical properties of sAUC. Experimental results on UCI data sets convincingly demonstrate the effectiveness of the new metric for classifier evaluation and selection in the case of limited validation data.

Item Type: Article
Subjects: H Social Sciences
H Social Sciences > HA Statistics > HA33 Management Science
Divisions: Faculties > Social Sciences > Kent Business School
Faculties > Social Sciences > Kent Business School > Management Science
Depositing User: Shaomin Wu
Date Deposited: 29 Oct 2012 12:24
Last Modified: 04 Jul 2014 13:19
Resource URI: http://kar.kent.ac.uk/id/eprint/32007 (The current URI for this page, for reference purposes)
ORCiD (Wu, Shaomin):
ORCiD (Flach, Peter A.):
ORCiD (Ferri, Cesar):
  • Depositors only (login required):

Downloads

Downloads per month over past year