Skip to main content
Kent Academic Repository

G-AUC: An improved metric for classification model selection

Sadafule, Shashank, Sarkar, Sobhan, Wu, Shaomin (2023) G-AUC: An improved metric for classification model selection. In: THE 26th INTERNATIONAL COMPUTER SCIENCE AND ENGINEERING CONFERENCE 2022. . (doi:10.1109/ICSEC56337.2022.10049319) (Access to this publication is currently restricted. You may be able to access a copy if URLs are provided) (KAR id:99277)

PDF Author's Accepted Manuscript
Language: English

Restricted to Repository staff only

Contact us about this Publication
[thumbnail of m54178-sadafule final.pdf]
Official URL:
https://doi.org/10.1109/ICSEC56337.2022.10049319

Abstract

The performance of classification models is often measured using the metric, area under the curve (AUC). The non-parametric estimate of this metric only considers the ranks of the test instances and fails to consider the predicted scores of the model. Consequently, not all the valuable information about the model’s output is utilized. To address this issue, the present paper introduces a new metric, called Gamma AUC (G-AUC) that can take into account both ranks as well as scores. The parameter G tackles the problem of overfitting scores into the metric. To validate the proposed metric, we tested it on 20 UCI datasets with 10 state-of-the-art models. Out of all the values of the parameter G that we tested, four of them got p-value less than 0.05 for the alternative hypothesis that, on the training sets, G-AUC has a greater correlation than AUC itself, with AUC on test sets. Furthermore, for all values of G considered, G-AUC always won majority of the times than AUC for selecting better models.

Item Type: Conference or workshop item (Paper)
DOI/Identification number: 10.1109/ICSEC56337.2022.10049319
Additional information: © 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Uncontrolled keywords: Classification, Performance metric, AUC, G-AUG
Subjects: Q Science > Q Science (General) > Q335 Artificial intelligence
Divisions: Divisions > Kent Business School - Division > Department of Analytics, Operations and Systems
Depositing User: Shaomin Wu
Date Deposited: 21 Dec 2022 18:11 UTC
Last Modified: 08 Mar 2023 15:34 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/99277 (The current URI for this page, for reference purposes)

University of Kent Author Information

  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.