Skip to main content
Kent Academic Repository

Testing whether a Learning Procedure is Calibrated

Cockayne, Jon and Graham, Matthew M. and Oates, Chris J. and Sullivan, T. J. and Teymur, Onur (2020) Testing whether a Learning Procedure is Calibrated. [Preprint] (doi:10.48550/arXiv.2012.12670) (The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided) (KAR id:90435)

The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided.
Official URL:
https://doi.org/10.48550/arXiv.2012.12670

Abstract

A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

Item Type: Preprint
DOI/Identification number: 10.48550/arXiv.2012.12670
Refereed: No
Other identifier: https://arxiv.org/abs/2012.12670
Name of pre-print platform: arXiv
Subjects: Q Science > QA Mathematics (inc Computing science) > QA273 Probabilities
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Mathematics, Statistics and Actuarial Science
Depositing User: Onur Teymur
Date Deposited: 28 Sep 2021 14:50 UTC
Last Modified: 10 Oct 2023 11:19 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/90435 (The current URI for this page, for reference purposes)

University of Kent Author Information

Teymur, Onur.

Creator's ORCID:
CReDIT Contributor Roles:
  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.