Skip to main content

Testing whether a Learning Procedure is Calibrated

Cockayne, Jon, Graham, Matthew M., Oates, Chris J., Sullivan, T. J., Teymur, Onur (2020) Testing whether a Learning Procedure is Calibrated. . (Submitted) (The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided) (KAR id:90435)

The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided. (Contact us about this Publication)
Official URL
https://arxiv.org/abs/2012.12670

Abstract

A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

Item Type: Article
Subjects: Q Science > QA Mathematics (inc Computing science) > QA273 Probabilities
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Mathematics, Statistics and Actuarial Science
Depositing User: Onur Teymur
Date Deposited: 28 Sep 2021 14:50 UTC
Last Modified: 11 Oct 2021 16:07 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/90435 (The current URI for this page, for reference purposes)
  • Depositors only (login required):