Cockayne, Jon,
Graham, Matthew M.,
Oates, Chris J.,
Sullivan, T. J.,
Teymur, Onur
(2020)
*
Testing whether a Learning Procedure is Calibrated.
*
.
(Submitted)
(The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided)
(KAR id:90435)

The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided. (Contact us about this Publication) | |

Official URL https://arxiv.org/abs/2012.12670 |

## Abstract

A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

Item Type: | Article |
---|---|

Subjects: | Q Science > QA Mathematics (inc Computing science) > QA273 Probabilities |

Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Mathematics, Statistics and Actuarial Science |

Depositing User: | Onur Teymur |

Date Deposited: | 28 Sep 2021 14:50 UTC |

Last Modified: | 11 Oct 2021 16:07 UTC |

Resource URI: | https://kar.kent.ac.uk/id/eprint/90435 (The current URI for this page, for reference purposes) |

- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV

- Depositors only (login required):