Cockayne, Jon and Graham, Matthew M. and Oates, Chris J. and Sullivan, T. J. and Teymur, Onur (2020) Testing whether a Learning Procedure is Calibrated. [Preprint] (doi:10.48550/arXiv.2012.12670) (The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided) (KAR id:90435)
The full text of this publication is not currently available from this repository. You may be able to access a copy if URLs are provided. | |
Official URL: https://doi.org/10.48550/arXiv.2012.12670 |
Abstract
A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.
Item Type: | Preprint |
---|---|
DOI/Identification number: | 10.48550/arXiv.2012.12670 |
Refereed: | No |
Other identifier: | https://arxiv.org/abs/2012.12670 |
Name of pre-print platform: | arXiv |
Subjects: | Q Science > QA Mathematics (inc Computing science) > QA273 Probabilities |
Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Mathematics, Statistics and Actuarial Science |
Depositing User: | Onur Teymur |
Date Deposited: | 28 Sep 2021 14:50 UTC |
Last Modified: | 10 Oct 2023 11:19 UTC |
Resource URI: | https://kar.kent.ac.uk/id/eprint/90435 (The current URI for this page, for reference purposes) |
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):