Goodness-of-fit of integrated population models using calibrated simulation

1. Integrated population modelling is proving to be an important and useful technique in statistical ecology.

However, there is currently no simple formal method for judging how well models fit data, when potentially sev-

eral different data sets described by different structured models are being analysed in combination.

2. We propose and evaluate a new approach, of calibrated simulation. Here, comparative data sets are obtained

from simulating data when model parameter values are obtained from the assumed asymptotic normal distribu-

tion of the maximum-likelihood estimators from the real data. The approach is motivated and justified by Baye-

sian P-values. Calibration of the resulting statistics is achieved as repeated data sets are easily simulated from the

fitted model. The method requires the specification of model discrepancy measures, and we show how different

measures can highlight different aspects of fit.

3. Calibration is only strictly necessary if the statistics proposed may appear to be extreme.

4. The approach of using calibrated simulation to check the goodness-of-fit of integrated population models is

demonstrated by application to data sets on lapwings and herons. In each case, there are two data sets involved

in the integrated analysis, and for each component data set, discrepancy measures of goodness-of-fit are

obtained. For the lapwing application, as replication is efficient, it is possible to calibrate the procedure simply by

using additional simulations. The heron application is shown to be feasible, but is substantially harder to cali-

brate, due to the presence of productivity thresholds that need to be estimated using profile likelihood methods.

We demonstrate the importance of taking more than one discrepancy measure for time-series data. Avenues for

future research are outlined. This article has supplementary materials on line.