Two-point methods for assessing variability in simulation output

In simulation experiments, the form of the distribution of input variables is often not known precisely. The simulation output then contains two sources of variation: that caused by uncertainty in estimating unknown parameters, and that caused by the inclusion of random variation within the simulation model itself. Cheng and Holland (1996) have shown how the classical method of statistical differential analysis (often called the delta-method) can be used to assess the degree of variability arising from each source. The disadvantage of the delta-method is that the computational effort needed for this increases linearly with the number of unknown parameters. In this paper it is shown that the method can be modified to assess the combined effect on the response output of variation in all the parameters by making most simulation replications at just two settings of parameter values, making the method substantially independent of the number of unknown parameters. Thus, for problems where this number is large, such two-point methods are substantially more efficient than the unmodified delta-method. For illustration, simulation results on the operation of two different computer networks are given. The workload in assessing the accuracy of estimates using the proposed two-point methods is compared with that using the delta-method, showing the large efficiency gains possible using the two-point method.