Reducing performance non-determinism via cache-aware page allocation strategies

Performance non-determinism in computer systems complicates evaluation, use, and even development of these systems. In performance evaluation via benchmarking and simulation, non-determinism requires long executions and more complex experiment design. Real-time systems are hard to dimension and tune with non-determinism. The slower benchmarking also slows down system development, as it takes developers longer to see performance implications of their modifications. Cache-unaware physical page allocation in an operating system is believed to be a significant cause of non-determinism, but there is no published empirical study that would confirm it. We provide such a study for the Linux operating system, comparing the default cache-unaware page allocation strategy to known cache-aware strategies, page coloring and bin hopping. We have implemented a framework for page allocation strategies in the Linux kernel, employed it for these two strategies, and measured the non-determinism on a large and diverse set of benchmarks. We propose a statistical technique which allows to classify different kinds of performance non-determinism and evaluate their magnitudes. Application of our technique reveals that the two strategies do reduce performance non-determinism without significantly increasing mean response time.