Reclaiming Profiler Ground Truth: Determining and Improving the Accuracy of CPU Profilers in the Presence of Dynamic Compilation

In the presence of highly optimising compilation and modern hardware, understanding where a program spends its time is complex. Profilers are used to understand program execution, but they introduce their own complexity through the observer effect, the phenomenon where measuring something changes the thing being measured. In this work, we empirically evaluate sampling- and instrumentation-based profilers. For sampling-based profilers, we evaluate how often results change across multiple runs, giving an indication of self-precision, and compare profilers against each other to determine their disagreement when profiling the same program execution. For instrumentation-based profilers, we investigate accuracy, overhead, and their effect on compilation through additional generated code and compilation statistics. To alleviate known issues such as the observer effect, we implement our own profiler, BuboC, that makes use of late-compiler-stage instrumentation to avoid interfering with compilation decisions and reduce the tool's overhead. We solve a key limitation in evaluating profilers: the absence of a known ground truth against which to compare results. Without knowing the correct execution profile, it is extremely difficult to determine whether a profiler is accurate, or to what degree. To address this, we introduce a novel approach based on deliberate machine-code slowdown. We insert additional instructions to induce a controlled slowdown, allowing us to evaluate a profiler based on its ability to correctly attribute this additional execution time. This provides a practical approach to measuring profiler accuracy and the degree of that accuracy. Finally, we evaluate a loop-level instrumentation profiler using late-compiler-stage instrumentation. We find that this loop-level profiler provides more accurate measurement of how much time the CPU spends in loops than sampling-based profilers, while maintaining practical overhead and reduced observer effect. Overall, this work provides a clear identification of the issues present in modern profiling under highly optimising compilation and hardware complexity, as well as an approach to alleviate these issues and improve profiler accuracy.