A new critical realist approach for evaluating multimethodological practice in operational research

Understanding why practitioners take the actions they do in multimethodological operational research (OR), and the effects these actions have, is fundamental for evaluating and improving practice. But it is widely recognized in OR literature and amongst practitioners that objective, comparative evaluation of complex, real-world interventions is challenging if not impossible. It is not clear how multimethodology should be evaluated. Critical realism has been advocated for exploring causation to evaluate OR practice holistically, but not fully operationalized. For our collaborative research evaluating multimethodological practice supporting strategic decision-making in non-academic settings, we therefore developed a new approach. We synthesized and operationalized empirical research and theories in OR and recent developments in critical realism and social science. In this paper, we present our theoretical framework – a new critical realist ontology of multimethodological OR practice and methodologies – and our empirical research design. We illustrate the approach with a case study of an OR analysis for a business case for government investment in a national capability. We demonstrate how we investigated the intervention context and the complex process of practitioners’ choice of methods using interviews and project documents, thematic and comparative analysis, rich description, and structural and causal analysis. We explore how this new operationalization of critical realism for identifying and comparing causal mechanisms across practitioner case studies can be used: (1) to examine mechanisms leading to other actions and outcomes, including intervention success; and (2) to theorize and evaluate mixed methods research projects across disciplines, including their causal complexity and practical utility.