Item Response Modeling of Situational Judgment Tests With Items Nested Within Scenarios

Situational judgment tests (SJTs) are widely used to evaluate decision-making and problem-solving skills. These tests present respondents with situations (scenarios), and possible actions in response to each situation, which they must rate based on effectiveness or appropriateness. Despite having high face and criterion-related validities, these tests involve complex cognitive processes, and their construct validity has been notoriously difficult to establish. This is particularly challenging when several questions relate to the same scenario, thus creating local dependencies in item responses after controlling for the overall ability. Finding a suitable measurement model for SJTs would address the operational challenges of test development, maintenance, scoring, and equating the scores across multiple test forms and/or testing years. We propose to use a variant of the continuous response model (CRM; Samejima, 1973) to describe points gained at the level of scenario, regardless of the number of questions asked about the scenario. A simulation study testing the performance of this scenario-level model on data generated from categorical items nested within scenarios found that CRM fits well under a wide range of realistic conditions and resembles the true score better than a summated test score. We illustrate the proposed approach with an SJT presenting clinical dilemmas with several possible actions, each scored using four ordered categories. The SJT was completed by N = 2,740 examinees in operational assessment settings. The proposed CRM variant fitted the data well and was effective for assessing dimensionality, modeling and scoring responses, and evaluating test reliability. Implications for other operational uses such as form linking and pre-equating are discussed.