Does test length have an impact upon performance in the UK GP, licensing Applied Knowledge Test?

Aims and objectives.

Assessment is a necessary part of the processes of medical education, but assessment also needs to ensure the test is fair.

Various methods are used to assess competence in postgraduate medical examinations, according to the model described by (Miller, 1990) and it is accepted that all methods of assessment have both strengths and intrinsic flaws. Competence is viewed in a hierarchical manner. Jolly (2010) highlights the evidence that knowledge is an important attribute, separating the novice from the expert.

There is a tension between increasing the face validity of the items by including more clinical information in the vignette and the impact on performance of increased candidate cognitive load.

Cognitive load theory is a lens through which to view assessment (Sweller, 2010). According to this model, there are three types of cognitive load. Intrinsic load relates to the inherent difficulty of the subject matter. Germane load is the memory related to integrating new information whilst extraneous load considers the portion of cognition not related to learning. From the perspective of assessment, test difficulty should relate to the inherent difficulty of the subject.

Content presentation

We used a retrospective study design examining data comprising candidates’ results from AKT 34-45 (taken between October 2018 and May 2022). The candidate scores (centred on the pass mark for each individual test sitting) were compared to the overall word count of the paper. Logistic regression was used to analyse the effect of test length, adjusting for candidate factors known to impact upon examination performance.

We included the results from 19287 candidates. Preliminary analysis indicates that there is no significant association between examination score and test word count.

Relevance/impact

An external review of the MRCGP examination recommended that clinical vignettes in AKT test items should include more clinical information to increase face validity. It is important to ensure that this design change has not impacted upon the cognitive load and fairness of the test.

Outcomes/discussion

These results will provide information as part of the Quality Assurance processes of the AKT. Given the high stakes nature of the test, this is important for all stakeholders including candidates, the GP Deaneries, the GMC as professional regulator and patients.