Introduction
Multiple-choice questions (MCQs) are one of the most widely used methods for assessing knowledge in medical education. While other formats, such as essays, short-answer questions, and extended matching items, effectively evaluate clinical reasoning, MCQs are preferred for their efficiency in sampling broad content domains, psychometric robustness, and compatibility with standard-setting methodologies (Douthit et al. Reference Douthit, Norcini, Mazuz, Alkan, Feuerstein, Clearfield, Dwolatzky, Solomonov, Waksman and Biswas2021). Their use spans undergraduate formative assessments to high-stakes postgraduate licensing examinations (Cizek and Bunch Reference Cizek and Bunch2007). At the University of Galway, fourth-year medical students undertake an end-of-year 100-item best-of-five summative MCQ examination accounting for 80% of their module marks, with a pass on this module compulsory for progression in their undergraduate medical degree. The exam contains a mixture of knowledge based and clinical scenario-based questions.
Standard-setting methods are classified as either absolute (criterion-referenced) or relative (norm-referenced) (George et al. Reference George, Haque and Oyebode2006). Absolute methods, such as the Angoff method, define a cut-score based on predefined expectations of minimal competence, independent of peer performance (Norcini Reference Norcini2003). The original Angoff method requires subject-matter experts to estimate the probability that a minimally competent (borderline-pass) candidate would answer each item correctly (Angoff Reference Angoff and Thorndike1971). The more widely used modified Angoff method incorporates structured group discussion and provision to standard setters of empirical performance data to improve inter-rater reliability and judgment calibration (Clauser and Margolis Reference Clauser and Margolis2006; Downing Reference Downing2004). These adaptations address the challenge of consistently conceptualising a borderline-pass candidate’s ability (Cizek and Bunch Reference Cizek and Bunch2007). Despite the modified Angoff method’s widespread adoption, few studies have empirically evaluated whether its’ item-level cut scores align with real-world performance, particularly for borderline-pass students (Boulet et al. Reference Boulet, van Zanten, de Champlain and Hawkins2003; Downing et al. Reference Downing, Tekian and Yudkowsky2006). To date, research has predominantly focused on whole-exam standard-setting, with little data examining how accurately standard setters’ estimates reflect actual achievement on individual questions. Moreover, the influence of item characteristics, such as questions describing clear clinical scenarios compared to more knowledge-based content or psychometric properties of MCQs, including difficulty and discrimination indices, on prediction accuracy remains unexplored.
Consequently, this study aims to evaluate if Angoff-derived item-level cut scores align with the actual performance of borderline students in a summative psychiatry exam and evaluate if item characteristics impact the accuracy of the standard-setting procedure.
Methods
Study design
Undergraduate medical students undertake their psychiatry module in the fourth (penultimate) year of the University of Galway medical degree and complete a summative end-of-year best-of-five item MCQ examination. All students who undertook the summative end-of-year psychiatry MCQ examination at the University of Galway over a 5-year period (2019–2023) were included in this study. The MCQ encompassed 50 questions in 2019, 2020, and 2021 and 100 questions in 2022 and 2023 resulting in 350 questions available for analysis. The pass mark is set at 50% after standard setting. Across the study period, a total of 754 students sat the summative examination, with a wide distribution of scores evident for the 350 MCQ questions (mean = 63.6%, SD = 23.8; range = 3.48–100%). Students who scored between 45% and 55% (post-standard setting) were classified as “borderline pass” candidates (minimally competent) and were included in this study. Repeat (autumn) examination candidates were not included given less available psychometric data (i.e. discrimination indices).
Each MCQ was categorised across a number of criteria including: (1) question type (clinical focus v. knowledge focus), (2) diagnostic category (addictions, organic, psychotic, mood, anxiety, neurodevelopmental, personality, eating, and somatic disorders), (3) specific content domains (psychopharmacology, psychotherapy, phenomenology, aetiology/epidemiology and medical ethics), (4) difficulty index, and (5) discrimination index.
The Angoff method of standard setting was undertaken in advance of these MCQ examinations. The standard setting for each examination was conducted using the modified Angoff method, involving a panel of 12 to 15 standard setters. All standard setters had clinical experience in psychiatry and academic experience in teaching psychiatry to medical students. Standard setters were selected based on this clinical and academic experience, and a number of panel members held, or currently hold, formal academic staff positions with responsibility for curriculum development and assessment. Prior to each standard-setting exercise, panel members received a structured briefing from an experienced academic staff member covering the conceptual definition of the minimally competent (borderline-pass) candidate and the mechanics of the modified Angoff procedure. Panel members subsequently independently estimated the probability that a minimally competent (i.e. borderline pass or just good enough) student would answer each question correctly. If their mark for any question was inconsistent with other standard setters (10% mean difference to other standard setters), they were invited to change their estimation for this question, thus improving inter-rater reliability. All data was anonymised prior to analysis.
Statistical analysis
All statistical analyses were conducted using the Statistical Package for Social Sciences, version 30.0 (SPSS, IBM Corp., Armonk, NY, USA). Descriptive statistics, including means and standard deviations, were attained for all relevant variables. Independent-samples t-tests and bivariate correlations were employed to compare the mean “borderline pass” students’ marks to the Modified Angoff predicted marks for different question types (clinical and non-clinical knowledge focused questions, different diagnoses, and specific content domains). We transformed Pearson’s r coefficient into Fisher’s z score to test the significance of the difference in correlations for knowledge and clinical questions between students and standard setters, where a Z observed ≥1.96 or a Z observed ≤1.96 was considered as significant (Pallant Reference Pallant2001). Chi-squared tests (or Fisher’s Exact test as appropriate) were utilised to compare differences between clinical and knowledge-based questions with difficulty index (easy, medium, hard questions). Questions were denoted as easy where >70% of all students (undertaking exam) attained the correct answer, as medium where 40–69% of all students attained the correct answer and hard where <40% of students attained the correct answer. Discrimination indices were determined by the point-biserial correlation coefficient, which measures the relationship between answering a specific MCQ correctly and the total score, and ranges from −1 to + 1, with scores of 0.3 or above determined as very good, 0.1–0.29 as acceptable, and <0.10 as poor. General linear models were applied to adjust for potential confounding variables, including item discrimination indices. To evaluate the sensitivity of study findings to the definition of the borderline-pass group, primary analyses were repeated using a narrower borderline band of 48–52% (n = 107). The threshold for statistical significance was set at p < 0.05.
Results
One-hundred and ninety-one students (25.3%) were categorised as “borderline pass” (post-standard setting mark of 45–55%). The mean student score for this cohort was 54.2% (SD = 24.5), compared to the mean Angoff score of 53.0% (SD = 7.0, t = 1.05, p = 0.30; r = 0.57). A comparison of the mean Angoff score and actual student score by question topic is displayed in Table 1. There was a wide variation in the actual student scores as indicated by the high standard deviation across topics. Standard setters significantly under-estimated student performance for anxiety disorder questions (t = 2.35, p = 0.03) and over-estimated student performance for eating disorder (t = 2.22, p = 0.05) questions. Standard setters significantly under-estimated student performance at questions relating to the content domains of psychopharmacology (t = 2.27, p = 0.03), and phenomenology (t = 3.54, p = <0.001), (see Table 1).
Comparison of mean Angoff score and actual student score by question topic
Table 1. Long description
A table comparing student scores and Angoff scores across various question topics. The table has 350 rows and 5 columns. Column headers are Question Topic, N, Student score, Angoff score, and Statistics. Row labels include All questions, Diagnostic category, and Specific domains. Under Diagnostic category, topics include Organic disorders, Addictions, Psychotic disorders, Mood disorders, Anxiety disorders, Neurodevelopmental disorders, Personality disorders, Eating disorders, and Somatic disorders. Under Specific domains, topics include Psychopharmacology, Psychotherapy, Phenomenology, Epidemiology / aetiology, and Medical ethics. Each row lists the number of questions (N), mean student score with standard deviation (SD), mean Angoff score with SD, and statistics including t and p values. Notable trends include significant differences in student scores and Angoff scores for Anxiety disorders, Eating disorders, Psychopharmacology, and Phenomenology.
SD = standard deviation.
The bold values denote those values that were statistically significant.
The difference in mean scores actually attained by borderline students and those predicted by modified Angoff estimates is displayed in Table 2. Students performed better than predicted by standard setters for clinical compared to knowledge-based questions, with a 6.2% (SD = 19.8) higher mark for clinical questions and a 2.9% (SD = 21.6) lower mark for knowledge-based questions (t = 4.07, p = <0.001) attained. This difference was evident across a number of diagnostic categories (psychotic, neurodevelopmental, and personality disorders) and for the content domain of phenomenology. Students performed better than standard setters predicted for knowledge-based rather than clinical-based psychotherapy questions. There was a high correlation between students and standard-setter mean marks for clinical (r = 0.61, p < 0.001) and a moderate correlation for knowledge-based (r = 0.49, p < 0.001) questions, with no statistically significant difference detected between these correlations (z = 1.60, Figure 1).
Correlation between mean Angoff and student scores for clinical and knowledge-based questions.
Mean differences (Students – Angoff) for clinical and knowledge domains
Table 2. Long description
A table comparing mean differences in clinical and knowledge domains for various question topics. The table has 18 rows and 7 columns. Column headers are Question topic, Clinical n, Knowledge n, Clinical Mean diff. (SD), Knowledge Mean diff. (SD), and t, p. Row labels include All questions, Diagnostic categories, and Specific domains. Diagnostic categories include Organic disorders, Addictions, Psychotic disorders, Mood disorders, Anxiety disorders, Neurodevelopmental disorders, Personality disorders, Eating disorders, and Somatic disorders. Specific domains include Psychopharmacology, Psychotherapy, Phenomenology, Epidemiology / aetiology, and Medical ethics. Each row lists the number of clinical and knowledge questions, their mean differences with standard deviations, and t and p values.
Positive mean difference scores indicate that students outperformed Angoff marks and negative Mean Difference marks suggest students performed less well than Angoff mark. SD = standard deviation.
The bold values denote those values that were statistically significant.
To assess the sensitivity of these findings to the definition of the borderline-pass group, analyses were repeated using a narrower band of 48–52% (n = 107). Findings were broadly consistent with the primary 45–55% analysis (see Supplementary Tables S1 and S2). Standard setters again under-estimated performance on anxiety disorder and psychopharmacology questions, although these differences were attenuated and no longer reached statistical significance in the narrower group. Students continued to outperform Angoff-predicted marks on clinical relative to knowledge-based questions overall (t = 4.02, p < 0.001), with this pattern again evident for neurodevelopmental and personality disorder categories and for phenomenology; the difference for psychotic disorder questions was attenuated below significance in this narrower group.
The overall mean discrimination index was 0.23 (SD = 0.13) (based on data for all students sitting the MCQ examinations) (clinical-based mean = 0.22, SD = 0.13; knowledge-based mean = 0.24, SD = 0.13) and was not associated with question difficulty. For knowledge-based questions, item discrimination correlated modestly with Angoff (r = 0.22, p = 0.002) and student marks (r = 0.23, p = 0.001). Lower item discrimination predicted Angoff marks being higher than student marks (B = –31.96, β = –0.19, t = 2.61, p = 0.01). No significant correlations were observed for clinical questions (Angoff: r = –0.09, p = 0.27; student: r = 0.05, p = 0.50).
The overall mean difficulty index was 0.64 (SD = 0.24) (based on data for all students sitting the MCQ examinations) (clinical-based: n = 158, mean = 65%, SD = 0.23; knowledge-based: n = 192, Mean = 62%, SD = 0.25). Item difficulty was not correlated with Angoff or student scores and was not a significant predictor of score differences between standard setters and students for either clinical (B = −5.97, β = −0.07, t = 0.87, p = 0.38) or knowledge-based questions (B = 7.34, β = 0.08, t = 1.17, p = 0.24).
Discussion
To our knowledge, this is the first study to examine if Angoff standard-set scores on individual MCQs align with the actual performance of “borderline-pass” or “just-good-enough” students, while also examining whether question types (clinical or knowledge-based), content domains, and psychometric properties were associated with differential performance in this group of students. Overall, we found no significant difference between the overall mean Angoff score and the actual mean borderline student score. However, there were differences between Angoff estimates and student performance regarding question type and content domains. Discrimination index was a significant predictor of the difference between Angoff estimates and student performance for knowledge-based questions. Difficulty index was not significantly associated with any difference between Angoff estimates and student performance.
The lack of a significant difference between the overall mean Angoff score and actual student score suggests that, on average, the standard-setting process using the modified Angoff method yields a reasonably accurate prediction of “borderline-pass” students’ scores. The moderate positive correlation between Angoff scores and actual performance (r = 0.57) suggests that expert-derived judgments of minimum competence reflect group-level student outcomes and was evident for both clinical and knowledge-based questions. This aligns with prior research demonstrating the Angoff method’s utility in high-stakes settings (Cizek and Bunch Reference Cizek and Bunch2007; Norcini Reference Norcini2003).
There were, however, differences between Angoff-derived estimates and student performance across question types (clinical and knowledge-based) and content domains. Student performance was underestimated in clinical questions and overestimated in knowledge-based questions with this pattern notable for questions relating to anxiety disorders, psychopharmacology, and phenomenology. A plausible explanation lies in the pedagogical environment of contemporary medical education, which increasingly emphasises integrated clinical exposures and clinical reasoning through case-based learning (CBL) and problem-based learning (PBL) methods (Plake and Cizek Reference Plake, Cizek and Cizek2012). These learning methods may better prepare students for clinically contextualised scenarios or vignettes, rendering such questions more relatable (i.e. anxiety disorders and phenomenology), and enabling borderline candidates to apply contextual clinical knowledge more effectively than in decontextualised knowledge-based factual recall questions (Baghdady et al. Reference Baghdady, Pharoah, Regehr, Lam and Woods2014; Schuwirth and van der Vleuten Reference Schuwirth and van der Vleuten2011). Additionally, students may perform above standard setters’ expectations on topics that are clinically salient and emphasised in teaching sessions particularly if standard setters are not fully cognisant of the entirety of the teaching attained by students (Eva et al. Reference Eva, Cunnington, Reiter, Keane and Norman2004; Schuwirth and van der Vleuten Reference Schuwirth and van der Vleuten2011). Overestimation by standard setters may occur when expert judges, relying on heuristic judgment, equate their own clinical familiarity with a topic or its perceived conceptual simplicity with a high likelihood of student success (Hurtz and Auerbach Reference Hurtz and Auerbach2003). An alternative, non-mutually exclusive explanation is that knowledge-based factual recall may receive comparatively less emphasis within increasingly integrated, clinically focused curricula. Consistent with dual-process accounts of clinical reasoning, in which analytic (knowledge-based) and non-analytic (pattern-recognition) processes draw on partially distinct cognitive processes (Eva Reference Eva2005), it is plausible that greater emphasis on clinically contextualised learning may differentially reinforce clinical application over isolated factual recall. Taken together, these discrepancies may reflect an interaction between curricular emphasis and well-recognised biases in expert judgement (Hurtz and Auerbach Reference Hurtz and Auerbach2003), rather than deficiencies in either the teaching programme or the assessment itself. If replicated in future studies, these findings may help identify content domains in which educators could review the alignment between intended learning outcomes, teaching approaches and assessment to ensure that foundational knowledge is reinforced alongside clinical application (Schuwirth and van der Vleuten Reference Schuwirth and van der Vleuten2011).
Lower discrimination MCQ indices was associated with larger differences (albeit modest) between Angoff and student scores, with this finding supportive of previous findings that MCQs with poor psychometric properties can reduce the accuracy of expert judgement (Hurtz and Auerbach Reference Hurtz and Auerbach2003; Jalili et al. Reference Jalili, Hejri and Norcini2011; Verheggen et al. Reference Verheggen, Muijtjens, van Os and Schuwirth2008). In the present study, all MCQs undergo review by three academic staff members following development and are subsequently piloted, typically within the autumn (repeat) examination, prior to first use in a summative summer examination; however, the comparatively small candidate numbers sitting the autumn examination limit the robustness of discrimination indices available at this pilot stage. Pooling pilot data across multiple cohorts or examination years, rather than relying on a single autumn sitting, may help generate more robust discrimination estimates prior to first summative use and represents a practical means of strengthening the piloting process. The absence of similar effects for clinical questions may reflect greater consistency in expert judgement when evaluating clinically contextualised content (Clauser and Margolis Reference Clauser and Margolis2006; Dwyer et al. Reference Dwyer, Wright, Kulasegaram, Theodoropoulos, Chahal, Wasserstein, Ringsted, Hodges and Ogilvie-Harris2016). Question difficulty was not significantly associated with Angoff scores, student scores, or score differences, indicating that the Angoff-derived estimates did not overestimate or underestimate questions based on their relative difficulty, consistent with previous data (Brandon Reference Brandon2004; Impara and Plake Reference Impara and Plake1998).
Strengths of this study is the utilisation of a relatively large cohort of participants (n = 191) across multiple examinations, and a sole focus on “borderline-pass” students in analysis. The examination of detailed item-level comparisons across clinical and knowledge-based questions, diagnostic categories, and other content domains provided a nuanced insight into where Angoff-derived estimates are most and least accurate. The study is also associated with a number of limitations. Study findings are drawn from a single institution and specialty area, psychiatry, which may limit generalizability to other modules, programmes, or settings, although the patterns observed are consistent with broader standard-setting research. The definition of “borderline pass” (45–55%) was necessarily arbitrary, albeit consistent with common operationalisation in standard-setting research (Cizek Reference Cizek1996; Homer et al. Reference Homer, Pell and Fuller2017). Alternative cut-points could arguably have been selected. However, sensitivity analyses using a narrower definition (48–52%) yielded broadly consistent findings (see Results), supporting the robustness of the study’s principal conclusions to the specific cut-points used to define the borderline-pass group. The study did not examine how panel members formed their judgments for individual items and formal inter-rater reliability statistics for the standard-setting panels were not calculated in this study. This was an exploratory study, and no correction of statistical significance was made for multiple comparisons, and thus the findings warrant independent replication. Qualitative research could elucidate why estimation accuracy varied across question types and content domains and will be a topic for future research. Finally, the focus on borderline candidates (45–55% scorers) provides targeted insight into standard-setting validity but does not provide insights for students attaining higher grades (i.e. first- or second-class honours marks).
Beyond these limitations, the choice of standard-setting method itself warrants consideration. Examinee-centred approaches, such as the Borderline Regression Method (Kramer et al. Reference Kramer, Muijtjens, Jansen, Düsman, Tan and van der Vleuten2003; Wood et al. Reference Wood, Humphrey-Murto and Norman2006), derive cut-scores by regressing examiners’ holistic ratings of candidates against their actual test scores rather than requiring item-level judgements. This is less resource-intensive and may more directly reflect real performance, but it has been validated mainly in small-cohort OSCEs to date and does not yield the item-level granularity central to the present study’s aims. The Bookmark method (Lewis et al. Reference Lewis, Mitzel and Green1996) orders items by empirical difficulty and asks experts to identify where a minimally competent candidate would cease answering correctly; while more empirically grounded, it requires pre-calibrated item pools and a full re-run whenever exam items change, unlike the more flexible item-level revision permitted by the modified Angoff method (Buckendahl et al. Reference Buckendahl, Smith, Impara and Plake2002). Given the item-level focus of this study, modified Angoff method remained the most appropriate choice. Comparative studies triangulating Angoff-derived cut-scores against Bookmark and Borderline Regression estimates using identical MCQ datasets would provide valuable evidence on their relative validity.
Conclusion
This study provided overall support for using the modified Angoff method to assist in determining the competence level for borderline students in a high stakes’ undergraduate psychiatry MCQ examination. However, subgroup analyses revealed subtle differences between clinical and knowledge-based questions, across diagnostic categories and content domains. Further education of standard setters on the taught curriculum and the inclusion of more diverse specialist input to improve standard-setting accuracy across less familiar domains might improve MCQ item-level standard setting accuracy.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ipm.2026.10225.
Data availability statement
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Author contributions
Kelechi Chukwudi, (MD, MSc, PDip (Clin Edu), PDip (Lead and Mgt), FRCPC, MRCPsych, MCPsychI), is a consultant psychiatrist in University Hospital Galway, Galway-Roscommon Mental Health Services, Ireland. He participated in the Angoff standard-setting process and led the study design, data collection, data analysis, interpretation of results, and manuscript preparation.
Brian Hallahan, (MD, MMedSci, MRCPsych), is a consultant psychiatrist in University Hospital Galway, Galway-Roscommon Mental Health Services, Ireland, and a Senior Lecturer in Psychiatry in the School of Medicine, University of Galway, Ireland. He participated in the Angoff standard-setting process and contributed to study conception, study design, data analysis, interpretation of results, and critical revision of the manuscript.
Colm McDonald, (MB, PhD, MRCPsych), is a consultant psychiatrist in University Hospital Galway, Galway-Roscommon Mental Health Services, Ireland, and Professor of Psychiatry in the School of Medicine, University of Galway, Ireland. He participated in the Angoff standard-setting process and contributed to study conception, study design, and final manuscript.
Funding statement
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interests
The authors declare none.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committee on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The study protocol was approved by the Ethics Committee of the University of Galway.
Open access
