Introduction
Paramedicine has experienced significant growth in recent years, extending traditional emergency or acute care roles with increases in breadth of practice and autonomy. At the same time, researchers have warned of the increasing risk to patient safety that this creates, especially as it relates to demands on clinical reasoning and decision making, but also where additional technical and other non-technical skills are required.Reference Tavares and Boet 1 - Reference Bigham, Buick, Brooks, Morrison, Shojania and Morrison 3 Safe paramedic practice has largely been dependent on clinical governance models (eg, medical oversight and/or the use of medical directives or clinical guidelines). However, these likely are limited in their ability to account for the degree of medical ambiguity, limited diagnostics, diverse contexts and circumstances, and growing patient groups paramedics increasingly are challenged with.Reference Fairbanks, Crittenden and O’Gara 4 Therefore, the extent to which paramedicine ultimately can be safe will be dependent not only on systems (eg, infrastructure, oversight, equipment, and process), but also on the safe clinical practice of those individuals serving within the profession.Reference Tavares and Boet 1 , Reference Bigham, Buick, Brooks, Morrison, Shojania and Morrison 3 As such, making accurate and trustworthy decisions regarding paramedic clinical proficiency, especially at the entry to practice level, can have significant implications for patient safety overall. While no one assessment method can be responsible for informing all aspects of clinical practice or competence,Reference Jensen, Croskerry and Travers 5 in many instances, performance-based assessments (PBAs; ie, where candidates exhibit behaviors in response to clinical challenges) have and continue to serve an integral role.Reference Van der Vleuten and Schuwirth 6 , Reference Miller 7 Therefore, the goal of this review was to provide the paramedic community with literature-based foundations by which to establish optimal PBA strategies.
Goals associated with PBAs can include: (a) differentiating between levels of performance; (b) making accurate determinations regarding the achievement of predefined competencies; (c) detecting the ability to adaptively apply those competencies; and/or (d) to make accurate predictions regarding future clinical performance in novel contexts. However, a review of the paramedicine literature reveals a paucity of research aimed at providing evidence for, or informing, best practices related to achieving these goals.Reference Holmboe, Sherbino, Long, Swing and Frank 8 What research does exists has been limited to identifying performance deficiencies in specific patient types (eg, pediatric resuscitation),Reference Regener 9 , Reference Su, Schmidt, Mann and Zechnich 10 as part of correlational studies involving a single PBA,Reference Lammers, Byrwa, Fales and Hale 11 studies assessing isolated technical skillsReference Studnek, Fernandez, Shimberg, Garifo and Correll 12 - Reference Rocca, Crosby, Maloney and Bryson 19 or non-technical skills,Reference Ruetzler, Roessler and Potura 20 or as part of outcome measures in research studying various interventions (eg, comparing intubation interventions, stress, and so on).Reference Lammers, Byrwa, Fales and Hale 11 , Reference Manser, Foster, Gisin, Jaeckel and Ummenhofer 21 - Reference Castle, Owen, Hann, Naidoo and Reeves 28 Furthermore, two systematic reviews in health professions education, the first identifying tools used for assessment of clinical competence and the other evaluating the use of simulation for the purpose of assessment, both failed to reveal any meaningful evidence related to paramedicine.Reference Hein, Owen and Plummer 29 , Reference Cook, Hatala and Brydges 30 Other recent research has been limited to scale development and validation, providing little to overall process and best practices.Reference Kogan, Holmboe and Hauer 31 - Reference Tavares, LeBlanc, Mausz, Sun and Eva 34
Despite the lack of profession-specific evidence, the broader health professions and assessment literature is extensive. For instance, a number of reviews have been conducted previously summarizing evidence and meaningful perspectives in the assessment of clinical competence.Reference Tavares, Boet, Theriault, Mallette and Eva 35 - Reference Regehr, Eva, Ginsburg, Halwani and Sidhu 50 This expansive body of literature can inform and contribute process, but context remains an important feature. Many assessment principles, including reliability, validity, and the role of “subjective” rater-based assessments (RBAs) may be directly applicable, yet are seldom applied in paramedicine. Therefore, this narrative review aimed to provide a synthesis of the relevant theoretical foundations and literature informing PBA as it might be applied to paramedicine. As such, this review was not based on a single, highly specific, or standardized search, nor was it an exhaustive review of the PBA literature. Instead, as per the narrative tradition, this study focused intentionally on various, but consistent, theoretically derived research, as well as paramedic-specific research (where applicable) that collectively can inform assessment best practices in paramedicine. Reviews of this type are well suited for comprehensive topic areas that require a holistic interpretation and integration of multiple and diverse existing theories and pluralities of scholarly work that are not amenable to quantitative analyses or results or highly specific search criteria. The authors of this report tried to be comprehensive in their review and limited bias by remaining broad in their inclusion of the literature and by providing readers with the information necessary to form their own interpretations. To provide boundaries in this search, this study focused mainly on summative assessments where the adaptive integration of multiple competencies for the purpose of making entry to practice decisions and optimizing predictions of future clinical performance in novel contexts is desired. The focus was on current understanding of various theoretical frameworks that support recommendations educators, employers, and/or licensing or certifying bodies may use (without being overly prescriptive) to evaluate or design their assessment processes, while also exploring emerging lines of inquiry in PBA literature.
Report and Discussion
The Act of Rater-based Assessments of Competence
Before exploring features associated with optimal PBA, it is helpful to explore the act of RBA of clinical competence. In its simplest form, RBA begins with presenting candidates with a clinical challenge. Depending on the context, the clinical interaction may be created deliberately, controlled, or selected (eg, simulation-based settings) or not (eg, field settings). Next, the candidate interacts with the clinical challenge, selecting and adaptively applying and demonstrating various competencies as the case warrants, exhibited as words, actions (or inactions), events, or interactions. Candidates do this fluidly in response to changing parameters (eg, changes in patient condition and implicit feedback). A rater must then consider the reams of information, process and interpret all of this in reference to some standard, then ultimately translate this information into a rating, categorical judgment, and/or some form of narrative. The degree to which assessment goals (described above) are achieved, or are even feasible, will depend on how these individual features are understood and applied.
Understanding Competence
In order to inform assessment of competence, it is helpful to understand the way in which competence has been conceptualized and discussed. Kane defined competence as “the degree to which an individual can use the knowledge, skills, and judgment (some have since added attitudes) associated with the profession to perform effectively in a domain of possible encounters defining the scope of professional practice.”Reference Schuwirth, Southgate and Page 51 Other definitions similarly capture the individual’s features, while drawing in the breadth and boundaries of the profession.Reference Kane 52 This definition suggests that understanding the degree to which a paramedic is competent requires that candidates demonstrate (ie, “use” or “perform”) various competencies adaptively across a number of contexts and patient types. Further, that knowledge, skill, and judgment not necessarily be limited to any one aspect of practice, rather that assessment designers optimally represent the profession in the assessment effort. Kane makes no distinction over technical or non-technical aspects of practice, and therefore, both may need to be included as they are important features of paramedic practice.Reference Frank, Mungroo, Ahmad, Wang, DeRossi and Horsley 53 , Reference Tavares and Mausz 54
In assessment terms, competence is related closely to the concept of a universe score (also referred to as a “true” score). A universe score refers to the long-run hypothetical mean a candidate would receive, absent of measurement error, across all possible observations.Reference Tavares 55 , Reference Brennan 56 In other words, if a candidate could be observed across all possible patient encounters representative of the field of paramedicine, one would essentially understand, or in effect, know that candidate’s “true” performance ability or degree of competence. However, this carries obvious feasibility challenges, and measurement error (ie, anything that results in deviations from this true score) is always present to some extent. In this universe score framework (which adopts a positivist perspective), “true” performance ability exists within every candidate to varying degrees, and it can be measured. However, “performance ability” is an abstract construct (like intelligence or motivation). Because of this, performance ability can only ever be inferred based on observation of behaviors in response to some form of stimuli. Using Kane’s views on competence and the concept of a universe score, what follows is a review of a number of conceptual frameworks with practice implications to consider when making decisions regarding assessment design. It is worth noting that the assessment community (and thus, this review) has been focused mainly, for over 50 years, on psychometric principles when thinking about, and on assessments practices in, health professions education. However, researchers have begun to explore a post-psychometric era in assessment, which is discussed below under the heading “Emerging Areas of Assessment Research.”
Millers Pyramid of Competence
In describing how best to conceptualize the assessment of competence, Miller proposed an assessment framework that consists of four levels of performance (often illustrated as a pyramid), each representing an increasingly complex stimulus and response format.Reference Van der Vleuten and Schuwirth 6 , Reference Van der Vleuten 36 At the base of the pyramid is Level 1, referred to as “knows,” followed by Level 2, or “knows how.” These levels involve assessment of cognition, mainly in the form of declarative and applied knowledge, respectively. These can be measured efficiently and effectively using stimuli and response formats (eg, texted-based questions with multiple-choice options) that do not require any complex performance or behaviors. For example, describing coronary blood flow and electrocardiogram (ECG) patterns related to various areas of the heart would be at the level of “knows.” Selecting care plans based on ECG findings and/or associated manifestations would be at the level of “knows how.” Level 3 and Level 4, referred to as “shows how” and “does,” respectively, require individuals to demonstrate behaviors in response to clinical challenges either in simulation (“shows how”) or with real patients in real clinical contexts (“does”).
Practice Implications
First, no one-assessment strategy can capture all levels of Miller’s Pyramid. This suggests the need for programmatic assessment with targeted strategies depending on areas of focus.Reference Van der Vleuten 36 Assess at higher levels of Miller’s pyramid what cannot be assessed more efficiently at lower levels.
Second, an underlying principle is that while performance at one level (eg, Level 2 “knows how”) may be dependent on the level below it (ie, Level 1 “knows”), that same level does not necessarily predict performance at the level above (ie, at Level 3 “shows how”).Reference Van der Vleuten and Schuwirth 6 , Reference Brennan 57
Third, competence involves a progression of knowledge, skills, and abilities, best measured in settings where the assessment context closely parallels the environment in which future clinical performance is expected to occur, mainly to minimize leaps in extrapolation in an inference-based model.Reference Hodges 40 Ideally, all clinical competence decisions ultimately would include assessment of behaviors exhibited in real clinical contexts with real patients (ie, at the “does” level).Reference Brennan 57 , Reference Crossley and Jolly 58 However, in paramedic contexts in particular, work-based assessments (WBAs) are often associated with a number of challenges, such as lack of control over content (potentially leading to inappropriate case variability and/or complexity or a situation in which the sample of cases used for summative decisions may represent inadequately the clinical domain), interruptions due to patient safety concerns, and undue influence by many uncontrollable contextual factors.Reference Norcini 59
As result of challenges associated with WBA, many have advocated for, and adopted, simulation-based assessments (SBAs;Reference Norcini 59 ie, the “shows how” level) where ecological validity (ie, similarity between assessment and work-based settings), standardization, elimination of patient safety barriers, and complete control over content can be exercised. However, SBA, only ever being a surrogate of reality, requires evidence of both reliability and validity.
Reliability
Ensuring a degree of reliability promotes trustworthiness in the PBA process. Reliability refers to the ability to differentiate consistently between candidates with both consistency and differentiation being important.Reference Boulet and Swanson 60 , Reference Eva 61 Consistency suggests that scores on a performance assessment would be relatively similar between raters (inter-rater reliability), within raters (intra-rater reliability), if assessed repeatedly (test-retest reliability), and/or among stations (inter-station reliability). Differentiation suggests the assessment process is designed and implemented such that differences between candidates (if present) can be detected. Reliability is also an indicator of the amount of error associated with the process.Reference Eva 61 , Reference Downing 62 It is calculated by taking the ratio of subject variance (eg, differences between candidates) to subject variance plus error, with a score of “1” indicating perfect reliability.Reference Haladyna and Downing 63 In an ideal sense, there would never be any measurement error in the assessment process and the reliability would always equal “1.” However, in reality, this is never the case. Therefore, a common strategy is to identify sources of measurement error and apply whatever strategies are available to mitigate them. Examples are provided below. Poor reliability suggests the risk of making classification errors is high. This renders the assessment process useless since incorrect decisions regarding a paramedic candidate can ultimately be made as a result, with significant dangerous downstream implications for the patients/public and profession.
As suggested above, two main sources of threat to reliability include factors contributing to poor differentiation and/or measurement error (defined above). First, differentiation (also understood as variance attributable to “subjects”) is achieved when the assessment design and implementation allow for a range of scores when they truly exist within and between candidates. Cases that result in ceiling or floor effects; raters who have difficulty differentiating between candidates, levels of performance, or dimensions of performance; and flawed rating tools can all reduce differentiation, and thus reliability. Second, measurement error can take many forms and is best-identified using generalizability theory.Reference Tavares 55 , Reference Streiner and Norman 64 Again, cases, raters, and items on a rating tool can all be sources of error. However, consistently the greatest source of error has been context specificity.Reference Van der Vleuten 36 , Reference Brennan 65 - Reference Newble and Swanson 69 Context specificity refers to the finding that a candidate’s performance on any one case is a poor predictor of performance on another. For example, context specificity would suggest that an individual’s clinical performance when presented with a patient experiencing an asthmatic exacerbation would serve as a poor predictor of performance when presented with a patient suffering from sepsis. To improve predictions regarding future clinical performance, and perhaps to get closer to the candidates “true” level of performance, additional/multiple observations are required.Reference Van der Vleuten 70
Practice Implications
First, promote differentiation; the sample of cases used to assess candidates overall should challenge the candidate pool adequately, avoiding ceiling or floor effects while staying true to the construct of interest. Then, ensure raters have the ability to detect differences between levels and dimensions of performance and candidates, as poor rater performance may mask or mitigate differences. Similarly, ensure rating tools (described in more detail below) allow for differentiation. Again, poorly designed tools may mask differences that may otherwise be present.
Second, minimize error; sample performance broadly across a number of contexts/cases. Williams et al suggest seven to 11 observations are required to achieve a reasonable level of reliability;Reference Epstein 47 however, this is only a useful starting point. Investigations of the effects of context specificity and other sources of error (eg, raters and items) need to be conducted initially and on an ongoing basis since the degree of relative contribution by each source can vary by context/assessment features and offering, and different mitigation strategies may apply (eg, rater training).
Validity
Even the most reliable assessment processes would be limited without evidence of validity. Validity refers to the degree of confidence one has in the inferences made based on scores generated by an assessment process.Reference Downing and Yudkowsky 71 - Reference Kane 76 Because competence is an abstract construct, it can only ever be inferred (not directly measured) based on observations of behaviors in response to various clinical challenges/stimuli. In a practical sense, validity then refers to the accuracy or appropriateness of those inferences, including proposed interpretations (eg, degree of competence) or proposed use of assessment scores (eg, certification, remediation, and advancement). Importantly, validity is never actually achieved, rather different levels or degrees of evidence to support claims of validity are achieved; some are stronger than others. For example, demonstrating that a PBA is perceived by experts, candidates, and other stakeholders to be a suitable measure of clinical performance (ie, face validity) is a much weaker argument, or source of evidence, than having evidence that the same assessment actually predicts performance in future clinical settings (ie, predictive validity).Reference Tavares and Mausz 54 The stronger the collection of evidence in support of the inferences, the more one can have confidence in and defend decisions or interpretations made based on scores generated from an assessment process.
There are many threats to validity.Reference Downing 62 These can include poor reliability (discussed above), lack of authenticity, under- or over- construct representation, and/or construct irrelevant variance.Reference Kane 77 These threats may be present in the stimuli (eg, the clinical cases the candidates are presented with) or the response format (eg, the rating tool used and/or raters themselves).Reference Kane 77 , Reference Downing and Haladyna 78 First, authenticity refers to the degree to which the assessment context closely matches or aligns with future clinical contexts or performance expectations. Asking candidates to intubate a task trainer without any of the contextual forces that may be present in real clinical contexts (eg, prioritization, data gathering, and clinical reasoning) is a threat to validity because of the significant differences that exist between settings. In this example, poor authenticity results in larger extrapolations when making inferences regarding performance in real clinical contexts based on the assessment context.
A second and third threat to validity involves construct under- and over-representation.Reference Kane 72 When considering the stimulus and assuming a performance exam at the “shows how” or “does” level, construct under-representation refers to under sampling of the domain of possible encounters/construct of interest. To use a hypothetical example, if there are 100 different skills or patient types in the domain of possible encounters, and the sampling strategy only includes 10 of each, the risk of construct under-representation is higher than if 20 of each were included. Of course, not all elements of the construct can be included in a given assessment process. Therefore, one must apply a sampling strategy using a structured blue print and appropriate framework (one or more that define the construct of paramedic practice) to demonstrate and ensure adequate/appropriate representativeness. Construct over-representation (a form of construct irrelevant variance – described below) would be essentially the opposite, albeit less common, problem. That is, including content or behavior expectations in a PBA that is not representative of paramedic practice. Importantly, these concepts can apply to the cases designed/selected for an assessment process, but the rating tools as well. For instance, when developing or using a rating tool, if the measurement tool does not represent adequately the construct, by either missing important items or dimensions, or including items or dimensions that should not be, then both construct under- and over-representation are again possible, just in a different way.
Finally, another threat to validity is referred to as construct irrelevant variance and involves any systematic influences not directly related to the construct of interest.Reference Downing 79 This may include flawed cases, poor rating scales, inappropriate rating items, various forms of rater biases (eg, leniency and stringency or restriction of range), inadequate sampling, poor case difficulty (either too easy or too difficult), and unfamiliar equipment or simulators.Reference Kane 77 Any of these may artificially lower or elevate scores, causing scores to deviate inappropriately from the “true” score, thereby contributing error and threatening the confidence one would have that the scores generated lead to appropriate inferences/decisions (ie, validity claims).
Practice Implications
When assessment at the “does” level (ie, at the workplace, obviously the most authentic stimulus) is not feasible, simulations designed to replicate physical, conceptual, and emotional realism should be employed to support eventual extrapolations.Reference Haladyna and Downing 80 When considering the adaptive integration of multiple competencies, this extends to ensuring cases involve full clinical encounters consistent with the field of paramedicine, as opposed to fragmented or decontextualized skills which are arguably less authentic.
First, ensure the construct of paramedic practice is defined adequately to then inform sampling strategies (and case development) and mitigate the risk of under- and over- construct representation in PBA. Then, use a blueprint to make informed decisions regarding representativeness (ie, how well the domain profession-specific knowledge, skills, judgments, and possible encounters are included/excluded). The concept of construct representation applies similarly for rating tool selection and/or development.
Second, identify and eliminate potential or real sources of construct irrelevant variance (ie, noise) – features of the assessment that might influence scores other than the candidate’s performance.
Multiple Observations Promote Both Reliability and Validity
The objective structured clinical evaluation (OSCE) aligns well with strategies to optimize reliability and validity. Originally developed in the 1970’s in response to the low reliability associated with oral-based exams,Reference Hodges, Hanson, McNaughton and Reghr 81 , Reference Harden 82 the OSCE has since taken on many variations across a variety of health professions and specialtiesReference Harden 83 - Reference Park, Chibnall, Blaskiewicz, Furman, Powell and Mohr 89 and has been adopted widely for high stakes licensing exams.Reference Manogue and Brown 90 - Reference Lee 93 Objective structured clinical evaluations involve having candidates rotate through a series of standardized stations that collectively represent a larger specified construct of clinical competence (see Hodges for a detailed outline of how to design and build an OSCE).Reference Haladyna and Downing 80 The strength associated with OSCEs are in their ability to effectively address context specificity by including multiple stations, while simultaneously promoting construct representation by expanding the sample of content included in the assessment process. Despite the variations in the number of stations needed, a recent review found that after more than 30 years of use, the OSCE produces reliable results,Reference Boulet, Smee, Dillon and Gimpel 94 has been reported as a gold standard for the assessment of competence in some settings,Reference Patrício, Juliao, Fareleira and Carniero 95 and was found to be predictive of future clinical performance,Reference Sloan, Donnelly, Schwartz and Strodel 96 including paramedic contexts.Reference Janing and Sime 33
Ruessler et al, in emergency medicine and Tavares et al, in paramedicine, provide two examples of PBA that are variations on the original OSCE.Reference Janing and Sime 33 , Reference Tamblyn, Abrahamowicz and Brailovsky 97 In both instances, the main structure of timed, sequential rotations, standardization, ecological validity, construct representation, and multiple stations leading to multiple observations by multiple observers remain. However, rather than measure isolated skills or separated components of a complete workflow, both Ruessler and Tavares opted for more authentic or complete clinical cases within stations requiring the adaptive application of multiple competencies. While Ruessler included a combination of full clinical cases and more traditional OSCE stations (eg, ECG acquisition and interpretation or isolated intraosseous insertion), Tavares opted for full clinical cases in all stations. One other difference is that Ruesseler used station-specific checklists with weighted items, while Tavares used a generic un-weighted 7-dimension global rating scale (GRS)Reference Tavares, LeBlanc, Mausz, Sun and Eva 34 across all stations (discussed in more detail below). Applying a generic rating tool allows researchers to assess each of the seven dimensions across a different context and rater, aggregating across stations rather than within stations. When considering only the full clinical cases in both the studies, reliabilities ranged from .55 (with five stations) to .79 (with six stations). Differences in reliability (even when using the same number of stations) may be associated with the rating tools, raters, the homogeneity of the group, and the cases used. For five of the seven dimensions, Tavares was able to demonstrate that performance in the simulation-based setting was associated with performance in real clinical contexts with real patients.Reference Janing and Sime 33
Practice Implications
The OSCE and its variations serve as an effective SBA strategy based mainly on its inclusion of multiple observations by multiple raters across multiple contexts. This multiple sampling associated with OSCEs results in improved reliability by addressing context specificity and validity by promoting construct representation.
In paramedic contexts, as in emergency medicine, where interactions may be relatively brief and where the adaptive integration of multiple competencies is desired, validity may be optimized by using full clinical cases in each station (as opposed to isolated decontextualized skills).
Rating Tools
In making decisions regarding measurement tools, there are a number of factors to consider. For instance, if the goal is to focus on isolated procedural skills, and optimal performance of these skills is relatively linear, it may be more appropriate to employ some form of checklist where each step can be itemized. However, checklists may be limited when considering the integration of multiple competencies and may provide a false sense of objectivity.Reference Ruesseler, Weinlich and Byhahn 98 However, selecting checklist purely for the pursuit of objectivity (defined as “a goal of measurement marked by freedom from subjective influences”) or objectification, (defined as a “set of strategies designed to reduce measurement error”), researchers have identified no significant gains in reliability over GRSs.Reference Ruesseler, Weinlich and Byhahn 98 - Reference Regehr, MacRae, Reznick and Szalay 101 Further, deconstructing clinical competence into its components parts, or assuming the linear accumulation of competencies results in accurate conclusions regarding competence, may be flawed. The challenges associated with checklists have led some to explore and adopt the use of GRSs. Global rating scales may have higher inter-station and inter-rater reliability, can be used across stations, and may better suited to capture “nuanced elements of expertise.”Reference Regehr, MacRae, Reznick and Szalay 101 - Reference Hodges, Regehr, McNaughton, Tiberius and Hanson 103
Practice Implications
Both checklists and GRSs have their place. Global rating scales have emerged based on research that suggests: (a) not all that can be measured should be; (b) that competence may not be deconstructed effectively into component parts; (c) that the linear accumulation of competencies may not necessarily equal competence (as might be suggested by some checklists); (d) that GRSs may be more appropriate when considering complex practice; and that (e) “subjectivity” associated with GRS is no longer considered a bad word (described in more detail below).
Summary
As the paramedic profession expands in scope and contributions to the health care system, safety will be in large part dependent on the clinical competence of those granted access to the profession. The profession thus has a responsibility to ensure only those that are truly ready for independent practice are offered the public’s trust. Doing this requires assessment strategies that optimize accuracy, trustworthiness, and defensibility when inferring competence based on behaviors exhibited in one context to future novel clinical contexts. In this narrative review, a number of foundational conceptual frameworks are highlighted, leading to practice implications aimed at meeting these assessment goals. In summary, an OSCE-like process (ie, using multiple stations/multiple views), including between seven and 11 simulation-based stations (as a starting point), involving full clinical cases and assessed using a GRS may promote defensibility when assessing paramedic clinical competence. The final number of stations can only be determined following a comprehensive blueprinting process using appropriate profession-specific frameworks, considerations of feasibility and logistical constraints, and a thorough statistical review using generalizability theory, or other methods, to identify sources of error, the data from which can be used to develop strategies aimed at improving psychometric and feasibility issues. The content for each station should include and require the adaptive integration of multiple competencies and maximum authenticity rather than isolated decontextualized skills. The performances can be evaluated using a GRS designed to represent the construct of paramedic clinical competence, and the scoring strategy can involve scoring across stations rather than within stations such that each relevant domain (eg, decision making) is evaluated across a variety of different contexts and raters, as opposed to passing or failing a station. Finally, any model must be followed with rigorous quality assurance/psychometric evaluation to determine and demonstrate evidence of both reliability and validity. A brief overview of these frameworks, along with the following emerging areas of research in assessment, which the paramedic community may consider monitoring, exploring further, and/or contributing to, is presented in Table 1.
Summary of Recommendations with Guiding Theoretical or Conceptual Framework
Abbreviations: GRS, global rating scale; OSCE, objective structured clinical examination; PBA, performance-based assessment; SBA, simulation-based assessment; WBA, work-based assessment.
Emerging Areas of Assessment Research
The assessment community continues to explore strategies aimed at optimizing the accuracy and utility of PBA. At a minimum, four areas of research have emerged as rich sources of study. These include: (a) a greater emphasis on programmatic assessment; (b) a movement away from a psychometric era to one that embraces the “subjective and collective;”Reference Malau-Aduli, Mulcahy and Warnecke 104 , Reference Eva and Hodges 105 (c) the use of narratives in addition to, or in place of, rating scales;Reference Hodges 106 and finally, (d) studies exploring rater cognition.Reference Hanson, Rosenberg and Lane 107 , Reference Gingerich, Kogan, Yeates, Govaerts and Holmboe 108 These relatively new areas of research are shaping the way the assessment community is viewing, implementing, and studying assessment practices. Reviewing each of these expanding bodies of literature in detail is beyond the scope of this report; however, brief summaries as they relate specifically to the growing field of PBA are provided.
Programmatic Assessments
Van der Vleuten and others suggest that assessments of individuals must encompass a variety of measures, carefully assembled such that a comprehensive understanding of the individuals degree of competence can be understood fully.Reference Van der Vleuten 36 This includes different stimuli (eg, multiple-choice questions, simulation, and WBAs) and response formats throughout Miller’s pyramid, but also within each level. Simulation and WBAs, for example, could be designed such that voids in WBA settings, based on the nature of the paramedic environment, could be complemented by SBA to contribute further to construct representation. Multiple sampling remains as an important feature, and strengths include having multiple sources of information usually from multiple perspectives and contexts collected longitudinally, each contributing a piece of the competence picture (when used for summative purposes). One of the methodological challenges that remain, however, with programmatic assessments of this kind includes how best to meaningfully assemble the data to form defensible decisions.
Practice Implications
Programmatic assessments should be considered in formative and summative assessment strategies. However, this, in effect, moves the decision point from the point-in-time assessments to either individuals or committees who must somehow formulate decisions regarding competence, often with conflicting information. This is not inherently flawed; however, it does require additional study to understand ways of addressing different defensibility challenges while remaining feasible.
Embracing the Subjective and Collective
Assessment of clinical competence has largely been dominated by a psychometric discourse, which is now being challenged.Reference Van der Vleuten 36 , Reference Eva and Hodges 105 The traditional psychometric view has been characterized by an unwavering pursuit of reliability and countless efforts to optimize “objectivity” while avoiding anything that might be classified as biased or subjective.Reference Eva and Hodges 105 This review of the literature and recommendations above are reflective, in part, of these principles mainly because of what has dominated in the literature. However, researchers recently have begun to better understand some of the limitations of this psychometric discourse and have argued that much of what was once considered noise (ie, deviations from a “true” score), particularly where rater views are concerned, may actually be viewed as “signal” (ie, meaningful rich variation). The main philosophical shift being that each sample of information is meaningful, important, informed, and/or influenced by context and the rater, and when taken together, collectively contributes to the understanding of the candidate’s readiness for independent practice. That is, there is no “true” score to capture, per se, rather an assembled co-constructed representation of the candidate’s abilities. In other words, subjectivity may no longer be equated with unfairness, and rather than make attempts to optimize standardization, supposed objectivity, and control of biases, the assessment community is being encouraged to embrace subjectivity and the idiosyncrasies of raters. However, some caution a complete abandonment of psychometric principles in place of more constructivist methodologies.Reference Tavares and Eva 109
Practice Implications
Subjective assessments of paramedic clinical competence have value, assuming that multiple views are obtained and processes for assembling and synthesizing the information have rigor.
The Use of Narratives
This growing emphasis on the value of subjectivity, in addition to the complexity with which rater judgment occurs, has led the assessment community to consider the role of narratives. When raters observe, and then translate their observations to numbers on rating tools, a significant amount of information may be lost; the use of narratives may be a solution.Reference Hodges 106 When used, the collection of narratives are then intended to be assembled and analyzed using qualitative strategies (eg, the identification of emerging themes in the data) to inform decisions or inferences regarding competence. The same concepts of validity apply, however, re-conceptualized to align with validity evidence that might be more consistent with constructivist perspectives or qualitative methodologies. Whereas in psychometric or positivist views, one of the questions researchers might ask may be “are the rater views the same enough” (promoting inter-rater reliability), in this emerging view, researchers may instead ask “are they different enough” (different enough to capture the construct in its entirety, as an example). Each RBA contributes only a piece of the larger construct, and as such, it is important that, collectively, the construct as defined by the profession is captured in some way, especially when narratives are unstructured. It is not entirely clear yet what the best method (taking into consideration feasibility) of assembling the narratives would be and/or how to work with the variable and interpretations that may arise.Reference Norman 110
Practice Implications
Given that raters are viewed as naturally part of the assessment process and the limitations associated with converting observations to rating tools, narratives, in addition to or in place of numbers, may provide an important way of assessing competence.
Rater Cognition
Finally, researchers are placing the rater, rather than the process (eg, OSCE) or rating tools as the object of study.Reference Hanson, Rosenberg and Lane 107 This is a complicated area of research with many theoretical frameworks driving research questions, including the role of first impressions,Reference Ginsburg, Regehr, Lingard and Eva 111 social contexts,Reference Wood 112 performance theories,Reference Gingerich, Regehr and Eva 113 and cognitive load.Reference Gingerich, Kogan, Yeates, Govaerts and Holmboe 108 Briefly, researchers are finding that subjectivity in RBAs cannot be avoided, and that raters are anything but objective measurement tools reliably collecting and transferring information onto rating tools. For instance, Yeates et al find that, even when observing the same performance, raters attend to and give greater importance to different aspects of performance.Reference Govaerts, Van de Wiel, Schuwirth, Van der Vleuten and Muijtjens 114 The assessment community now views raters as cognitive filters actively detecting and selecting information, processing the information while often influenced by a host of factors (eg, fatigue, previous performances, and inherent human capacity) and engaging in a translation process when forming categorical decisions or formulating feedback. Further, raters may be engaging in internal strategies that satisfy, rather than optimize, their performance.
Practice Implication
While researchers continue to explore rater cognition, the emerging message seems to be to accept that objectivity in the assessment of clinical competence context may not exist and that rater judgment should be further understood, valued, and harnessed.
There is growing discussion in the assessment literature and support for the value of rater judgment.Reference Malau-Aduli, Mulcahy and Warnecke 104 Rather than making attempts to objectify assessment processes, efforts should be made to align raters’ natural thinking with assessment processes.Reference Brennan 57 , Reference Yeates, O’Neill, Mann and Eva 115 , Reference Ginsburg, McIlroy, Oulanova, Eva and Regehr 116 Assessment designers should embrace rater judgment and be less concerned about rater “subjectivity” when appropriate processes are in place.
Conclusions
Assessment of clinical competence is a complex process requiring an appreciation of a broad and ever-expanding literature. A number of practice implications have been identified for the community to explore in more detail, and perhaps to integrate. Inherent in these practice implications is a (not so hidden) research agenda for the paramedic community to consider as the profession evolves. Those responsible for decisions regarding competence will need to engage in the scientific discussion to support safety in the professions through best practices in assessment. The conceptual frameworks discussed (eg, Kane’s conceptualization of competence, Brennan’s concept of a universe score, Miller’s pyramid, reliability, validity, and multiple sampling) and the emerging programs of research (eg, programmatic assessment, harnessing inevitable “subjectivity,” the use of narratives, and understanding how raters think) provide a foundation by which to structure, and perhaps to expand, assessments of paramedic competence. While this review has focused mainly on summative assessments at the entry to practice level, similar challenges exists in formative models, or even in the on-going maintenance of competence issue.
Acknowledgments
The authors would like to thank ORNGE Transport Medicine (Mississauga, Ontario, Canada), McMaster University (Hamilton, Ontario, Canada), Centennial College Paramedic Program (Toronto, Ontario, Canada), the Paramedic Association of Canada (Ottawa, Ontario, Canada), York Region Paramedic and Seniors Services (Sharon, Ontario, Canada), and the University of Ottawa (Ottawa, Ontario, Canada) for supporting this work.
