Introduction
Awareness of personal abilities and limitations in a clinical setting is an important part of professional practice and development. Self-assessment of thinking and reasoning is part of metacognition, which is the capacity to reflect on personal thinking patterns.Reference Schwartz and Perfect1 The resurgence of academic interest into how people’s metacognitions affect their judgment and decision-making processes may be dated to the late 1970s and early 1980s, when scholars like Daniel Kahneman began to look at people’s decisions in both ‘real world’ and laboratory-based studies (leading to Kahneman’s winning of the Nobel Prize in Economics in 2002). It has been found that people overestimate how good they are on assigned decisions or tasks in the studies of everyday activities across education, health perception and work.Reference Dunning, Heath and Suls2, Reference Lichtenstein, Fischhoff, , Phillips, Kahneman, Slovic and Tversky3 Several studies have shown that doctors and other health professionals are often poor at rating their own level of knowledge performance. For example, Weiss, Koller, Hess and WasserReference Weiss, Koller, Hess and Wasser4 found that residents in obstetrics/gynaecology training were poor at judging their own technical skills and clinical problem-solving. The confidence ratings nurses gave in their knowledge of core life-support protocols did not correlate with their actual knowledge base.Reference Marteau, Johnston, Wynne and Evans5 In a seminal early paper, clinical psychologists were found to overestimate the likelihood that their assessments of documentation relating to clients will lead to accurate judgments.Reference Oskamp6 Within academia,Reference Trafimow and Sniezek7 politicsReference Tetlock, Hardman and Macchi8 and medicine,Reference Wallstein9 it has been found that the self-rated confidence of both novices and experts does not predict good task performance. Clearly, health professionals’ self-ratings on clinical decision-making are pertinent for both patient outcomes and good professional development.
In this study, self-ratings of problem-solving were examined in radiation therapists (RTs) at different levels of post-qualification experience (PQE): basic, senior and clinical specialist. RTs were chosen for two reasons: first, to redress the overrepresentation of medical practitioners in the clinical decision-making literature (cf.Reference Chapman and Sonnenberg10); and second, as the population of RTs in Ireland is approximately 160, a high proportion of the total number of RTs could be sampled, thus boosting generalisability of the findings. A previous study found that years of experience did affect the self-rated confidence of RTs when judging the accuracy of electronic portal image review, but was not found to affect the actual accuracy of anatomy contouring and matching.Reference Rybovic, Halkett and Williams11
Two categories of problem were used: ambiguous clinical scenarios, where there is no unequivocally correct answer; and critical thinking items (CTIs). CTIs test the process of sifting through, synthesising and evaluating knowledge and data in order to reach a logically derived decision.Reference Cottrell12 They are increasingly being used as a selection tool in professions, and are part of the BioMedical Admissions Test (BMAT)Reference Butterworth and Thwaites13 for entry to medicine, veterinary medicine and some related courses in several UK institutions. The inclusion of CTIs allows self-ratings to be assessed in terms of both magnitude and calibration outside of the clinical workplace, thus giving a fuller picture of the respondents’ metacognition. This paper examined whether the magnitude of self-ratings of task performance differs according to the PQE and the type of problem encountered.
METHOD
Participants
Sixty RTs working at various hospitals in the Republic of Ireland volunteered for the study.
Materials
Based on the results of a pilot study of academic RTs, three written clinical scenarios each varying in difficulty were devised (see Appendix A). These were reviewed by two academic RTs. The CTIs were randomly selected from the BMAT (see Appendix B). All six items were presented as a questionnaire that was distributed as an e-mail attachment, with the clinical scenarios preceding the CTIs.
Procedure
Prior ethics approval for the study was granted by the Research Ethics Committee of the Faculty of Health Sciences at Trinity College, Dublin. Once they had consented to take part in the study, participants were informed that they could generate as many clinical scenario solutions as they saw fit. Because the solutions generated were self-evident, there was no need for an inter-rater reliability coefficient calculation. For the purpose of face validity of the study, participants were told about the scoring systems for the clinical scenarios and the CTIs. However, as self-ratings of task performance were the focus of the study, these data were not recorded. Participants were instructed to generate as many solutions to the clinical scenarios as they saw fit, and to answer the CTIs. Participants then rated their confidence on all clinical scenarios and on all CTIs by circling one number from 1 (not very confident) to 10 (very confident) on a Likert scale.
Design
A once-off survey was carried out. Participants were assigned to one of the three groups dependent upon the level of PQE: RTs (n = 29), senior radiation therapists (SRTs) (n = 17), and clinical specialists (CSs) (n = 14). A 2 (problem type) × 3 (PQE) mixed factorial between- and within-groups design was used, with problem type as a repeated measure.
Results
In terms of PQE, participants classified themselves as follows (mean number of months of experience in parentheses): 29 RTs (21 months); 17 SRTs (70 months); 14 CSs (134 months).
Means (and standard deviations) of confidence ratings per problem type and PQE level are presented in Table 1.
Mean (and 95% confidence interval) confidence ratings per level of PQE and problem type
The data of the confidence ratings per problem type and PQE level were normally distributed with homogeneity of variance among the conditions. These data were subjected to a 2 (problem type) × 3 (PQE level) analysis of variance (ANOVA), which produced a main effect of problem type, F(1, 57) = 50.62, MSE = 0.45, p < 0.001, η 2 = 0.47, with participants being more confident overall on the clinical scenarios. The ANOVA also yielded a main effect of PQE level, F(2, 57) = 4.07, MSE = 2.11, p < 0.001, η 2 = 0.13, which revealed that, regardless of problem type, ratings were highest among SRTs and lowest among RTs. The ANOVA also yielded an interaction, F(2,57) = 33.55, MSE = 0.45, p < 0.001, η 2 = 0.54, which was further investigated in the following manner: a one-factor ANOVA on the clinical scenarios produced a main effect, F(2,57) = 17.72, MSE = 1.15, p < 0.001, η 2 = 0.38, with RTs being less confident than SRTs who were less confident than CSs (p < 0.05); a one-factor ANOVA on the CTIs produced no significant main effect, F(2,57) = 2.32, MSE = 1.41, p > 0.10, η 2 = 0.08, with no reliable difference among the PQE groups.
DISCUSSION
For the clinical scenarios, the results revealed that self-rated confidence increased in line with PQE. This is a logical finding, as it would be expected that the greater store of successful decisions in the careers of more experienced RTs would serve as a benchmark in the self-rating task. Indeed, it could be argued that it is helpful for basic grades to be less confident, because this makes it more likely that they will ask for help and so not make hasty or poor-quality decisions.
For the critical thinking problems, the results revealed that self-rated confidence did not differ according to PQE. This finding suggests that the greater self-confidence of more experienced staff is domain-specific, and does not necessarily translate to other problem-solving scenarios. Furthermore, less experienced staff are more likely to be recent graduates, and so might be more confident at abstract problem-solving as a function of their more recent exposure to academic course content.
The limitations of the study include the fact that actual task performance was not measured; one possible methodological difficulty here is that an assessment of actual task performance would require an agreed measure. Drawing up such a measure was outside the expertise of the current authors. If an agreed measure with good metric properties was devised, then future work could examine the correlation between self-confidence ratings and task performance.
APPENDIX A: CLINICAL SCENARIOS
For the each of following scenarios, a more senior member of staff is unavailable for consultation, so you have to make the decision yourself. Remember that there is no correct or incorrect answer. Please write as little or as much as you wish for each scenario.
Scenario 1
The treatment machine is just back in service after a breakdown period of 2 hours. One of the patients receiving treatment for prostate cancer is demanding that he is treated next as he is afraid that he will not be able to wait longer with a full bladder. However, there are two other patients ahead of him in the queue. Additionally, he is already late for another appointment in a different hospital. What do you decide to do?
Scenario 2
You and one other radiation therapist are covering for staff breaks on a particular unit. A patient who is in his second week of treatment for a head and neck malignancy arrives for his treatment. However, when you set him up in the BDS, you notice that the shielding in the form of customised lead is inaccurate in the left–right axis. All port films and diode measurements have been acceptable to date. What do you decide to do?
Scenario 3
A patient who is being treated for breast cancer comes for treatment at 10 pm on a Thursday. She is to receive the penultimate fraction of her radiotherapy course. Since the team last saw her, her skin reaction has progressed to a grade four (EORTC/RTOG) in the infra-mammary fold. She is really looking forward to finishing her treatment on Friday and returning to her family home on Saturday to attend her son’s wedding. What do you decide to do?
APPENDIX B: CRITICAL THINKING ITEMS
Problem 1
Finally, it was time to snack on a sandwich. The maths teacher removed her sandwich and remarked that the coating of mould had now completely covered the surface of the bread. It was 30th June. The sandwich was first packed for a field trip on 1st June. If the area occupied by the mould doubled every day, on what day was the surface half-covered with mould?
Problem 2
Every branchiopod is a crustacean and every crustacean is an arthropod. No insect is a crustacean. Which two of the following are true?
i. Every branchiopod is an arthropod.
ii. No insect is an arthropod.
iii. No branchiopod is an insect.
iv. Some crustaceans are insects.
Problem 3
In a class of 30 students, all must study at least one language, but no more than three languages. 70% study French, 40% study German, 20% study Spanish and 10% study Italian. Which two of the following must be true?
a. At least three study both French and German
v. No more than 12 study French and German
vi. At least nine do not study either French or German
