Introduction
Every physician who makes decisions about diagnosis or treatment runs the risk of a potentially fatal decision error, by sticking to a wrong preliminary decision and not re-evaluating it appropriately. This error can occur because physicians, like all other individuals, have the tendency to confirm their preconceived ideas and neglect contradictory or unsupportive information (Nickerson, Reference Nickerson1998; Dawson, Reference Dawson, Chapman and Sonnenberg2000). For example, a physician who has made a preliminary diagnosis of migraine in a patient subsequently searches mainly for symptoms that confirm the preliminary diagnosis. In doing so, the physician neglects to obtain or ignores conflicting information suggesting another diagnosis (e.g. memory loss/ataxia indicating a brain tumor). In the case of a wrong preliminary diagnosis, the physician would not detect their mistake and the final diagnosis would be incorrect. Thus, in this example, the physician overlooks the presence of a brain tumor because of a biased information search.
Such a tendency to confirm a favored hypothesis is called ‘confirmation bias’. This bias may occur in the interpretation of available information or in the search for new information subsequent to a favored or initial hypothesis (Klayman, Reference Klayman1995; Gurmankin et al. Reference Gurmankin, Baron, Hershey and Ubel2002; Gambrill, Reference Gambrill2005). Confirmation bias is expected to be highly influential on reasoning and decision making, and is held responsible for many decisional errors (Nickerson, Reference Nickerson1998; Croskerry, Reference Croskerry2002). Some publications have raised the awareness of confirmation bias even in the lay public (Crichton, Reference Crichton2007; Groopman, Reference Groopman2007), and medical literature contains frequent warnings against confirmation bias in the context of making diagnoses (Klein, Reference Klein2005; Pines, Reference Pines2006). However, this warning is supported mainly by studies on students or non-physicians (Kern & Doherty, Reference Kern and Doherty1982; Frey, Reference Frey and Berkowitz1986; Friedman et al. Reference Friedman, Connell, Olthoff, Sinacore and Bordage1998; Jonas et al. Reference Jonas, Schulz-Hardt, Frey and Thelen2001). The few studies involving physicians (e.g. Wolf et al. Reference Wolf, Gruppen and Billim1985; Krems & Zierer, Reference Krems and Zierer1994) have, to our knowledge, not systematically studied the influence of confirmation bias on physicians' information search and on the quality of the diagnosis (as described in the example of the patient with migraine/tumor). In addition, no research on this issue has been reported from the field of mental health.
Therefore, we have conducted an experimental study to investigate: (1) whether and to what extent psychiatrists and medical students are prone to confirmation bias when searching for new information after a preliminary diagnostic decision; and (2) whether confirmation bias in the information search negatively impacts the quality of diagnostic decision and subsequent poor treatment recommendations.
Method
To study psychiatrists' information search and decision making, we chose an experimental study design. The experiment was carried out by two trained experimenters (both graduate students) in one-on-one sessions lasting approximately 30 min.
Experiment
At the start of the experiment, a case vignette (describing a 65-year-old male patient exhibiting depressive symptoms who had been admitted to a psychiatric hospital due to an overdose of a sedative agent) was presented to the subjects (see Fig. 1). Participants were requested to make a preliminary diagnosis in which they were given the choice between ‘Alzheimer's disease’ (dementia) or ‘severe depressive episode’ (i.e. ‘major depression’). The symptoms described in the case vignette were selected such that the majority of participants tended to make the (incorrect) preliminary diagnosis of ‘depression’. This procedure was chosen because we were interested in studying whether participants are able to correct their initial wrong diagnosis depending on how they search for subsequent information (i.e. confirmatory, balanced or disconfirmatory search).
Procedure.
Subsequent to the preliminary diagnosis, the participants were informed that they could obtain additional information about the patient to confirm or revise their initial diagnosis. They were offered 12 items of information, six of which spoke in favor of ‘Alzheimer's disease’ and six that spoke in favor of a ‘severe depressive episode’. These 12 items of information (each comprised approximately 150–175 words) were presented to the doctors, summarized as to their main thesis. A summary thesis in favor of depression was, for example, ‘The diagnosis “severe depressive episode” is supported by the patients’ statements about death'; and a summary thesis in favor of Alzheimer's disease was, for example, ‘A hint of the presence of “Alzheimer's disease” could be that the patient shows memory problems’. The doctors were asked to decide on obtaining detailed information corresponding to the respective summary thesis by making a checkmark. The detailed information requested by the participants was then supplied by the experimenter in written form. The summaries of the detailed information were not informative enough to enable a reliable diagnosis, so that the participants were forced to ask for the full information if they thought they needed it to reach a diagnosis. The information search was serial, that is doctors requested detailed information one by one. Participants were able to request as many pieces of detailed information as they wanted. Subsequent to the information search, the participants were requested to make their final diagnosis by choosing between ‘Alzheimer's disease’ and ‘severe depressive episode’. In addition, they were to state which treatment they would recommend to the patient. Finally, participants were asked questions with regard to their age, gender and professional experience.
Pretests
The case vignette and the additional information available to the participants were developed in conjunction with six experts on dementia/depression, and it was confirmed that the case vignette and the additional information were compatible with a diagnosis of Alzheimer's disease and not with the diagnosis of a severe depression according to ICD-10 criteria. Moreover, the case vignette and the additional information were checked in extensive pretests with 50 physicians (different from those participating in the main experiment). Emphasis was placed on the construction of the 12 summary theses. Here, it was ensured that they spoke equally strongly in favor of depression and of Alzheimer's disease and that subjects were still interested in requesting the detailed information after having read the summary thesis. Unlike the summary theses, the detailed pieces of information were not balanced, but contained strong evidence for Alzheimer's disease and only weak evidence for a depressive episode.
Overall, the experimental design of our study was similar to the information search paradigm of Frey (Reference Frey1981), which is used in many psychological information search experiments (e.g. Friedman et al. Reference Friedman, Connell, Olthoff, Sinacore and Bordage1998; Fischer et al. Reference Fischer, Jonas, Frey and Schulz-Hardt2005).
Participants
As there is evidence that more experienced people differ from inexperienced people with regard to confirmation bias (Kern & Doherty, Reference Kern and Doherty1982), we decided to apply the experiment not only to psychiatrists but also to a sample of medical students (‘subinterns’). To recruit the physicians (consecutive recruitment), the staff of three psychiatric hospitals (two state hospitals and one university hospital) were asked to participate. Medical students were recruited consecutively during a week-long internship in a psychiatric university hospital.
The study was approved by the Institutional Review Board of the Technical University of Munich.
Data analysis
Sociodemographic data of participants were described as frequencies or as means and standard deviations. Preliminary and final diagnoses were reported as percentages. The participant's information search (i.e. number and type of requested items of information) was described in terms of means and standard deviations and was also analyzed by unpaired and paired t tests. Items of information chosen by study participants speaking in favor of the selected preliminary diagnosis are designated as confirmatory information. Conversely, items of information speaking in favor of the non-selected diagnosis are designated as disconfirmatory information. Furthermore, an information search was labeled as a ‘balanced information search’ if a participant searched for the same amount of confirmatory information as disconfirmatory information. An information search containing at least one more item of confirmatory information than the items of disconfirmatory information was named ‘confirmatory information search’; and an information search incorporating more disconfirmatory than confirmatory items of information was named ‘disconfirmatory information search’.
To determine whether professional experience has an influence on information search, a χ2 test and a correlation analysis were conducted. The influence of the participants' information search and sociodemographic variables on the quality of diagnostic decisions was analyzed by using a logistic regression model to account for possible interactions of the variables of interest. The influence of the diagnosis on treatment decisions was calculated by χ2 tests and Fisher's exact tests. Data were analyzed using SPSS version 16.0 (SPSS Inc., USA).
All statistical tests were two-tailed, and a p value <0.05 was considered significant.
Results
A total of 75 psychiatrists (44 men, 31 women) from the three participating psychiatric hospitals took part in this study. Their mean age was 35.7 years (s.d.=5.4) and their professional experience averaged 6.0 years (s.d.=4.0). Medical students (31 men, 43 women) were aged between 21 and 37 years (mean=25.0, s.d.=2.9); they were currently in their fourth year of medical studies. None of the physicians and only one female student (out of the 76 students approached) refused to take part in this study.
Preliminary diagnoses and information search
A total of 97% of the physicians (95% of the students) chose ‘severe depressive episode’ and not ‘Alzheimer's disease’ as the suspected preliminary diagnosis (see Table 1).
Diagnostic decisions, information search and treatment decisions
s.d., Standard deviation.
Out of a maximum of 12 items of detailed information, physicians selected a mean of 8.4 items (s.d.=3.3); medical students (mean=8.0, s.d.=3.1) did not differ from psychiatrists (unpaired t test: t=0.64, p=0.52). Overall, psychiatrists requested more disconfirmatory items of information (mean=4.6, s.d.=1.6) than confirmatory items of information (mean=3.8, s.d.=2.0; paired t test: t=−3.98, p<0.001) whereas the students' information search was balanced (mean of disconfirmatory information=4.1, s.d.=1.7; mean of confirmatory information=3.9, s.d.=1.7; paired t test: t=−1.21, p=0.23). A total of 13% of the psychiatrists (students: 25%) showed a confirmatory information search. A balanced information search was conducted by 43% of the psychiatrists (students: 44%) and a disconfirmatory information search by 44% of the psychiatrists (students: 31%). The difference between psychiatrists and students regarding the information search was not significant (χ2=4.59, p=0.10). However, a correlation analysis including only psychiatrists (n=75) revealed that the less professionally experienced the physicians were, the more often they searched for information in a confirmatory way (r=−0.30, p=0.01).
Influence of participants' information search on the accuracy of the final diagnosis
After the information search, more than half of the participants changed their preliminary diagnosis and indicated ‘Alzheimer's disease’ as the final diagnosis, which was the correct answer (Table 1). A total of 41% of the psychiatrists (medical students: 36%) chose the wrong diagnosis (‘depression’). A total of 73% of the psychiatrists searching in a disconfirmatory manner and 53% of the psychiatrists searching in a balanced manner made the correct final diagnosis (‘Alzheimer's disease’). However, only 30% of psychiatrists (medical students: 37%) who showed a confirmatory information search ultimately made the correct diagnosis (see Fig. 2).
Frequencies of correct diagnosis in relation to information search.
To test whether type of information search had a significant influence on the accuracy of the final diagnosis, we performed a logistic regression analysis with accuracy of diagnosis as the dependent variable and other variables that might influence diagnostic accuracy (e.g. number of information items searched, professional experience) as covariables (Table 2).
Logistic regression analysis: influence of information search and sociodemographic variables on the accuracy of final diagnosis
OR, Odds ratio; CI, confidence interval; Ref, reference.
According to the model, participants conducting a confirmatory information search were significantly less likely to make the correct diagnosis compared to participants searching in a balanced or disconfirmatory way. Additionally, the number of information items searched significantly predicted diagnostic accuracy, with participants selecting six or less items showing poorer diagnostic accuracy compared to participants selecting more than six items.
Influence of diagnostic decision on therapy choice
Most of the psychiatrists (93%) who diagnosed Alzheimer's disease recommended anti-dementia drug therapy, whereas none of the doctors who diagnosed depression proposed such a treatment (χ2=63.72, p<0.001). There were also significant differences regarding treatment with antidepressants: 77% of psychiatrists who made the correct diagnosis (Alzheimer's disease) recommended an antidepressant drug therapy, whereas all of the psychiatrists who incorrectly diagnosed depression proposed the use of an antidepressant (χ2=8.13, p=0.004). The students behaved similarly to the physicians (see Table 1): students who made the correct diagnosis recommended anti-dementia drug therapy more often (χ2=66.18, p<0.001) and antidepressant drug therapy less often (χ2=10.12, p=0.001) than students who made the incorrect diagnosis.
Discussion
Our study has two important findings: (1) confirmation bias is present in some psychiatrists' and medical students' information search; and (2) confirmation bias in information search leads to poorer diagnostic accuracy.
Confirmation bias is present in some psychiatrists' and medical students' information search
In our study, about one out of eight physicians and every fourth medical student showed a confirmation bias when searching for new information subsequent to a preliminary decision. That is, physicians and medical students are not immune to confirmation bias. These results are similar to the findings of numerous studies in psychology showing that individuals often search for confirmatory evidence when making decisions (Frey, Reference Frey1981, Reference Frey1986; Jonas et al. Reference Jonas, Schulz-Hardt, Frey and Thelen2001; Fischer et al. Reference Fischer, Jonas, Frey and Schulz-Hardt2005). Furthermore, our results are in accordance with former studies on ‘pseudodiagnosticity’ (a related field to confirmation bias), which found that subjects sometimes tend to select diagnostically irrelevant information (Kern & Doherty, Reference Kern and Doherty1982).
Our study also showed that less experienced decision makers might be affected by confirmation bias more often than more experienced decision makers. This is congruent with results of Krems & Zierer (Reference Krems and Zierer1994), who found that novices exhibit a higher confirmation bias than experts, in so far as novices stick to their wrong assumption longer and correct it later than experts. Knowledge and experience contribute to integrating conflicting information more readily (Arocha et al. Reference Arocha, Patel and Patel1993), which might lead to less avoidance of disconfirming information and thus to experts being less prone to confirmation bias.
Confirmation bias in information search leads to poor diagnostic accuracy
Many experts warn against confirmation bias, but this study is, to our knowledge, the first investigation showing that confirmation bias in information search has an impact on the quality of a decision: psychiatrists and medical students who conducted a confirmatory information search made a wrong diagnosis more frequently than their colleagues who showed a balanced or disconfirmatory information search. This is because someone who fails to consider disconfirming information is unlikely to recognize the incorrectness of their preliminary decision and will subsequently tend to stick to it (Nickerson, Reference Nickerson1998). But why do individuals show confirmation bias in information search, although it can lead to wrong decisions? Dissonance theory (Festinger, Reference Festinger1964; Frey, Reference Frey1986; Jonas et al. Reference Jonas, Schulz-Hardt, Frey and Thelen2001) can help explain this phenomenon: after having made a (preliminary) decision, aspects speaking against the chosen alternative are present in a decision-maker's mind and lead to cognitive dissonance. Individuals try to overcome this aversive state by searching for confirmatory information and ignoring information speaking against the chosen alternative. Another explanation for confirmation bias is that this bias, like many other biases, occurs as a result of ‘cognitive limitations’ (Klayman, Reference Klayman1995): individuals tend to use ‘simpler rather than more complex cognitive strategies’ (Elstein & Schwarz, Reference Elstein and Schwartz2002), and it is simpler and involves lower cognitive costs to stick to a preliminary diagnosis than to falsify it by searching for disconfirmatory information.
A confirmatory information search not only entailed wrong diagnostic decisions but also, not surprisingly, influenced subsequent therapeutic decisions. Thus, psychiatrists who had chosen the wrong diagnosis (depression) less often prescribed anti-dementia drugs than psychiatrists who had chosen the correct diagnosis (Alzheimer's disease).
Implications of findings for clinical practice
Considering our results, the question arises how dangerous a bias is if ‘only’ 13% of the psychiatrists (and 25% of the medical students) are affected. Because wrong diagnoses have far-reaching consequences, every patient who receives a wrong diagnosis due to confirmation bias is one too many. Furthermore, it should be noted that confirmation bias is only one bias among many other cognitive biases (e.g. omission bias, hindsight bias; Croskerry, Reference Croskerry2003a). If confirmation bias alone affects more than 10% of physicians, and if other biases that are detrimental to medical reasoning to a comparable extent also play a role, then a considerable proportion of medical decisions (i.e. diagnoses and therapies) may be flawed. Because diagnostic decisions are considered to be among the most frequent error-prone decisions made by physicians (Weingart et al. Reference Weingart, McL Wilson, Gibberd and Harrison2000; Newman-Toker & Pronovost, Reference Newman-Toker and Pronovost2009), factors influencing the quality of diagnoses (such as confirmation bias) are not only of scientific interest but also have a major impact on the quality of medical care. Thus, highly sophisticated diagnostic techniques [e.g. magnetic resonance imaging (MRI) scan, the liquor test] have insufficient utility if they are not requested or considered because of confirmation bias. Furthermore, Dawson (Reference Dawson, Chapman and Sonnenberg2000) suggested that confirmation bias might lead to unnecessary laboratory tests being carried out by physicians only to meet the desire to confirm a favored hypothesis. This might result in ‘increasing inefficiency and costs’ of the diagnostic process (Dawson, Reference Dawson, Chapman and Sonnenberg2000).
Strategies against confirmation bias
Because disconfirming evidence helps to detect incorrect preliminary decisions (Nemeth et al. Reference Nemeth, Connell, Rogers and Brown2001), it might be assumed that a disconfirmatory information search could prevent wrong decisions. However, a disconfirmatory information search does not inevitably guarantee the highest diagnostic accuracy because in the case of a correct preliminary decision, a disconfirmatory information search might lead to a denial of the correct prior decision and would thus be disadvantageous. Therefore, if physicians want to make sure that their decisions are of high quality, irrespective of whether a preliminary decision is correct or incorrect, they should consider confirmatory and disconfirmatory information in a balanced way.
There are several concrete debiasing strategies that could be used to overcome confirmation bias: for example, warning physicians about confirmation bias or encouraging doctors to think systematically of alternatives (e.g. using checklists) to the preliminary hypothesis. The results of several studies on the effectiveness of these single strategies are contradictory, however (Mynatt et al. Reference Mynatt, Doherty and Tweney1978; Arkes, Reference Arkes1981; Wolf et al. Reference Wolf, Gruppen and Billi1988; Hirt & Markman, Reference Hirt and Markman1995; Parmley, Reference Parmley2006). A comprehensive approach to counteracting diagnostic errors (such as confirmation bias) is the use of cognitive forcing strategies, that is ‘ways of actively monitoring and modifying decision making through insight into one's clinical thinking’ (Gallagher, Reference Gallagher2003). These comprise training in meta-cognition (i.e. thinking about thinking), inform about cognitive errors, identify contexts in which these errors occur, and help in selecting appropriate cognitive strategies (Croskerry, Reference Croskerry2003b). Experts criticize the fact that this meta-cognitive training is, like many other debiasing techniques, insufficiently validated (Berner, Reference Berner2007). Further debiasing techniques are algorithms and computer-aided diagnostic systems (Berner et al. Reference Berner, Maisiak, Cobbs and Taunton1999). Although several studies have demonstrated a positive influence of decision support systems on physicians' performance, it is argued that these techniques are often too impractical and time-consuming to be incorporated into daily practice (Berner et al. Reference Berner, Maisiak, Cobbs and Taunton1999; Graber & VanScoy, Reference Graber and VanScoy2003; Ramnarayan et al. Reference Ramnarayan, Cronje, Brown, Negus, Coode, Moss, Hassan, Hamer and Britto2007). In summary, there are several promising debiasing strategies against confirmation bias, but to date most of them have not been sufficiently evaluated, nor have they been implemented in medical practice or education.
Limitations
The experimental design of our study is based on the experimental paradigm of Frey (Reference Fischer, Jonas, Frey and Schulz-Hardt1981), which has been used successfully in many psychological information search experiments. To date, its validity has not been proven in the medical context, and thus the results of our study should be interpreted cautiously. Additionally, limitations of our study arise from the experimental setting, which inevitably differs in several ways from the clinical settings in which medical decisions are normally made. Thus, in our experiment we presented only two diagnostic alternatives to choose from, which clearly differs from everyday diagnostic decisions, where more than two possible alternatives are usually present. However, Jonas & Frey (Reference Jonas and Frey2003) have shown (for non-medical decisions) that a confirmatory information search is also present if there are more than two decisional alternatives. Another limitation is that we only supplied information about symptoms in written form. This differs from making medical diagnoses in real life, where physicians examine real-life patients and obtain important information from different sources (e.g. from visual cues or interaction with the patient). In addition, contrary to our study, real-life diagnoses are often made under time pressure. Several studies from psychology suggest that time pressure increases confirmation bias (Ask & Granhag, Reference Ask and Granhag2007; D. Frey, unpublished observations). Therefore, this bias may occur even more frequently under natural conditions.
Conclusions
We found that psychiatrists and medical students, and also other individuals, can be prone to confirmation bias. We were able to show that psychiatrists who conducted a confirmatory information search showed poorer diagnostic accuracy. Given that confirmation bias increases the risk of a chain of wrong decisions (e.g. wrong treatment decisions, requisition of unnecessary laboratory tests), more emphasis should be put on identifying and teaching effective debiasing techniques.
Acknowledgments
We thank H.-J. Möller (Department of Psychiatry, Ludwig-Maximilians-Universität, München); H. Pfeiffer (Bezirkskrankenhaus Augsburg); M. Riedel (Department of Psychiatry, Ludwig-Maximilians-Universität, München); M. Schmauß (Isar-Amper-Klinikum, Haar) and A. Kurz and I. Maerz (Department of Psychiatry, Technische Universität München) for their support in recruiting participants. We also thank all participating psychiatrists and medical students for their help.
Declaration of Interest
None.
