Introduction
Depression is common in primary care settings with prevalence estimates ranging between 5–20% (Mitchell et al. Reference Mitchell, Vaze and Rao2009). In recent years, primary care physicians (PCPs) have come under increasing scrutiny for both underrecognition and overdetection (Höfler & Wittchen, Reference Höfler and Wittchen2000; Mitchell et al. Reference Mitchell, Vaze and Rao2009) of depression due to poor concordance with various gold standard diagnostic criteria (Ani et al. Reference Ani2008; Cepoiu et al. Reference Cepoiu2008; McGrady et al. Reference McGrady2010). This highlights the challenges associated with making mental health diagnoses in primary care settings (Carey et al. Reference Carey2014). Whilst underrecognition raises concern due to the high societal costs associated with disability, poor quality of life, morbidity and potentially mortality (Üstün et al. Reference Üstün2004; Cornelius et al. Reference Cornelius2014), over-detection can also be problematic due to unnecessary treatments and the potential for adverse drug effects (Mitchell et al. Reference Mitchell, Rao and Vaze2011). Despite this, most PCPs do not use screening instruments or symptom inventories to identify depression, preferring to make decisions based on their own clinical assessment and judgment (Bermejo et al. Reference Bermejo2005). One of the key challenges in deciding whether depression is present or not is the clinician's ability to differentiate whether the patient has a physical problem, a psychological problem, or both, in a setting where patients are more likely to present with somatic symptoms even when the problem is psychological (Tylee & Gandhi, Reference Tylee and Gandhi2005; Bekhuis et al. Reference Bekhuis2015). A large multi-center study found that 45–95% of primary care patients with depression presented with physical symptoms and 11% did not report any psychological symptoms (Simon et al. Reference Simon1999). This has led some to speculate whether the criteria used for diagnosing depression in primary care should be different from those used in specialist settings (Armstrong & Earnshaw, Reference Armstrong and Earnshaw2004; Gask et al. Reference Gask2008; Malhi et al. Reference Malhi2014; Goldberg et al. Reference Goldberg2017).
Although many studies have explored the factors associated with depression recognition, most have used logistic regression analyses (Barbui & Tansella, Reference Barbui and Tansella2006; Akhtar-Danesh & Landeen, Reference Akhtar-Danesh and Landeen2007; Cepoiu et al. Reference Cepoiu2008; Henriques et al. Reference Henriques2009; Chin et al. Reference Chin2014). Regression modeling, however, is limited because the results can be difficult to interpret when a large list of variables are assessed together. It also does not provide information about how factors interplay (Lemon et al. Reference Lemon2003). Decision tree (also known as classification tree) analysis is a non-parametric statistical procedure that identifies mutually exclusive subgroups within a study population that share common characteristics associated with an outcome. The analysis examines all possible independent or splitting variables and selects the one that results in binary groups that are most different in respect to the dependent variable. The tree continues to grow by assessing each of the remaining independent variables to determine which variable results in the best split. The process continues until a terminal node is reached. The end product is a visual multi-level output resembling the branches of a tree with the probabilities of having the dependent measure for each node (Lemon et al. Reference Lemon2003). Tree analyses have long been used for financial forecasting, but it can have useful applications to exploring large healthcare datasets (Podgorelec et al. Reference Podgorelec2002; Dowrick et al. Reference Dowrick2011; Kuhn et al. Reference Kuhn2014).
The aim of this study was to analyze the statistical associations between patient self-reported depressive symptoms and PCP diagnosis of depression to develop a model that might simulate the decision-making process involved in diagnosing depression. Insights gained from the model can help further our understanding of how doctors make a diagnosis of depression in primary care and identify areas for quality improvement. The findings can also further our understanding of what PCPs are diagnosing as depression.
Methods
Study design
This was a secondary analysis of the baseline data obtained from a cohort study examining the epidemiology of depressive disorders in Hong Kong's primary care setting. Baseline subject recruitment occurred between October 2010 and January 2012. The study protocol and findings from the primary analyses have been previously published (Chin et al. Reference Chin2012; Chin et al. Reference Chin2014; Chin et al. Reference Chin2015a, Reference Chinb; Reference Chin2016a, Reference Chin, Choi and Wanb).
Setting and subjects
PCPs working in clinics across Hong Kong were invited to collaborate as part of a practice-based research network. Doctors were identified using the mailing list of the Hong Kong College of Family Physicians and comprised PCPs working in various private practice settings (solo, group, out-patient departments of private hospitals), government-funded General Outpatient Clinics of the Hong Kong Hospital Authority, and non-profit, non-governmental organizations, representative of how primary care is delivered in Hong Kong. Characteristics of the 59 doctors who participated in the study and their clinics have been previously described (Chin et al. Reference Chin2014). All consecutive, eligible patients presenting on one randomly allocated day each month over a 12-month recruitment period were approached to complete a self-administered survey. Patients were excluded if they were <18 years, did not understand English, Cantonese or Mandarin, had cognitive or communication difficulties, had previously been recruited to the study, or did not consult the study doctor.
Study instruments
The Patient Health Questionnaire-9 (PHQ-9) is a ten-item questionnaire commonly used to screen, monitor, diagnose, and measure the severity of depressive symptoms in primary care and as an outcome instrument in treatment studies in specialist care settings. (Spitzer et al. Reference Spitzer1999; Guo et al. Reference Guo2017). It was used to collect the patients’ self-reported symptoms of depression. The PHQ-9 scores each of the nine diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (DSM-IV) for depression from 0 (not at all) to 3 (nearly every day) over the previous 2 weeks, with a tenth item on functional impairment which does not contribute to the total score (Kroenke et al. Reference Kroenke, Spitzer and Williams2001). The Chinese version of the PHQ-9 used for this study has been validated in Hong Kong (Yu et al. Reference Yu2012).
Questions on patient socio-demographics, recent health service use and co-morbidities were adapted from previously performed health services studies (Lam et al. Reference Lam2011). Information collected included: age (continuous); gender (Male; Female); marital status (Married; other including Single/Separated/Divorced/Widowed); employment status (Yes; No); household income (⩽HK$ 30 000; >HK$ 30 000); presence of chronic diseases (None; One; Two or more); self-reported past history of doctor-diagnosed depression or mental illness (Yes; No); family history of mental illness (Yes; No); and number of doctor visits in the past month (continuous).
PCP diagnosis of depression was collected using a case record form completed by the study doctor during the index consultation. Doctors were instructed to document whether they thought the patient had a clinically significant depressive disorder (yes/no). The PCP's diagnosis of depression was made spontaneously without the aid of any screening or diagnostic tools. Doctors were blinded to their patient's PHQ-9 responses. Only the doctor's diagnosis was recorded, and the contents of the medical consultation were not collected, so it is not known whether or not the patient's self-reported PHQ-9 symptoms were elicited during the clinical encounter.
Data analysis
Patient characteristics and PHQ-9 total scores stratified by depression diagnosis status were analyzed using descriptive statistics. Differences between groups were evaluated by independent t test for continuous variables and χ2 for categorical variables. The association between PHQ-9 item responses and diagnosis of depression were examined using a logistic regression model. A second model was performed using PHQ-9 total scores in lieu of the individual item scores. Analyses were controlled for socio-demographic characteristics. Only complete data were used. Odds ratios (OR) for each factor were reported with a 95% confidence interval (CI). The goodness of fit was examined by the Hosmer and Lemeshow test (Paul et al. Reference Paul, Pennell and Lemeshow2013) and presence of multicollinearity was checked using variance inflation factors.
For a more in-depth exploration of the correlations, a conditional inference decision tree model was developed through binary recursive partitioning of potential predictors that maximized between-group differences allowing for interactions, using the ‘ctree’ function in the party package in R (Hothorn et al. Reference Hothorn2010). At each node of the tree, the recursive partitioning algorithm was reapplied to select the classifier and the split in this classifier that allowed the maximal difference in the rate of diagnosis of depression between the two subgroups. The algorithm was reported recursively until the tree was grown to an optimal number of terminal leaves. Conditional inference tree is an extension of a classification and regression tree analysis where selection procedures are based on the associations between classifiers and the rate of diagnosis of depression using a significance test using a quadratic form of statistic rather than impurity functions such as Gini Index and entropy. This method ensures that the right sized tree is developed requiring no form of pruning or cross-validation (Hothorn et al. Reference Hothorn, Hornik and Zeileis2006). The model also allows for all data to be used with no need to deal with missing data. Unlike multivariable logistic regression, this method can maximize between-group differences, allowing for higher-order interactions between classifiers and a graphical display of the results. Two tree models were generated: the first using PHQ-9 symptoms, and the second using PHQ-9 total scores.
All significance tests were two-tailed and those with p values <0.05 were considered statistically significant. Statistical analyses were performed in Stata Version 13.0 and R Version 3.3.1.
Ethics
The Institutional Review Board of the University of Hong Kong/Hospital Authority Hong Kong West Cluster (reference number UW 09–305), and all other relevant regional ethics committees approved this study.
Results
Overall, 10 179 patients consented to participate with a response rate of 81% of all eligible subjects approached. Of these, 9263 subjects who had complete data for both the doctor's diagnosis and all PHQ-9 items were included for further analysis. Amongst these, 594 patients (6.4%) received a diagnosis of depression. Among those with a PHQ-9 total score >9, 23.2% received a PCP diagnosis of depression. Amongst those with a PCP diagnosis of depression, 41.2% had PHQ-9 scores >9. Table 1 displays the subject characteristics categorized by depression diagnosis status. A comparison of the subjects who were included and excluded from the analyses found no statistical differences in characteristics between the two groups (Appendix 1).
Subject characteristics of depression diagnosis status
Bold indicates that the p-value is significant (i.e. p < 0.05).
a Diagnosis made at the time of the index visit and reported by the PCP on case record forms.
b ‘Over the last 2 weeks, how often have you been bothered by’; Response option = 0 to 3 (0 = Not present; 1 = Infrequent; 2 = Frequent; 3 = Very frequent).
c Response option ranges from 0 to 4 (0 = Not applicable; 1 = No difficulty; 2 = Some difficulty; 3 = Difficult; 4 = Very difficult).
*Significant differences (p < 0.05) between groups by independent t test or by χ2 as appropriate.
Note: Missing value categories are omitted.
Table 2 shows the results of the logistic regression analysis of the patient characteristics associated with a PCP diagnosis of depression. History of mental illness had the largest effect size with an OR of 10.97. Significant PHQ-9 items included depressed mood (OR 1.33), sleep disturbance (OR 1.17), appetite change (OR 1.15), worthlessness (OR 1.26) and functional impairment (OR 1.35). The total PHQ-9 score was significant but had a small effect size (OR 1.11). Other significant factors included female gender (OR 1.47), increasing age (OR 1.02), employment (OR 0.7), family history of mental illness (OR 1.69), doctor visit in the past month (OR 1.1). Of note, anhedonia and suicidal ideation were not significantly associated with a depression diagnosis. Variance inflation factors ranged from 1.03 to 2.18 indicating no multicollinearity. Hosmer and Lemeshow's test demonstrated the model fitted the data well with a p value of 1.00.
Factors associated with doctor diagnosis of depression by non-linear mixed effect model
Bold indicates that the p-value is significant (i.e. p < 0.05).
Hosmer and Lemeshow Tests suggested adequate model fit using PHQ-9 symptom items: χ2 = 147.8, p value = 1.00; Incorporating PHQ-9 total score: χ2 = 176.4, p value = 0.98.
a Diagnosis made at the time of the index visit and reported by the PCP on case record forms.
b ‘Over the last 2 weeks, how often have you been bothered by’.
*Statistically significant at p value <0.05 by linear mixed effects model.
Note: Doctor Intraclass Correlation Coefficient (ICC) in the model using PHQ-9 symptom items = 18.7%; in the model incorporating PHQ-9 total score = 18.6%.
Figure 1 shows the PHQ-9 symptom tree model. The root of the tree was formed by past history of depression (yes/no), with the frequency of depressed mood forming the parent branch. In patients with past depression, subsequent branching occurred with the presence of fatigue and feelings of worthlessness. In patients with no past depression, subsequent branching occurred with a frequency of sleep disturbance and severity of functional impairment. There were a total of 17 terminal nodes/groups with diagnosis rates ranging from 0.5% to 55%. For example, in patients with past depression and frequent to very frequent depressed mood and fatigue present, 55% were diagnosed as having depression. In patients with no past depression, frequent depressed mood and moderate to severe functional impairment, 27.3% were diagnosed as having depression.
Decision tree for depression diagnosis by doctors modeling PHQ-9 symptoms.
Figure 2 shows the PHQ-9 total score tree model. Once again, past history of the depression formed the root of the tree with the PHQ-9 total score forming the parent branch. In patients with a past history of depression, branch splitting occurred at a PHQ-9 total score of >12. In patients with no past history of depression, a four-way split was observed at PHQ-9 total scores of ⩽5, 6–9, 10–15 and >15. There were 12 terminal nodes with depression diagnosis rates ranging from 0.4 to 70%. For example, in patients with past depression, 61.2% of subjects with PHQ-9 total score >12 were diagnosed with depression. In patients with no past depression, 26.8% of patients with PHQ-9 total scores >15 were diagnosed with depression.
Decision tree for depression diagnosis by doctors modeling PHQ-9 total score.
Factors identified in the logistic regression were slightly different to the tree classifiers. This was expected as interaction effects between factors are fully taken into account in the decision tree analysis, whereas no interaction effect is assumed in the logistic regression.
Discussion
This is the first study to use an inference tree analysis to model the factors associated with depression diagnosis in primary care. Our models demonstrated the key criteria associated with PCP diagnosis of depression (including past history of depression, depressive symptoms and symptom severity), in the form of a tree diagram, illustrating how the variables interact. Our findings provide new insights into the complex processes involved in diagnosing depression in primary care. Although the study did not provide any evidence regarding the accuracy of PCP diagnosis, it revealed information on real-world clinician behaviors related to diagnosing depression.
From the waiting room questionnaire, only 23.2% of subjects with PHQ-9 total score >9 received a diagnosis of depression, while in those who received a diagnosis of depression, only 41.2% had PHQ-9 scores >9. This suggests that PCP diagnosis of depression in our setting has both low sensitivity and poor predictive value (if using the PHQ-9 scores with a cut-off of >9 as the diagnostic gold standard), which may have repercussions on the adequacy of depression care.
It was observed from the logistic regression that a past history of depression or other mental illness was strongly associated with a diagnosis of depression, whilst all other significant factors including depressive symptoms and depression severity had much smaller effect sizes with OR <2.0. The tree models further demonstrated this with a history of depression emerging as the parent branch.
The complexities associated with depression diagnoses in primary care were demonstrated in the PHQ-9 total score tree analysis where it was observed that many patients who had received a PCP diagnosis of depression had PHQ-9 total scores <9 with some diagnoses made at quite low PHQ-9 scores ⩽5. It is possible that these are cases of diagnostic inaccuracy, or alternately, they may be cases of remitted depression.
Our interpretation of the tree analysis models is that PCPs appear to use a hypothetical-deductive problem-solving approach incorporating pre-test probability with different diagnostic pathways for patients with and without past depression. The tree analyses found that in patients with past depression, fewer PHQ-9 items were associated with PCP diagnosis with diagnoses occurring at lower PHQ-9 symptom severities. This seems to indicate that PCPs have a higher pre-test probability and lower thresholds for diagnosing depression in these patients. This may mean that PCPs are either more willing or confident to diagnose patients with known past depression or that patients with a past history of depression are much more willing to disclose their symptoms to the PCP, making it more likely for a positive diagnosis to occur. One potential risk is that PCPs may prematurely converge on a diagnosis of depression without performing sufficient further assessment and possible misdiagnoses. Future studies to assess diagnostic accuracy using decision tree analyses could help to clarify if this is happening.
In contrast, for patients without a history of depression, the tree models had more branches suggesting that PCPs may take more factors into consideration before making a judgment on the diagnosis. This illustrates how diagnosing new cases of depression is a more complex clinical process. It was observed that in patients with no previous known depression, diagnoses were made at higher symptom severity levels and using more criteria. This seems to suggest that PCPs exercise greater caution when diagnosing new cases. As mental illness remains highly stigmatizing in Asian cultures, one possibility is that PCPs in our setting are wary about excessively burdening patients experiencing distress and may only diagnose depression when they feel active treatment is absolutely necessary. Alternatively, our patients may be reluctant to disclose the extent of their depressive symptoms, requiring the doctor to ‘fish’ for symptoms before coming to a decision about whether or not the patient has depression. One potential risk for patients with no history of depression is that if diagnostic thresholds are too high, many patients who may benefit from treatment may be missed.
In keeping with previous studies, a range of psychological, somatic, and other symptoms was found to be associated with the diagnosis of depression (Malhi et al. Reference Malhi2014), however, the model found that physical symptoms (sleep, fatigue) and functional impairment occurred higher in the tree. The PHQ-9 symptom model demonstrated a hierarchy of symptoms associated with the diagnosis of depression, with low mood, sense of worthlessness, fatigue, sleep disturbance, and functional impairment branching earlier. Assuming that such symptoms were elicited during the consultation, this suggests that these criteria may be of greater diagnostic value to PCPs. One possibility is that some depressive symptoms such as appetite change or poor concentration may be of limited value because of their relatively low specificity in generalist settings, where clinicians often prefer to rule out physical disease first (Dew et al. Reference Dew2005). This finding was supported by a British study comparing PCPs to psychiatrists which found that PCPs tend to use a more limited number of constructs to establish psychological diagnoses, more commonly basing their judgment on low mood, sleep disturbance and functional impairment (Armstrong & Earnshaw, Reference Armstrong and Earnshaw2004). The findings of our model were also supported by earlier studies that found physician sensitivity to depression is higher when the functional impairment is present and patient presentations with sleep disturbance promote depression assessment and diagnosis in primary care (Ani et al. Reference Ani2008).
Two key diagnostic criteria for depression were notably absent from our PHQ-9 symptom model namely anhedonia and suicidal ideation. Although anhedonia is traditionally considered a cardinal diagnostic feature of depression, it appears that PCPs in our setting either do not elicit this symptom or find it less useful for diagnosing depression. In a fast-paced and workaholic environment such as Hong Kong, loss of motivation or loss of pleasure from activities may well be masked or considered less significant. Doing things even when they are not pleasurable is often considered a positive attribute in Chinese culture (Shek et al. Reference Shek2003).
Whilst many studies have found high concordance rates between the presence of suicidality and depression recognition, this was not observed in our study sample (Ani et al. Reference Ani2008; Henriques et al. Reference Henriques2009). One potential reason may be that the spectrum of severity in our primary care sample was relatively mild, and there may have been too few patients with significant suicidal symptoms for the model to identify this criterion as a classifier. Another possibility is that our study doctors may not be sufficiently sensitive to their patients’ suicidal communications or did not encourage them to talk about it (Riihimäki et al. Reference Riihimäki2014; Younes et al. Reference Younes2015), raising potential concerns regarding the quality of suicide prevention in this setting. An Australian study found that PCPs are more likely to use suicidality as a measure of depression severity (rather than for diagnosing depression), and may not enquire about suicidal thoughts unless they perceive the patient is severely depressed (Malhi et al. Reference Malhi2014). Thirdly, it is possible that many people in our setting will have suicidal ideas but do not appear depressed. Although suicidal ideation is most often associated with depression, other social circumstances such as marital discord, financial problems or sense of hopelessness may also be associated with suicidal thoughts (Cheung et al. Reference Cheung2006). Nevertheless, what our findings show is that patients with suicidal thoughts was not an associated factor with PCP diagnosis of depression, which may have potential implications on whether the quality of care regarding suicide detection is adequate in our setting.
Limitations
This study has several limitations. Firstly, data on patient characteristics including PHQ-9 symptoms of depression were all self-reported, and are susceptible to self-report bias. It is not known how many symptoms the doctor elicited during the consultation and it is entirely possible that some of the patient's self-reported symptoms did not get conveyed. It must be stressed that the findings of our models only represent the statistical relationship between the presence of symptoms (as reported by the patient) and PCP diagnosis of depression, and these have been used to simulate the diagnostic pathways. A further study where the symptoms elicited by the doctor are collected independently would help to validate our findings.
Secondly, we only examined the factors associated with PCP diagnosis of depression without considering diagnostic accuracy and there was no gold standard assessment to validate the doctor's clinical diagnosis. The scope of this current study was limited to exploring PCP behavior patterns in relation to diagnosing depression in a real-world setting. Further studies where the PCP's performance is benchmarked against a gold standard are needed to compare the differences in pathways leading to accurate and inaccurate recognition to determine whether clinical assessments are adequately performed, particularly in those with past depression.
Third, our models were based on the diagnostic behaviors of 59 PCPs on approximately 10 000 predominantly Chinese primary care patients in Hong Kong and may not be representative of depression diagnosed in other settings. Differences in age, gender, education, culture, and health systems of both patients and doctors will affect diagnostic behavior. Our findings provide valuable information on the direction of effects, the relative importance of classifiers and the utility of the decision tree methodology, however, further studies in other settings and using other instruments are needed to help build further knowledge in this field and confirm, amend or refute our findings. This study needs to be replicated in other settings for cross-cultural comparison and to examine how diagnostic behaviors differ among other types of mental health providers.
Conclusions
We found that decision tree analysis is an informative method to explore the factors associated with a clinical diagnosis of depression. Our findings provide new insights regarding the associations between PHQ-9 symptoms, symptom severity and depression diagnoses. Data on these associations were used to model how primary care doctors diagnose depression and the potential factors influencing their decisions. Our tree diagrams illustrated how primary care doctors use a hypothetical-deductive problem-solving approach incorporating pre-test probability with different criteria and diagnostic pathways for patients with and without past depression. The models suggested that there is a hierarchical structure to the PHQ-9 criteria, demonstrating that some depressive symptoms may have greater diagnostic value in primary care. We found that many patients experiencing suicidal ideation are not diagnosed as having depression, which raises concerns regarding the quality of suicide detection and prevention in our setting. Our methods could be replicated in other settings or on other data to compare if different types of mental health providers diagnose depression differently to build further knowledge on how depression is diagnosed. Future studies using decision tree analyses to compare the decision-making pathways between accurate and inaccurate diagnoses are needed to inform better quality of care.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291718001058.
Acknowledgements
The authors wish to acknowledge all the doctors of our Primary Care Research Network for their contribution to this study and Ms. Kit Chan for her project coordination. This study was supported by the Hong Kong Food and Health Bureau's Commissioned Research on Mental Health Policy and Service Grant Reference No. SMH-27. The funding body played no role in the study design; data collection, data analysis, data interpretation; or in the writing of the manuscript. The authors are completely independent of the funding source.
Declaration of interest
All authors declare no conflicts of interest.
