Introduction
The United States’ population is aging and is placing the geriatric mental health services delivery system under considerable strain (Bartels and Naslund, Reference Bartels and Naslund2013). According to the Institute of Medicine, the “burden of mental illness and substance use disorders in older adults in the United States borders on a crisis” (Institute of Medicine, 2012, pp. ix). Among those aged 65 years and older, 14%–20% have a mental health or substance use condition, and older adult men have the highest suicide rates (Institute of Medicine, 2012). In addition to the considerable morbidity associated with late-life mental illness, the cost to society can be substantial, with late-life depression being associated with a 47%–51% increase in medical costs in ambulatory and inpatient settings (Katon et al., Reference Katon, Lin, Russo and Unutzer2003). There is a pressing need for innovative approaches that can be implemented broadly and cost-effectively to address late-life mental healthcare needs. Many innovations are occurring in telehealth, which involve the use of telephones, smartphones, and mobile wireless devices helping millions of patients manage acute and chronic conditions while aiming to increase access to and reduce the cost of care (Dorsey and Topol, Reference Dorsey and Topol2016).
One approach is the Behavioral Health Laboratory (BHL), which is a cost-effective approach that can improve mental illness identification and management (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006). The BHL was developed for use in the Department of Veterans Affairs (VA) primary care clinics and often uses structured telephone assessments to facilitate care decisions and to reduce the burden of in-person visits. When a primary care physician (PCP) has mental health concerns about a patient, the PCP can request a BHL assessment. BHL technicians subsequently conduct a telephone (or in-person)-based interview and provide the PCP a report summarizing the test results with recommendations for clinical care. The BHL assesses several domains: demographics, cognition (for those 55 and older), mania, psychosis, panic disorder, generalized anxiety disorder (GAD), post-traumatic stress disorder (PTSD), alcohol misuse, depression, suicidal ideation, and antidepressant use.
As described by Oslin and colleagues (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006), the BHL is cost effective and increases identification of alcohol misuse, drug misuse, suicidal ideation, and depression. While telephone assessments can be an efficient use of resources, effective for tracking mental health symptoms longitudinally, and convenient for older adults (Senior et al., Reference Senior, Kunik, Rhoades, Novy, Wilson and Stanley2007; Claassen et al., Reference Claassen2009), the instruments used in the BHL, including the MINI Neuropsychatric Interview (Sheehan et al., Reference Sheehan1998), the Patient Health Questionnaire-9 (PHQ-9) (Kroenke et al., Reference Kroenke, Spitzer and Williams2001), the Blessed Orientation-Memory-Concentration Test (Katzman et al., Reference Katzman, Brown, Fuld, Peck, Schechter and Schimmel1983), and the Paykel Scale for suicide ideation (Paykel et al., Reference Paykel, Myers, Lindenthal and Tanner1974), were validated in-person. Their reliability and validity in the BHL telephone-based assessment, to our knowledge, have yet to be confirmed. Some data suggest that telephone administration of the PHQ-9 yields comparable scores to in-person administration (Pinto-Meza et al., Reference Pinto-Meza, Serrano-Blanco, Penarrubia, Blanco and Haro2005; Ryan et al., Reference Ryan, Bailey, Fearon and King2013), but neither study was conducted in an older adult population. Differences between telephone and in-person administration may exist, with one study reporting that individuals completing the GAD-7 via telephone endorsed more anxiety than those completing it in-person (Ryan et al., Reference Ryan, Bailey, Fearon and King2013). Furthermore, given that the BHL elicits sensitive information regarding alcohol and drug use as well as suicidal ideation, the increased rapport possible with a face-to-face interview may facilitate elicitation of data on these sensitive topics. Alternatively, telephone administration may reduce factors such as social desirability (Senior et al., Reference Senior, Kunik, Rhoades, Novy, Wilson and Stanley2007) thus facilitating more valid responses. There also may be differences in assessing cognitive impairment between telephone and in-person modalities as the in-person interview offers an opportunity for a more comprehensive assessment (e.g. can observe an individual's presentation and environment).
A key limitation of the BHL thus far is that its telephone assessments have yet to be validated with in-person assessments for which the individual survey instruments were originally designed. To test the BHL's procedural validity, we therefore conducted a cross-sectional study to examine the concordance between BHL data gathered by telephone and in-person. We conducted this study in the Aging Services Network (ASN). The ASN is a national system of ~30,000 local and state agencies that annually provide human services to over 10 million older adults and their caregivers to help maintain the independence of older adults (services range from providing information and referral to intensive care management) (O'Shaughnessy, Reference O'Shaughnessy2008). Consistent with previous research on the concordance of telephone and face-to-face interviews for psychiatric syndromes and symptoms (Fenig et al., Reference Fenig, Levav, Kohn and Yelin1993; Lee et al., Reference Lee, Tsang, Mak, Lee, Lau and Ng2010), we hypothesize that there will be strong evidence of equivalence between telephone and in-person interview data for the BHL. Strong evidence of equivalence would suggest that endorsement of psychiatric symptoms does not differ when evaluated via telephone and in-person and would help confirm the BHL's procedural validity.
Methods
Aging services clients aged 60 years and older in Monroe County, NY were referred to the study if case managers (CMs) believed the clients to be “depressed, anxious, or confused” during routine interactions. The CMs, at the time of client intake, assessed clients for potential study eligibility with the sole inclusion criterion at the intake interview being whether the client appeared depressed, anxious, or confused. From 4/1/2009 to 9/15/2010, these clients were then referred to research staff who called the clients. Clients were given a brief description of the study and research staff scheduled in-person meetings for those interested in participating. During the in-person meeting, the study was described in further detail and informed consent was obtained. Individuals were excluded if they were younger than 60 years, unable to communicate in English, or did not have capacity to provide informed consent. Subjects were randomly assigned to an in-person interview conducted in the subject's home (n = 55) or to a telephone interview (n = 53). The randomization procedure used block randomization and stratified by gender, interviewer, and age. Participants had an average age of 74.2 years, 72.2% were women, 85.2% were white, 20.4% were currently married, and 58.3% lived alone (Table 1).
Sociodemographic characteristics of the aging services clients who participated in the in-person or phone interview
a Two-sided p values determined by t-test if variable is continuous with degrees of freedom equal to 106 or χ 2 test if variable is categorical with degrees of freedom equal to one.
Trained research personnel performed interviews in a manner consistent with the procedures employed by previous BHL studies (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006). Following the BHL assessment, a standard letter summarizing the combined results of the BHL and CM assessments was sent to the subjects, their primary care providers, and CMs. University of Rochester's Research Subjects Review Board approved this study.
Measures
All measures were administered in a uniform manner on the phone and in-person (e.g. using the same language and research personnel). Scores on a battery of assessments were then compared for agreement using the Food and Drug Administration standard for equivalence, 80% to 125% (Food and Drug Administration, 2015).
Sociodemographics
Information on age, gender, race, marital status, and living arrangement (e.g. living alone) was collected.
Mental health
We used instruments specified by the BHL (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006). The Mini-International Neuropsychiatric Interview (i.e. MINI) is a short structured diagnostic interview for DSM-IV and ICD-10 psychiatric disorders (Sheehan et al., Reference Sheehan1998). Our study included MINI modules that assessed mania, psychosis, substance misuse, PTSD, GAD, and panic disorder. The MINI has been shown to have good reliability and validity when compared to the Structured Clinical Interview for DSM and is less burdensome (Sheehan et al., Reference Sheehan1998). To assess overall mental health, we relied on the SF-12 which is a 12-item instrument that includes a mental health subscale, has demonstrated test–rest validity, and correlates strongly with the longer 36-item survey, the SF-36 (Ware et al., Reference Ware, Kosinski and Keller1996). Depression was assessed with the PHQ-9, a nine-item instrument that measures depression severity and has a sensitivity and specificity of 88% for detecting major depression (Kroenke et al., Reference Kroenke, Spitzer and Williams2001); we used the pattern of responses on the PHQ-9 to determine if major or minor depression DSM-IV criteria were met. Cognitive impairment was evaluated with the Blessed Orientation-Memory-Concentration Test, a six-item test that includes one memory task, two calculation tasks, and three orientation questions and has adequate sensitivity and specificity for detecting cognitive impairment (Woodford and George, Reference Woodford and George2007). The Paykel five-item scale was used to assess severity of suicidal ideation (Paykel et al., Reference Paykel, Myers, Lindenthal and Tanner1974). All participants were also asked, “During the past month, did you take any medications, prescribed or not, for depression, anxiety or nerves?” For participants endorsing medication use, their medications were reported and subsequently grouped by antidepressant or anxiolytic medication classifications.
Statistical analysis
Differences for continuous and categorical variables in sociodemographic characteristics of the aging services clients who participated in the in-person or phone interview were evaluated with t-tests and χ 2 tests, respectively. Equivalence of assessments for continuous outcomes were evaluated employing two one-sided t-tests (TOSTs) where p values were based on an assumption of a lognormal distribution of the ratios of geometric means, using pooled error terms, with 95% confidence limits established about these ratios. These limits were then compared with the FDA standard for equivalence (0.80 to 1.25) (Food and Drug Administration, 2015) to determine if the confidence interval associated with the telephone-to-in-person ratio of geometric means fell within this interval. p values < 0.05 were deemed evidence of equivalence. Categorical data utilized TOST χ 2 tests with 95% equivalence limits and margins of ± 0.2. For the depressive disorders category, the equivalence of the trend between methods of administration was determined by examining Cochran–Armitage Trend Test (a p value of 1.000 indicates strong equivalence). To correct for inflated type I error rates due to multiple comparisons or testing, the false discovery rate (FDR) was used to ensure a correct overall type I error. FDR overcomes the limitation of classic methods such as the Bonferroni correction, which are too conservative for adjusting the type I error for a large number of comparisons or testing as in this study (Benjamini and Hochberg, Reference Benjamini and Hochberg1995). Thus, for BHL performance in Table 2, we also report FDR results. All analyses were carried out using SAS/STAT software, Version 9.4 of the SAS System (Copyright © 2002- 2012, SAS Institute Inc.) on a Windows 7 platform.
Comparison of the BHL assessment by in-person and telephone interview modalities
a p values were based on ratios determined with geometric means. To evaluate equivalence between groups, these ratios were used to construct a 95% confidence interval, which was then compared with the FDA standard for equivalence (0.80 to 1.25) to determine if the confidence interval associated with the telephone-to-in-person ratio of geometric means fell within this interval.
b These p values were determined by using the false discovery rate (FDR) to correct for inflated type I error rates due to multiple comparisons or testing.
c For the depressive disorders category, the equivalence of the trend between methods of administration was determined by examining the difference between groups than equivalence (a p value of 1.000 indicates strong equivalence).
Some variables were missing information for the p value calculation: d 55, 52; f 54, 53.
e Data were exponentiated prior to analyses.
Results
Sociodemographics
Age, gender, race, marital status, and living arrangement did not differ between those interviewed in-person and by telephone (Table 1).
Mental health
Key components of the interview were analyzed for equivalence and results appear in Table 2. With regard to the prevalence of psychiatric syndromes, major depression was highly prevalent (about one in five participants). There was strong equivalence between the in-person and telephone interviews for major and minor depression as well as for diagnoses of GAD, PTSD, and panic disorder and the presence of substance misuse and psychosis (as evaluated by the MINI).
The SF-12 mental health subscale demonstrated strong equivalence with the in-person interviewees having an average score of 44.5, while those interviewed on the telephone had an average score of 43.5. The PHQ-9 mean scores also showed relatively high levels of depression (mean scores were just under the moderate severity threshold) and indicated marginal equivalence between the in-person and telephone interviews. The assessment of cognitive impairment showed higher scores in the telephone interviews (5.2) compared to the in-person (3.1) interviews, which were not equivalent.
With regard to psychotropic medications, there was marginal equivalence across interview methods for antidepressants with 38.2% of the in-person and 34.0% of the telephone interviewees reporting antidepressants use. Anxiolytic medication use, however, was not equivalent across interview methods (in-person: 14.6%; telephone: 26.4%).
Suicidal ideation, as assessed by the Paykel Scale, showed strong equivalence in four of the five categories of past year suicidality. The exception was the death ideation question (i.e. subjects wished they were dead in the past year), which had marginal equivalence and was endorsed by 32.7% of the in-person and 24.5% of the telephone interviewee. Additionally, 13.0% of the in-person and 13.2% of the telephone interviewees endorsed a lifetime history of a suicide attempt, indicating strong equivalence across assessment modalities.
Discussion
The current study, as an examination of BHL's procedural validity, suggests that dissemination and integration of the BHL into a variety of settings (including the ASN) may represent an efficient approach for strengthening linkages between primary care, mental healthcare, and social service providers. For example, having this service available to ASN providers could meaningfully enhance their ability to screen for, assess, and potentially triage mental health issues. Congruent with our hypothesis, with a few exceptions (marginal equivalence in PHQ-9 total scores and one of the six questions assessing suicidal ideation, and no evidence of equivalence between interview modalities for assessing cognitive impairment), the analyses indicate that assessment of key mental health domains such as psychiatric syndromes, overall mental health, and suicidality is largely equivalent whether evaluated by the BHL in-person or on the telephone. Furthermore, these data can be readily summarized in a report to PCPs and are likely to be highly practical in determining the need further referral, examination, and/or treatment and represent another approach to integration of primary and mental healthcare.
Marginal evidence for equivalence was found for the PHQ-9, antidepressant use, and death ideation item. The reasons for the lack of strong equivalence are unclear. Possibilities include the increased ability to seek clarification and inclusion of non-verbal cues in face-to-face interviews (Irvine et al., Reference Irvine, Drew and Sainsbury2013), which may be relevant to death ideation, a concept that can be interpreted in different ways (Van Orden et al., Reference Van Orden, Simning, Conwell, Skoog and Waern2013). With regard to the PHQ-9, the small difference between means (in-person: 9.8; telephone: 9.6), while enough to result in marginal statistical equivalence, does not represent a difference that is clinically meaningful.
For the Blessed memory test and anxiolytic use there was no evidence of equivalence. A consideration for the Blessed memory test is that functional issues such as hearing loss may have been magnified by the telephone (telephone interviewees had worse scores) and contributed to the discrepancy between the interview modalities. For the prescribed anxiolytic drugs, a category of medications that is often abused and/or diverted (McLarnon et al., Reference Mclarnon, Monaghan, Stewart and Barrett2011), it is possible that there was decreased reporting in-person as people may be less likely to disclose sensitive information in face-to-face interviews (Reddy et al., Reference Reddy, Fleming, Howells, Rabenhorst, Casselman and Rosenbaum2006).
As with any telephone-based approach, the differences in face-to-face interaction (and assessment) and telephone assessment confer limitations upon the BHL. Most prominently, opportunities for gathering data are more limited during telephone-based assessments (e.g. fewer cues to probe for more information, no ability to use visual cues). Additionally, especially in older adults, telephone interviews may be affected by communication difficulties related to hearing loss and/or cognitive decline. Nonetheless, others have found that psychiatric illness can be equivalently evaluated by phone versus in-person (Rohde et al., Reference Rohde, Lewinsohn and Seeley1997; Lee et al., Reference Lee, Tsang, Mak, Lee, Lau and Ng2010) or internet assessments (Hedman et al., Reference Hedman2013).
In primary care VA settings, the BHL has been shown to be a relatively low-cost approach to assess, monitor, and plan treatment for those with mental health needs (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006). Likewise, the BHL has considerable potential for being a cost-effective method to foster mental health services integration in a variety of settings. Given that most mental health screening is conducted by a physician or other treating provider, using the BHL in settings such as ASN, home health, rural/underserved areas, and non-VA outpatient clinics could increase identification, referral, and, possibly, indicated treatment of older adults with mental illness and suicidal ideation. To highlight the BHL's potential, we will use the ASN as a case example. Older adults at risk for suicide are unlikely to seek out help from traditional venues (Conwell and Thompson, Reference Conwell and Thompson2008), and many older adults who seek ASN services experience thoughts of death and suicide (O'Riley et al., Reference O'Riley, Van Orden, He, Richardson, Podgorski and Conwell2014), in addition to having high levels of anxiety and depression (Simning et al., Reference Simning, Richardson, Friedman, Boyle, Podgorski and Conwell2010). This is especially pertinent since factors that may cause an older adult to seek help from a community-based provider such as an ASN agency (e.g. functional impairment, social isolation) are associated with an increased risk of completed suicide (Conwell et al., Reference Conwell2000; Fassberg et al., Reference Fassberg2012). Yet few ASN clients have seen a mental health specialist prior to the initial ASN intake (Simning et al., Reference Simning, Richardson, Friedman, Boyle, Podgorski and Conwell2010). The ASN therefore represents a promising point of contact to bridge the aging services, primary care providers, and mental healthcare systems.
Our study has several limitations that should be considered when evaluating the validity of the BHL. First, due to the sample size, we did not assess how the BHL performed in different sociodemographic subgroups. Second, this study occurred in one ASN agency and only included older adults fluent in English who had sufficient cognitive capacity to provide informed consent, which imposes some limitations on its generalizability. For instance, there are possible differences in access to healthcare or community resources that may affect the operationalization and utility of this model in other regions. Third, we did not exclude participants based on their Blessed score (as was done in the initial testing of the BHL (Oslin et al., Reference Oslin, Ross, Sayers, Murphy, Kane and Katz2006)); not excluding these participants is likely to have a minimal impact, however, as our study only included an additional four participants that would have been excluded by the original BHL protocol. Fourth, the BHL was conducted by research staff highly trained in administering mental health measures, whereas BHL technicians who would carry out a “real world” assessment may require additional training to ensure the validity of information gathered via telephone. Fifth, we did not perform formal inter-rater reliability assessments of our two research coordinators, both of whom conducted in-person and telephone-based assessments, which could affect our findings and subsequent interpretations. Lastly, this study was designed to compare interview modalities, and prevalence estimates should be interpreted cautiously because their generalizability is unclear as the aging service's care managers did not record which clients were asked to participate in our study.
Future research is needed to further elucidate the utility and limitations of the BHL and its potential for use in various settings. Because it represents a cost-effective model for mental health issue recognition, referral, decision support, and ongoing monitoring, further research should focus on its effectiveness in different populations and associated outcomes (e.g. reducing hospital readmissions and use of acute services, facilitating treatment adherence) as well as its possible application in the rapidly developing field of telehealth (e.g. increasing access to mental health services in areas that are otherwise limited by scarce resources).
Conclusions and implications
In many countries the population is aging rapidly and innovative models addressing mental health in community-residing older adults is an important component to the improvement of public health. Building upon prior research, our study's findings show that the BHL is able to gather nearly equivalent information via telephone as compared to in-person interviews (although some caution should be exercised with possible over-reporting of cognitive impairment and under-reporting of wishing for one's own death when assessed by telephone). These findings suggest that the BHL may be a cost-effective, innovative model appropriate for dissemination to more fully integrate mental health services in a wide variety of settings. Additional research is warranted, however. In particular, examining the cost-effectiveness of disseminating the BHL into other settings as well as its validity and reliability for tracking changes in mental health outcomes over time would be of high utility. Dissemination of the BHL could help prevent older adults with mental illness who may, because of barriers, stigma, and immobility, otherwise “fall through the cracks” in our current systems of care.
Conflicts of interest
None.
Description of authors’ roles
Y. Conwell designed the study, supervised data collection, and helped write the article. A. Simning, N. Driffill, and D. Oslin assisted with the research questions, interpreting the results, and writing the article. Y. Xia, X. Tu, and S.P. Messing assisted with the statistical design of the study and its data analyses.
Acknowledgments
This work was supported in part by National Institute of Mental Health (R24 MH071604) and the VISN 4 Mental Illness Research Education and Clinical Center. Dr. Simning is supported through the Empire Clinical Research Investigator Program, sponsored by the New York State Department of Health.
