The course of psychiatric disorders is complex and heterogeneous (Marquand, Wolfers, Mennes, Buitelaar, & Beckmann, Reference Marquand, Wolfers, Mennes, Buitelaar and Beckmann2016). Clinicians and stakeholders have a pronounced desire to anticipate clinical progression to target intervention selection and intensity appropriately (McMahon, Reference McMahon2014). Therefore, statistical attempts have been made to identify shared predictors of illness outcome, most frequently utilizing regression-based inferential statistics. Yet, the results of these investigations have yielded unreliable or insensitive predictors (Franklin et al., Reference Franklin, Ribeiro, Fox, Bentley, Kleiman, Huang and Nock2017; Suvisaari et al., Reference Suvisaari, Mantere, Keinänen, Mäntylä, Rikandi, Lindgren and Raij2018).
A number of issues associated with conventional statistics limit their utility in predicting complex psychiatric phenomena. Traditional regression-related approaches can only accommodate a restricted number of predictors or else encounter low power, multicollinearity, and poor interpretability (Burke, Ammerman, & Jacobucci, Reference Burke, Ammerman and Jacobucci2019). Additionally, such statistics often rely upon the assumption of linearity, which may not capture the relational patterns between varied predictors and outcomes of interest (Jordan & Mitchell, Reference Jordan and Mitchell2015). These issues may conspire to produce artificially simplistic prediction models inappropriate for describing most psychiatric disorders (Linthicum, Schafer, & Ribeiro, Reference Linthicum, Schafer and Ribeiro2019). An additional risk of traditional analytic techniques is over-fitting the data to the measured sample (Jordan & Mitchell, Reference Jordan and Mitchell2015), which could account for the poor between-study correspondence in predictors (Vall & Wade, Reference Vall and Wade2015).
Emerging computational approaches utilizing machine-learning (ML) methods hold promise for optimizing prediction of psychiatric illness course. ML encompasses a collection of data-driven techniques that permit computer algorithms to identify and iteratively refine the optimal parameters to fit complicated patterns between variables (Bzdok & Meyer-Lindenberg, Reference Bzdok and Meyer-Lindenberg2018; Linthicum et al., Reference Linthicum, Schafer and Ribeiro2019). These approaches are well suited to overcome the historical limitations of predictive analytics. ML approaches accommodate intricate inter-dependent relations between a large number of variables without formal assumptions regarding the importance or structure of the data, allowing more rigorous predictive models. Emerging data demonstrate the advantage of ML over traditional methods in predicting the course of serious mental illnesses, such as major depressive disorder (Kessler et al., Reference Kessler, van Loo, Wardenaar, Bossarte, Brenner, Cai and Zaslavsky2016), obsessive–compulsive disorder (Askland et al., Reference Askland, Garnaat, Sibrava, Boisseau, Strong, Mancebo and Eisen2015), and psychotic disorders (Koutsouleris et al., Reference Koutsouleris, Kahn, Chekroud, Leucht, Falkai and Wobrock2016); however, this field is nascent and outcomes from many mental health concerns have not been examined using these novel methods.
Eating disorders (EDs), including anorexia nervosa (AN), bulimia nervosa (BN), and binge-eating disorder (BED), are among the psychiatric disorders most in need of the improved prediction derived through ML techniques. These illnesses are associated with significant medical and psychological morbidity (Marucci et al., Reference Marucci, Ragione, De Iaco, Mococci, Vicini, Guastamacchia and Triggiani2018; Mitchell & Crow, Reference Mitchell and Crow2006) and among the highest mortality rates of any psychiatric disorder (Crow et al., Reference Crow, Peterson, Swanson, Raymond, Specker, Eckert and Mitchell2009). There is considerable unexplained heterogeneity in the lifetime course of EDs. Naturalistic estimates suggest that 50–70% of adults with an ED achieve remission, whereas 30–33% improve, but remain symptomatic, and 20–25% maintain chronic illness or die prematurely (Eddy et al., Reference Eddy, Tabri, Thomas, Murray, Keshaviah, Hastings and Franko2017; Steinhausen, Reference Steinhausen2002; Steinhausen & Weber, Reference Steinhausen and Weber2009). Conventional regression-based approaches have yielded a wide range of predictors of ED outcomes, including ED diagnosis, symptoms, and behaviors, body mass index (BMI), and comorbid psychiatric symptoms (Carter et al., Reference Carter, Mercer-Lynn, Norwood, Bewell-Weiss, Crosby, Woodside and Olmsted2012; Franko et al., Reference Franko, Tabri, Keshaviah, Murray, Herzog and Thomas2018; Lock et al., Reference Lock, Agras, Le Grange, Couturier, Safer and Bryson2013). However, these individual risk factors account for very little variance in clinical outcomes and fail to replicate across samples (Vall & Wade, Reference Vall and Wade2015). Given the potential severity of the consequences associated with EDs, there is a critical need to improve prediction in these populations.
Therefore, we tested the ability of ML models to enhance the prediction of clinical outcomes among individuals with EDs. Specifically, we used elastic net regularized regression models to longitudinally predict ED diagnoses and key ED features (e.g. binge eating, compensatory behaviors, and underweight BMI) at 1- and 2-year follow-ups from baseline predictors, including demographic (e.g. age, ethnicity), clinical (e.g. ED behaviors, depressive symptoms), and treatment (e.g. prior hospitalization) variables among a large, transdiagnostic ED sample. We compared predictive accuracy between this ML method and a traditional inferential statistics approach (logistic regression). As further tests of model robustness, we examined whether findings replicated when the most important predictor was removed, or a different ML method (random forests) was applied. We hypothesized that the elastic net models would improve the prediction of long-term ED outcomes compared to logistic regression models, and that these findings would replicate even with the above described tests of model strength. We also conducted an exploratory investigation into the strongest predictor variables in each ML model to inform future investigations. This study constitutes the first to examine an ML approach to improve the prediction of long-term ED outcomes. The results can provide insight into the utility of these innovative computational approaches for advancing research on EDs and other severe psychiatric illnesses in the ultimate direction of data-derived personalized medicine.
Methods
Participants and procedure
Data were used from a longitudinal study conducted across sites in California, New York, and Minnesota (Agras, Crow, Mitchell, Halmi, & Bryson, Reference Agras, Crow, Mitchell, Halmi and Bryson2009; Crow, Agras, Halmi, Mitchell, & Kraemer, Reference Crow, Agras, Halmi, Mitchell and Kraemer2002). Participants were recruited from ED clinics and research studies and local media. Inclusion criteria included being female, between 14 and 50 years old, and meeting full- or sub-threshold diagnoses of AN, BN, or BED according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV; APA, 2000). Males were excluded due to the elevated ED prevalence in females (Smink, van Hoeken, & Hoek, Reference Smink, van Hoeken and Hoek2012). Restrictions were not placed on whether participants could be enrolled in treatment. Full study details are described elsewhere (Crow et al., Reference Crow, Agras, Halmi, Mitchell and Kraemer2002).
Eligible participants completed interviews and self-report questionnaires assessing demographics, ED symptoms, and comorbid psychopathology at baseline and Year 1, 2, 3, and 4 post-baseline follow-ups. At baseline, 415 participants completed assessments. For follow-up, the sample size consisted of 320 participants at Year 1 (77% retention rate), 277 at Year 2 (67% retention rate), 248 at Year 3 (60% retention rate), and 254 at Year 4 (61% retention rate). Because of the higher attrition in Years 3 and 4, we made an a priori decision to analyze only Year 1 and 2 data. Institutional Review Boards at all sites approved the study and all participants provided informed consent.
Measures
Predictor variables
Prior investigations have identified links between long-term ED outcomes and baseline demographics (e.g. age, education level), ED symptoms (e.g. binge eating, compensatory behaviors), Axis-I (e.g. depressive disorders) and Axis-II (e.g. borderline personality disorder) diagnoses, treatment history, and psychological symptoms (e.g. depression, self-esteem) (Carter, Blackmore, Sutandar-Pinnock, & Woodside, Reference Carter, Blackmore, Sutandar-Pinnock and Woodside2004; Franko et al., Reference Franko, Tabri, Keshaviah, Murray, Herzog and Thomas2018; Thompson-Brenner et al., Reference Thompson-Brenner, Franko, Thompson, Grilo, Boisseau, Roehrig and Wilson2013; Vall & Wade, Reference Vall and Wade2015). Thus, the 33 baseline features in our model were selected to reflect these previously identified predictors available within the dataset.
Demographics and psychiatric treatment. Self-reported demographic information included age, living situation (alone, with family, with friends, with significant other), ethnicity (White, Hispanic, Black, American Indian, and Asian),Footnote 1Footnote † marital status (single, married once, divorced, divorced/remarried, widowed, widowed/remarried, significant other), number of children, education level (less than grade school, grade 7–12, graduated high school, part college, graduated 2-year college, graduated 4-year college, part graduate school, graduated graduate school), and employment status (employed, unemployed). Participants also reported whether they had received past psychiatric treatment or hospitalization. Although not all demographics included in the model reflected previously identified predictors of ED outcomes, a broad range of demographics were included due to the ease of collection and under-investigation of such variables in prior research (Thompson-Brenner et al., Reference Thompson-Brenner, Franko, Thompson, Grilo, Boisseau, Roehrig and Wilson2013).
ED symptoms and behaviors. Previous research has shown that different measures designed to assess the same ED symptom can predict distinct outcomes (Stice, Fisher, & Lowe, Reference Stice, Fisher and Lowe2004). Therefore, a number of ED symptom measures, including some aiming to assess the same construct, were selected as model features. BMI was calculated using baseline measured height and weight. The Eating Disorder Examination-12 (EDE; Cooper & Fairburn, Reference Cooper and Fairburn1987) assessed baseline ED presentation and diagnosis. The EDE is an investigator-based interview measuring ED psychopathology over the past 28 days, and longer intervals corresponding to diagnostic criteria. The EDE assesses psychological aspects of EDs and the frequency of behavioral symptoms (e.g. binge eating). In this study, we specified baseline EDE subscales (restraint, eating concern, weight concern, shape concern) and behavioral frequency items, including the number of episodes in the past 3 months of objective binge eating and compensatory behavior (vomiting, laxative use, and diuretic use), as ED cognitive and behavioral predictors. The EDE has excellent psychometrics, including inter-rater and test-retest reliability (Berg, Peterson, Frazier, & Crow, Reference Berg, Peterson, Frazier and Crow2012a). In this study, inter-rater reliability for the EDE scores and behavioral items ranged from 0.90 to 0.99.
Two questionnaires further measured ED symptoms. The Three-Factor Eating Questionnaire (TFEQ; Stunkard & Messick, Reference Stunkard and Messick1985) is a 51-item questionnaire assessing restraint (cognitive control over food intake), disinhibition (loss of control over food intake), and hunger (susceptibility to hunger cues), with higher scores indicating greater impairment. In this study, Cronbach's α ranged from 0.86 to 0.89 for these subscales. The Binge Eating Scale (BES; Gormally, Black, Daston, & Rardin, Reference Gormally, Black, Daston and Rardin1982) is a 16-item self-report questionnaire that dimensionally assesses binge eating, with higher scores indicating greater severity. The BES has good reliability, convergent validity, and discriminant validity (Greeno, Marcus, & Wing, Reference Greeno, Marcus and Wing1995). In this study, Cronbach's α was 0.93.
Psychiatric diagnoses and symptoms. The Structured Clinical Interview for DSM-IV Axis-I Disorders (SCID-I; First, Spitzer, Gibbon, & Williams, Reference First, Spitzer, Gibbon and Williams1995) and the Structured Clinical Interview for DSM-IV Axis-II Personality Disorders (SCID-II; First, Gibbon, Spitzer, Williams, & Benjamin, Reference First, Gibbon, Spitzer, Williams and Benjamin1997) identified baseline comorbid psychopathology. The SCID-I assessed whether participants had a current or past diagnosis of any mood, substance use, obsessive–compulsive, and/or post-traumatic stress disorder or current diagnosis of generalized anxiety disorder. We also used the SCID-II to determine the presence of cluster A (odd/bizarre), cluster B (dramatic/erratic), and cluster C (avoidant/fearful) personality disorders (APA, 2000).
The Beck Depression Inventory (BDI; Beck, Steer, & Brown, Reference Beck, Steer and Brown1996) assessed depressive symptomology. The BDI is a 21-item questionnaire evaluating symptoms of major depressive episodes, with higher scores indicating greater depression severity. The BDI has high reliability and internal consistency (Beck, Steer, & Carbin, Reference Beck, Steer and Carbin1988). In this study, Cronbach's α was 0.92. In addition, the Rosenberg Self-Esteem Scale (RSES; Rosenberg, Reference Rosenberg1965) assessed global self-esteem. The RSES is a 10-item questionnaire, with higher scores indicating more self-esteem. In this study, Cronbach's α for the RSES was 0.93.
Outcome variables
We examined the prediction of the several outcomes at follow-up: (1) meeting diagnostic criteria for any ED using validated DSM-IV EDE diagnostic algorithms (Berg et al., Reference Berg, Stiles-Shields, Swanson, Peterson, Lebow and Le Grange2012b). According to these algorithms, absence of an ED was determined by no objective or subjective binge-eating episodes or compensatory behaviors in the past 3 months, and scores ⩽2 out of 6 on items assessing ‘Fear of Weight Gain,’ ‘Importance of Weight,’ and ‘Importance of Shape’; (2) presence of objective binge eating over the past 3 months on the EDE; (3) presence of compensatory behaviors over the past 3 months on the EDE; and (4) underweight BMI (<18.5 kg/m2), based on measured height and weight. All outcome variables were assessed at Year 1 and 2 follow-ups.
Data analyses
Sample characteristics
To maximize statistical power, Year 1 outcome analyses included all 320 participants who completed both baseline and Year 1 assessments and Year 2 analyses included 277 participants who completed both baseline and Year 2 assessments. Casewise missing data were not imputed due to the mathematical equivalence of complete-case and imputed data when information is only missing for dependent variables in regression contexts (van Buuren, Reference van Buuren2018; Von Hippel, Reference Von Hippel2017). There was a minimal amount of missing item-level data in the initial EDE interview (1.5% unavailable) due to specific questions not being asked. As the reasons for omitting these questions were not documented by study assessors and these data were not missing at random, we did not impute these data points. To test whether follow-up samples differed from the original sample on baseline characteristics, participants were categorized according to whether they completed follow-up at Year 1 only (n = 16), Year 2 only (n = 56), both (n = 261), or neither (n = 82). These groups were compared on baseline predictors using separate linear regression or χ2 analyses with Bonferroni corrections (p = 0.002) for multiple comparisons.
Modeling approach
Analyses were performed in R (R Development Core Team, 2013) via glm in base R, and the caret (Kuhn, Reference Kuhn2008) and boot (Canty & Ripley, Reference Canty and Ripley2019) packages. To test inferential statistical models, we conducted non-regularized multiple logistic regression models including all baseline variables to predict each outcome. To compare the performance of traditional logistic regressions to ML, we conducted elastic net regularized logistic regressions. Elastic net models (Zou & Hastie, Reference Zou and Hastie2005) combine two other regularization/ML methods: ridge (which shrinks coefficients in a model toward zero; Tikhonov, Reference Tikhonov1963) and lasso (which shrinks some coefficients completely to zero; Tibshirani, Reference Tibshirani1996). This algorithm was chosen given its well-established accuracy and robustness (Ogutu, Schulz-Streeck, & Piepho, Reference Ogutu, Schulz-Streeck and Piepho2012; Zou & Hastie, Reference Zou and Hastie2005) and an ability to maintain clinical result interpretability compared to less transparent ML algorithms (e.g. neural networks). However, to ensure that our conclusions about the accuracy of ML models were not idiosyncratic to the elastic net, we also tested our models using a random forests algorithm. Although a complete description of these ML models is beyond the scope of this paper, we refer readers to Kuhn and Johnson (Reference Kuhn and Johnson2013) for a full introduction to these models and general ML concepts.
Following recommendations (Kuhn & Johnson, Reference Kuhn and Johnson2013), we used 10-fold cross-validation with three repetitions to select the optimal λ (a shrinkage parameter reflecting the degree of coefficient regularization) and α [a mixing parameter reflecting the balance between ridge (α = 0) and lasso (α = 1)] parameters for elastic net models, and the optimal mtry parameter (number of random predictors to consider at each split in the model) for random forests. This allowed us to obtain a cross-validation estimate of model performance for non-regularized logistic regression, elastic net, and random forest models. Whereas splitting a dataset into a single training (model building) and testing (model evaluation) set can have limited ability to accurately characterize uncertainty in results for smaller sample sizes, repeating the training and testing process in this manner can provide more reasonable estimates of model accuracy in predicting outcomes in new datasets. We calculated a 95% bias-corrected and accelerated confidence interval of the Area Under the Receiving Operating Characteristics Curve (AUC) model performance metric, reflecting model fit, using 5000 bootstrapped resamples, and refitted a model using the optimal λ and α parameters to examine variable importance/coefficients for the best-fitting elastic net model. To estimate variable importance for random forest models, we used the varImp() function in the caret package. For models with unbalanced outcomes (e.g. more than 70% of cases belonging to one outcome class), we used upsampling in the caret package (Kuhn, Reference Kuhn2008) to improve the balance across classes. We used this subsampling technique within the resampling process to ensure that the proportions of outcomes in test data sets were not altered (Kuhn, Reference Kuhn2008).
Model fit
We calculated the mean cross-validation estimate of AUC as a metric of model performance. AUC is a discrimination metric that ranges from 0.5 = chance to 1.0 = perfect. Guidelines for AUC scores range from extremely poor (0.5–0.59), poor (0.60–0.69), fair (0.70–0.79), good (0.80–0.89), to excellent (>0.90).
Results
Sample characteristics
See Table 1 for baseline characteristics. At Year 1, 299 (93.4%) participants continued to meet the criteria for any ED, 171 (53.4%) reported binge-eating episodes, 118 (36.9%) reported compensatory behaviors, and 46 (14.3%) had an underweight BMI. At Year 2, 242 (87.4%) participants met the criteria for any ED, 119 (43.0%) reported binge-eating episodes, 75 (27.1%) reported compensatory behaviors, and 31 (11.2%) had an underweight BMI. No significant baseline differences were detected between participants who did or did not complete one or both follow-ups (online Supplementary Table S1).
Baseline sample characteristics (n = 415)
BDI, Beck Depression Inventory (Beck et al., Reference Beck, Steer and Brown1996); BES, Binge Eating Scale (Gormally et al., Reference Gormally, Black, Daston and Rardin1982); EDE, Eating Disorder Examination (Cooper & Fairburn, Reference Cooper and Fairburn1987); RSES, Rosenberg Self-Esteem Scale (Rosenberg, Reference Rosenberg1965); TFEQ, Three-Factor Eating Questionnaire (Stunkard & Messick, Reference Stunkard and Messick1985).
Model performance
Model performance indices for logistic regression and elastic net models are reported in Table 2. At Year 1 and 2 follow-ups, elastic net provided better accuracy in predicting all outcomes (binge eating, compensatory behaviors, ED diagnosis, underweight BMI) than the logistic regression models, with average cross-validated AUC for elastic net models ranging from poor/fair (ED diagnostic status and binge eating) to good (compensatory behaviors and underweight BMI at Year 2) and excellent (underweight BMI at Year 1).
Model performance of non-regularized and elastic net regularized logistic regressions
AUC, Area Under the Receiving Operating Characteristics Curve; BMI, body mass index; CI, confidence interval.
The λ (penalization) and α (mixing) parameters were identified through 10-fold cross-validation repeated three times for the elastic net models.
Elastic net model performance did not deteriorate even when the most important predictor was removed (online Supplementary Table S2), demonstrating that the ability of the ML model to improve predictive accuracy was not dependent on any specific input variable. Finally, the accuracy of the random forest models (online Supplementary Table S3) was comparable to elastic net models for all outcomes at both time points, demonstrating that the superior performance of ML compared to standard approaches was not restricted to the selection of elastic net models.
Predictor importance
Coefficients for the variables in each elastic net model are shown in Table 3. Corresponding coefficients for the random forests model are included in online Supplementary Table S4 for comparison.
Coefficients for variables in elastic net models
AN, anorexia nervosa; BDI, Beck Depression Inventory (Beck et al., Reference Beck, Steer and Brown1996); BED, binge-eating disorder; BES, Binge Eating Scale (Gormally et al., Reference Gormally, Black, Daston and Rardin1982); BN, bulimia nervosa; ED, eating disorder; EDE, Eating Disorder Examination (Cooper & Fairburn, Reference Cooper and Fairburn1987); RSES, Rosenberg Self-Esteem Scale (Rosenberg, Reference Rosenberg1965); TFEQ, Three-Factor Eating Questionnaire (Stunkard & Messick, Reference Stunkard and Messick1985).
Elastic net coefficients represent the final model, identified through 10-fold cross-validation repeated three times to identify the optimal λ (penalization) and α (mixing) parameters. Dashes represent values of zero. Reference groups for categorical variables are: anorexia nervosa (baseline eating disorder diagnosis), white (ethnicity), married (marital status), alone (living situation), less than grade school (education level), and employed (employment status).
ED diagnosis
Elastic net models indicated that the most important variables for predicting ED diagnosis at Year 1 were baseline BN diagnosis, no baseline diagnosis of partial BED (this diagnosis was protective), not being of American Indian descent (this ethnicity was protective), and education level (<grade school or >graduate education). At Year 2, the most important variables predicting ED diagnosis included higher baseline EDE restraint, EDE shape concern, and TFEQ disinhibition scores, not being of American Indian descent, lower education level, and living alone relative to living with friends. Some overlap in predictor importance with the random forest model was found; both models identified EDE restraint, EDE shape concern, and TFEQ disinhibition scores to be of high importance to predicting Year 2 ED diagnosis. Evaluation of these metrics should take into account the poor fit of the models predicting ED diagnosis.
Binge eating
The most important variables for predicting binge eating in the elastic net model at Year 1 were baseline BN diagnosis, higher baseline BES and TFEQ disinhibition scores, and lower education level. Similarly, at Year 2, the most important predictors of binge eating were higher baseline BES and TFEQ disinhibition scores, Asian ethnicity, and no history of psychiatric treatment or cluster B personality disorder (these variables were protective). The random forest model also identified BES and TFEQ disinhibition scores as the variables of high importance to predicting binge eating at Years 1 and 2.
Compensatory behavior
The most important variables for predicting compensatory behavior in the elastic net model at Year 1 were baseline diagnosis of sub- or full-threshold BN, no baseline diagnosis of sub- or full-threshold BED (these diagnoses were protective), and living with friends. Similarly, at Year 2, the most important predictors of compensatory behavior were baseline diagnosis of sub- or full-threshold BN, no baseline diagnosis of sub- or full-threshold BED, and baseline diagnosis of generalized anxiety disorder. Similar to the elastic net model, the random forests models identified baseline BN and sub- or full-threshold BED diagnoses to be of high importance to predicting compensatory behavior at Years 1 and 2.
Underweight BMI
The most important variables for predicting underweight BMI in the elastic net model at Year 1 were lower baseline BMI, baseline diagnosis of sub-threshold AN, American Indian ethnicity, lower education level, and history of psychiatric hospitalization. Likewise, at Year 2, the most important predictors were lower baseline BMI, baseline diagnosis of sub-threshold AN, no baseline diagnosis of BED (this diagnosis was protective), history of psychiatric hospitalization, lower baseline TFEQ disinhibition, and current or past diagnosis of obsessive–compulsive disorder. The random forests model also identified baseline BMI to be of high importance to predicting underweight BMI at Years 1 and 2, baseline diagnosis of sub-threshold AN as important to predicting underweight BMI at Year 1, and baseline diagnosis of BED and TFEQ disinhibition score as important to predicting underweight BMI at Year 2.
Discussion
Advances in computational capacity have led the behavioral sciences to increasingly embrace ML techniques to improve the prediction of complex psychiatric phenomena (Askland et al., Reference Askland, Garnaat, Sibrava, Boisseau, Strong, Mancebo and Eisen2015; Fox et al., Reference Fox, Huang, Linthicum, Wang, Franklin and Ribeiro2019; Kessler et al., Reference Kessler, van Loo, Wardenaar, Bossarte, Brenner, Cai and Zaslavsky2016; Koutsouleris et al., Reference Koutsouleris, Kahn, Chekroud, Leucht, Falkai and Wobrock2016; Ribeiro, Huang, Fox, Walsh, & Linthicum, Reference Ribeiro, Huang, Fox, Walsh and Linthicum2019). However, gaps in the adoption of these approaches remain. Despite the urgent need to improve prognosis related to EDs given their high morbidity and mortality rates (Crow et al., Reference Crow, Peterson, Swanson, Raymond, Specker, Eckert and Mitchell2009; Mitchell & Crow, Reference Mitchell and Crow2006), heterogeneous outcomes (Eddy et al., Reference Eddy, Tabri, Thomas, Murray, Keshaviah, Hastings and Franko2017; Steinhausen & Weber, Reference Steinhausen and Weber2009), and modest treatment response (Berkman et al., Reference Berkman, Bulik, Brownley, Lohr, Sedway, Rooks and Gartlehner2006), modeling techniques have not been used to maximize prediction for these populations. In this study, we conducted an initial investigation to determine whether ML models (e.g. elastic net) could improve upon traditional analytical techniques (logistic regression) in predicting key ED outcomes, such as continued ED diagnosis, binge eating, compensatory behaviors, and underweight BMI. Results demonstrated that, for each outcome at both Year 1 and 2 follow-ups, the elastic net model outperformed the logistic regression model, improving classification by up to 19%. In fact, in all but one comparison, the elastic net analyses moved the AUC estimate into a qualitatively higher accuracy category (e.g. fair to good). These results highlight the promise of ML for enhancing the ability to accurately predict complex outcomes for individuals with serious psychiatric disorders using only a set of mostly simple self-report measures.
Although the logistic regression analyses occasionally produced good models (e.g. correctly classifying 83% of Year 1 compensatory behaviors), overwhelmingly these models yielded fair to extremely poor AUCs. In fact, the ability to classify whether an individual would maintain an ED diagnosis at Year 1 and 2 follow-ups was no better than chance in these models. These findings are somewhat unsurprising, given that logistic regression is not designed to accommodate multiple, highly correlated predictors (Burke et al., Reference Burke, Ammerman and Jacobucci2019). However, the results highlight the insufficiency of the traditional analytic techniques that comprise the majority of predictive research in addressing a critical question facing psychiatric care: how likely are individuals with different clinical presentations to improve? In contrast, an elastic net produced fair to excellent models in all but two cases (ED diagnosis at Years 1 and 2). In the strongest models, predicting compensatory behaviors and underweight, 85–93% of cases were correctly classified. This is especially important given established links between low weight and purging and premature mortality in EDs (Crow et al., Reference Crow, Peterson, Swanson, Raymond, Specker, Eckert and Mitchell2009). The AUCs reflected in these findings parallel the accuracy of ML models for other psychiatric phenomena (Fox et al., Reference Fox, Huang, Linthicum, Wang, Franklin and Ribeiro2019; Ribeiro et al., Reference Ribeiro, Huang, Fox, Walsh and Linthicum2019). Further, the fit of each model was not altered drastically with the removal of the most impactful variable, and was replicated using an alternative ML algorithm, highlighting the flexibility, reliability, and power of these approaches.
Improved prediction of psychiatric outcomes could have important long-term clinical implications. More individuals are affected with psychiatric disorders, including EDs, than the current care model can accommodate (Kazdin, Fitzsimmons-Craft, & Wilfley, Reference Kazdin, Fitzsimmons-Craft and Wilfley2017), thus healthcare access for these disorders remains limited (Guarda, Wonderlich, Kaye, & Attia, Reference Guarda, Wonderlich, Kaye and Attia2018). However, a sizable portion of individuals can improve with lower intensity interventions, such as guided self-help (Traviss-Turner, West, & Hill, Reference Traviss-Turner, West and Hill2017). Enhanced ability to detect the likelihood of clinical improvement for a particular individual could lead to optimized personalized decision-making regarding the appropriate intervention intensity. As such, ML models hold significant promise as scalable approaches to providing highly accurate, less burdensome, guidance in clinical settings in the future. However, it is important to acknowledge that the specific models developed in this study are not expected to yield direct practice utility and that additional research is needed before ML approaches can inform individualized clinical judgment (Challen et al., Reference Challen, Denny, Pitt, Gompels, Edwards and Tsaneva-Atanasova2019). Rather, this study provides proof of concept for a class of analytic approaches with the potential to improve the models of psychiatric outcomes in subsequent research and, perhaps, clinical care.
However, even our ML models could be improved. Few produced excellent prediction, and prediction of ED diagnosis at Years 1 and 2 remained quite poor. This limitation could result from several factors. First, despite the potency of data-driven analytics, these approaches are constrained by their input. Although the predictors in this study reflect many of the most common psychiatric and ED assessments, they are not diverse and may have excluded important factors. For instance, clinical prognosis decisions often incorporate a number of variables not examined in this study (e.g. motivation, medical stability, support). Additionally, many of the indices in this study reflected symptoms or diagnoses, despite the increasing gravitation toward dimensional, mechanistic variables for predicting mental health outcomes (Bilder, Howe, & Sabb, Reference Bilder, Howe and Sabb2013). Further, all the measures relied upon participant self-report. Acknowledging self-report limitations, there is a movement to include multi-modal assessment, incorporating behavioral (e.g. neuropsychological tasks) and biological (e.g. neuroimaging) features into ML models (Bilder et al., Reference Bilder, Howe and Sabb2013). Because this study collected naturalistic data of long-term outcomes, all predictors were derived at baseline and dependent variables were collected infrequently. Future ML models may be improved by incorporating frequent and continuous data collection, including information gathered throughout clinical care. Finally, although behavioral indices often define ED outcomes, converging evidence suggests that cognitive symptoms (e.g. weight and shape overvaluation) may be of greater centrality to ED pathology (Bardone-Cone, Hunt, & Watson, Reference Bardone-Cone, Hunt and Watson2018; DuBois, Rodgers, Franko, Eddy, & Thomas, Reference DuBois, Rodgers, Franko, Eddy and Thomas2017). More research is needed to determine if different or more comprehensive measures could further enhance the prediction of a variety of ED outcomes.
We also conducted exploratory analyses into which variables contributed the greatest variance to the ML models. There is a significant debate about the degree to which individual ML predictors should be interpreted given the complexity of the interactions modeled in ML (Ribeiro et al., Reference Ribeiro, Huang, Fox, Walsh and Linthicum2019). However, data-driven approaches reduce reliance on subjective interpretations that sometimes bias hypothesis-driven investigations (Burke et al., Reference Burke, Ammerman and Jacobucci2019) and may permit the identification of variables with unanticipated influences on outcomes that warrant examination in future predictive research. In this study, many of the important model features, especially those replicating between ML methods, had strong face validity (e.g. disinhibition predicting binge eating; sub-threshold AN predicting underweight BMI). However, other unexpected patterns (e.g. level of educational attainment contributing to most elastic net models) may warrant investigation in future research. Although these findings highlight the ability of ML to construct models that overcome clinical biases in the service of accurately calculating risk, these approaches hold the greatest promise for precise prediction, as opposed to mechanism identification. Additionally, particular caution is encouraged in interpreting feature importance for predicting ED diagnosis at Years 1 and 2, given the poor fit of these models.
Our study strengths include multi-site recruitment of a large transdiagnostic sample, extended length of follow-up, and collection of a sizeable number of assessments that could be readily administered in a real-world setting. However, there were also limitations. The sample size was adequate for an elastic net model, but was somewhat small for ML, which often requires very large sample sizes for the most rigorous prediction (Bzdok & Meyer-Lindenberg, Reference Bzdok and Meyer-Lindenberg2018). Additionally, there are numerous ML approaches besides elastic net and random forests, each with a unique set of strengths and limitations (Cho, Yim, Choi, Ko, & Lee, Reference Cho, Yim, Choi, Ko and Lee2019); it is possible that other models could yield better accuracy. It is also possible that alternative modeling choices may have yielded different outcomes. For instance, although cross-validation is an accepted ML practice for investigations with smaller sample sizes and we conducted upsampling within the training (but not test) datasets (Kuhn & Johnson, Reference Kuhn and Johnson2013), it has the potential to yield overly optimistic models (Smith, Seaman, Wood, Royston, & White, Reference Smith, Seaman, Wood, Royston and White2014). Similarly, AUC is a frequently-used metric for testing model performance; however, some recommend the use of precision-recall curves when outcomes are imbalanced (He & Garcia, Reference He and Garcia2009). Further, as with any longitudinal investigation, this study experienced attrition. No baseline differences were found between individuals who did and did not complete follow-ups, but it is possible that these participants varied in unmeasured ways. The sample also lacked diversity in certain demographics (e.g. gender, race/ethnicity) and may not be generalizable to the broader ED population. Additionally, some model features were derived from relatively time-consuming assessments requiring extensive training (e.g. diagnostic interviews), which may limit the ease in clinical dissemination. Finally, the method of determining ED diagnosis, though based on established algorithms (Berg et al., Reference Berg, Stiles-Shields, Swanson, Peterson, Lebow and Le Grange2012b), may have been outdated (based on DSM-IV) and biased toward inflating clinical cases due to the high standard for cognitive symptom improvement. Further research is needed to test multiple ML models with larger, more diverse ED samples and objective markers of clinical outcome.
Conclusion
This investigation confirmed that ML can enhance the long-term prediction of persistent ED symptoms and may also identify novel markers of risk. These results encourage future testing of novel ML algorithms to predict illness course, as well as to determine treatment response and to develop adaptively tailored psychiatric interventions. The complexity, flexibility, and accuracy of ML make these approaches well suited to ultimately advance precision medicine for serious mental health concerns, including EDs, and thereby to contribute toward mitigating the severe and life-threatening consequences of these illnesses.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291720000227.
Acknowledgements
The investigators express gratitude to the McKnight Longitudinal Study research staff at the University of Minnesota, Cornell University, and Stanford University for their contributions to this investigation. The investigators also thank Dr Patrick Mair, who provided statistical guidance and consulting on machine-learning models.
Financial support
This work was supported in part by the McKnight Foundation; National Institute of Mental Health of the National Institutes of Health (Award numbers: K23MH112867, T32MH082761); National Science Foundation (Award number DGE-1745303); Klarman Family Foundation; Hilda and Preston Davis Foundation; and Minnesota Obesity Center. These funding agencies did not influence the design of the study, collection, analysis, and interpretation of data, or writing of the manuscript
Conflict of interest
None.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
