Introduction
Major depression disorder (MDD) affects 280 million people globally and ranks among the top reasons for disability (World Health Organization, Reference World Health Organization2021). Dozens of antidepressants, augmentation pharmacological agents, psychological therapies, and brain stimulation procedures are effective for depression (Cipriani et al., Reference Cipriani, Furukawa, Salanti, Chaimani, Atkinson, Ogawa and Geddes2018; Kennedy et al., Reference Kennedy, Lam, McIntyre, Tourjman, Bhat, Blier and Uher2016; Milev et al., Reference Milev, Giacobbe, Kennedy, Blumberger, Daskalakis, Downar and Ravindran2016; Parikh et al., Reference Parikh, Quilty, Ravitz, Rosenbluth, Pavlova, Grigoriadis and Uher2016), but the efficacy of these treatments varies across individuals. Fewer than half of people with MDD achieve remission with the first treatment (Trivedi et al., Reference Trivedi, Rush, Wisniewski, Nierenberg, Warden, Ritz and Fava2006). Many have to try multiple treatments before finding an effective one (Malone, Reference Malone2007; Rush et al., Reference Rush, Trivedi, Wisniewski, Nierenberg, Stewart, Warden and Fava2006). Each treatment trial takes between 6 and 12 weeks and the delays are associated with the risk of adverse outcomes, including loss of employment and suicide (Al-Harbi, Reference Al-Harbi2012; Crown et al., Reference Crown, Finkelstein, Berndt, Ling, Poret, Rush and Russell2002). If we could predict response to a specific treatment from individual characteristics, we could reduce the duration of depression and improve long-term functional outcomes (Oluboka et al., Reference Oluboka, Katzman, Habert, McIntosh, MacQueen, Milev and Blier2018). Multiple features have been identified as potential predictors of treatment outcomes (Fava, Reference Fava2009; McGrath et al., Reference McGrath, Kelley, Holtzheimer, Dunlop, Craighead, Franco and Mayberg2013; Uher et al., Reference Uher, Perlis, Henigsberg, Zobel, Rietschel, Mors and McGuffin2012a; Uher, Tansey, Malki, & Perlis, Reference Uher, Tansey, Malki and Perlis2012b; Zisook et al., Reference Zisook, Lesser, Stewart, Wisniewski, Balasubramani, Fava and Rush2007). None of them has been adopted for treatment selection in clinical practice (Perlis, Reference Perlis2016). Reasons for the lack of adoption may be that no single characteristic provides a prediction that is accurate enough to be clinically meaningful or differential prediction of outcomes with alternative treatments (Simon & Perlis, Reference Simon and Perlis2010). Since depression is a complex and heterogeneous disorder (Fried, Reference Fried2017; Wray et al., Reference Wray, Ripke, Mattheisen, Trzaskowski, Byrne, Abdellaoui and Sullivan2018), multiple features will likely have to be considered in a multivariate model to allow accurate prediction of treatment outcomes (Gillan & Whelan, Reference Gillan and Whelan2017; Kautzky et al., Reference Kautzky, Baldinger-Melich, Kranz, Vanicek, Souery, Montgomery and Kasper2017; Kessler, Reference Kessler2018).
Machine learning is defined as a combination of algorithms that explore how computer systems can learn rules from multiple examples with no need for explicit programming (Samuel, Reference Samuel2000). The computer gradually improves its performance of a task through learning from an increasing amount of data. Machine learning methods can build a model that classifies individuals into predefined categories (e.g. treatment response) or estimates a level of a continuous concept (e.g. degree of reduction in depression severity). The last decade has seen an expansion of machine learning applications in health care, including the prediction of depression treatment outcomes (Lee et al., Reference Lee, Ragguett, Mansur, Boutilier, Rosenblat, Trevizol and McIntyre2018). In this article, we will synthesize and critically examine the applications of machine learning to depression treatment outcome prediction, evaluate the potential of these methods to inform treatment selection, and propose directions for further research.
Methods
Literature search
We conducted a search of PubMed, Google Scholar, ScienceDirect, and PsychINFO following the PRISMA guidelines (Moher et al., Reference Moher, Liberati, Tetzlaff, Altman, Altman, Antes and Tugwell2009), for articles and reports on MDD, treatment outcomes, and machine learning, published from database inception to 12 October 2020. We used a combination of terms tagging machine learning (statistical learning OR machine learning OR predictive analytics OR deep learning) with terms tagging depression treatment [antidepressant OR depression OR major depressive disorder (MDD)] and its outcomes (treatment outcome OR response OR remission).
Two study authors (M.S. and R.U.) screened the studies and applied the following inclusion criteria: (1) participants with a diagnosis of MDD; (2) clinical assessment with rating scales before and after treatment or historical assessment of treatment resistance; (3) use of a validate machine learning method. The literature search and selection of eligible reports are shown in Fig. 1.
Data extraction
We extracted the size of training, testing, and validation datasets, type of treatment, type and number of predictor variables (clinical variables, demographical variables, treatment history, rating scales for depression, etc.), outcome definition [response, remission, treatment-resistant depression (TRD)], methods used for prediction, missing data, feature selection, validation methods (leave-n-out, k-fold cross-validation, nested cross-validation, holdout or external validation). We recorded the results as accuracy, balanced accuracy, or area under the receiver operating characteristics curve. We transformed available results to the common metrics of balanced accuracy (the average of the reported sensitivity and specificity), which is independent of the proportion of individuals with an outcome of interest.
Study quality assessment
In the absence of a validated quality measure for machine learning studies, we applied minimal requirements for aspects of methodology that have been linked to the replicability of results: sample size and validation procedure. Larger samples are more likely to generate replicable results because they reduce the problems of dimensionality and underfitting (Vabalas, Gowen, Poliakoff, & Casson, Reference Vabalas, Gowen, Poliakoff and Casson2019). Estimates of the minimal sample size for a machine learning study range from 100 to 300 (Beleites, Neugebauer, Bocklitz, Krafft, & Popp, Reference Beleites, Neugebauer, Bocklitz, Krafft and Popp2013; Luedtke, Sadikova, & Kessler, Reference Luedtke, Sadikova and Kessler2019). A validation procedure that separates training and testing sets is essential to avoid overfitting. Non-nested cross-validation procedures where feature selection and/or parameter tuning occur in the same loop as predictive accuracy test leads to overfitting (Cawley & Talbot, Reference Cawley and Talbot2010). Therefore, we required either nested cross-validation or an external validation in a held-out sample with feature selection separated from prediction. We designated studies with a sample size of 100 or more and adequate validation methods as ‘adequate-quality’. In addition, a detailed quality assessment following published guidelines (Yusuf et al., Reference Yusuf, Atal, Li, Smith, Ravaud, Fergie and Selfe2020) is reported in Supplementary Table S1. All of the adequate-quality papers reported on data sources, data split method, etc., however, none of them reported on the distribution of treatment outcome scores. Moreover, none of the adequate-quality studies used reporting guidelines.
Data analysis
Most studies used more than one machine learning method and reported multiple estimates of predictive accuracy. We used linear mixed-effects models to estimate prediction accuracy and test the effects of methodological features while accounting for the non-independence of multiple estimates with a random effect of the study. In data visualization (e.g. Fig. 2), we plot a single mean estimate of balanced accuracy for each study.
Literature search and selection of eligible recoreds for the systematic review and meta-analysis.
Results
Literature search results
Our literature search retrieved 7732 non-duplicate records. We retained 59 eligible reports that matched our inclusion criteria (Fig. 1). These 59 eligible studies varied in focus, size, and method. The predicted outcomes were remission (19 studies), response (35 studies), and TRD (six studies). The number of individuals ranged from 6 to 36 902 (mean 410, median 115). The predictive variables included demographic, clinical, cognitive, neuroimaging, and molecular genetic variables. The number of features used in prediction ranged from 1 to 4 241 701 (mean 131 660, median 92.5). The ratio of participants to features ranged from 1:1432 to 3690:1.
Balanced accuracy and participant-to-feature ratio in published machine learning studies of outcome prediction in the treatment of MDD. The X-axis plots the ratio of participants to predictive features. Y-axis plots the mean balanced accuracy within each study. Studies predicting response, remission, and TRD are plotted as circles, diamonds, and triangles respectively. Adequate-quality studies are highlighted with large, filled symbols. The dark gray horizontal dashed line shows the mean balanced accuracy of the eight adequate-quality studies. The pale gray horizontal dashed line shows the average balanced accuracy of the other 45 studies.
Accuracy of prediction
Fifty-four of the 59 eligible studies provided accuracy or other estimates (sensitivity and specificity) that allowed the calculation of balanced accuracy. Across these studies, we extracted 364 estimates of balanced accuracy, ranging from 0.39 to 1.00. Mean accuracy across estimates within study ranged from 0.48 to 0.91 (mean 0.74, 95% CI 0.71–0.77).
Treatments
The treatments included antidepressant medication (36 studies (61%)), neurostimulation (18 studies (32%)), psychological treatments (four studies (7%)), and other treatments (exercise, psilocybin, blended treatment delivery model; three studies (5%)). Two studies used a combination of two treatment modalities (psychotherapy and antidepressants, neurostimulation, and antidepressant) (Guilloux et al., Reference Guilloux, Bassi, Ding, Walsh, Turecki, Tseng and Sibille2015; Kambeitz et al., Reference Kambeitz, Goerigk, Gattaz, Falkai, Benseñor, Lotufo and Brunoni2020).
The outcome of neurostimulation treatment was predicted with greater accuracy (mean 0.79, 95% CI 0.74–0.84) than treatment with antidepressants (mean accuracy 0.70, 95% CI 0.67–0.74) or other treatments (mean accuracy 0.69, 95% CI 0.65–0.73). Most studies predicted outcomes within a single group of participants who received the same treatment. Three studies probed the treatment-specificity of outcome prediction through testing predictive models in groups of participants who received either the same or a different treatment (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018; Kambeitz et al., Reference Kambeitz, Goerigk, Gattaz, Falkai, Benseñor, Lotufo and Brunoni2020). In one study, a model based on clinical variables developed in a study of treatment with the antidepressant citalopram significantly predicted outcomes among individuals who received citalopram, but not among those who received a combination of venlafaxine and mirtazapine (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016). Another study used a combination of clinical and genetic variables to derive two models predicting outcomes with escitalopram and nortriptyline respectively, which demonstrated treatment-specificity in a held-out test sample (Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018). A third study used clinical and cognitive variables to develop models predicting outcomes of antidepressant and neurostimulation treatment and demonstrated the specificity of predicting escitalopram vs. transcranial direct current stimulation (tDCS) outcomes (Kambeitz et al., Reference Kambeitz, Goerigk, Gattaz, Falkai, Benseñor, Lotufo and Brunoni2020). In summary, while most studies investigated only one treatment group, three studies suggest that multivariate prediction of outcome is treatment-specific (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018; Kambeitz et al., Reference Kambeitz, Goerigk, Gattaz, Falkai, Benseñor, Lotufo and Brunoni2020).
Features contributing to the prediction
The eligible studies used a variety of features as predictors of depression treatment outcomes. Most used neuroimaging (n = 35), followed by clinical and demographic variables (n = 30). Relatively few studies used molecular (n = 8), and cognitive (n = 6) measures. Eighteen studies combined predictors from two modalities: most commonly neuroimaging and clinical (n = 9) (Bartlett et al., Reference Bartlett, DeLorenzo, Sharma, Yang, Zhang, Petkova and Parsey2018; Jaworska, De La Salle, Ibrahim, Blier, & Knott, Reference Jaworska, De La Salle, Ibrahim, Blier and Knott2019). One study employed a combination of predictors from three modalities of clinical, cognitive, and neuroimaging features (Patel et al., Reference Patel, Andreescu, Price, Edelman, Reynolds and Aizenstein2015). For further details, please see Supplementary Tables S2 and S4.
There was a significant relationship between feature modality and sample size. Studies that used neuroimaging had small samples (mean 85, median 50 individuals), studies using genetic variables had intermediate sample sizes (mean 307, median 254), and studies using clinical variables had the largest samples (mean 950, median 276). All studies with data on 1000 or more individuals were limited to clinical and demographic variables (Cepeda et al., Reference Cepeda, Reps, Fife, Blacketer, Stang and Ryan2018; Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Delgadillo & Salas Duhne, Reference Delgadillo and Salas Duhne2020; Nie, Vairavan, Narayan, Ye, & Li, Reference Nie, Vairavan, Narayan, Ye and Li2018; Perlis, Reference Perlis2013).
There was a significant relationship between data modality and reported balanced accuracy. Studies using neuroimaging or genetic data reported significantly higher balanced accuracies (β = 0.13, 95% CI 0.07–0.18, p < 0.001; β = 0.13, 95% CI 0.08–0.18, p < 0.001, respectively) than studies using clinical and demographic variables.
No study tested the added value of neuroimaging to clinical variables within the same sample. One analysis reported improved prediction of treatment outcome with the inclusion of a large number of genetic variables compared to using clinical variables alone (Iniesta, Stahl, & McGuffin, Reference Iniesta, Stahl and McGuffin2016; Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018), but a study that used genetic information without clinical features reported prediction not significantly better than chance (Maciukiewicz et al., Reference Maciukiewicz, Marshe, Hauschild, Foster, Rotzinger, Kennedy and Geraci2018). Overall, the use of data from multiple modalities was not associated with reported prediction accuracy (β = 0.01, 95% CI −0.02 to 0.04, p = 0.641).
A minority of studies reported on the contribution of specific features. In three analyses of the same large trial sample, initial depression severity and race were among the variables that contributed the most to the predictive models (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Nie et al., Reference Nie, Vairavan, Narayan, Ye and Li2018; Perlis, Reference Perlis2013). Symptoms of reduced interest and activity have also ranked among the most strongly contributing variables, consistent with the results of univariate analyses (Iniesta et al., Reference Iniesta, Stahl and McGuffin2016, Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018; Uher et al., Reference Uher, Perlis, Henigsberg, Zobel, Rietschel, Mors and McGuffin2012a, Reference Uher, Tansey, Malki and Perlis2012b, Reference Uher, Frey, Quilty, Rotzinger, Blier, Foster and Kennedy2020).
The number of features used for prediction and the feature-to-observation ratio were unrelated to the reported accuracy of prediction (Fig. 2).
Treatment of missing values
In real-world data, missing values appear due to the loss of participants to follow-up, missed assessments, and intentional or accidental failure to complete items or instruments. Depending on the relationship of missing values to dependent and independent variables of interest, the mechanisms underlying missing values can be classified as missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Many machine learning methods do not support missing values and, consequently, investigators take various options to deal with missing values outside the machine learning algorithms. Ways of handling missing data in machine learning studies include list-wise deletion, replacing with mean/median/mode, predicting the missing values, or using algorithms that support missing values imputation, such as k-nearest neighborhood and random forest. With an increasing number of features, the proportion of individuals with missing data points increases, leading to the loss of a substantial part of the sample at the cost of reduced power. Besides, the deletion of observations with missing values reduces external validity unless the data are MCAR. Imputation of missing values with the prediction by machine learning methods performs better than replacing with mean/median/mode, but care has to be taken to completely separate the imputation between training and testing sets (Bertsimas, Pawlowski, & Zhuo, Reference Bertsimas, Pawlowski and Zhuo2018; Schmitt, Reference Schmitt, Mandel and Guedj2015; Zhang, Reference Zhang2016). Among the eligible articles, the majority of studies did not address the handling of missing values (43 studies) and other studies used case-wise deletion or mean/mode imputation. Only one study used a machine-learning-based method of handling missing data, the bagged tree imputation (Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018). Supplementary Tables S3 and S4 provide detailed information on the treatment of missing values.
Feature selection
Feature selection helps avoid the curse of dimensionality and reduces training time by decreasing the number of features. It also provides information on feature importance and increases generalizability by reducing overfitting (Bermingham et al., Reference Bermingham, Pong-Wong, Spiliopoulou, Hayward, Rudan, Campbell and Haley2015; James, Witten, Hastie, & Tibishirani, Reference James, Witten, Hastie and Tibishirani2013). Feature selection methods are divided into three main classes: wrapper, filter, and embedded (Guyon, Elisseeff, & Kaelbling, Reference Guyon, Elisseeff and Kaelbling2003). Wrapper methods employ a predictive model, including the interactions between variables. However, these methods risk overfitting if the number of observations is small and is computationally intensive if the number of variables is large. Filter methods are efficient in calculation time, but often select redundant variables as they do not consider the relationship between features. Embedded methods combine the advantages of wrapper and filter methods. Irrespective of which feature selection method is used, it must be implemented in the training set only to avoid overfitting. Of the 59 included studies, the majority (n = 33) did not use any feature selection. We found no relationship between the feature selection method and reported prediction accuracy. For details, please see Supplementary Figs S1 and S2.
Choice of machine learning method
The majority of included studies reported a single machine learning method, but 10 compared multiple methods (Supplementary Fig. S6). The most used methods were regression-based models and support vector machines (Supplementary Fig. S7). Only one study used a deep learning method to predict treatment response (Lin et al., Reference Lin, Kuo, Liu, Yu, Yang and Tsai2018). Within a study, the various methods often gave moderately consistent estimates of balanced accuracy (intraclass correlation 0.62, 95% CI 0.51–0.73). Across the included studies, we found no systematic relationship between the type of machine learning method and the balanced accuracy (Supplementary Tables S4 and S5).
Validation procedures
Validation procedures in machine learning assess the performance of the classification model and its stability across data sets. This is typically achieved by repeatedly dividing the available observations into multiple non-overlapping training and testing sets, an approach known as cross-validation. Details of cross-validation determine the stability of results and the degree of protection against overfitting. Methods with a large overlap between training datasets (e.g. leave-n-out cross-validation) are known to provide less stable estimates than methods with random subsampling (e.g. k-fold cross-validation) whereas the former is less-biased than the latter assuming all other factors are controlled i.e. this is a bias-variance trade-off (Kuhn & Johnson, Reference Kuhn and Johnson2013). Overfitting will occur if the imputation of missing values or feature selection is performed with the entire dataset because the information from the testing set is used in feature selection. Significant overfitting also occurs when feature selection is performed in the same cycle of cross-validation as parameter tuning, because of information leakage between feature selection and parameter tuning (Cawley & Talbot, Reference Cawley and Talbot2010). Nested cross-validation offers adequate protection against overfitting through separating feature selection from model parameter tuning of the inner and outer cross-validation loops. Another method that offers an adequate test of generalizability is holdout validation, which uses an additional ‘unseen’ testing dataset that was not used in any way in the model development. Of the 59 eligible studies, 18 reported described validation methods with adequate separation of training and testing sets, including nested cross-validation, external validation in a holdout and/or a separately collected test dataset. The remaining 41 studies used validation methods that may not adequately protect against overfittings (Supplementary Table S4 and Fig. S8).
Study quality and the accuracy of prediction
We defined study quality as a combination of an adequate sample size of 100 or more observations and an adequate validation method with complete separation of training and testing sets at all stages including feature selection (e.g. nested cross-validation or external validation). Of the 59 included studies, 26 had 100 or more participants and 17 reported adequate validation methods. The eight studies that had more than 100 participants and reported adequate validation methods were designated as adequate-quality studies (Table 1, Table 2, and Supplementary Table S4). The adequate-quality designation was significantly negatively related to reported accuracy (b = −0.05, 95% CI −0.10 to −0.004, p = 0.035). Among the adequate-quality studies, the mean balanced accuracy was 0.63 (95% CI 0.56–0.71). Among the remaining 46 studies, the mean balanced accuracy was 0.75 (95% CI 0.72–0.78). The difference in accuracy between adequate-quality and other studies was primarily driven by sample size. The 33 studies with samples smaller than 100 reported a mean balanced accuracy of 0.76 (95% CI 0.73–0.80). The 21 studies with samples of 100 or greater reported a mean balanced accuracy of 0.68 (0.63–0.72). Sample size greater than 100 was significantly negatively related to reported accuracy (b =−0.05; 95% CI −0.08 to −0.01, p = 0.005). The relationship between adequate validation method and reported balanced accuracy was not significant (−0.02, 95% CI −0.07 to 0.03, p = 0.469). Moreover, the adequate-quality studies reported the following range of accuracy for each depression treatment outcome (the confidence intervals of these estimates are relatively broad because of the small number of contributing studies):
(a) response with mean balanced accuracy 0.56 (95% CI 0.43–0.68) based on 17 estimates from three studies;
(b) remission with mean balanced accuracy 0.60 (95% CI 0.51–0.70) based on 16 estimates from five studies;
(c) treatment resistance with mean balanced accuracy 0.69 (95% CI 0.60–0.77) based on 26 estimates from three studies.
Methods for construction of the machine learning model of the eight adequate-quality papers
Description summary of the eight adequate-quality papers
Replicability of classification
The likelihood that a prediction will generalize to individuals who were not included in model derivation can be inferred from differences in prediction accuracy between internal cross-validation and external validation or from independent replication in new samples. Only five studies reported accuracy from both internal validation and external validation (Athreya et al., Reference Athreya, Neavin, Carrillo-Roa, Skime, Biernacka, Frye and Bobo2019; Browning et al., Reference Browning, Kingslake, Dourish, Goodwin, Harmer and Dawson2019; Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Crane et al., Reference Crane, Jenkins, Bhaumik, Dion, Gowins, Mickey and Langenecker2017; Guilloux et al., Reference Guilloux, Bassi, Ding, Walsh, Turecki, Tseng and Sibille2015). In these studies, the mean balanced accuracy in internal validation was 0.77 and the mean balanced accuracy in external validation was 0.69. The relatively small internal−external drop in accuracy would suggest adequate generalizability, but only a small minority of studies reported relevant data. The preferred way to assess generalizability is independent replication. We found only one attempt at replication in the published literature. One study (Browning et al., Reference Browning, Bilderbeck, Dias, Dourish, Kingslake, Deckert and Dawson2021) replicated previous work by the same authors predicting antidepressant treatment outcome from measures of symptoms and attentional bias after 1 week of treatment (Browning et al., Reference Browning, Kingslake, Dourish, Goodwin, Harmer and Dawson2019). The prediction in replication was statistically significant, but the accuracy of the prediction was reduced from 0.80 in the first study to 0.67 in replication (Browning et al., Reference Browning, Kingslake, Dourish, Goodwin, Harmer and Dawson2019, Reference Browning, Bilderbeck, Dias, Dourish, Kingslake, Deckert and Dawson2021).
There is an available benchmark dataset that researchers can use to test the generalizability of their algorithms, for example, the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) (Sinyor, Schaffer, & Levitt, Reference Sinyor, Schaffer and Levitt2010).
Discussion
This review synthesizes the rapidly expanding literature on the implementation of machine learning to predict treatment outcomes in depression. Some studies reported promising results, including increased prediction accuracy with the inclusion of multi-modal data, treatment-specific predictions, and positive results from external validation data sets. However, a pattern observed across studies suggests that smaller studies and studies using inadequate validation methods tend to report higher predictive accuracy. This systematic relationship between method and result, coupled with a lack of independent replication, suggests caution in interpreting existing results and the need for careful methodological development.
Several studies suggest that it is possible to derive a multivariate predictive model that is both replicable and treatment-specific. A model developed in a study of nearly 2000 participants based on demographic and clinical variables significantly predicted outcomes in an independent sample of 151 individuals from a study using similar treatment and assessment procedures (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016). While the accuracy of less than 0.60 may not be sufficient for clinical application (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016), other studies suggest that prediction may be improved if data from more modalities are included. In a study of 280 individuals randomly allocated to one of two antidepressant medications, a combination of clinical and molecular genetic features allowed the development of drug-specific prediction models that replicated in a held-out sample of 150 individuals with prediction accuracy over 0.70 (Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018). In both studies, the algorithms predicted outcomes in individuals treated with the same type of antidepressant but not among individuals treated with a different type of antidepressant than was used in model development (Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Iniesta et al., Reference Iniesta, Hodgson, Stahl, Malki, Maier, Rietschel and Uher2018). These promising results of two adequate-quality studies suggest that treatment-specific prediction can be achieved and may be applied to a personalized selection of treatment (Kessler, Reference Kessler2018). However, only a minority of reviewed studies included multiple treatments, limiting the application of results to personalized treatment selection.
While the results of individual studies may be promising, it is important to examine patterns in the literature and consistency across studies. Notably, the machine learning method, feature selection, or features-to-observations ratio were not associated with the reported prediction accuracy. Instead, the sample size and validation design proved essential to the understanding of differences among published studies. We found that some of the highest accuracy estimates had been reported from studies with fewer than 100 participants and/or studies using methods prone to overfitting. While an individual study with fewer than 100 participants may well achieve replicable results, a systematic relationship between study size or quality, and the strength of reported results may indicate bias. The distribution of study size and quality may be partly a result of an early stage in the applications of machine learning methodology to clinical problems and lack of access to large datasets. The largest available datasets are limited to demographic and clinical data (Cepeda et al., Reference Cepeda, Reps, Fife, Blacketer, Stang and Ryan2018; Chekroud et al., Reference Chekroud, Zotti, Shehzad, Gueorguieva, Johnson, Trivedi and Corlett2016; Delgadillo & Salas Duhne, Reference Delgadillo and Salas Duhne2020; Nie et al., Reference Nie, Vairavan, Narayan, Ye and Li2018; Perlis, Reference Perlis2013). Therefore, it is not possible to separate the effect of bias due to low study quality from the potential advantages of additional data modalities. In the next decade, it will be essential to establish large datasets with optimized multimodal assessments that will allow examining the contribution of biomarkers to prediction with adequate methodology.
The success of any machine learning model is defined by its ability to generalize and replicate on a truly independent sample. In recent years, the inability to reproduce the results of many studies has turned into a growing concern among researchers (Baker, Reference Baker2016). In this context, it is worrying that no full independent replication attempt has been reported for machine learning prediction of depression treatment outcomes. The findings of the present review should make replication a priority for the field of depression treatment outcome prediction.
Criteria for the applicability of machine learning approaches in healthcare
The recent years have seen rapid growth in the publications of studies using machine learning approaches in the prediction of treatment outcomes. However, the methodological rigor of these studies is variable. The present review raises a concern that highly optimistic results might be correlated with insufficient scrutiny of machine learning procedures. The applicability of machine learning algorithms in healthcare will depend on multiple factors, including predictive performance, robustness in calibration across a variety of samples, and proof of an impact on relevant outcomes in practice. Tutorials on how to develop an efficient and reliable machine learning algorithm are now available (Faes et al., Reference Faes, Liu, Wagner, Fu, Balaskas, Sim and Denniston2020; Tohka & van Gils, Reference Tohka and van Gils2021). In addition, the essential role of using an external validation dataset to prevent overfitting in high-dimensional classification algorithms should be taken into consideration (Park & Han, Reference Park and Han2018). Consensus criteria and checklists are now available that allow assessing the adequacy of predictive model development and its applicability (Scott, Carter, & Coiera, Reference Scott, Carter and Coiera2021; Vollmer et al., Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson and Hemingway2020).
Future directions
The next decade is expected to see an expansion in open data sharing. Coupled with mature machine learning methodology, the availability of large samples with multimodal measurements will allow separating potential information advantage of adding objective measurement modalities, such as neuroimaging, from publication bias. New data collection in large samples with multiple alternative treatments will improve the clinical applicability of results. Replicability and generalizability are essential features of clinical research and prerequisite to implementation. External validation of a predictive algorithm in a sample that was not available at the time of model development is needed to prove that a machine learning prediction model is reproducible and generalizable.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291721003871.
Acknowledgements
This work was conducted as part of the Canadian Biomarker Integration Network in Depression (CAN-BIND), an Integrated Discovery Program, supported by the Ontario Brain Institute.
Financial support
This work has been funded by the Ontario Brain Institute, which is an independent non-profit corporation, funded partially by the Ontario Government. The opinions, results, and conclusions are those of the authors and no endorsement by the Ontario Brain Institute is intended or should be inferred. Additional funding was provided by the Canadian Institutes of Health Research (CIHR), Lundbeck, Bristol-Myers Squibb, and Servier. Funding and/or in-kind support was also provided by the investigators' universities and academic institutions. Dr Uher has received additional support from the Canada Research Chairs Program (award number 231397), the Canadian Institutes of Health Research (Grant reference number 148394), and the Dalhousie Medical Research Foundation.
Conflict of interest
Drs. Sajjadian, Foster, Turecki, Farzan, and Uher report no conflict of interest. Dr Lam has received honoraria or research funds from Allergan, Asia-Pacific Economic Cooperation, BC Leading Edge Foundation, CIHR, CANMAT, Canadian Psychiatric Association, Hansoh, Healthy Minds Canada, Janssen, Lundbeck, Lundbeck Institute, MITACS, Ontario Brain Institute, Otsuka, Pfizer, St. Jude Medical, University Health Network Foundation, and VGH-UBCH Foundation. Dr Milev has received honoraria or research funds from Ontario Brain Institute, Allergan, Janssen, Lallemand, Kye, Lundbeck, Nubiyota, Otsuka, Pfizer, and Sunovion. Dr Rotzinger holds a patent ‘Teneurin C-Terminal Associated Peptides (TCAP) and methods and uses thereof. Inventors: David Lovejoy, R.B. Chewpoy, Dalia Barsyte, Susan Rotzinger.’ Dr Frey had a research grant from Pfizer. Dr Müller is funded by a CIHR operating grant 428404. Dr Soares has acted as a consultant for Servier, Sunovion, Lundbeck, and Otsuka. Dr Parikh has received honoraria or research funds from Assurex, Takeda, Janssen, Mensante, Aifred, Sage. Dr Strother is the Chief Scientific Officer of ADMdx, Inc., which receives NIH funding. Dr Kennedy has received honoraria or research funds from Abbott, Alkermes, Allergan, BMS, Brain Canada, CIHR, Janssen, Lundbeck, Lundbeck Institute, Ontario Brain Institute, Ontario Research Fund, Otsuka, Pfizer, Servier, Sunovion, Xian-Janssen, and Field Trip Health.
