Introduction
Given the often progressive nature of schizophrenia and related psychoses, as well as the abundant evidence supporting the benefits of early detection and intervention in improving outcomes and prognosis (Marshall et al., Reference Marshall, Lewis, Lockwood, Drake, Jones and Croudace2005; Perkins et al., Reference Perkins, Gu, Boteva and Lieberman2005), researchers have sought to develop means of identifying individuals in the pre-psychotic or clinical high-risk (CHR) phase, characterized by attenuated positive symptoms, which typically antedate full-blown psychotic illness (Correll et al., Reference Correll, Hauser, Auther and Cornblatt2010; Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Richer-Rossler, Schultze-Lutter, Keshavan, Wood, Ruhrmann, Seidman, Valmaggia, Cannon, Velthorst, De Haan, Cornblatt, Bonoldi, Birchwood, McGlashan, Carpenter, McGorry, Klosterkotter, McGuire and Yung2013). Initial studies found that approximately 30% of CHR individuals convert to a psychotic disorder, usually within 2 years, while more recent studies by the NAPLS group and others have reported declining rates of conversion (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016; Hartmann et al., Reference Hartmann, Yuen, McGorry, Yung, Lin, Wood, Lavoie and Nelson2016).
In North America, CHR is defined primarily by the Structured Interview for Psychosis-Risk Syndromes (SIPS), a semi-structured interview (McGlashan et al., Reference McGlashan, Miller, Woods, Hoffman, Davidson and Miller2001; Miller et al., Reference Miller, McGlashan, Rosen, Somjee, Markovich, Stein and Woods2002; Rosen et al., Reference Rosen, Woods, Miller and McGlashan2002) that probes for past and current attenuated v. threshold psychotic symptoms. The Attenuated Positive Symptom Psychosis-Risk Syndrome (APSS), delineated in the SIPS, served as the basis for Attenuated Psychosis Syndrome (APS) included in the appendix section of proposed diagnoses in the DSM-5, with the sole additional requirement that CHR individuals be help-seeking.
The major reasons that prevented APS from being approved as an official diagnostic category, were the high false-positive rates (Haroun et al., Reference Haroun, Dunn, Haroun and Cadenhead2006; Corcoran et al., Reference Corcoran, First and Cornblatt2010; Fusar-Poli et al., Reference Fusar-Poli, Cappucciati, Rutigliano, Schultze-Lutter, Bonoldi, Borgwardt, Riecher-Rössler, Addington, Perkins, Woods, McGlashan, Lee, Klosterkötter, Yung and McGuire2015); the small sample sizes of studies (Woods et al., Reference Woods, Addington, Cadenhead, Cannon, Cornblatt, Heinssen, Perkins, Seidman, Tsuang, Walker and McGlashan2009; Van der Gaag et al., Reference Van der Gaag, Smit, Bechdolf, French, Linszen, Yung, McGorry and Cuijpers2013); the low annual incidence of CHR cases (less than 1/10 000 persons) (Addington et al., Reference Addington, Cadenhead, Cannon, Cornblatt, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Heinssen2007); and the variable and declining conversion rates (or increasing false positive rates) (Yung et al., Reference Yung, Yuen, Berger, Francey, Hung, Nelson, Phillips and McGorry2007; Simon et al., Reference Simon, Umbricht, Lang and Borgwardt2014). Further, at the present time, there are few treatments available that are specific to the CHR phase beyond treating concomitant symptoms (e.g. anxiety, depression) (Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Richer-Rossler, Schultze-Lutter, Keshavan, Wood, Ruhrmann, Seidman, Valmaggia, Cannon, Velthorst, De Haan, Cornblatt, Bonoldi, Birchwood, McGlashan, Carpenter, McGorry, Klosterkotter, McGuire and Yung2013).
Despite these limitations, there is an emerging consensus about the clinical characteristics that predict conversion to psychosis among CHR individuals, including specific positive symptoms from the SIPS (e.g. P.1. Unusual Thought Content) as well as other clinical/demographic factors (Brucato et al., Reference Brucato, Masucci, Arndt, Ben-David, Colibazzi, Corcoran, Crumbley, Crump, Gill, Kimhy, Lister, Schobel, Yang, Lieberman and Girgis2017; Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Richer-Rossler, Schultze-Lutter, Keshavan, Wood, Ruhrmann, Seidman, Valmaggia, Cannon, Velthorst, De Haan, Cornblatt, Bonoldi, Birchwood, McGlashan, Carpenter, McGorry, Klosterkotter, McGuire and Yung2013; Addington et al., Reference Addington, Liu, Buchy, Cadenhead, Cannon, Cornblatt, Perkins, Seidman, Tsuang, Walker, Woods, Bearden, Mathalon and McGlashan2015). Based upon these and other data, Cannon et al. developed a risk calculator that was published online and allows a clinician to obtain individualized 2-year conversion risks by way of an assessment battery that includes the SIPS, neurocognitive tests, and several clinical measures inclusive of stressful life events (Carrión et al., Reference Carrión, Cornblatt, Burton, Tso, Auther, Adelsheim, Calkins, Carter, Niendam, Sale, Taylor and McFarlane2016). In addition, Fusar-Poli et al. published on a transdiagnostic risk calculator that utilizes basic demographic and diagnostic information to identify individuals who are at highest risk of becoming psychotic over 6 years (Fusar-Poli et al., Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017). These efforts seek to emulate, in psychiatry, a paradigm that was successfully implemented in other areas of medicine, such as cardiovascular disease (D'Agostino et al., Reference D'Agostino, Vasan, Pencina, Wolf, Cobain, Massaro and Kannel2008) and oncology (Nam et al., Reference Nam, Kattan, Chin, Trachtenberg, Singal, Rendon, Klotz, Sugar, Sherman, Izawa, Bell, Stanimirovic, Venkateswaran, Diamandis, Yu, Loblaw and Narod2011).
In light of these efforts, we examined our data from previously published articles on a CHR sample to determine whether clinical characteristics readily obtained from only the SIPS and accompanying clinical interview, could predict conversion to psychosis. Importantly, we recently identified SIPS-based measures that are highly associated with conversion to psychosis but have not traditionally been scored on the SIPS by other groups – namely, specific perceptual abnormalities (i.e. auditory and visual perceptual abnormalities) (Lehembre-Shiah et al., Reference Lehembre-Shiah, Leong, Brucato, Abi-Dargham, Lieberman, Horga and Girgis2017) and violent ideation and violent behavior (Brucato et al., Reference Brucato, Appelbaum, Lieberman, Wall, Feng, Masucci, Altschuler and Girgis2018). Therefore, our goal was to examine whether our SIPS data, enhanced with SIPS-based measures of violent ideation, violent behavior, and auditory and visual perceptual abnormalities, could predict conversion to psychosis, and hypothesized that these measures would be particularly informative. To construct a predictive model based on these measures, we employed a modern machine-learning algorithm known as LASSO-logistic regression that simultaneously selects relevant predictors and estimates their joint effects on the outcome (i.e. conversion to psychosis). The model can be used to compute subject-specific risk estimates for conversion to psychosis in CHR individuals. We assess the relative importance of the predictor variables that are included in the model and assess the utility of our predictive model with respect to overall accuracy by using evaluation methods that reduce the risk of obtaining overly optimistic performance metrics.
Methods
Subjects
We have previously reported on our procedures for subject recruitment/enrollment as well as data collection (Brucato et al., Reference Brucato, Masucci, Arndt, Ben-David, Colibazzi, Corcoran, Crumbley, Crump, Gill, Kimhy, Lister, Schobel, Yang, Lieberman and Girgis2017). In summary, we recruited help-seeking individuals age 13–30 participating at the early detection and intervention for mental disorders program at the Center of Prevention and Evaluation (COPE) at the New York State Psychiatric Institute (NYSPI) at Columbia University Medical Center, all meeting criteria for both the APSS syndrome defined in the SIPS and the APS defined in the DSM-5. Written informed consent was provided by those aged ⩾18. Minors gave written assent, with written informed consent provided by a parent/legal guardian. All research procedures were approved by NYSPI's Institutional Review Board.
Exclusion criteria included lack of proficiency in English; any history of full-blown psychotic illness; a DSM disorder better accounting for the clinical presentation; I.Q. < 70; medical conditions affecting the central nervous system; imminent risk of harm to self or others; unwillingness to participate in research; geographic distance; or active current substance use disorder.
Clinical assessments
The SIPS (McGlashan et al., Reference McGlashan, Miller, Woods, Hoffman, Davidson and Miller2001; Miller et al., Reference Miller, McGlashan, Rosen, Somjee, Markovich, Stein and Woods2002; Rosen et al., Reference Rosen, Woods, Miller and McGlashan2002) includes a semi-structured interview which probes for past and current signs and symptoms of attenuated v. threshold psychotic states and contains the Scale of Prodromal Symptoms which includes 19 items, divided into Positive, Negative, Disorganization, and General Symptom subsections. CHR criteria are met if one meets criteria for the APSS, the Genetic Risk and Deterioration Psychosis-Risk Syndrome, or the Brief Intermittent Psychotic Psychosis-Risk Syndrome, but, notably, all COPE patients have met at least APSS criteria. The SIPS also includes a checklist of DSM-IV symptoms of Schizotypal Personality Disorder (American Psychiatric Association, 2000), a survey of family history of mental illness (Andreasen et al., Reference Andreasen, Endicott, Spitzer and Winokur1977), and a modified version of the DSM-IV Global Assessment of Functioning scale (GAF) (Hall, Reference Hall1995).
At baseline, participants also completed either the Diagnostic Interview for Genetic Studies (Nurnberger et al., Reference Nurnberger, Blehar, Kaufmann, York-Cooler, Simpson, Harkavy-Friedman, Severe, Malaspina and Reich1994) or Structured Clinical Interview for DSM-IV Axis-I Disorders, Patient Edition (SCID-I/P) (First et al., Reference First, Spitzer, Gibbon and Williams2002). Those aged <16 completed the Kiddie Schedule for Affective Disorders and Schizophrenia, Present and Lifetime Version (K-SADS-PL) (Kaufman et al., Reference Kaufman, Birmaher, Brent, Rao and Ryan1996).
Participants were seen for follow-up with the SIPS every 3 months for 2 years when possible, or else whenever conversion was suspected. For some patients who would not come in for SIPS, we used clinical notes, family or physician report, and/or telephone notes to determine the outcome. Post-conversion diagnoses were established by COPE psychologists and/or psychiatrists. Administrators were certified in the measures and established scoring consensus.
The Global Functioning Scale: Social (GF: Social) and Global Functioning Scale: Role (GF: Role) were assessed at baseline and each follow-up visit (Cornblatt et al., Reference Cornblatt, Auther, Niendam, Smith, Zinberg, Bearden and Cannon2007). Violent behavior and violent ideation (Brucato et al., Reference Brucato, Appelbaum, Lieberman, Wall, Feng, Masucci, Altschuler and Girgis2018), suicidal ideation and behavior and history of trauma (Grivel et al., Reference Grivel, Leong, Masucci, Altschuler, Arndt, Redman, Yang, Brucato and Girgis2018), the SIPS P.4. subcomponents of auditory and visual perceptual abnormalities (Lehembre-Shiah et al., Reference Lehembre-Shiah, Leong, Brucato, Abi-Dargham, Lieberman, Horga and Girgis2017), and the SIPS P.1. subcomponents (Crump et al., Reference Crump, Arndt, Grivel, Horga, Corcoran, Brucato and Girgis2017) were all extracted from the baseline SIPS and accompanying the clinical interview, as previously described.
Statistical methods
All analyses were conducted in R version 3.3.2. Demographic and clinical characteristics of the entire analytic sample, as well as those for only converters and only non-converters, were summarized using means, standard deviations, and ranges for continuous measures, and frequencies and percentages for categorical measures. Baseline characteristics of converters and the non-converters were compared using two-sided two-sample t tests for continuous measures and Fisher's exact test for categorical measures. The criteria used for statistical significance is a p value <0.05.
In constructing the predictive model for conversion, we sought to use all of the baseline clinical and demographic measures available. Some subjects were missing one or more of these baseline measures, though the amount of missing data was small. We used the k-nearest neighbors (kNN) algorithm, with k = 5 implemented using the VIM package in R, to ‘fill in’ the values for those with missing baseline measures prior to fitting the predictive model (Kowarik and Templ, Reference Kowarik and Templ2016). We constructed the predictive model using binary logistic regression with a LASSO penalty (Tibshirani, Reference Tibshirani1996) where the outcome was a binary indicator of conversion status within the 2 years of follow-up and the predictors consisted of the baseline clinical and demographic characteristics. LASSO-logistic regression is ideal in this setting because it allows us to perform both estimations of the logistic model parameters and selection of relevant baseline variables simultaneously (i.e. it can screen out those baseline variables that are not predictive of conversion status by setting their corresponding parameter estimates to zero). Estimating the predictive model parameters using the LASSO penalty requires the selection of a tuning parameter, λ, which controls the number of model coefficients that are estimated to be non-zero. We selected an optimal λ from a range of plausible values using 5-fold cross-validation, selecting the λ value that maximized the average cross-validated area under the receiver operating characteristic (ROC) curve (AUC). The folds for the cross-validation procedure were selected via stratified random sampling to reflect the approximate 30% conversion rate in the entire sample. Selection of λ and fitting of the LASSO logistic model were carried out using the GLMNET package in R (Friedman et al., Reference Friedman, Hastie and Tibshirani2010). The final predictive model was internally validated using 1000 bootstrap resamples (also sampled to reflect the approximate 30% conversion rate) to compute an optimism-adjusted performance measure (Harrell et al., Reference Harrell, Lee and Mark1996) of AUC, or as it is also known, concordance index (c-index) (ranging from 0.5 for no ability to discriminate converters from non-converters to 1 for perfect ability to discriminate them).
Results
The analytic sample consisted of 199 subjects who had conversion status (converted to psychosis v. did not convert to psychosis) available in the 2-year follow-up period. In the entire sample, 64 (32.16%) converted to psychosis. Their mean age is 20.08 (s.d. = 3.82). Fifty-three (27.00%) are female (three subjects were missing gender information) and 106 (54.90%) are non-Caucasian (six subjects were missing information on race). Table 1 shows the descriptive statistics for all baseline demographic and clinical variables that are considered in the predictive model.
Characteristics of the full sample, converters, and non-converters
* ES column provides Cohen's d statistic value comparing Converters and non-Converters for continuous measures or odds ratios for binary measures.
** P column provides p values from t tests for continuous measures or Fisher's Exact test for binary measures.
Table 1 also shows descriptive measures and statistical tests for the converters and non-converters. At baseline, converters appear to score higher on average than non-converters on Total Positive Symptoms (Cohen's d (d) = 0.36, p = 0.02) and the corresponding subscales of P.1. Unusual Thought Content (d = 0.47, p < 0.01) and P.5. Disorganized Communication (d = 0.47, p < 0.01); Total Negative Symptoms (d = 0.44, p < 0.01) and the corresponding subscales of N.1. Social Anhedonia (d = 0.47, p < 0.01), N.3. Expression of Emotion (d = 0.40, p = 0.01), and N.5. Ideational Richness (d = 0.42, p = 0.01); total Disorganized Symptoms (d = 0.40, p = 0.01) and the corresponding subscales of D.1. Odd Behavior (d = 0.50, p < 0.01) and D.3. Trouble with Focus and Attention (d = 0.40, p = 0.01); G.3. Motor Disturbance (d = 0.45, p < 0.01) and SIPS Total Scores (d = 0.43, p = 0.01). Converters appear to score lower on average than non-converters with respect to P.4. Visual Perceptual Abnormalities (d = −0.50, p < 0.01) and G.2. Dysphoric Mood (d = −0.35, p = 0.02), as well as the GAF (d = −0.44, p = 0.01). A higher proportion of converters described violent ideation at baseline (44.40% v. 21.50%, p < 0.01), endorsed violent behavior at baseline (12.70% v. 3.10%, p = 0.02), and were non-Caucasian (68.30% v. 48.5%, p = 0.02). Converters and non-converters did not differ on any other measured characteristics.
Only 3.21% of the overall data were missing from the analytic sample. Table 1 shows the number of observations (N) of each variable that was available in the data set. Social and role functioning scores had the highest amount of missing data, with each having 20.60% of the values missing. Family history was missing for 17.10% of the sample while all other variables had less than 10% missing values. Missing values were imputed using the kNN algorithm with k = 5. This allowed us to use all 199 observations to construct and validate the predictive model.
All of the clinical and demographic variables shown in Table 1 were considered as potential predictors of conversion status, except the Total Sum Scores (i.e. Total Positive, Negative, and Disorganized Symptoms, as well as the SIPS Sum of Total Scores) as they were linear combinations of other measures. We eliminated the sum scores to allow for better ability to identify sub-symptom scores that are predictive of conversion status.
The LASSO logistic prediction model identified 17 of the 40 baseline variables as having predictive value based on cross-validation. Table 2 lists these predictors in order of the magnitude of their standardized coefficient estimates (log odds ratios). The last column of Table 2 shows the proportion of times that the corresponding variable was selected to remain in the model across all 1000 bootstrap resamples. A high value provides evidence that the corresponding variable is predictive of the outcome. The P.4. sub-symptom score for P.4. Visual Perceptual Abnormalities had the largest predictive value, reflected by the magnitude of the standardized coefficient and was also selected to remain in the largest proportion of LASSO logistic models (99%) across all 1000 bootstrap resamples. Other predictors with relatively large magnitude standardized coefficient estimates that were also selected in 90% or more of the LASSO logistic models across all 1000 bootstrap resamples were G.2. Dysphoric Mood, P.1. Unusual Thought Content, P.5. Disorganized Communication, Violent Ideation, race, N.1. Social Anhedonia, and Violent Behavior.
Estimated unstandardized and standardized model parameters (ORs) from the LASSO logistic model
Coefficients are ordered by magnitude of the standardized coefficient estimate.
a The value for the intercept is −1.65.
b Proportion of times that each variable was selected to remain in the model across all 1000 bootstrap resamples.
Figure 1 displays the frequency distributions of the model-based predicted risks for converters and non-converters in the sample. There tends to be a higher proportion of converters than non-converters in each risk class corresponding to risk scores above 44%, with the exception of the 54–59% risk class. Table 3 shows the in-sample statistics for prediction of conversion to psychosis for increasing thresholds of predicted risk. The in-sample values in the table are computed using predicted risk scores for the full sample derived from the fitted model and are therefore overly optimistic as they are essentially using the same data twice, but are included for completeness and for comparison with the similar published values from the North American Prodrome Longitudinal Study (NAPLS) (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016). The bootstrap validation procedure produced an optimism-adjusted AUC of 0.73 for our model, in comparison with a similarly adjusted AUC of 0.71 provided by NAPLS.
Frequency distributions of the model-based predicted risks for the converters and non-converters.
In-sample prediction statistics for conversion to psychosis for different levels of model-based risk scores
Figure 2 provides ROC curves for the present model (in-sample) and from published (in-sample) values from the NAPLS study [Table 3 in the original publication (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016)]. Figure 2 shows our prediction model to have superior sensitivity and specificity at every level of risk. These in-sample ROC curves are overly optimistic though, as they do not take into account that the data have been used to both fit the models and evaluate their performance. The in-sample AUC of our model is 0.84 in comparison with 0.74 for the NAPLS model (obtained by calculation from tabulated numbers in the original publication). Additionally, we calculated an in-sample Brier score of 0.15 for our model.
ROC curves for New LASSO-logistic model (in-sample) and NAPLS model (in-sample).
Discussion
The goal of this study was to construct a practical and clinically useful predictive model for conversion to psychosis among individuals at CHR for schizophrenia and related psychotic disorders as per the SIPS research measure and the APS diagnosis proposed in the DSM-5 utilizing only clinical and demographic characteristics as obtained using the SIPS and accompanying baseline clinical interview. This technique was intended to facilitate assessment of risk for psychotic illness in clinical settings, where the use of the various batteries requiring specialized expertise, such as required by the NAPLS calculator, might be impractical. We applied LASSO-logistic regression with 5-fold cross-validation to data from 199 CHR patients, (135 non-converting and 64 converting to psychosis within the 2-year follow-up period) to construct our predictive model and then performed a bootstrap validation procedure to assess model performance.
The estimated predictive model performed well, with an optimism-adjusted AUC of 0.73. Our predictive model bears some similarities with other published predictive models for conversion to psychosis, namely the recently published NAPLS model (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016). The NAPLS model includes unusual thought content (SIPS P.1.) and Suspiciousness (SIPS P.2.), decline in social functioning, lower verbal learning and memory performance, slower speed of processing, age, stressful life events, trauma, and family history of psychosis as predictors of conversion, with the combined factor of P1 + P2 being the strongest predictor. By comparison with respect to the ROC curves, the in-sample performance of our proposed predictive model is superior to the in-sample performance of the NAPLS model. Our optimism-adjusted AUC of 0.73 was also comparable with that from the NAPLS model, which had a bootstrap-validated AUC of 0.71. Similar to the NAPLS model and to other literature, SIPS symptoms P.1. and P.5. have been shown to be reliable predictors of conversion to psychosis (Addington et al., Reference Addington, Liu, Buchy, Cadenhead, Cannon, Cornblatt, Perkins, Seidman, Tsuang, Walker, Woods, Bearden, Mathalon and McGlashan2015). A finding unique to our model is that the subcomponent of P.4. Visual Perceptual Abnormalities had the largest magnitude standardized coefficient indicating non-risk for conversion, and that Violent Behavior and Violent Ideation were also strong predictors of conversion. Importantly, these are variables that are already collected by the SIPS and accompanying clinical interview but have not been assigned their own scores as they are subcomponents of other scores and therefore had not, in this form, been examined for their predictive abilities until recently by our group (Brucato et al., Reference Brucato, Appelbaum, Lieberman, Wall, Feng, Masucci, Altschuler and Girgis2018; Lehembre-Shiah et al., Reference Lehembre-Shiah, Leong, Brucato, Abi-Dargham, Lieberman, Horga and Girgis2017). It is possible that these variables conferred the robust predictive ability of our model and are supported by previous studies which examine similar variables and methodologies (Perkins et al., Reference Perkins, Jeffries, Cornblatt, Woods, Addington, Bearden, Cadenhead, Cannon, Heinssen, Mathalon, Seidman, Tsuang, Walker and McGlashan2015; Marshall et al., Reference Marshall, Deighton, Cadenhead, Cannon, Cornblatt, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods, Bearden, Mathalon and Addington2016).
We could not directly apply the NAPLS predictive model to our sample because our patients were not assessed with the Hopkins Verbal Learning Test nor the BACS symbol coding as part of standard admission, nor do we routinely perform the specific life events and trauma scales performed by NAPLS. We believe that our predictive model has potential as an aid for clinicians to assess prognosis in CHR patients and to assist in guiding clinicians in the selection of treatment. However, before these clinical applications can be realized, the proposed model needs to be applied to and validated in a large independent set of CHR individuals, by our group as well as in other independent cohorts. The potential effectiveness of this model could also be enhanced by incorporating specific, biological measures of conversion that are readily available, inexpensive, and highly replicated.
It is also possible that the reasons for our findings and differences with other predictor models like NAPLS are due to the patient samples as reflected by the conversion rates. The NAPLS calculator was based on data derived from a patient sample with a substantially lower conversion rate, indicating possibly greater heterogeneity in the disturbances for which they sought treatment or who had milder forms or were at an earlier stage of their onset of illness. Alternatively, it could be that our sample was more atypical than other CHR cohorts in terms of its high conversion rate, in comparison with many recently published CHR samples, particularly those in recent clinical trials (McGorry et al., Reference McGorry, Nelson, Markulev, Yuen, Schäfer, Mossaheb, Schlögelhofer, Smesny, Hickie, Berger, Chen, de Haan, Nieman, Nordentoft, Riecher-Rössler, Verma, Thompson, Yung and Amminger2017). For example, NAPLS-2 (Addington et al., Reference Addington, Liu, Buchy, Cadenhead, Cannon, Cornblatt, Perkins, Seidman, Tsuang, Walker, Woods, Bearden, Mathalon and McGlashan2015) noted a conversion rate of 15.2%, or 25.3% of 367 participants who either converted or completed 2-year follow-up.
There are also differences between our prediction model and the risk calculator of Fusar-Poli et al. (Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017), which disallow direct comparisons. While the study of Fusar-Poli et al., utilized individuals with non-organic, non-psychotic disorders, we utilized a purely CHR sample. In addition, the study of Fusar-Poli et al. calculated 6-year conversion rates, as opposed to two for our model. Given the robust accuracy of the Fusar-Poli et al. calculator as well as the NAPLS calculator, we would suggest that these calculators/prediction models serve to complement each other and together provide a more complete assessment/study of the variables that predict conversion to psychosis in different populations.
One criticism of using the LASSO, as with many other regression fitting procedures, is that if the predictors are correlated, then their individual effects on the outcome may be poorly estimated. However, the predictions derived from the estimated model need not be poor. Simulation studies have shown that LASSO-logistic regression models can perform well with respect to prediction of the outcome, even when the predictors are highly correlated (Lu and Petkova, Reference Lu and Petkova2014). Since the primary goal of this study is to construct a prediction model rather than to derive estimates of individual predictor effects, we believe that it is appropriate to apply the LASSO logistic model in the present context.
There are several limitations of this study. First, we noted that some of the subjects were missing information on one or more of the baseline predictors. Though the amount of missing data is very small, there is a possibility that the estimated prediction model, based on the combined complete and imputed data, is different from what the estimated prediction model would have been, had we had complete data on all subjects. We conducted several sensitivity analyses to assess the stability of the estimated model under various patterns for the missing covariates and found the estimated model was robust to the substitution of a range of realistic values for the missing data. Hence, we are not particularly concerned with the effects of missing data in this setting. Second, we used the COPE data to perform both model selection and estimation. Though we used LASSO-logistic regression with 5-fold cross-validation to avoid overfitting and a bootstrap procedure to compute optimism-adjusted measures of model performance so as not to suggest that the model performs better than it actually does, the real test of our model's performance will entail conducting a large study to assess how well the model classifies new CHR patients as converters or non-converters within 2 years of follow-up. Generalizability of the model would be supported if the CHR patients in the aforementioned study come from a diverse collection of subjects who represent the population of all CHR patients, not just those with characteristics represented in the present sample of 199 patients from NYSPI. Third, using LASSO-logistic regression to estimate the prediction model does not allow for a precision estimate of a single predicted risk score for a new CHR individual to whom we wish to apply our prediction model. Though, as pointed out by Cannon et al. (Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016), the individual predicted risk scores generated from the predictive model are more meaningful to the individual patient whereas the confidence and prediction intervals may not be of specific interest since they relate to likelihoods under future sampling.
In summary, this study demonstrates a practical and potentially clinically useful model that uses baseline data obtained exclusively from the SIPS and accompanying clinical interview to predict conversion to psychosis within 2 years of follow-up for CHR patients. The primary advantage of the proposed predictive model is that all of the information can be obtained by a single, 1–2 h interview, and does not require other lengthy assessments (including cognitive assessments) requiring special expertise (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016). We believe that our predictive model has potential as an aid for clinicians to assess prognosis in CHR patients and to assist in guiding clinicians in the selection of treatment. Our goal is to make our model available online in the form of a ‘risk calculator.’ However, before these clinical applications can be realized, the proposed model needs to be applied to and validated in a large independent set of CHR individuals, by our group as well as in other independent cohorts. Future work should focus on replicating this model in independent samples as well as determining if its effectiveness/accuracy can be enhanced by incorporating biological measures that are predictive of conversion to psychosis.
Acknowledgements
The project was supported by the NIH: Center for Research Resources and the National Center for Advancing Translational Sciences, UL1TR000040 and 2KL2RR024157; R21MH110700; K23MH101637; K23MH066279; R21MH086125; R01P50MH086385; R01MH093398-01; R01 MH107558, and the Brain and Behavior Research Foundation; Lieber Center for Schizophrenia Research; New York State Office of Mental Hygiene; and K23MH106746.
R. Girgis acknowledges receiving research support from Otsuka, Allergan/Forest, BioAvantex, and Genentech. S. Schobel reports that he is an employee for F. Hoffman-LaRoche. J. Lieberman has received support administered through his institution in the form of funding or medication supplies for investigator initiated research from Denovo, Taisho, Pfizer, Sunovion and Genentech, and for company sponsored phase II, III and IV studies from Alkermes and Allergan, and is a is a consultant to or member of the advisory board of Intracellular Therapies, Lilly, Pierre Fabre and Psychogenics. He neither accepts nor receives any personal financial remuneration for consulting, speaking or research activities from any pharmaceutical, biotechnology or medical device companies. He has received honoraria for serving on an advisory board for Clintara, a clinical research organization, and holds a patent from Repligen that yields no royalties. A. Ciarleglio, G. Brucato, M. Mascucci, R. Altschuler, T. Colibazzi, C. Corcoran, F. Crump, G. Horga, E. Lehembre-Shiah, W. Leong, L. Yang, and M. Wall report no competing interests.
