Introduction
The American Psychiatric Association (APA) has recently adopted a continuous improvement model for the Diagnostic and Statistical Manual of Mental Disorders (DSM) revisions and has established a new DSM web portal (http://www.dsm5.org) to receive and solicit comments on proposed changes (First, Reference First2016). There are five types of substantive proposals that can be submitted: type 1 proposals involve changes to an existing diagnostic criteria set, type 2 proposals involve the addition of a new diagnostic category, type 3 proposals entail deleting an existing category, type 4 proposals comprise clarifications and corrections, and type 5 proposals involve text changes (First, Reference First2016; Moran, Reference Moran2017). In this paper, the focus will be on type 1 DSM revisions where the primary concern is assessing proposed modifications to existing diagnostic criteria set.
There are at least two circumstances that warrant criteria modifications within the DSM – a conceptual problem with the criteria or identification of a superior or simpler way of defining a disorder (Zimmerman et al., Reference Zimmerman, Chelminski, McGlinchey and Young2006). An example of a conceptual problem that was previously modified can be seen in the transition from DSM-IV to DSM-5 for substance use disorders (SUDs), where two distinct diagnoses for abuse and dependence (in DSM-IV) were combined into one SUD (DSM-5). This change was supported by a large amount of empirical evidence including large-scale literature reviews and new data analyses conducted by Hasin et al. (Reference Hasin, O'Brien, Auriacombe, Borges, Bucholz, Budney, Compton, Crowley, Ling and Petry2013). This extensive work totaled 39 studies and included over 200 000 study participants in the USA and elsewhere, all indicating a single dimensional construct for SUDs (Hasin et al., Reference Hasin, O'Brien, Auriacombe, Borges, Bucholz, Budney, Compton, Crowley, Ling and Petry2013).
While the conceptual problem of DSM-IV's distinction between substance abuse and dependence was eliminated, this integration led to DSM-5 SUDs having 11 criteria, more than was previously used in DSM-IV dependence (7 criteria) or abuse (4 criteria). The problem of DSM criteria sets being too lengthy for clinicians and researchers to implement has been documented, particularly in major depressive disorder (MDD) with 9 criteria (Zimmerman et al., Reference Zimmerman, Chelminski, McGlinchey and Young2006). Even longer lists of criteria discourage clinicians from learning and using them. For example, when general practitioners were asked to list which criteria they used to diagnose depression, only one quarter listed at least five of the nine MDD criteria, while even after an educational program, only two-thirds indicated they used the formal diagnostic criteria (Krupinski and Tiller, Reference Krupinski and Tiller2001; Learman et al., Reference Learman, Gerrity, Field, van Blaricom, Romm and Choe2003).
Another potential problem with lengthy criteria sets is the increased heterogeneity of possible presentations being diagnosed. Combining the two diagnoses of dependence and abuse into one may have simplified overall diagnosis, but in fact, it increased the total number of criteria combinations that can lead to a diagnosis. Specifically, DSM-IV's dependence and abuse taken together had 1485 possible combinations of criteria that could result in a diagnosis, which increased in the DSM-5 to 2036 possible combinations. This problem of heterogeneity has been articulated well for post-traumatic stress disorder (PTSD), for which there are 636 120 different possible presentations, based on the requirement for a minimum of 8 of 19 possible criteria in DSM-5 (Galatzer-Levy and Bryant, Reference Galatzer-Levy and Bryant2013). Hence, lengthy criteria sets encompass more heterogeneous presentations that may limit the field's ability to target the disorder for research and treatment.
The development and investigation into short-form methodology is a substantial subfield of psychometrics and has contributed to many advances in test shortening and curtailment (e.g. Singh et al., Reference Singh, Matson, Mouttapa, Pella, Hill and Thorson2009; Bohlmeijer et al., Reference Bohlmeijer, ten Klooster, Fledderus, Veehof and Baer2011; Finkelman et al., Reference Finkelman, He, Kim and Lai2011, Reference Finkelman, Smits, Kim and Riley2012, Reference Finkelman, Smits, Kulich, Zacharoff, Magnuson, Chang, Dong and Butler2017; Smits and Finkelman, Reference Smits and Finkelman2015; Black et al., Reference Black, McCaffrey, Villapiano, Jamison and Butler2017; Gibbons et al., Reference Gibbons, Kupfer, Frank, Moore, Beiser and Boudreaux2017). More traditional ‘static’ test-shortening designs, and the focus of our methodology require all respondents to receive the same full set of items, with short-forms identified from the original full set using a specified methodology. For example, Finkelman et al. (Reference Finkelman, Smits, Kulich, Zacharoff, Magnuson, Chang, Dong and Butler2017) developed a short-form version of the 24-item Screener and Opioid Assessment for Patients with Pain-Revised (SOAPP-R). Their methodology first established a list of potential short-forms from the original 24 items through regression techniques and then identified an optimal 12-item short-form utilizing measures of diagnostic accuracy (i.e. maximize Youden J index among short-forms where sensitivity ⩾ specificity). Additional advances in short-form methodology are an alternative to traditional test-shortening designs and utilize CAT (computerized adaptive testing) that allows for customizing the assessment at the individual level (Lord, Reference Lord1980). Test shortening for computer-based assessments includes curtailment and stochastic curtailment, which are sequential analysis techniques that classify respondents into categories. These methods cease testing at the point at which additional items cannot or are unlikely to change the respondent's classification (Finkelman et al., Reference Finkelman, He, Kim and Lai2011, Reference Finkelman, Smits, Kim and Riley2012; Smits and Finkelman, Reference Smits and Finkelman2015; Gibbons et al., Reference Gibbons, Kupfer, Frank, Moore, Beiser and Boudreaux2017).
Given potential problems associated with lengthy criteria sets and previous advances in test shortening, the present study develops an empirically based method to systematically identify subsets of the criteria and associated diagnostic cut-offs (i.e. diagnostic short-forms) that will yield diagnoses as similar as possible to using the full-length criteria set. We propose an alternative method for static test shortening and apply it to develop the best diagnostic short-forms for alcohol use disorder (AUD) based on data collected on subjects who were evaluated on the full set of 11 DSM-5 AUD criteria. Our methodology can be used for other diagnoses and extends for use in other types of scale construction; however, AUD provides an ideal working example as it is an established and reliable (American Psychiatric Association, 2013; Hasin et al., Reference Hasin, O'Brien, Auriacombe, Borges, Bucholz, Budney, Compton, Crowley, Ling and Petry2013), fixed-length set of unidimensional criteria, and there is receptivity to improvements (Hales et al., Reference Hales, Yudofsky and Roberts2014; First, Reference First2016).
Methods
Data source
To develop empirically a diagnostic short-form derived from existing DSM criteria, a data source is needed with reliably measured criteria across all subjects. Ideally, to maximize generalizability, the sample population should include subjects ranging from very healthy non-disordered to mild, moderate and severely disordered. The sample also must be large enough to statistically delineate between different short-form alternatives. While many factors influence the statistical power of the screening steps described below, generally we recommend the margin of error for 95% confidence intervals should be small for the proportions (e.g. ±5%) and correlations (±0.10). Hence, we recommend samples with at least 300 individuals with the disorder and 300 without to achieve this precision.
In the present study, we use a large general population sample, the National Epidemiologic Survey on Alcohol and Related Conditions-III (NESARC-III) conducted by the National Institute on Alcohol Abuse and Alcoholism (NIAAA) from 2012 to 2013. The NESARC-III is a nationally representative sample of 36 309 non-institutionalized US civilians, 18 years or older (Grant et al., Reference Grant, Chu, Sigman, Amsbary, Kali, Sugawara, Jiao, Ren and Goldstein2014, Reference Grant, Goldstein, Saha, Chou, Jung, Zhang, Pickering, Ruan, Smith, Huang and Hasin2015a). Criteria for AUD were measured using the AUD and Associated Disabilities Interview Schedule (AUDADIS-5), which has been shown to have high reliability and validity (Hasin et al., Reference Hasin, Greenstein, Aivadyan, Stohl, Aharonovich, Saha, Goldstein, Nunes, Jung, Zhang and Grant2015; Grant et al., Reference Grant, Goldstein, Smith, Jung, Zhang, Chou, Pickering, Ruan, Huang, Saha, Aivadyan, Greenstein and Hasin2015b).
The analytic sample used comprises those NESARC-III participants who reported having at least one drink of an alcoholic beverage within the past year (n=25 778). Among current drinkers, 51% were male; the racial breakdown was: 69% White, non-Hispanic; 11% African American, non-Hispanic; 2% American Indian/Alaska Native, non-Hispanic; 5% Asian/Native Hawaiian/Other Pacific Islander, non-Hispanic; 14% Hispanic, any race. Table 1 lists each of the 11 DSM-5 alcohol use disorder criteria and their prevalence in this sample. Thirty-two percent of current drinkers met at least 1 criterion and, based on the current DSM-5 requirement for two or more out of 11 criteria being met, 19.1% (95% CI 18.5–19.7%) had AUD.
DSM-IV and DSM-5 diagnostic criteria for substance use disorders
a The order of AUD diagnostic criteria presented follows the order listed in the DSM-5 manual (National Institute on Alcohol Abuse and Alcoholism, 2015) and the conceptual domains provide useful groupings (Horvath et al., Reference Horvath, Misra, Epner and Cooper2016).
b DSM-IV Dependence diagnosed with three or more dependence criteria within a 12-month period.
c Craving was not present in DSM-IV and was added to the DSM-5 criteria.
d DSM-IV Abuse diagnosed with one or more abuse criteria within a 12-month period and no dependence diagnosis.
e Legal problems were in the DSM-IV abuse and were dropped as a DSM-5 diagnostic criterion.
Steps for identifying optimal diagnostic short forms
The goals of the procedure are to obtain a subset (or multiple subsets) of the original criteria set and associated diagnostic cut-offs that (1) maximize the association between the sum score of all original criteria with the newly constructed subscales from subsets of criteria; (2) optimize the similarity between the prevalence of diagnosis using the new diagnostic short form and the original DSM-5 rule; (3) maximize sensitivity and specificity of the newly constructed diagnostic short forms against the current DSM-5 rule; and (4) minimize any differences in the accuracy of the short-form across chosen covariates.
A flowchart of the empirical steps to accomplish these goals is shown in Appendix A (Figure A1), where steps 1–3 deal with choosing the optimal subset of criteria (i.e. subscale) and steps 4–8 focus on determining the optimal threshold. Within the procedure, there are screening steps that make comparisons between different short-forms or comparisons to a pre-fixed value. When making comparisons, 95% confidence intervals are derived with non-parametric bootstrapping (Efron and Tibshirani, Reference Efron and Tibshirani1986), so that random variability is statistically taken into account when making conclusions. A brief description of the procedure for each step is provided below and then specific details applying the method to DSM-5 AUD are provided in the results section.
Step 1: Assess the dimensionality of full-length criteria set
A common diagnostic algorithm structure within DSM-5 is to have a single list of P different criteria and determine that a person is diagnosed with the disorder if they have at least M out of those P criteria present. This method of counting P criteria implicitly assumes that the P items represent a unidimensional construct (Bond and Fox, Reference Bond and Fox2007). Bollen and Lennox (Reference Bollen and Lennox1991) demonstrated that when a criteria set can be considered a reflective measure of a unidimensional construct, any subsets of those items are essentially interchangeable. Thus, if the original set of criteria are unidimensional, then subscales constructed from the full set are unidimensional. To verify unidimensionality, we consult prior literature as well as examine the eigenvalues obtained from tetrachoric correlations of the full-length criteria set using the Kaiser criterion (Nunnally and Bernstein, Reference Nunnally and Bernstein1994) and the model fit indices for a confirmatory one-factor model (i.e. root-mean-square error of approximation (RMSEA) ⩽0.06 and comparative fit index (CFI) ⩾0.95) (Hu and Bentler, Reference Hu and Bentler1999). Dimensionality analyses were conducted in Mplus v7.3.
However, if the dimensionality of the full-criteria set is determined to be greater than one, this would suggest that the original set of items should be organized into separate sets of items corresponding to each dimension. In the case of DSM-5, this separation of criteria sets by domains or dimensions exists for several disorders (e.g. ADHD, PTSD). While additional considerations beyond the scope of the present method may be warranted for how best to combine across dimensions, the currently proposed method could be applied to each of the distinct subsets of criteria to identify short-forms that represent each of the separate dimensions of the disorder.
Step 2: Form all possible subscales from subsets of full-length criteria set
Given P criteria in the full-length criteria set, the number of subsets of size P, where p ⩽ P, is
$$\left( {\displaystyle{P \over p}} \right) = \displaystyle{{P!} \over {\,p!\left( {P - p} \right)!}}\;. $$Thus, considering subsets of all possible sizes from p = 1 up to the full-length criteria set P we have,
$$\mathop \sum \limits_{\,p = 1}^P \left( {\matrix{ P \cr p \cr}} \right) = \mathop \sum \limits_{\,p = 1}^P \left( {\displaystyle{{P!} \over {\,p!\left( {P - p} \right)!}}} \right),\; {\rm where\;} p = 1,\; \ldots, \; P$$different subsets of items. For every subset of items, we then create corresponding subscales by taking the simple sum score of the respective items in the subset for each individual in the dataset.
Step 3: Identify optimal subscales using correlation with full-length criteria set
Calculate the correlation and 95% confidence interval between each subscale from step 2 and the sum score of the full-length criteria set. The motivation is that subscales which have higher correlations with the full-length scale at this step, will lead to diagnostic short forms constructed in the next step that are most similar to the full-length criteria set. Depending on the goals of the researcher, it may be desirable to identify diagnostic short forms of each size p < P. The current method identifies the top k ranked subscales for each size P and any other subscales that are not significantly different from them. Different types of correlation can be considered (e.g. Pearson, Spearman, and polychoric) but generally do not differ in terms of how they rank order subscales.
Step 4: Form all possible diagnostic short forms from optimal subscales
Once optimal subscales have been identified across all P where p = 1, …, P − 1, the next step is to introduce the threshold component (m) that allows for the subscales to be cut at a specified point to determine whether a diagnosis should be made. To construct an exhaustive list of all possible diagnostic short-forms from the optimal subscales in the previous step, we let M vary from m = 1 to p for each subscale.
Step 5: Identify optimal diagnostic short-forms using comparative prevalence estimates
Calculate prevalence estimates of disorder and 95% confidence intervals for each of the newly constructed diagnostic short-forms from step 4. Screen in all those that fall within a small percentage point difference of the full-length rule prevalence. The choice of how small or how large to allow the percentage point difference to be depended on the goal of the clinician/researcher.
Step 6: Identify optimal diagnostic short-forms using sensitivity/specificity estimates
Taking the DSM-5 diagnosis to be the target, we next examine the sensitivity and specificity of each diagnostic short-form (DSF) against that target, i.e. sensitivity = Pr(DSF + |DSM5+), specificity = Pr(DSF – |DSM5–). In general, there is a trade-off between sensitivity and specificity such that the more sensitive a diagnostic short-form, the higher the rate of false positives (i.e. the lower the specificity) and vice versa. For screening in diagnostic short-forms, a Youden's index (Sensitivity + Specificy − 1) above a certain cut-off can be used, or simply diagnostic short-forms with both sensitivity and specificity estimates higher than a cut-off (e.g. 0.90) can be kept. Hence, optimal diagnostic short-forms will be identified when 95% confidence intervals of estimates of sensitivity and specificity both contain values greater than or equal to 0.90. This cutoff value can be adjusted or more weight can be placed on either measure depending on the goals of the clinician/researcher.
Step 7: Identify optimal diagnostic short-forms minimizing differential functioning across covariates
A desirable psychometric property for any measurement tool is that it should not exhibit differential test functioning (DTF) (Raju et al., Reference Raju, van der Linden and Fleer1995; Kleinman and Teresi, Reference Kleinman and Teresi2016). That is, it should not exhibit systematic differences in the way it measures the target construct in different subgroups than what is currently being captured by the full-length rule. In our setting, we are concerned with reducing the systematic differences with the entire diagnostic short-form rather than with each item within the short-form. Inherently, there may be systematic differences in the way a specified item behaves in certain subgroups [e.g. ‘Social/interpersonal problems’ (C6) could depend on cultural differences across races]. However, we are concerned with assessing DTF of the entire measure rather than specific item-level DIF, because taken as a whole, these item-level differences can cancel each other out.
To assess DTF among the diagnostic short-forms, we performed logistic regression with the diagnostic short-form as the outcome predicted by the full-length diagnostic rule (i.e. DSM-5) and relevant covariates (e.g. age, sex, race, diagnosis of any mood and/or anxiety disorder). If any covariate is found to be significant at the α = 0.05 level, this indicates that the log odds of the diagnostic short-form being positive (or negative) after controlling for the DSM-5 being positive (or negative) differs across levels of the covariate and hence is less desirable and is screened out since it exhibits DTF. The choice of covariates to examine depends on the goals of the clinician/researcher based on the relevant subgroups in the population.
Step 8: Provide a final list of optimal diagnostic short-forms and comparative information for decision making
A list of optimal diagnostic short-forms that successfully satisfy each optimization step is provided, along with estimates and 95% confidence intervals for correlation, prevalence, sensitivity, and specificity.
Extension of steps to take into account severity cut-offs
There has been recent movement towards a dimensional approach to psychiatric disorders (Kozak and Cuthbert, Reference Kozak and Cuthbert2016; Kose and Cetin, Reference Kose and Cetin2017) and DSM-5 AUD specifically indicates a severity spectrum with four categories: None (0–1 criteria), Mild (2–3 criteria), Moderate (4–5 criteria), and Severe (6–11 criteria). Our new diagnostic short-form identification methodology easily extends to identify optimal severity measures. Rather than compare each potential diagnostic short-form to the current DSM-5 rule for AUD (2 + criteria), we can re-run our algorithm two additional times to obtain the optimal diagnostic short-forms for Moderate AUD (4 + criteria) as well as Severe AUD (6 + criteria). Once the optimal diagnostic short-forms for each severity level are obtained, we can identify those that overlap across each run of the algorithm and assess the agreement to the current severity dimensions for DSM-5 AUD.
Replicability on split halves and on a different data source
To demonstrate replicability of our results, we (1) re-ran the empirical steps on random split halves of the NESARC-III data source and (2) re-ran the empirical steps on a new data source (Wave 2 NESARC) and then compared results. In 2004–2005, approximately three years after the initial NESARC (Wave 1), n = 34 653 of the Wave 1 participants were re-sampled in Wave 2 representing a cumulative response rate of 86.7% (Ruan et al., Reference Ruan, Goldstein, Chou, Smith, Saha, Pickering, Dawson, Huang, Stinson and Grant2007). Of note, NESARC-III was a survey conducted in 2012–2013 on a completely different sample of participants than the sample included in NESARC Waves 1 and 2. We applied our new method to the subsample of Wave 2 NESARC participants who reported they were current drinkers (n = 22 177). Participants’ current drinker status was defined in the same manner as in NESARC-III and the 11 criteria for DSM-5 AUD were measured with only minor differences (see Appendix B and C for survey questions for NESARC-III and Wave 2 NESARC, respectively). Note that the ‘Craving’ (C4) criterion was not available on the first wave of the NESARC, which is why that data source was not also used.
Results
Results from each step using the NESARC-III data
Step 1: The DSM-5 AUD criteria set previously has been shown to be unidimensional in Wave 2 NESARC (Saha et al., Reference Saha, Chou and Grant2006) and other samples (Hasin et al., Reference Hasin, O'Brien, Auriacombe, Borges, Bucholz, Budney, Compton, Crowley, Ling and Petry2013) and we found similar results for the NESARC-III sample. The first eigenvalue of the tetrachoric correlation is 8.16 while the rest are <0.55 and the one factor model has adequate model fit by CFI ⩾0.95 (CFI = 0.998) and RMSEA ⩽0.06 (RMSEA = 0.013), all indicating unidimensionality.
Step 2: A total of 2046 possible subsets of criteria of varying size, where p = 1, …, 10 were enumerated (Table 2) and subscales created from them.
Subscales selected and diagnostic short-forms constructed for DSM-5 AUD
Step 3: We calculated the Pearson correlations between all 2046 subscales and the full-length criteria set. As expected, on average, the correlation coefficients increase as the number of criteria included in the subset increases (Appendix Figure A2). Nevertheless, variability within each set of subscales indicates some subscales with smaller P are better than others with larger P. Optimal subscales were identified within each P by selecting the one with the highest correlation (i.e. k = 1) and all other subscales that were not significantly different from it, resulting in 86 subscales that pass this first screening step (Table 2, column 3). The correlations between the top chosen subscales at each size P and the full 11 criteria sum score range from 0.718 to 0.998 (Table 2, column 4).
Step 4: All possible diagnostic short-forms that can be constructed from each of the 86 optimal subscales from step 3 are then created and enumerated, totaling 401 DSFs (Table 2, column 5).
Step 5: We next assess the differences in the prevalence of diagnosis between each diagnostic short-form and DSM-5 AUD. Because the prevalence of AUD using the DSM-5 is 19.1% in this sample of current drinkers, we implemented a tolerance of ±5% as the threshold for the difference, although this could easily be made smaller or larger. For a disorder with lower prevalence, a smaller tolerance for the difference may be desirable. There were 72 out of the 401 DSFs screened in by exhibiting an absolute difference in prevalence <5% compared with the DSM-5 AUD (Appendix Figure A3). Of note, the optimal diagnostic short-forms with smaller P (p = 1, 2, 3, 4) were found to have a threshold of m = 1, whereas the larger diagnostic short-forms (p = 6, 7, 8, 9, 10) had a threshold of m = 2. These findings show the impact of maintaining prevalence of disorder at ±5% on the threshold; when fewer criteria are included, the optimal threshold requires fewer criteria compared to when more criteria are included.
Step 6: Treating DSM-5 AUD as the target diagnosis, the sensitivity and specificity for each of the 72 diagnostic short-forms were calculated; 25 DSFs had confidence intervals for sensitivity and specificity both >0.90 (Appendix Figure A4). Of note, all diagnostic short-forms with a threshold of m = 2 have 100% specificity due to DSM-5 also having a threshold of m = 2, and thus there can be no false positives. Also, there are diagnostic short forms with >90% sensitivity and specificity that have a threshold of m = 1 identified when p = 4, 5. We reviewed the literature on the diagnostic accuracy of other alcohol screening tools (i.e. CAGE, TWEAK, AUDIT) to determine the 90% cut-off used in this step (Cherpitel, Reference Cherpitel1995; Bush et al., Reference Bush, Kivlahan, McDonell, Fihn and Bradley1998).
Step 7: The distribution of the covariates examined in relation to DTF were: age (52.0% of sample <45 years old), sex (50.7% male), race/ethnicity (80.8% White, Hispanic/non-Hispanic), and diagnosis of any mood and/or anxiety disorder (22.3%, with rule out on substance-induced). Among the 25 optimal diagnostic short-forms considered at this step, four were highly accurate (i.e. with sensitivity >97% and specificity = 100%) with p = 8, 9, and 10 and were bypassed from explicit testing as there was quasi- or complete separation in the logistic model due to there being no variability left between the diagnostic short-form and the DSM-5 to be explained by covariates. Of the remaining 21 optimal diagnostic short forms, there were 14 with p = 4 criteria, only two of which did not have any differential functioning across levels of the specified covariates (Appendix Table A1). Of the four p = 5 and 6 diagnostic short-forms, none screened past this step. One of the two p = 7 diagnostic short-forms and one p = 8 diagnostic short-form was found not to exhibit any differential functioning. Hence in total, eight of the 25 optimal diagnostic short-forms passed this screening step.
Step 8: The final eight optimal diagnostic short-forms are shown in Fig. 1 and the associated descriptive statistics (i.e. correlations, prevalence, sensitivity, specificity) in Table 3 (and Figure A5). Optimal rules were found for size p = 4, 7, 8, 9, and 10 out of the 11 original criteria, with a threshold of m = 1 for the p = 4 rule, and m = 2 for the rest. Nearly perfect sensitivity and specificity were found for the p = 8, 9, and 10 rules, with a p = 4 rule that reached 92% sensitivity and specificity. Among the diagnostic short-forms identified for DSM-5 AUD using NESARC-III, ‘Neglected major roles to use’ (C5) and ‘Activities given up to use’ (C7) are the first two criteria to be excluded. ‘Used larger amounts/longer’ (C1), ‘Repeated attempts to quit/control’ (C2), ‘Craving’ (C4), ‘Hazardous use’ (C8), ‘Physical/psychological problems’ (C9), ‘Tolerance’ (C10), and ‘Withdrawal’ (C11) are consistently included in the optimal diagnostic short-forms, which may indicate their primary importance to the diagnosis of AUD.
Criteria included/excluded in the final eight optimal diagnostic short-forms.
Fit statistics with 95% CIs on optimal diagnostic short-forms for DSM-5 AUD
a DSM-5 AUD prevalence in sample is 0.191 (95% CI 0.185–0.197).
The number after each DSF is a label identifier given by the algorithm when all original subscales are enumerated and do not encode any specific information. p = # of criteria, m = threshold.
Diagnostic short-forms for severity of AUD
When the methodology was re-run using Moderate (4+) and Severe (6+) DSM-5 diagnosis as the targets, the same p = 9 subscale that excluded both ‘Neglected major roles to use’ (C5) and ‘Activities given up to use’ (C7) was consistently identified as optimal. Based on only these 9 criteria, the optimal threshold for Moderate and Severe was found to be 4+ and 5+ criteria, respectively. Appendix Table A2 provides the concordance of the NESARC-III participants in each AUD Severity group using the DSM-5 AUD Severity rule compared to the optimal one, with p = 9corresponding to DSF 89. The agreement has a weighted κ = 0.97, which indicates an almost perfect level of agreement (Viera and Garrett, Reference Viera and Garrett2005). Furthermore, there are only 2% misclassified participants, most being persons who were Moderate according to DSM-5 classified as Severe by the short-form. These findings demonstrate that the methodology can be extended to identify diagnostic short-forms that perform well not just for diagnosis but also for disorder severity.
Replicability
Demonstrating the performance of the method on random split halves of the data
Using simple random sampling, we split the NESARC-III data into two datasets: split half 1 and 2 (n = 12, 889 for each) (Appendix Table A3). All eight of the optimal diagnostic short-forms identified when using the full NESARC-III data are found across the two split halves (Appendix Figure A6). Additionally, we find the same pattern of criteria inclusion/exclusion that was found when using the full NESARC-III data.
Demonstrating the performance of the method on Wave 2 NESARC
Applying our proposed method to Wave 2 NESARC, we identified 16 optimal diagnostic short-forms for DSM-5 AUD (see Appendix Table A3 for descriptive statistics and Appendix Figure A7 for criteria selected). Again, we find the same patterns of criteria being included/excluded as we found using NESARC-III. The same five diagnostic short-forms were identified in Wave 2 NESARC that were identified in NESARC-III when p ⩾ 8. Further, these five diagnostic short-forms identified across both sets of data all performed very well, with ⩾0.987 correlation with total, ⩾96.4% sensitivity and 100% specificity.
Discussion
Our data-driven method can be used to shorten DSM criteria sets for those disorders that are diagnosed by having a certain number of criteria present (e.g. m out of p) based on a simple count of criteria. The method searches the space of all possible diagnostic short-forms and narrows down the selection based on specific optimization steps. The method has been programmed as a SAS Macro (Appendix D) that can be applied to any dataset with any number of criteria. In our application to DSM-5 AUD using NESARC-III, there were more than 11 000 possible diagnostic short-forms that could be created from the 11 criteria; using our method we were able to narrow the possible choices down to eight. Moreover, the same diagnostic short forms were found with 8, 9, and 10 criteria when the method was applied to Wave 2 of the NESARC and in split halves of the NESARC-III, demonstrating very strong replicability of the method and findings.
Our primary finding for AUD is that excluding ‘Neglected major roles to use’ (C5) and/or ‘Activities given up to use’ (C7) created no marked change in ability of these diagnostic short-forms to diagnose AUD or indicate its severity when compared to the current DSM-5 diagnostic criteria or to DSM-5 severity criteria (κ = 0.97). Further, our method was able to identify two diagnostic short-forms for AUD with only four criteria (p = 4) and a threshold of one (m = 1) that performed very well, with prevalence estimates near that of DSM-5 AUD (21% and 25% v. 19.1%), correlations above 0.90 with the total sum score using all 11 criteria, and measures of sensitivity and specificity higher than 90%.
The final selection of optimal diagnostic short-forms is dependent on the optimization parameters fixed within the steps which can be adjusted depending on the goal of the user. These decisions will affect the number of diagnostic short-forms that make it to the subsequent steps in the algorithm. For example, we accept those diagnostic short-forms that yield prevalence in the sample within 5% of the DSM-5 AUD diagnosis prevalence. Depending on the importance of maintaining current prevalence, this range can be adjusted to be more liberal or conservative in terms of how many diagnostic short-forms are selected to move to the next step. We consider this flexibility to be a strength of the methodology and would argue that for the purpose of making recommendations for changes to the DSM-5, specific standards could be set by a governing body. A limitation of the current algorithm is that it only allows binary covariates to be tested for DTF in step 7; though this can easily be extended in future versions of the method. We used a median split to dichotomize continuous age, which is the most common method for dichotomization (MacCallum et al., Reference MacCallum, Zhang, Preacher and Rucker2002).
Although we were able to replicate our findings in a separate general population sample, we would recommend that before any changes are suggested to the DSM-5, these results be replicated on clinical samples. It is also important to point out that each step in our methodological process uses some type of comparison to the current DSM-5 rule as if it is the best diagnostic target for AUD. Hence, in terms of recommendations, we cannot say that we are deriving a more valid criteria-based definition of AUD than the DSM-5, because implicitly the method is assuming the current DSM-5 rule for AUD is already the most valid we have. Future work could consider inclusion of external validators of AUD to act as surrogates to the full set of criteria.
Constructing short-forms has been criticized on various grounds including the problem that removing items has the potential to narrow the content, leading to an inadequate representation of the breadth of the construct (Smith et al., Reference Smith, McCarthy and Anderson2000). For AUD we found that diagnostic short-forms excluding ‘Neglected major roles to use’ (C5) and/or ‘Activities given up to use’ (C7) perform as well as the current rule in terms of correlation, diagnostic accuracy, and prevalence. The 11 AUD criteria have been described as falling into four conceptual domains (Table 1) (Horvath et al., Reference Horvath, Misra, Epner and Cooper2016) and excluding these two criteria still allows for at least one criterion to be included from each of these four domains. An additional criticism of short-form creation is the reliance on using the same data source twice which can lead to overlapping variance from correlated measurement error. At the correlation step 3, we incorporate 95% confidence intervals when screening but in future work could consider applying Levy's (Reference Levy1967) correction to account for the overlap of measurement errors using the same data source (Bohlmeijer et al., Reference Bohlmeijer, ten Klooster, Fledderus, Veehof and Baer2011). Importantly though, we were able to demonstrate in a separate data set the good performance of the short-forms hence reducing concerns that they were unduly influenced by spurious correlations.
Our proposed methodology differs from other test-shorting methods. Some methods have used regression techniques like LASSO (Black et al., Reference Black, McCaffrey, Villapiano, Jamison and Butler2017; Finkelman et al., Reference Finkelman, Smits, Kulich, Zacharoff, Magnuson, Chang, Dong and Butler2017) to identify shorter item sets that best predict an externally measured outcome but this does not apply in our context without external variables and does not necessarily produce optimal rules based on simple sum (non-weighted) scores. Other methods for shortening have used factor analysis techniques such as EFA and CFA (Singh et al., Reference Singh, Matson, Mouttapa, Pella, Hill and Thorson2009; Bohlmeijer et al., Reference Bohlmeijer, ten Klooster, Fledderus, Veehof and Baer2011) but these are limited to continuous scale construction and not optimizing diagnostic ability.
Each of the optimization steps in our proposed methodology is not in themselves new ideas, but the order in which we implement them in our algorithm, the combination of all steps as a macro, and the flexibility we allow provides a novel approach to investigating possible short-form alternatives. With a changing environment for DSM revisions, criteria selection methods need to evolve as well. Our intention with this new methodology is to provide a data-driven tool that identifies diagnostic short-form alternatives that perform as well as the current rule in a systematic and reproducible way.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291718001551.
