Quantitatively derived dimensional models of psychopathology enjoy overwhelming empirical support, and a large and active community of psychopathology researchers has been establishing an empirically based dimensional hierarchical taxonomy of psychopathology (or HiTOP) as a strong candidate replacement for the current categorical classification system (Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017). The hierarchical nature of this taxonomy implies that different levels of detail are likely to be optimal for different research and clinical purposes (e.g. Zald and Lahey, Reference Zald and Lahey2017).
This implication is the focus of the current study. We begin with an overview of hierarchical dimensional models of psychopathology. We then discuss the implications of this hierarchical framework for questions of validity and relevant research and clinical applications. We conclude by describing the current study, in which we test the relative validity of different levels of a hierarchical model of diagnostic covariance in a large sample of psychiatric outpatients.
Hierarchical dimensional models of psychopathology
Categorical diagnostic systems imply that mental illnesses are qualitatively distinct (a) from normal-range functioning and (b) from one another (Andrews et al., Reference Andrews, Goldberg, Krueger, Carpenter, Hyman, Sachdev and Pine2009). However, there is overwhelming evidence that the reliability and validity of these diagnostic entities are inadequate for many scientific purposes and of limited clinical utility in many cases (e.g. Haslam, Holland, and Kuppens, Reference Haslam, Holland and Kuppens2012; Widiger and Samuel, Reference Widiger and Samuel2005; Wright et al., Reference Wright, Krueger, Hobbs, Markon, Eaton and Slade2013). In addition, psychiatric diagnoses co-occur in individuals far more often than not (Andrews, Slade, & Issakidis, Reference Andrews, Slade and Issakidis2002; Bijl, van Zessen, Ravelli, de Rijk, & Langendoen, Reference Bijl, van Zessen, Ravelli, de Rijk and Langendoen1998; Grant et al., Reference Grant, Stinson, Dawson, Chou, Dufour, Compton and Kaplan2004; Kessler, Chiu, Demler, & Walters, Reference Kessler, Chiu, Demler and Walters2005). This suggests that common liabilities are represented across multiple diagnoses, such that it would be far more effective and parsimonious to address these liabilities instead (Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017).
Research using diagnostic covariance to model the implied common liabilities (e.g. Eaton, Rodriguez-Seijas, Carragher, and Krueger, Reference Eaton, Rodriguez-Seijas, Carragher and Krueger2015) has served as the basis for the development of dimensional factor-analytic models of psychopathology metastructure (Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017). These latent liability dimensions improve upon diagnoses in terms of reliability and utility and in facilitating research on mechanisms of psychopathology (Kim & Eaton, Reference Kim and Eaton2015; Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017; Krueger, Caspi, Moffitt, & Silva, Reference Krueger, Caspi, Moffitt and Silva1998; Markon, Reference Markon2015; Sanislow, Reference Sanislow2016). Importantly, these dimensions dramatically outperform diagnoses in predicting key clinical outcomes such as recovery and role functioning (Martin et al., Reference Martin, Jonas, Lian, Foti, Donaldson, Bromet and Kotov2021).
Research on dimensional models of psychopathology focused initially on determining the optimal number of latent dimensions, related to a longstanding debate in individual-differences research between emphasizing broad (‘lumping’) v. narrow (‘splitting’) constructs (e.g. Carragher, Krueger, Eaton, and Slade, Reference Carragher, Krueger, Eaton and Slade2015; Krueger, Reference Krueger2005). However, contemporary metastructural research since has shifted to organizing psychopathology hierarchically, with the understanding that different levels of resolution are likely to be optimal for different research and clinical questions (Kotov et al., Reference Kotov, Foti, Li, Bromet, Hajcak and Ruggero2016; Krueger, Reference Krueger2005; Krueger & Piasecki, Reference Krueger and Piasecki2002). This then raises the question of how to identify the optimal level of detail for a given purpose. For this type of methodological work, there are significant advantages to using sequential exploratory factor or principal component analysis (commonly referred to as bass-ackwards analysis; Goldberg, Reference Goldberg2006), because this method allows for relatively straightforward comparison of the validity of dimensions at different levels of hierarchy (Michelini et al., Reference Michelini, Barch, Tian, Watson, Klein and Kotov2019).
Implications of a hierarchical framework for research and clinical applications
There is already a small body of research using hierarchical regression to compare the relative validity of different levels of bass-ackwards models of psychopathology in terms of significant R 2 change. Kotov et al. (Reference Kotov, Foti, Li, Bromet, Hajcak and Ruggero2016) computed a 12-level bass-ackwards model of psychosis symptoms using data from patients with history of hospitalization for psychosis. They determined that a four-factor solution (reality distortion, disorganization, inexpressivity, and apathy/asociality) provided the optimal level of detail for predicting concurrent and future dysfunction. Using a similar approach in community adults diagnosed with eating disorders, Forbush et al. (Reference Forbush, Hagan, Kite, Chapa, Bohrer and Gould2017) found that a 15-factor solution in a bass-ackwards model of internalizing and eating pathology symptoms explained the most variance in eating disorder-related clinical impairment; moreover, this level explained substantially more variance than did a multiple regression model using mood, anxiety, and eating disorder diagnoses as predictors. In a bass-ackwards model of psychopathology data from 9- and 10-year old children, Michelini et al. (Reference Michelini, Barch, Tian, Watson, Klein and Kotov2019) found that general psychopathology (i.e. a one-factor solution) explained the bulk of variance in familial aggregation of psychopathology, whereas the lowest level of their model (a five-factor solution comprising internalizing, detachment, somatoform, neurodevelopmental, and externalizing dimensions) provided maximal explanatory power with regard to most other social, cognitive, and academic functioning indices. These studies represent significant contributions to the literature on the validity of hierarchical dimensional models, while also highlighting the need for further research in this area that includes (a) a wider range of psychopathology coverage and (b) data from diagnostically heterogeneous clinical samples to improve generalizability.
Generalizability is of particular importance here, because these validation efforts have downstream implications for the translation of hierarchical dimensional models of psychopathology into assessment tools for research and clinical applications. For example, if a relatively broader level of detail (e.g. overarching general psychopathology, or internalizing/externalizing) is optimal (i.e. provides maximal validity) for a given purpose (e.g. identifying need for more intensive services), scale development and selection should focus on strong measurement of those broader domains, as measurement at a more finely grained level would be a poor use of time and effort. On the other hand, if a relatively fine-grained level of detail is optimal, that suggests that each of these narrower dimensions may be providing different and important pieces of information. It would then follow that it is worth the time and effort to develop and implement measurement at that level, as measurement at a broader level would be missing important information. These considerations are likely to be particularly valuable for research designs and clinical settings for which lengthy assessments are not feasible (e.g. ecological momentary assessment studies, measurement-based care). In these cases, the utility of dimensional models of psychopathology requires knowledge about the most appropriate level of measurement for the purpose at hand so as to strike the right balance between comprehensiveness and brevity.
The current study
Investigations into the optimal level of hierarchy for a given purpose are a necessary link between the articulation of quantitative dimensional models of psychopathology and the application of these models to relevant research and clinical questions (including more detailed examinations of the concurrent and predictive validity of individual dimensions). As noted, current research in this area is limited in terms of (a) breadth of psychopathology coverage and (b) generalizability to adult clinical populations, which merits a focus on foundational methodological questions (i.e. the optimal level of hierarchy for relevant clinical variables) using data that can address these gaps.
In the current report from the Rhode Island Methods to Improve Diagnostic Assessment and Services (MIDAS) project, we use data from a sample of 2900 psychiatric outpatients to compare different levels from bass-ackwards analysis in relation to indices of psychosocial morbidity. Our chief aim was to identify which level of hierarchy is likely to provide optimal validity and explanatory power with regard to relevant clinical variables, thus contributing to a necessary bridge between quantitative dimensional models of psychopathology and research and clinical applications.
Method
Sample
Data come from the first wave (n = 2900) of the MIDAS project (see Zimmerman, Reference Zimmerman and First2003, Reference Zimmerman2016 for additional details), which is part of an ongoing integrated research and clinical program at a community-based outpatient psychiatric practice. Participants ranged in age from 18 to 85 (M = 38.5, s.d. = 13.0), and the sample was predominately female (61.0%; 39.0% male) and white (87.5%; 4.4% Black, 2.6% Hispanic/Latinx, 0.9% Asian, 4.5% other). The research protocol was approved by the Rhode Island Hospital institutional review committee. All participants provided informed written consent. Potential participants were excluded only if they were younger than 18 years, were unable to understand English, or exhibited severe cognitive impairment (e.g. dementia).
Variables
Interviewers made lifetime Diagnostic and Statistical Manual for Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000) diagnoses using the Structured Clinical Interview for DSM-IV (SCID; First, Spitzer, Gibbon, and Williams, Reference First, Spitzer, Gibbon and Williams1995) for clinical disorders and the Structured Interview for DSM-IV Personality (SIDP-IV; Pfohl, Blum, and Zimmerman, Reference Pfohl, Blum and Zimmerman1997) for personality disorders (PD; see, e.g. Zimmerman, Reference Zimmerman and First2003 for further details regarding interviewer training and diagnostic reliability). In line with other metastructural research using the first wave of MIDAS data (Forbes et al., Reference Forbes, Kotov, Ruggero, Watson, Zimmerman and Krueger2017; Kotov et al., Reference Kotov, Ruggero, Krueger, Watson, Yuan and Zimmerman2011), thresholds for SIDP-IV PD diagnoses were relaxed to require one criterion less than DSM-IV-TR thresholds, and our analyses included only the diagnoses that (a) were modified to ignore hierarchical exclusion rules, (b) were frequent enough (⩾20 cases) to provide reliable covariance estimates, and (c) correlated <0.80 with all other diagnoses (see Table 1).
Lifetime DSM-IV diagnoses
Note: n = 2900. Percentages estimated for available cases. Personality disorders include cases with one less criterion than DSM-IV-TR thresholds. Individuals could be given more than one diagnosis.
a Excluded from main analyses due to redundancy (r ⩾ 0.80) with social phobia.
This diagnostic interview also included several ratings of psychosocial morbidity and functioning, many of which came from the Schedule for Affective Disorders and Schizophrenia (Endicott and Spitzer, Reference Endicott and Spitzer1978). Clinicians used the DSM-IV Global Axis of Functioning (GAF) scale to rate overall functioning for each participant on a scale of 0–100. Based on patient report, clinicians rated patients on the following: suicidal ideation over the past two weeks (0 = not at all, 6 = very extreme), social functioning over the past five years (1 = superior, 7 = grossly inadequate), time missed from work due to psychiatric problems over the past five years (1 = virtually no time at all, 9 = worked none or practically none of the time; those not expected to work for reasons unrelated to psychopathology were treated as missing), lifetime history of inpatient psychiatric hospitalizations (coded as 0, 1, 2, 3, 4, or 5+), and lifetime history of suicide attempts (coded as 0, 1, 2, 3, 4, or 5+).
Analyses
Analyses were conducted using the R psych package (Revelle, Reference Revelle2022). We constructed a bass-ackwards model (Goldberg, Reference Goldberg2006) of the diagnoses using a series of principal component models with increasing numbers of components and correlating component scores across levels. Following Forbes et al.'s (Reference Forbes, Kotov, Ruggero, Watson, Zimmerman and Krueger2017) bass-ackwards model of these data, we extracted seven levels of hierarchy. In line with other bass-ackwards research (Forbes et al., Reference Forbes, Kotov, Ruggero, Watson, Zimmerman and Krueger2017; Loehlin & Goldberg, Reference Loehlin and Goldberg2014), we interpreted very high cross-level correlations (>0.90) as indicating the perpetuation of the same dimension across levels, cross-level correlations between 0.30 and 0.90 as indicating lower order dimensions nested within higher order dimensions, and relatively small correlations (<0.30) with all preceding dimensions as indicating the emergence of novel (rather than nested) dimensions.
We ran a series of hierarchical regression models to test for incremental increases in R 2 at more complex levels. Each step of the model contained all components from the level being tested, as well as all components from preceding levels. Because significant increases in R 2 can occur simply due to adding more variables, we also computed adjusted R 2 change at each step. All regression models controlled for age and sex. We then examined the unique variance accounted for by each dimension within the optimal level of hierarchy.
Results
The bass-ackwards model
Our bass-ackwards results were highly consistent with similar models of these data (Forbes et al., Reference Forbes, Kotov, Ruggero, Watson, Zimmerman and Krueger2017). Level 1 General Psychopathology split into Broad Internalizing [e.g. social phobia, generalized anxiety disorder (GAD), paranoid PD] and Externalizing [e.g. antisocial PD, histrionic PD (HPD)] at level two. At level 3, Broad Thought Disorder [e.g. psychosis, schizotypal PD (STPD)] emerged as a novel dimension (correlated <0.30 with both level 2 dimensions, strongest correlation was 0.31 with level 1 General Psychopathology). At level 4, Broad Internalizing split into Core Internalizing (e.g. GAD, major depression) and Somatoform (e.g. somatoform disorder, pain disorder). At level 5, Broad Thought Disorder split into Detachment (e.g. schizoid PD, STPD) and Core Thought Disorder (e.g. mania, psychosis). At level 6, Externalizing split into Antagonism (e.g. HPD, narcissistic PD) and Disinhibition (e.g. alcohol and substance use disorders). At level 7, Compulsivity (e.g. obsessive-compulsive PD) emerged as a novel dimension (correlated <0.30 with all preceding dimensions, strongest correlation with level 6 was 0.20 with Antagonism). See Fig. 1 and online Supplementary Tables S1–S7.
The bass-ackwards model. Component score correlations ⩾0.30 between adjoining levels are depicted.
The definition of each component was generally consistent with the overall HiTOP model (Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017) and recent meta-analytic work synthesizing metastructural analyses (Ringwald, Forbes, & Wright, Reference Ringwald, Forbes and Wright2021). However, it is noteworthy that posttraumatic stress disorder (PTSD) split between Internalizing and Thought Disorder at lower levels of the model. Although PTSD's location on Thought Disorder is unusual in metastructural research, this result is consonant with other literature on the overlap between PTSD and the psychosis spectrum (e.g. Levin-Aspenson, Watson, Ellickson-Larew, Stanton, and Stasik-O'Brien, Reference Levin-Aspenson, Watson, Ellickson-Larew, Stanton and Stasik-O'Brien2021).
Functioning relations
Level 1 General Psychopathology was significantly related to all functioning indices, and more complex levels of the bass-ackwards model provided significant incremental validity in all cases (Fig. 2). By and large, results for adjusted R 2 were virtually identical to those for raw R 2, indicating that this increase in explanatory power was not due to simply adding more variables. However, the level providing maximal validity varied across indices.
Associations between psychopathology dimensions from the bass-ackwards model and indices of psychosocial morbidity and impairment. The black solid line shows cumulative explanatory power (R 2) for a given level plus all preceding levels. The gray dashed line presents adjusted R 2, which controls for spurious increases in R 2 due to adding more predictors to the model. Asterisks indicate significant change in R 2 for a given level over and above all preceding levels combined. *p < 0.05, ***p < 0.001.
In terms of global functioning (GAF), the two-, three-, four-, five-, and seven-component models showed significant incremental validity over and above all preceding levels. There was a steady increase in explanatory power that maximized at level 7 (adjusted R 2 = 0.27).
For inability to work for psychiatric reasons, broader levels of the model did not provide significant incremental validity (other than a slight increase at level 3; adjusted R 2 change = 0.005). However, there was a clear increase in explanatory power at level 5 and a steady increase thereafter. As with GAF, explanatory power maximized at level 7 (adjusted R 2 = 0.16). An opposite pattern emerged for social functioning over the past five years. There was a steady increase in incremental validity through level 5 (adjusted R 2 = 0.14). After level 5, the line flattened out.
For current suicidal ideation, level 2 provided significant incremental validity (adjusted R 2 = 0.10). After that, more complex levels did not provide additional explanatory power, other than a very small increase at level 4 that may be due to chance (i.e. adding more predictors; adjusted R 2 change = 0.002).
A different pattern emerged for history of suicide attempts. All levels provided significant incremental validity, although the increase from level 6 to level 7 was small and likely due to adding more predictors (adjusted R 2 change = 0.001). Explanatory power maximized at level 6 (adjusted R 2 = 0.14). Similarly, all levels through level 6 provided significant incremental validity for history of inpatient psychiatric hospitalizations. Explanatory power maximized at level 6 (adjusted R 2 = 0.15).
We then conducted follow-up analyses examining the unique R 2 attributable to each dimension within the optimal level of hierarchy (controlling for all other dimensions within that level, as well as age and sex; Figure 3). All dimensions made significant incremental contributions, with the exception of compulsivity for inability to work. Not surprisingly, different dimensions explained the most variance in different indices of functioning. For example, internalizing explained the most variance in global functioning (and more variance in suicidal ideation relative to externalizing), Detachment explained the most variance in social functioning, and thought disorder explained the most variance in inability to work, history of suicide attempts, and history of hospitalization.
Unique associations between psychopathology dimensions from the bass-ackwards model and indices of psychosocial morbidity and impairment. Results are shown for the level of hierarchy that provided optimal explanatory power for a given index of functioning (see Fig. 2). Asterisks indicate significant change in R 2 for a given component over and above all other components within the same level, controlling for age and sex. ***p < 0.001. SOM, Somatoform; INT, Internalizing; DET, Detachment; THO, Thought disorder; COMP, Compulsivity; ANT, Antagonism; DIS, Disinhibition; EXT, Externalizing.
Discussion
The hierarchical aspect of the HiTOP model implies that different levels of detail will be appropriate for different research and clinical purposes. In the current study, we explored this implication by examining which level of hierarchy is likely to provide optimal validity and explanatory power with regard to relevant clinical variables. We examined this question using (a) a diagnostically heterogeneous clinical sample and (b) a relatively wide range of psychopathology indicators – both of which support the potential generalizability of our results. These findings can contribute to the development and implementation of HiTOP-consistent assessment in other research and clinical settings.
We tested the incremental validity of increasingly complex levels of a bass-ackwards model with regard to six key clinical variables: GAF, suicidal ideation, inability to work for psychiatric reasons, social functioning, history of psychiatric hospitalization, and history of suicide attempt. All functioning indices were significantly associated with general psychopathology, which is consistent with other research showing that it is a potent predictor of functional impairment (Forbes et al., Reference Forbes, Greene, Levin-Aspenson, Watts, Hallquist, Lahey and Krueger2021a; Michelini et al., Reference Michelini, Barch, Tian, Watson, Klein and Kotov2019). However, in all cases, more complex levels of the bass-ackwards model provided significant incremental validity, and in almost all cases, each dimension within these more complex levels made a significant incremental contribution. Taken together, these findings demonstrate that there is also benefit to accounting for multidimensionality.
Notably, the optimal level of complexity varied widely across functioning indices. This finding is consistent with the rationale for hierarchical taxonomy, in that there is no single ‘best’ level for understanding relations between psychopathology and functioning – instead, different levels appear to be suitable for different purposes. Pending further research, this finding complicates straightforward recommendations for a single optimal level of hierarchy to use in applied research and practice. At the same time, there are advantages to compromising on a suitable level for the purposes of utility. Based on results from this study, measurement at more complex levels (e.g. levels 6 or 7 of the model from this study) is likely to be fruitful. Although dimensions within more complex levels differed in their explanatory power for various indices of functioning, in almost all cases each component within these more complex levels made a significant incremental contribution. This suggests that it will be worthwhile to measure each dimension within the selected level once such instruments are available. Further actionable measurement implications are limited pending the development of instruments that can measure these dimensions directly.
More research is needed to see if there are systematic reasons for variability of optimal levels. Consistent with prior literature (Forbush et al., Reference Forbush, Hagan, Kite, Chapa, Bohrer and Gould2017; Kotov et al., Reference Kotov, Foti, Li, Bromet, Hajcak and Ruggero2016), general functioning was best explained by the most complex level of our model. This suggests that it is worthwhile to consider the different contributions of narrower dimensions of psychopathology to broad indices of dysfunction, as these types of indices likely have many potential causes (i.e. equifinality). If broader functioning indices tend to be better explained by more complex levels of psychopathology hierarchy, it follows that more specific indices of psychosocial morbidity may be adequately explained by less complex levels. For example, results from this study suggest that higher levels of hierarchy (e.g. a model differentiating internalizing and externalizing psychopathology) provide the most effective and parsimonious account for psychopathology's relations with suicidal ideation, such that there is little-to-no validity gained from making more finely grained distinctions. However, given the complexity and severity of suicidal ideation and behavior, further research that uses more detailed suicidality assessment is needed before translating these findings into concrete assessment recommendations.
It also could be the case that this variability reflects time frame (e.g. concurrent v. retrospective) rather than type of impairment. For example, history of suicide attempt was best explained by more complex levels of the model, whereas these levels did not provide significant incremental validity with regard to current suicidal ideation. More systematic coverage of functioning and psychosocial morbidity across different time frames would allow for clearer identification of patterns of variability in associations between functioning and different levels of psychopathology hierarchy.
In addition, researchers and clinicians are likely to be interested in the utility of dimensional models of psychopathology for predicting dysfunction and psychosocial morbidity over time – for example, in the service of clarifying etiological mechanisms or informing treatment recommendations. Indices of psychosocial morbidity in this study were limited to cross-sectional or retrospective report; results may differ when using hierarchical dimensional models of psychopathology to predict future clinically relevant outcomes (see, e.g. Kotov et al., Reference Kotov, Foti, Li, Bromet, Hajcak and Ruggero2016). Further investigation into the relative validity of different levels of hierarchy should include prospective data to further address these utility considerations.
Another limitation to the current study involves the use of categorical diagnoses. As has been noted extensively in the taxonomic literature (e.g. Forbes et al., Reference Forbes, Sunderland, Rapee, Batterham, Calear, Carragher and Krueger2021b), categorical diagnoses provide crude characterizations of psychopathology dimensions due to suboptimal measurement properties such as within-diagnosis heterogeneity. Moreover, the implied (or required) severity and functional impairment of diagnoses may inflate relations between diagnosis-based dimensions and indices of functioning. If so, it would be worth examining whether any such distortion varies in magnitude across different indices of functioning or otherwise alters findings regarding incremental validity. Results may differ when dimensions are modeled without the discussed limitations of categorical diagnoses (e.g. when using symptom-level data). Symptom-level data also allows for the modeling of more finely grained dimensions, which (a) may provide a superior level of resolution for some research and clinical applications and (b) would facilitate relevant measure development.
Finally, the sample used in the current study is relatively homogeneous demographically, particularly in terms of race/ethnicity. More research is therefore needed to demonstrate the extent to which results from our study generalize to populations underrepresented in these data. Notably, the hierarchical organization of psychopathology offers a potentially useful framework for encompassing diverse presentations of psychopathology (Cicero & Ruggero, Reference Cicero and Ruggero2021) and contextualizing psychopathology in social identity (Langwerden, Thompson, & Wagner, Reference Langwerden, Thompson and Wagner2021). We therefore see the methodological approach of the current study as a promising avenue for long-overdue work in this area.
Results from this study support the hierarchical organization of psychopathology dimensions with regard to validity considerations and downstream implications for applied assessment. Our finding that different levels of psychopathology hierarchy appear to be suitable for different purposes suggests that it would be fruitful to develop and implement measurement of these dimensions at the appropriate level for the purpose at hand. This highlights the need for further research focused on translating results from quantitative models of psychopathology into efforts to develop assessment tools that are optimized for key research and clinical applications.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291722003324.
Financial support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflict of interest
None.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
