Introduction
Mental disorders affect one in seven young Australians in any given year and are responsible for the largest burden of disease in this age group (Erskine et al., Reference Erskine, Moffitt, Copeland, Costello, Ferrari, Patton and Scott2015). Epidemiological data on population samples have consistently identified a high rate of comorbidity (or co-occurring disorders) between mood, anxiety, psychotic, and substance use disorders with the presence of any disorder associated with increased odds of experiencing another disorder (Slade, McEvoy, Chapman, Grove, & Teesson, Reference Slade, McEvoy, Chapman, Grove and Teesson2015). Higher rates of comorbidity have also been linked to increased levels of distress, higher disorder severity, service use, and poor treatment response (Lubman, Allen, Rogers, Cementon, & Bonomo, Reference Lubman, Allen, Rogers, Cementon and Bonomo2007; Teesson, Slade, & Mills, Reference Teesson, Slade and Mills2009). In younger populations, comorbidity is the rule rather than the exception and many disorders first manifest as transdiagnostic indicators or common symptoms that further develop into more defined and recognizable disorders (McElroy, Belsky, Carragher, Fearon, & Patalay, Reference McElroy, Belsky, Carragher, Fearon and Patalay2018). These findings have led researchers to speculate that there might be several underlying common or transdiagnostic factors that better account for co-occurring multiple disorders and their association with other clinically relevant factors or outcomes, such as suicide or treatment response (Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019). More recently, the literature has placed considerable attention on identifying an underlying “p-factor” that accounts for the broad tendency to experience any psychopathology, which might manifest in numerous specific ways (Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel and Moffitt2014). Indeed, identifying and targeting broad levels of the p-factor might result in more efficient treatment and prevention efforts to reduce the sizable burden of psychopathology (Caspi & Moffitt, Reference Caspi and Moffitt2018).
Primarily based on empirical data, the Hierarchical Taxonomy of Psychopathology (HiTOP) consortium has proposed that psychopathology might be more efficiently captured and organized through a system of continuous latent factors that are structured within a larger hierarchy, ranging from specific symptom clusters at the lowest level through to a broad p-factor at the superspectrum level (Kotov et al., Reference Kotov, Waszczuk, Krueger, Forbes, Watson, Clark and Zimmerman2017). Previous studies have identified and validated various levels of the HiTOP hierarchy and employed bifactor models to identify general and specific factors related to adolescent psychopathology (Afzali, Sunderland, Carragher, & Conrod, Reference Afzali, Sunderland, Carragher and Conrod2017; Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod and Slade2016; Castellanos-Ryan et al., Reference Castellanos-Ryan, Brière, O'Leary-Barrett, Banaschewski, Bokde and Bromberg2016; Haltigan et al., Reference Haltigan, Aitken, Skilling, Henderson, Hawke, Battaglia and Andrade2018; Laceulle, Vollebergh, & Ormel, Reference Laceulle, Vollebergh and Ormel2015; Tackett, Reference Tackett2006; Waldman, Poore, van Hulle, Rathouz, & Lahey, Reference Waldman, Poore, van Hulle, Rathouz and Lahey2016). However, these models assume the structure underlying psychopathology is inherently continuous and normally distributed, relying on the use of variable-centered latent modeling. Other potential models can relax and test these strict assumptions by fitting person-centered categorical latent variables (in a latent class modeling framework) or a mixture of continuous and categorical latent variables (factor mixture models) (Lubke & Muthén, Reference Lubke and Muthén2005). Additional insights into the nature of psychopathology can be obtained via detailed model comparisons; for example, whether the latent distribution might be considered continuous but nonnormally distributed. These investigations seek to clarify the manner in which the latent structure is modeled and has fundamental implications for understanding individual differences at the latent level as well as how differences in trajectories, antecedents, and consequences of psychopathology are derived (Masyn, Henderson, & Greenbaum, Reference Masyn, Henderson and Greenbaum2010). Among adult samples, comprehensive model comparisons have indicated that the structure of symptom-based data across multiple mental and substance use disorders is best represented by continuous and normally distributed factors (Wright et al., Reference Wright, Krueger, Hobbs, Markon, Eaton and Slade2013).
In addition to identifying the latent structure of psychopathology, studies have demonstrated the various degrees of validity and utility associated with latent models when capturing the association between multiple individual mental disorders (i.e., comorbidity) and other key indicators of interest that have been hypothesized to be associated with multiple disorders (Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019). If the common or broad latent factors (either categorical, continuous, or both) adequately capture the total observed variance between individual indicators of psychopathology and other key health and behavioral validators, it may be inferred that the association is primarily driven by commonalities across multiple disorders and/or symptoms rather than information that is specific to individual disorders and/or symptoms. Among adult samples, the shared and specific associations between psychopathology and numerous additional clinical validators have been investigated, including: suicidality, treatment seeking, functional impairment/disability, traumatic events, intergenerational transmission, and adverse childhood events (Albott, Forbes, & Anker, Reference Albott, Forbes and Anker2018; Keyes et al., Reference Keyes, Eaton, Krueger, McLaughlin, Wall, Grant and Hasin2012; Starr, Conway, Hammen, & Brennan, Reference Starr, Conway, Hammen and Brennan2014; Sunderland et al., Reference Sunderland, Carragher, Chapman, Mills, Teesson, Lockwood and Slade2016; Sunderland & Slade, Reference Sunderland and Slade2014). These studies have utilized latent variable and structural equation models to parse the total association between disorders, signs, and symptoms of psychopathology and additional clinical validators into shared and specific variance. Additional modeling enables the size and significance of the specific associations to be evaluated after accounting for the shared association (via a reduced set of latent factors). These results show that broad latent factors, such as internalizing and externalizing, can adequately capture multiple associations between additional clinical indicators and individual disorders that comprise the latent factors. Indeed, these results provide additional evidence of criterion validity associated with a more parsimonious and empirically informed conceptualization of psychopathology, potentially leading to more efficient assessments, diagnosis, prevention programs, early interventions, and treatments that target the commonalities shared across multiple putatively distinct disorders. Moreover, these studies provide further evidence for the utility of the HiTOP model as an organizational structure for research in psychopathology (Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019). Yet much of this structural work has been confined to adults. Adolescence is a developmental period marked by significant neurobiological, behavioral, and social changes. It is also a time when substance use disorders, psychosis, personality disorders, and eating disorders first emerge (Paus, Keshavan, & Giedd, Reference Paus, Keshavan and Giedd2008), with critical implications for adult development and health.
In adolescents, there are several key health and behavioral validators, including suicidality, self-harm, sleep, self-esteem and risky sexual behaviors, that have been previously associated with numerous mental health and substance use conditions. Moreover, these validators have been linked to increased levels of distress, hospitalization, mortality, and in the case of risky sexual behavior, sexually transmitted infections and unwanted pregnancies (Borschmann, Stark, Prakash, & Sawyer, Reference Borschmann, Stark, Prakash and Sawyer2018; Prendergast, Toumbourou, McMorris, & Catalano, Reference Prendergast, Toumbourou, McMorris and Catalano2019). Often these validators further contribute to and exacerbate psychopathology and negative outcomes (Baglioni et al., Reference Baglioni, Nanovska, Regen, Spiegelhalder, Feige, Nissen and Riemann2016; Daraganova, Reference Daraganova2017; Mars et al., Reference Mars, Heron, Crane, Hawton, Lewis, Macleod and Gunnell2014; Masselink, Van Roekel, & Oldehinkel, Reference Masselink, Van Roekel and Oldehinkel2018). Further empirical evidence is required to better understand the shared and specific relationships that might exist between psychopathology and these key health and behavioral validators. Indeed, specific associations would indicate there is something unique about a pairwise association between a symptom domain/disorder and a key validator, with implications for the development of transdiagnostic and/or tailored interventions that seek to reduce these harmful and inter-related behaviors.
As such, the current study: (a) examined the fit and compared a range of alternative and theoretically plausible models of psychopathology from a large sample of Australian adolescents; and (b) investigated if these broad models of psychopathology captured the total association among specific indicators of psychopathology (e.g., major depression) and other hypothesized key health and behavioral validators implicated across multiple disorders (e.g., poor sleep, self-harm, suicidal ideation, suicide attempts, multiple sexual partners, condom use, and low self-esteem).
Methods
Sample
Data for the current study were drawn from the Second Australian Child and Adolescent Survey of Mental Health and Wellbeing (Young Minds Matter; YMM) (Hafekost, Johnson et al., Reference Hafekost, Johnson, Lawrence, Sawyer, Ainley, Mihalopoulos and Zubrick2016), a stratified, multistage area probability sample of Australian households with at least one child aged 4–17 years. The survey was administered by the Telethon Kids Institute at the University of Western Australia in partnership with Roy Morgan Research. The survey received ethical approval from the Australian Government Department of Health. Moreover, parents or carers and young people provided informed consent prior to participating in the survey and the self-report measures. The survey data have been weighted to represent the population of all Australian young people aged 11–17 years old. Weights were calculated using the generalized raking procedure and were calibrated by sex, age, family size, and household income (Deville & Särndal, Reference Deville and Särndal1992; Hafekost, Lawrence et al., Reference Hafekost, Lawrence, Boterhoven de Haan, Johnson, Saw, Buckingham and Zubrick2016a). The clustered nature of the sample design was incorporated into the analyses when estimating standard errors and confidence intervals. The current study restricted the analysis to adolescents aged 14 years or older given those younger than 14 were not administered detailed self-report questions on alcohol, tobacco, drug use, risky sexual behavior, and psychotic-like experiences. More detail on the survey methods and technical details can be found elsewhere (Hafekost, Lawrence et al., Reference Hafekost, Lawrence, Boterhoven de Haan, Johnson, Saw, Buckingham and Zubrick2016b). Given the current study reports on a secondary analysis of publicly available survey data, a priori power analyses specific to the reported analyses were not conducted to inform the choice of sample size.
Measures
Indicators of psychopathology
All diagnostic variables included in the analyses were assessed using the parent-reported Diagnostic Interview Schedule for Children version IV (DISC-IV) developed by the US National Institute of Mental Health (Shaffer, Fisher, Lucas, Dulcan, & Schwab-Stone, Reference Shaffer, Fisher, Lucas, Dulcan and Schwab-Stone2000). The parent version of the DISC-IV has previously demonstrated acceptable reliability compared to the self-report version (Shaffer, Fisher, & Lucas, Reference Shaffer, Fisher, Lucas, Helsenroth, Segal and Hersen2004). DSM-IV criteria were assessed in the past 12 months prior to the survey and diagnostic variables were generated for eight disorders: separation anxiety disorder (SA), social anxiety disorder (SAD), generalized anxiety disorder (GAD), major depressive disorder (MDD), attention deficit hyperactivity disorder (ADHD), obsessive-compulsive disorder (OCD), conduct disorder (CD), and oppositional defiance disorder (ODD). Preliminary inspection of the data indicated zero cells in the bivariate correlation matrix when running the latent variables models. As such, the diagnoses of ODD and OCD were removed from the analysis.
Indicators of substance use were measured using a combination of self-report items (e.g., frequency and quantity of alcohol use) and single-use questions (e.g., have you ever smoked cigarettes daily?). More specifically, binge drinking in the past 30 days was measured by calculating the frequency of single-occasion drinking with binging classified as greater than or equal to four standard drinks in a row (using the Australian definition of a standard drink equivalent to 10 grams of alcohol). Tobacco use was determined by asking if the respondent had ever smoked cigarettes daily. Cannabis use and other illegal drug use were assessed by combining multiple binary variables about whether the respondent had ever used cannabis or other illegal drugs, including sniffing petrol, glue, aerosols, solvents, paints, or nitrous.
Psychotic-like experiences were measured using a five-item self-report screening questionnaire that obtained information on feelings of paranoia (“In the last year, have you ever believed that people were spying on you?”), unusual beliefs (“In the last year, have you ever believed that you were being sent special messages through the TV or radio or that a program had been arranged for you alone and no one else?”, “In the last year, have you ever believed that someone was using special powers to read your mind?”), visual hallucinations (“In the last year, have you ever seen something or someone that other people who were present could not see, that is had a vision when you were completely awake?”), or auditory hallucinations (“In the last year, have you heard things other people could not hear, such as a voice?”).
Key health and behavioral validators
The YMM survey collected data on a number of health- and wellbeing-related variables associated with child and adolescent development. As such, the choice of health and behavioral validators to utilize for the current study was informed by selecting validators from the available data that have (a) previously been found to be associated with individual mental and substance use disorders and (b) compound or exacerbate mental and substance use disorders and overall levels of poor health, distress, and impairment in adolescent samples.
Poor sleep was measured using two self-report questions to generate the typical number of hours of sleep obtained by the respondent on weeknights (Sunday–Thursday) and weekend nights (Friday–Saturday). The two variables were dichotomized into poor sleep (<7 hr or >11 hr) or adequate sleep (between 7–11 hr) based on established sleep guidelines (Hirshkowitz et al., Reference Hirshkowitz, Whiton, Albert, Alessi, Bruni, DonCarlos and Adams Hillard2015). Self-harm was measured by a single dichotomous (yes/no) self-report question “Have you deliberately harmed or injured yourself without intending to end your own life during the past 12 months?”. Suicidal ideation and suicide attempts were measured using two dichotomous (yes/no) self-report questions: “During the past 12 months, did you ever seriously consider attempting suicide?” and “Did you attempt suicide during the past 12 months?”. Risky sexual behavior was measured by two indicators representing (a) multiple sexual partners dichotomized into those who have had four or more sexual partners across their lifetime and those with less than four sexual partners and (b) condom use by a single dichotomous (yes/no) question “Have you ever had sexual intercourse without using a condom?”. Finally, low self-esteem was measured using the Adolescent Self-esteem Questionnaire (ASQ) developed specifically for the YMM survey (Hafekost, Boterhoven de Haan, Lawrence, Sawyer, & Zubrick, Reference Hafekost, Boterhoven de Haan, Lawrence, Sawyer and Zubrick2017). Scores were dichotomized to indicate the presence of low self-esteem (or not) based on a previously validated cut-point (Hafekost et al., Reference Hafekost, Boterhoven de Haan, Lawrence, Sawyer and Zubrick2017).
Statistical analysis
Delineating the structure of psychopathology
To determine the structure underlying the selected indicators of psychopathology in the current study, a series of latent variable models were estimated with different underlying assumptions regarding the distribution of the latent factors. Confirmatory factor analysis (CFA) was used to estimate latent models with continuous and normally distributed latent variables. Latent class analysis (LCA) was used to estimate latent models with categorical latent variables with an increasing number of classes. Finally, factor mixture models (FMM) were used to estimate latent models with nonnormal continuously distributed latent variables. All models were estimated using data weighted to the sex and age characteristics of the young Australian population and standard errors were adjusted for the clustered nature of the sampling design (Muthen & Muthen, Reference Muthen and Muthen2015).
The current study proceeded by fitting four CFA models: (a) a one-factor model with all indicators loading on a single latent factor representing general psychopathology; (b) a correlated trait model with three factors representing internalizing (MDD, SD, SAD, GAD, ADHD), externalizing (binge drinking, daily smoking, cannabis use, illegal drug use, and conduct disorder), and psychotic-like experiences (paranoia, receiving special messages, believing people can read your mind, visual hallucinations, and auditory hallucinations); (c) a bifactor model with all indicators of psychopathology loading on a single factor representing general psychopathology as well as indicators of internalizing, externalizing, and psychotic-like experiences (as described in the correlated latent trait model); and (d) a higher-order model with three lower-order factors representing internalizing, externalizing, and psychotic-like experiences, and a single higher-order factor representing general psychopathology that accounts for correlations among lower-order factors.Footnote 1
All models were estimated using a robust maximum likelihood estimator (MLR) with numerical integration, suitable for binary indicators, as implemented in Mplus (Muthen & Muthen, Reference Muthen and Muthen2015). Models were compared based on the Bayesian information criterion (BIC) and the sample size adjusted BIC (ssaBIC), with lower values indicating better model fit (Raftery, Reference Raftery1995). Additional bifactor model fit indices were calculated including the explained common variance (ECV), percent of uncontaminated correlations (PUC), and the OmegaH coefficient (derived from standardized factor loadings) (Rodriguez, Reise, & Haviland, Reference Rodriguez, Reise and Haviland2016). Moreover, we compared the bifactor and correlated factor models using the H coefficient given recent simulations demonstrating that model fit statistics such as BIC and ssaBIC tend to be biased towards selection of bifactor models (Greene et al., Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Kotov2019). The H coefficient provides an indication of the replicability of the latent factors with values ranging from 0 to 1 and estimates >0.8 indicating well-defined latent variables (Rodriguez et al., Reference Rodriguez, Reise and Haviland2016). Finally, the CFA models were re-run using a weighted least squares estimator (WLSMV) to generate the comparative fit index (CFI), Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). Prior empirically derived cut-points were used to determine excellent fit (CFI and TLI > 0.95 and RMSEA < 0.05) (Hu & Bentler, Reference Hu and Bentler1999).
A series of latent class models were then estimated on the indicators of psychopathology with an increasing number of latent classes included until either there was no decrease in BIC values between successive models or there was insufficient information to fit additional classes. The models were fit using the MLR estimator with increasing the number of random starting values to ensure multiple replications of the best log-likelihood value. The LCA models were compared using BIC and ssaBIC values as well as the Lo–Mendell–Rubin log-likelihood difference test used to compare a model with k classes with a model with k−1 classes (Nylund, Asparouhov, & Muthén, Reference Nylund, Asparouhov and Muthén2007). A nonsignificant result indicates that the k class model does not significantly improve model fit over the k−1 class model. An entropy value was calculated to determine the classification accuracy of the latent classes with scores ranging from 0 to 1 and higher scores indicating greater classification accuracy.
Finally, a series of FMMs that contain both categorical and continuous latent variables were estimated based on the results of the CFA and increasing the number of latent classes until BIC values no longer decreased or there was insufficient information to model more classes. In the current study, the FMMs were estimated in which the only parameters of the factor model that varied across classes were the factor means. Item thresholds and loadings were held invariant across classes and the factor variance and covariance matrix was fixed to zero (Clark et al., Reference Clark, Muthén, Kaprio, D'Onofrio, Viken and Rose2013). This model can be interpreted as a nonlinear factor analysis with differences across latent classes representing differences in the severity location on the underlying dimensional continuum of each factor. FMMs were compared using BIC and ssaBIC as well as Lo–Mendell–Rubin log-likelihood difference test and entropy. The best-fitting model across CFA, LCA, and FMMs was selected by comparing BIC and ssaBIC values, with the lowest value indicating better fit.
Associations between psychopathology and key health and behavioral validators
The best-fitting model of psychopathology (identified using the method described above) was then used to examine the association between broad latent variables of psychopathology (that account for the shared variance across multiple indicators) and the key validators of interest. The results of the structural modeling dictated the specific analytic approach utilized to address the current aim. As such, the current study applied latent variable indirect effect models to decompose the total effect associated with each indicator of psychopathology and key validator into direct effects (e.g., the association remaining after partialling out the variance explained by the higher and lower-order latent factors) and indirect effects (the association explained via the higher-order and lower-order latent factors). The models can be interpreted in a similar manner to a measurement invariance analysis or a multiple indicator multiple causes (MIMIC) model approach to differential item functioning (DIF), in that the indirect effect provides an indication of the association between indicators of psychopathology (e.g., depression) and key validators (e.g., self-harm) assumed by the structural model, whereas significant direct effects indicate the need for additional information specific to the indicators to either amplify or reduce the assumed association via the higher-order factors. Significant and large direct effects suggest that the structural model does not adequately capture the association (i.e., measurement noninvariance), whereas nonsignificant and small direct effects suggest that the model does adequately capture the association and therefore might provide a highly useful and parsimonious model that accounts for multiple pairwise associations (i.e., measurement invariance). To evaluate the magnitude of significant direct effects, an effect size estimate was calculated based on previous literature using MIMIC models for DIF detection (for models associated with binary indicators as the dependent variable). The MIMIC-Effect size (MIMIC-ES) can be interpreted as quantifying the change, on a normally distributed latent trait (mean of 0 and standard deviation of 1), where the probability of endorsing a factor indicator equals 0.5 between those with and without the validator of interest. Therefore, an effect size of 0.3 indicates that the difference between endorsing or not endorsing a factor indicator (e.g., major depression) across different levels of a validator (e.g., those with and without poor sleep), while holding levels of the underlying latent trait constant represents a potential bias of 0.3 standard deviations on the underlying latent scores (e.g., internalizing). For the current study, MIMIC-ES values of 0.3, 0.5, and 0.7 were considered to reflect small, medium, and large effects, respectively (Jin, Myers, Ahn, & Penfield, Reference Jin, Myers, Ahn and Penfield2013).
The indirect effect models estimated in the current study are described further in Figure 1. Each of the observed factor indicators and the lower-order factors (internalizing, externalizing, psychotic-like experiences) are regressed on a key validator while controlling for age, sex and the higher-order factors. For example, the path labelled c1 estimates the total effect of the covariate “poor sleep” on mean latent general psychopathology scores (estimated with mean 0 and standard deviation 1) controlling for age and sex. The path labelled c2 represents the total effect of “poor sleep” on mean latent internalizing scores (estimated with mean 0 and standard deviation 1), whereas the path c2′ represents the direct effect of “poor sleep” on mean latent internalizing after accounting for the indirect association via the higher-order factor (paths a1 × b1) from the total effect. Finally, the path labelled c3 represents the total effect of “poor sleep” on the probability of major depression whereas the path c3′ represents the direct effect of “poor sleep” on the probability of major depression after accounting for the indirect association via the higher-order latent factors (paths a1 × b1 × b2) + (a2 × b2). This set of models enables the examination of potential specific effects between key validators and observed indicators of psychopathology (e.g., major depression) within lower-order factors (e.g., internalizing) as well as specific effects between validators and lower-order factors (e.g., internalizing) within higher-order factors (e.g., general psychopathology). Significant regression path estimates were determined using a more conservative p < .01 given the large sample size and number of statistical tests.
Path diagram of example model examining total, direct and indirect effects between poor sleep and the structural model of psychopathology.
Results
Descriptive statistics
Of the 2,003 young Australians aged 14–17 years included in the survey, 1,029 (51.4%, SE = 1.19) were male with an average age of 15.5 (SE = 0.03) years old. The unweighted frequencies, weighted percentages, and standard errors associated with the indicators of psychopathology and the key validators are provided in Table 1. The indicators of psychopathology ranged from 1.9% (SE = 0.4) for CD through to 15.9% (0.91) for auditory hallucinations. Tetrachoric correlations for all the binary indicators of psychopathology are provided in Table 2 with results indicating moderate to high correlations for all conceptually related indicators. There was evidence of a moderate correlation between ADHD and CD (r = .539); however, ADHD exhibited low correlations (rs < .268) with other externalizing indicators.
Sample descriptive statistics
Tetrachoric correlation matrix for indicators of psychopathology
Notes: SAD = social anxiety disorder, SA = separation anxiety, GAD = generalized anxiety disorder, MDD = major depressive disorder, ADHD = attention deficit hyperactivity disorder, CD = conduct disorder, BINGE = binge drinking, TOB = daily smoking, CAN = cannabis use, DRG = other illegal drug use, PSY1 = paranoia, PSY2 = unusual beliefs, PSY3 = mind reading, PSY4 = visual hallucinations, PSY5 = auditory hallucinations. Bold indicate correlations across disorders from conceptually related domains. Italics indicate moderate to large correlations (>0.3) across disorders from different domains
Structural analysis
The model fit statistics generated by the CFA, LCA, and FMMS are provided in Table 3. The best-fitting CFA model, according to BIC and ssaBIC, was the correlated three factor model and therefore by extension the higher-order model with three lower-order factors and one higher-order factor. The path diagram for the best-fitting model and standardized factor loadings are provided in Figure 2. Loadings ranged from 0.44 (INT by ADHD) to 0.90 (EXT by Cannabis) and all loadings were significant at the p < .01 level. The bifactor model generated excellent model fit according to the CFI, TLI, and RMSEA values but was considered worse than the correlated three factor/higher-order model according to BIC and ssaBIC. The additional fit indices (ECV = 0.366, PUC = 0.714, and OmegaH = 0.59) suggest that the presence of multidimensionality warrants the use of multidimensional models as opposed to a unidimensional model of psychopathology (Rodriguez et al., Reference Rodriguez, Reise and Haviland2016). Moreover, the H coefficients for the bifactor model (General factor = 0.87, Internalizing = 0.81, Externalizing = 0.78, Psychotic-like experiences = 0.82) were acceptable but slightly lower than the H coefficients for the correlated factor model (Internalizing = 0.90, Externalizing = 0.93, Psychotic-like experiences = 0.90).
Path diagram of the best-fitting factor model. SA = separation anxiety, SAD = social anxiety disorder, GAD = generalized anxiety disorder, MDD = major depressive disorder, ADHD = attention deficit hyperactivity disorder, BINGE = binge drinking, TOB = daily smoking, CAN = cannabis use, DRG = other illegal drug use, CD = conduct disorder, PSY1 = paranoia, PSY2 = unusual beliefs, PSY3 = mind reading, PSY4 = visual hallucinations, PSY5 = auditory hallucinations, INT = internalizing, EXT = externalizing, PSY = psychotic-like experiences, p = general psychopathology. All loadings are standardized and significant at p < .05.
Fit statistics for confirmatory factor analysis (CFA), latent class analysis (LCA), and factor mixture models (FMMs)
Notes: aEstimated using WLSMV in Mplus otherwise all models were estimated using MLR. All FMM models were estimated with variances and covariances fixed to 0 and factor loadings and thresholds invariant across classes. Bold text indicates best fitting model in each modeling framework.
The best-fitting LCA model contained five classes according to the BIC and ssaBIC values with good classification accuracy (entropy = 0.88). However, the 3-factor 7-class FMM evidenced substantially lower BIC and ssaBIC values indicating superior fit relative to all LCA models. The predicted class probabilities for the 3-factor 7-class FMM are provided in Figure 3. The largest class representing 74% of the sample (Class 4) were relatively free from psychopathology across the three factors. The second largest class representing 9% of the sample (Class 3) had relatively moderate levels of externalizing, low internalizing and low psychotic-like experiences. Class 7 comprised 8% of the sample and were relatively high in psychotic-like experiences but low on internalizing and externalizing. Class 6 comprised 4% of the sample and demonstrated relatively high levels of internalizing, low externalizing, and low psychotic like experiences. Class 1 comprised 2% of the sample and demonstrated high levels of comorbidity between externalizing and psychotic-like experiences but low levels of internalizing. Finally, Classes 5 and 2 both comprised 1% of the population and represented relatively moderate to high levels of psychopathology across the three factors. However, Classes 1, 5, and 2 comprised relatively low proportions (≤2%) and suggests that a more parsimonious solution might be considered; for example, the 3-factor 4-class solution. Moreover, inspection of the BIC and ssaBIC values over all models indicates the 3-factor correlated model/higher-order model provided better fit relative to all FMMs.
Best-fitting 3-factor 7-class FMM predicted probabilities. SA = separation anxiety, SAD = social anxiety disorder, GAD = generalized anxiety disorder, MDD = major depressive disorder, ADHD = attention deficit hyperactivity disorder, Binge = binge drinking, TOB = daily smoking, CAN = cannabis use, DRG = other illegal drug use, CD = conduct disorder, PSY1 = paranoia, PSY2 = unusual beliefs, PSY3 = mind reading, PSY4 = visual hallucinations, PSY5 = auditory hallucinations.
Latent variable indirect effects models
The standardized path coefficients for the latent variable indirect effect models (i.e., total effects, total indirect effects, and direct effects) are provided in the online supplementary material. The MIMIC-ES (for binary dependent variables) and standardized direct effect estimates (for continuous dependent variables) as well as 99% confidence intervals for each model separately for key validators are presented in Figure 4.
Effect sizes and 99% confidence intervals for direct effect of each indicator of psychopathology on key validity indicators in the higher-order model. Effect sizes for direct effects associated with binary outcomes (MIMIC-ES) were calculated using formula provided in Jin et al. Reference Jin, Myers, Ahn and Penfield2013. Effect sizes for direct effects associated with continuous outcomes were calculated as y-standardized (STDY) regression coefficients. SA = separation anxiety, SAD = social anxiety disorder, GAD = generalized anxiety disorder, MDD = major depressive disorder, ADHD = attention deficit hyperactivity disorder, Binge = binge drinking, TOB = daily smoking, CAN = cannabis use, DRG = other illegal drug use, CD = conduct disorder, PSY1 = paranoia, PSY2 = unusual beliefs, PSY3 = mind reading, PSY4 = visual hallucinations, PSY5 = auditory hallucinations, INT = internalizing, EXT = externalizing, PSY = psychotic-like experiences. Vertical dotted lines represent an effect size of 0.7 indicating a large effect.
For sleep, the association with each indicator of psychopathology was accounted for by the higher-order latent variables with nonsignificant direct effects between weeknight and weekend night poor sleep and observable indicators of psychopathology. However, the general psychopathology factor did not account for the total association between internalizing and poor sleep with a significant effect size of 0.60 and 0.43 for weeknights and weekend nights, respectively. This indicates that adolescents reporting poor sleep patterns on either weeknights or weekend nights had significantly higher internalizing scores than those with good sleep patterns, holding underlying levels of the p-factor constant.
For self-harm, the associations with 13 out of 15 observable indicators of psychopathology were accounted for by the higher-order latent factors, as the direct effects were nonsignificant. Notably, the higher-order factors failed to account for the total association between MDD and self-harm with a significant and medium to large direct effect (b = 1.39, p < .01, MIMIC-ES = 0.55), holding underlying levels of internalizing and p-factor constant.
For suicidality (suicidal ideation and suicide attempt), the associations with 14 out of 15 observable indicators of psychopathology were accounted for by the higher-order latent factors. The exception was between suicidal ideation and MDD with a significant and positive direct effect representing a medium to large effect size of 0.64. The direct association between suicide attempt and MDD also represented a large effect size (MIMIC-ES = 0.75) but the 99% confidence intervals contained a value of zero indicating nonsignificance. Similarly, there were medium to large direct effects for the association between internalizing and suicidality, with evidence of a significant direct effect between internalizing and suicide attempts (b = 0.82, p < .01).
For risky sexual behavior, there was no evidence for significant direct effects associated with the observable indicators of psychopathology. There were large, positive, and significant direct effects between externalizing and multiple sexual partners and ever had sex without using a condom (b = 1.00, p < .01 and b = 0.90, p < .01, respectively), holding levels of the p-factor constant.
Finally, for self-esteem there were no significant direct effects associated with self-esteem and the observable indicators of psychopathology, holding the corresponding higher-order factors constant. However, there were large, positive, and significant direct effects associated with internalizing and self-esteem (b = 0.75, p < .01), holding levels of the p-factor constant.
Discussion
The current study utilized a large population-based sample of Australian adolescents aged 14–17 years old to: (a) compare alternative and theoretically plausible models of psychopathology; and (b) investigate the shared and specific associations among multiple indicators of psychopathology (e.g., major depression, internalizing, etc.) and other key health and behavioral validators: poor sleep, self-harm, suicidal ideation, suicide attempts, multiple sexual partners, condom use, and low self-esteem. The study found evidence that the best-fitting model among this sample comprised three latent factors representing internalizing, externalizing, and psychotic-like experiences. The correlations between these three factors might be represented by a single higher-order factor indexing the overall liability to experience psychopathology. Moreover, the best-fitting CFA model provided substantially better fit according to BIC and ssaBIC values in comparison to the best-fitting FMM. This provides some evidence that the latent factors underlying psychopathology could be sufficiently modeled as continuous and normally distributed in this population.
The study indicated that the hierarchical model sufficiently accounted for the majority of the pairwise associations between psychopathology and the key validators except for MDD and suicidal ideation. Importantly, looking at the associations between first-order latent factors and the key validators, the overall p-factor could not sufficiently explain the total association between internalizing and self-esteem, poor sleep, and suicide attempts, as well as externalizing and condom use, and multiple sexual partners. Borrowing terminology from the field of psychometrics, these direct effects provide some indication of measurement noninvariance of the higher-order psychopathology model in relation to the identified validators (Vandenberg & Lance, Reference Vandenberg and Lance2000). There are potentially additional factors unique to the identified psychopathology indicators (e.g., internalizing) that increase the association with the key validators (e.g., self-esteem) than would be expected if estimated via the higher-order latent factors alone (e.g., p-factor). Indeed, these results also provide evidence of the incremental validity of the lower-order factors over and above the general p-factor with respect to these key validators. Additional research is needed to focus on identifying the specific mechanisms driving these direct associations.
The results of the current study have several implications regarding the meaning and interpretation of the higher-order latent factors, particularly the p-factor and its relative importance with lower-order factors and specific indicators of psychopathology. For example, the large direct associations between internalizing and self-esteem, poor sleep, and suicide attempts indicates that knowledge of an individual's proclivity to experience psychopathology in general (e.g., scores on the p-factor) will not provide enough detail regarding their likelihood of also experiencing low self-esteem, poor sleep, or suicide attempts. This also suggests that the internalizing factor might be specifically characterized and differentiated from general psychopathology by associations with low self-esteem, poor sleep, and suicide attempts. Similarly, the study demonstrated that risky sexual behaviors were associated with higher externalizing scores over and above general psychopathology, reflecting and supporting previous findings that externalizing might be differentiated from general psychopathology by increased rates of sensation seeking and the tendency to engage in risky or impulsive behaviors (Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod and Slade2016). However, it was interesting that at the individual indicator level, the associations between a large majority of the indicators and all seven of the validators was adequately captured by the externalizing factor, which suggests these indicators have relatively homogeneous associations with the key validators included in the current study. Importantly, the findings further contribute to the utility of specifying multiple levels of the psychopathology hierarchy simultaneously rather than focusing research on aspects of psychopathology that could potentially be too broad or too narrow, missing part of the picture. Finally, the fact that the current study demonstrates these outcomes among adolescents, a time when symptoms of psychopathology first emerge, is a considerable strength of the current study and offers several avenues for clinical application.
Regarding clinical implications, by estimating and comparing associations with key validators at varying levels of the psychopathology hierarchy the current study provides some guidance regarding the development of broad or targeted prevention or early intervention programs. For example, interventions that broadly target internalizing disorders may also improve sleep, and self-esteem, and reduce suicide attempts in a more efficient and effective manner than an intervention that broadly targets all psychopathology (i.e., p-factor, including externalizing and psychotic-like experiences). In contrast, an intervention that narrowly focuses on features specific to MDD may not efficiently or effectively address levels of suicidal ideation in comparison to a transdiagnostic intervention that targets commonalities across all internalizing disorders (including anxiety and pathological worry). Conversely, interventions to improve sleep and self-esteem may have broad positive impacts on internalizing disorders, but with less impact on externalizing or psychotic disorders. Indeed, these findings highlight the potential of efficient stepped or staged care models for the early intervention or prevention of psychopathology in adolescents based on the hierarchical organization proposed in the HiTOP model (Forbes, Rapee, & Krueger, Reference Forbes, Rapee and Krueger2019). Such interventions might begin with broad components that adequately address commonalities associated with general psychopathology before stepping up into more focused interventions for those with higher internalizing to further address factors that have additional specific associations with those conditions (e.g., poor sleep, self-esteem, etc.).
There are several limitations of the current study that should be considered when interpreting the results. First, the indicators of psychopathology used in the current study represent a mix of self-report and parent-reported indicators as well as a mix of different reference timeframes (e.g., 30-day/12-month/no reference). Moreover, there were relatively fewer robust indicators available in the survey to measure psychotic-like experiences and externalizing, with the current study relying on the use of screening questions to infer the presence of more severe psychopathology rather than clinically assessed symptoms of psychopathology. The high prevalence of auditory hallucinations observed in the current study might suggest that the question “hearing voices no one else can hear” is too ambiguous or interpreted as referring to high acuity of hearing. As such, additional research is required to replicate this work utilizing more pathological indicators of externalizing and psychosis/psychotic-like experiences. Indeed, substance use was considered as an indicator of the externalizing dimension in the current study in line with previous evidence demonstrating that substance use forms a large component of the externalizing spectrum model (Krueger, Markon, Patrick, Benning, & Kramer, Reference Krueger, Markon, Patrick, Benning and Kramer2007), but these indicators might be considered more normative relative to substance use disorders in this population. Second, the current study used data from a cross-sectional general population survey and therefore could only provide an indication of association between variables rather than inferring any causality or temporality between psychopathology and the validity indicators. In addition, the lack of significant direct effects at the observable indicator level might reflect a lack of power given the relatively low number of participants in this population with high rates of psychopathology. Any replication of this work in the future would benefit from the use of large clinical samples to maximize power to detect direct effects. Third, the current study estimated factor mixture models by constraining all parameters except for latent means as equivalent across latent classes. Prior studies have indicated that these restrictive models rarely fit real data and additional models are required to freely estimate additional parameters; however, the alternative recommended models failed to converge in the current sample and could not generate usable estimates (Clark et al., Reference Clark, Muthén, Kaprio, D'Onofrio, Viken and Rose2013). Additional independent samples are required to replicate the findings demonstrated here as well as examine the possibility of less restrictive factor mixture models. Fourth, the current study drew a somewhat artificial distinction between indicators of psychopathology and health and behavioral validators of the higher-order psychopathological model. Validators for the current study were selected given evidence demonstrating associations with mental and substance use disorders but also based on evidence regarding the wide-spread health and behavioral issues (other than mental and substance use disorders) that can arise from poor sleep, suicidality, low self-esteem, and risky sexual behaviors. However, the proposed validators could also be conceptualized as indicators of psychopathology, particularly given previous evidence suggesting that risky sexual behaviors serve as indicators of externalizing (Rodriguez-Seijas, Arfer, Thompson, Hasin, & Eaton, Reference Rodriguez-Seijas, Arfer, Thompson, Hasin and Eaton2017), and additional research regarding an expanded structural model of multimorbidity is warranted.
In conclusion, among a large population-based sample of Australian young people (14–17 years), psychopathology might be best conceptualized and organized using a hierarchical structure of latent factors that differ in their specificity. These higher-order latent factors can adequately and more efficiently account for a majority of the multiple pairwise associations evident between indicators of psychopathology and additional health and behavioral validators including poor sleep, self-harm, suicidality, risky sexual behavior, and low self-esteem. However, there were significant and large direct effects identified predominately between the validators and factors at the first-order level (i.e., with internalizing and externalizing). These findings provide further evidence for the utility of estimating multiple levels of the psychopathology hierarchy simultaneously. This is of importance among adolescent populations where early intervention and prevention programs offer the greatest potential to reduce the burden of psychopathology.
Supplementary material
Supplementary material available online at https://doi.org/10.1017/S0954579420000437
