Factor analytic methods suggest that mental health symptoms assessed across the lifespan can be combined into several broad dimensions, including primarily externalizing (e.g., disruptive behavior disorders) and internalizing (e.g., depressive disorders and anxiety disorders) spectra as well as thought disorder/psychosis (e.g., schizophrenia spectrum disorder, mood disorders with psychosis), somatoform (e.g., somatic symptom disorder), and detachment (e.g., avoidant, dependent, and schizoid PD; Achenbach & Rescorla, Reference Achenbach and Rescorla2001; Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby, Brown, Carpenter, Caspi, Clark, Eaton, Forbes, Forbush, Goldberg, Hasin, Hyman, Ivanova, Lynam, Markon, Miller, Moffitt, Morey, Mullins-Sweatt, Ormel, Patrick, Regier, Rescorla, Ruggero, Samuel, Sellbom, Simms, Skodol, Slade, South, Tackett, Waldman, Waszczuk, Widiger, Wright and Zimmerman2017; Krueger, Reference Krueger1999; Tuvblad et al., Reference Tuvblad, Zheng, Raine and Baker2009). Given the later age of onset of clinically significant symptoms associated with thought disorder/psychosis, somatoform, and detachment spectra (DSM-5; American Psychiatric Association, 2013), internalizing and externalizing dimensions are particularly relevant for describing child and adolescent psychopathology, have been studied extensively in these age groups, and are the focus of this paper.
Overall, the evidence for the existence of latent internalizing and externalizing factors comes from large-scale factor analyses of symptom co-occurrence (e.g., Achenbach & Rescorla, Reference Achenbach and Rescorla2001) and multivariate behavior genetic studies (e.g., Tuvblad et al., Reference Tuvblad, Zheng, Raine and Baker2009). Based on the extant literature on the internalizing and externalizing correlated spectra in youth, different types of anxiety, including generalized anxiety, social anxiety, and OCD, depression, and somatic complaints have been identified as markers of the internalizing factor, while ADHD, ODD, and CD typically load on the externalizing factor (e.g., Achenbach & Rescorla, Reference Achenbach and Rescorla2001; Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby, Brown, Carpenter, Caspi, Clark, Eaton, Forbes, Forbush, Goldberg, Hasin, Hyman, Ivanova, Lynam, Markon, Miller, Moffitt, Morey, Mullins-Sweatt, Ormel, Patrick, Regier, Rescorla, Ruggero, Samuel, Sellbom, Simms, Skodol, Slade, South, Tackett, Waldman, Waszczuk, Widiger, Wright and Zimmerman2017; Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015). This pattern of correspondence between indicators and latent factors was expected in the correlated two-factor model in our analysis. At the same time, internalizing and externalizing spectra are typically strongly positively correlated (e.g., .52–.58 in youth reported by Gjone & Stevenson, Reference Gjone and Stevenson1997; .65 reported in adults by Slade, & Watson, Reference Slade and Watson2006), likely due to shared causes (e.g., genetic and environmental influences, Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011). Correlations of this magnitude between internalizing and externalizing spectra can make it difficult to understand components of risk that uniquely predict specific sets of symptoms. For example, if internalizing symptoms are included in the model but not externalizing symptoms, risk factors that are related to externalizing symptoms might appear to be associated with internalizing symptoms because of the large overlap between internalizing and externalizing symptoms. As a result, some (e.g., Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel, Meier, Ramrakha, Shalev, Poulton and Moffitt2014) have questioned whether there is one factor at the apex of psychopathology with some symptom variability left over that may form specific subdimensions of psychopathology.
One way to test this question is using a bifactor model to create a general factor (or “p”) that indexes the overall propensity to develop psychopathology and unique factors (internalizing and externalizing) that are interpreted as residual covariation among the observed symptom indicators not captured by the general factor (i.e., bifactor model; Lahey et al., Reference Lahey, Krueger, Rathouz, Waldman and Zald2017; Ronald, Reference Ronald2019). Bifactor models are frequently compared to second-order factor models, which are also comprised of a general factor, that is a function of the correlation between lower-order latent factors (e.g., internalizing and externalizing spectra). Since the general factor in a bifactor model captures the variance of all observed indicators, rather than just the correlation between lower-order latent factors, it could be more theoretically meaningful (Markon, Reference Markon2019). Overall, second-order factor models represent a more constrained version of a bifactor model, and the two are equivalent under certain constraints (Bader & Moshagen, Reference Bader and Moshagen2022; Markon, Reference Markon2019; Reise, Reference Reise2012). As a result, bifactor models tend to fit the data better than the second-order factor models (e.g., Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016) due to the number of parameters being estimated, their vulnerability to overfitting, and violations of the proportionality assumption (see Bader & Moshagen, Reference Bader and Moshagen2022; Bonifay et al., Reference Bonifay, Lane and Reise2017). Numerous studies have found that bifactor models fit well across development, starting as young as 2 years of age, using multiple informants and measures in clinical and community samples (e.g., Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel, Meier, Ramrakha, Shalev, Poulton and Moffitt2014; Haltigan et al., Reference Haltigan, Aitken, Skilling, Henderson, Hawke, Battaglia, Strauss, Szatmari and Andrade2018; Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003; Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2015, Reference Laceulle, Vollebergh and Ormel2019; Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011, Reference Lahey, Applegate, Hakes, Zald, Hariri and Rathouz2012; Lahey et al., Reference Lahey, Krueger, Rathouz, Waldman and Zald2017, Reference Lahey, Zald, Perkins, Villalta‐Gil, Werts, Van Hulle, Rathouz, Applegate, Class, Poore, Watts and Waldman2018). We present a comprehensive review of studies on bifactor models of psychopathology across the lifespan in Supplemental Materials. Below we emphasize the composition and nomological network for each factor supported in the literature in middle childhood and adolescence given the age range of our sample while pointing out outstanding conceptual and methodological issues, which informed the aims of our paper.
Child and adolescent psychopathology: content of general and specific factors
Keiley and colleagues (Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003) were among the first to derive a bifactor model of youth mental health symptoms, a decade prior to Caspi et al.’s (Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel, Meier, Ramrakha, Shalev, Poulton and Moffitt2014) seminal paper on a general factor of adult psychopathology. Most research on bifactor models of psychopathology has been conducted with children and adolescents (Levin-Aspenson et al., Reference Levin-Aspenson, Watson, Clark and Zimmerman2021), with most of these studies deriving general, internalizing, and externalizing specific factors; however, there is marked variability in factor composition and the number of specific factors extracted (Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021). Bifactor models in samples of children (age<9) are far less common than those derived in samples of youth in middle childhood and adolescence, making it difficult to draw firm conclusions about the composition of the factors in the younger age range (see Supplemental Materials for additional details).
The composition of the general psychopathology factor in older youth (middle-childhood and adolescence) varies even more extensively than in young children given the wider range of studies and marked differences in assessed content (Levin-Aspenson et al., Reference Levin-Aspenson, Watson, Clark and Zimmerman2021). In particular, some studies report that the general factor is defined primarilyFootnote 1 by the internalizing markers (e.g., Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015), while others have found more substantial contributions of externalizing indicators (Castellanos-Ryan et al., Reference Castellanos-Ryan, Brière, O’Leary-Barrett, Banaschewski, Bokde, Bromberg, Büchel, Flor, Frouin, Gallinat, Garavan, Martinot, Nees, Paus, Pausova, Rietschel, Smolka, Robbins, Whelan, Schumann and Conrod2016; Clark et al., Reference Clark, Hicks, Angstadt, Rutherford, Taxali, Hyde, Weigard, Heitzeg and Sripada2021; Hamlat et al., Reference Hamlat, Snyder, Young and Hankin2019; Hankin et al., Reference Hankin, Poggi, Snyder, Young, Glynn and Sandman2017; McElroy et al., Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020; Pettersson et al., Reference Pettersson, Lahey, Larsson and Lichtenstein2018; Snyder et al., Reference Snyder, Young and Hankin2017). In our own prior work, we tested a bifactor model at the item level of the Youth Self Report form (YSR; Achenbach & Rescorla, Reference Achenbach and Rescorla2001) in late childhood (M age = 12.1) as a measurement model to predict later substance use (Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013). Results suggested that the general factor was primarily indexed by a combination of internalizing and externalizing symptoms, such as depression, rule breaking, and aggression symptom items (e.g., “unhappy, depressed,” “cutting school” and “run-away” had the largest loadings); similar findings were obtained by others (e.g., Afzali et al., Reference Afzali, Sunderland, Carragher and Conrod2018; Black et al., Reference Black, Panayiotou and Humphrey2019; Murray et al., Reference Murray, Eisner and Ribeaud2016). Additional studies have reported a combination of externalizing and autism spectrum indicators (e.g., Martel et al., Reference Martel, Pan, Hoffmann, Gadelha, do Rosário, Mari, Manfro, Miguel, Paus, Bressan, Rohde and Salum2017; Noordhof et al., Reference Noordhof, Krueger, Ormel, Oldehinkel and Hartman2015), and psychotic experiences (Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015; Stochl et al., Reference Stochl, Khandaker, Lewis, Perez, Goodyer, Zammit, Sullivan, Croudace and Jones2015) as the main contributors to the general factor. Although extensive variability of observed indicators creates challenges for identifying patterns within the content of the general factor of psychopathology, results from studies with older youth indicate presence of more markers compromised of self-regulation (i.e., externalizing symptoms, such as inattention, impulsivity, aggression, and rule-breaking) than in younger children.
Specific psychopathology factors in youth have demonstrated even worse replicability across studies than the general factor. For older youth (middle childhood/adolescence), the internalizing specific factor has shown strong loadings from primarily markers of anxiety, such as social, specific phobias, separation anxiety, and panic disorder with lower loadings for general worried/anxious symptoms (Black et al., Reference Black, Panayiotou and Humphrey2019; Castellanos-Ryan et al., Reference Castellanos-Ryan, Brière, O’Leary-Barrett, Banaschewski, Bokde, Bromberg, Büchel, Flor, Frouin, Gallinat, Garavan, Martinot, Nees, Paus, Pausova, Rietschel, Smolka, Robbins, Whelan, Schumann and Conrod2016; Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013; Haltigan et al., Reference Haltigan, Aitken, Skilling, Henderson, Hawke, Battaglia, Strauss, Szatmari and Andrade2018; Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015; Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011; McElroy et al., Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020; Neumann et al., Reference Neumann, Pappa, Lahey, Verhulst, Medina-Gomez, Jaddoe, Bakermans-Kranenburg, Moffitt, van IJzendoorn and Tiemeier2016; Olino et al., Reference Olino, Dougherty, Bufferd, Carlson and Klein2014, Reference Olino, Bufferd, Dougherty, Dyson, Carlson and Klein2018; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015; Petersen et al., 2018; Tackett et al., Reference Tackett, Lahey, Van Hulle, Waldman, Krueger and Rathouz2013; Vine et al., Reference Vine, Byrd, Mohr, Scott, Beeney and Stepp2020). Fewer studies report primary markers of the specific internalizing factor characterized by depressive symptoms (Afzali et al., Reference Afzali, Sunderland, Carragher and Conrod2018; Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Murray et al., Reference Murray, Eisner and Ribeaud2016; Neumann et al., Reference Neumann, Pappa, Lahey, Verhulst, Medina-Gomez, Jaddoe, Bakermans-Kranenburg, Moffitt, van IJzendoorn and Tiemeier2016; Noordhof et al., Reference Noordhof, Krueger, Ormel, Oldehinkel and Hartman2015). Although the content of the specific internalizing factor in youth appears to be mostly comprised of anxious symptoms, it has been significantly influenced by lack of available indicators for depression and constraints placed on the models (e.g., having certain indicators load only on the general factor).
Variability in the number and content of specific externalizing factors reported in the context of bifactor models is even more extensive. For youth in middle childhood/adolescence, some have derived a single specific externalizing factor with aggressive and rule-breaking behaviors as primary markers (e.g. Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013; Black et al., Reference Black, Panayiotou and Humphrey2019; Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015; Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011; Martel et al., Reference Martel, Pan, Hoffmann, Gadelha, do Rosário, Mari, Manfro, Miguel, Paus, Bressan, Rohde and Salum2017; Neumann et al., Reference Neumann, Pappa, Lahey, Verhulst, Medina-Gomez, Jaddoe, Bakermans-Kranenburg, Moffitt, van IJzendoorn and Tiemeier2016; Noordhof et al., Reference Noordhof, Krueger, Ormel, Oldehinkel and Hartman2015; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015; Tackett et al., Reference Tackett, Lahey, Van Hulle, Waldman, Krueger and Rathouz2013; Vine et al., Reference Vine, Byrd, Mohr, Scott, Beeney and Stepp2020), while others have derived additional specific factors (inattention/poor concentration- Clark et al., Reference Clark, Hicks, Angstadt, Rutherford, Taxali, Hyde, Weigard, Heitzeg and Sripada2021; Haltigan et al., Reference Haltigan, Aitken, Skilling, Henderson, Hawke, Battaglia, Strauss, Szatmari and Andrade2018; McElroy et al., Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020; Murray et al., Reference Murray, Eisner and Ribeaud2016; Petersen et al., 2018; and impulsivity - Petersen et al., 2018). As a result, there is no consensus if, in addition to the general and specific internalizing factors, distinct inattention, impulsivity, and aggression/rule-breaking specific factors are necessary to best capture the structure of psychopathology in middle childhood/adolescence.
Finally, it is notable that the addition of psychotic symptoms as observed indicators does not consistently result in emergence of another specific factor in youth samples (e.g., Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015, Reference Laceulle, Chung, Vollebergh and Ormel2019 vs. Afzali et al., Reference Afzali, Sunderland, Carragher and Conrod2018; Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Haltigan et al., Reference Haltigan, Aitken, Skilling, Henderson, Hawke, Battaglia, Strauss, Szatmari and Andrade2018; Stochl et al., Reference Stochl, Khandaker, Lewis, Perez, Goodyer, Zammit, Sullivan, Croudace and Jones2015). Overall, based on the extant literature on the bifactor models of psychopathology in youth, we expected to observe a combination of internalizing (anxiety and depression symptoms), externalizing symptoms (ADHD, ODD, and CD), and psychotic symptoms as markers of the general factor (i.e., loadings ≥ .32) in our sample comprised of youth in middle childhood and early to middle adolescence. Our predictions of the content of the specific factors were more speculative with symptoms of anxiety being the most likely primary markers of the internalizing specific factor and aggressive and rule-breaking behaviors as primary markers of the externalizing specific factor.
Child and adolescent psychopathology: correlates of general and specific factors
Given the poor replicability of the general and specific factors of psychopathology across studies with youth, it is important to situate the most consistently derived factors (general, internalizing and externalizing specific) within a nomological network to provide clarity regarding their interpretation. Numerous variables at different levels of analysis have been identified as risk factors for all psychopathology (general factor) and more specifically for internalizing and externalizing psychopathology in youth (e.g., Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021). The patterns of associations observed in the extant literature and described below form a foundation for our hypotheses presented in Table 2, which is organized by groups of validators that are predicted to show similar associations with psychopathology: demographic and maturation characteristics, dimensions of temperament (fear/sadness, anger/aggression, positive emotionality/surgency, and effortful control), environmental factors (peer behavior), and maladaptive outcomes (substance use).
In particular, a gradual increase in both internalizing and externalizing psychopathology is observed across childhood and adolescence with a particular spike observed after puberty (Costello et al., Reference Costello, Copeland and Angold2011). Some studies (e.g., Hamlat et al., Reference Hamlat, Snyder, Young and Hankin2019; Platt et al., Reference Platt, Colich, McLaughlin, Gary and Keyes2017) have demonstrated relevance of pubertal timing for both internalizing and externalizing specific factors, as well as the general factor, indicating its transdiagnostic nature of risk. As a result, we expected positive associations between all psychopathology factors and maturational variables (age and puberty).
Among demographic variables, higher family SES has been identified as a negative correlate of externalizing and a positive correlate of the internalizing problems in youth (e.g., Black et al., Reference Black, Panayiotou and Humphrey2019), while its association with the general factor appears to be inconsistent and potentially dependent on the informant (e.g., Dodge et al., Reference Dodge, Pettit, Bates and Valente1995; Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003). A similar pattern was identified for sex differences. Female sex has been consistently identified as risk factor for internalizing psychopathology (Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021). Therefore, we predicted that higher SES and female sex will show positive associations with internalizing, negative associations with externalizing, and likely small and not statistically significant associations with the general factor.
Child temperament, or individual differences in reactivity and self-regulation, emerge early in life and increase later risk for psychopathology (Kotelnikova et al., Reference Kotelnikova, Mackrell, Jordan and Hayden2015; Rothbart, Reference Rothbart2012). While negative emotionality (NE) has been identified as a risk factor for all psychopathology (Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021), its facets, fear, anger, and sadness show different patterns of associations with psychopathology. Fear and sadness described in the context of bio-behavioral models of temperament (Rothbart, Reference Rothbart2012) overlaps with the behavioral inhibition system (BIS; gray, Reference Gray, Strelau and Angleitner1991), involving sensitivity to punishment, novelty, and fear stimuli. Olino et al. (Reference Olino, Dougherty, Bufferd, Carlson and Klein2014) reported observed fear/inhibition as a specific risk factor for internalizing problems in youth. At the same time, anger/frustration is an interstitial trait relevant to both NE and disinhibition dimensions (Rothbart, Reference Rothbart2012); as a result, it is not surprising that anger/ frustration has been positively linked to general and externalizing specific but not internalizing specific factors in adolescence (Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2019). Therefore, we expected associations with opposite signs for the fear/BIS and anger/frustration variables with internalizing and externalizing specific factors.
Several dimensions of temperament (e.g., surgency, high intensity pleasure, sensation seeking) are rooted in neural systems that support behavioral activation (BAS; gray, Reference Gray, Strelau and Angleitner1991) and are responsible for reward responsiveness. These dimensions have been positively associated with specific externalizing, negatively associated with specific internalizing, and not associated with the general factor in middle childhood/adolescence (Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Castellanos-Ryan et al., Reference Castellanos-Ryan, Brière, O’Leary-Barrett, Banaschewski, Bokde, Bromberg, Büchel, Flor, Frouin, Gallinat, Garavan, Martinot, Nees, Paus, Pausova, Rietschel, Smolka, Robbins, Whelan, Schumann and Conrod2016; Clark et al., Reference Clark, Hicks, Angstadt, Rutherford, Taxali, Hyde, Weigard, Heitzeg and Sripada2021; Hankin et al., Reference Hankin, Poggi, Snyder, Young, Glynn and Sandman2017; Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020; Neumann et al., Reference Neumann, Pappa, Lahey, Verhulst, Medina-Gomez, Jaddoe, Bakermans-Kranenburg, Moffitt, van IJzendoorn and Tiemeier2016; Olino et al., Reference Olino, Dougherty, Bufferd, Carlson and Klein2014). Shyness/behavioral inhibition is a temperament trait that is negatively related to surgency (Kagan, Reference Kagan, Damon and Eisenberg1998; Rothbart, Reference Rothbart2012). Not surprisingly, it was negatively associated with the specific externalizing, positively associated with specific internalizing, and not associated with the general factor in youth (Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003; Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2019; Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021). As a result, we predicted positive associations with externalizing, negative associations with internalizing, and small and not statistically significant associations between the general factor and the BAS and low shyness variables.
Effortful control (EC) represents a top-down, regulatory aspect of temperament that exerts modulating effects on its reactive components (Rothbart, Reference Rothbart2012). Poor executive functioning, a construct relevant to EC, indexed by performance on multiple cognitive tasks in the ABCD study, has been established as a risk factor for externalizing problems in youth (Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020). Further, a negative association between parent-reported EC (including attentional, inhibitory and activation control facets) with externalizing and general factors, and no associations with the internalizing factor have also been reported (Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2019; Olino et al., Reference Olino, Dougherty, Bufferd, Carlson and Klein2014; Wang et al., Reference Wang, Eisenberg, Valiente and Spinrad2016). However, Lynch et al. (Reference Lynch, Sunderland, Newton and Chapman2021) pointed out some inconsistent associations between the internalizing factor and EC. Therefore, our predictions included negative associations between EC variables with externalizing and general factors, and possibly no remarkable associations with the specific internalizing factor.
Further, affiliation domain of temperament emerges in middle childhood reflecting individual differences in the desire for warmth, closeness, and getting along with others (Rothbart, Reference Rothbart2012). Affiliation has not shown any consistent associations with any of the psychopathology factors in adolescence, and we predicted the same (Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2019).
Aside from demographic, maturational, and temperament variables, several environmental factors have been identified as conferring risk for psychopathology in youth. As youth mature, their peer groups exert increasingly important influence; as a result, older children and adolescents who are rejected by peers tend to have more internalizing and externalizing symptoms than their non-rejected peers (e.g., Coie et al., Reference Coie, Lochman, Terry and Hyman1992; Panak & Garber, Reference Panak and Garber1992). Peer rejection and peer problems, constructs relevant to peer victimization and exclusion, have consistently shown moderate to strong positive associations with the general and both specific factors (e.g., Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015). As a result, we predicted that exclusion and victimization by peers will correlate positively with all psychopathology variables. In addition to exclusion and victimization by peers, associating with delinquent and aggressive peers has been identified as a risk factor for externalizing and general factors of psychopathology in particular and to some extent for internalizing symptoms as well (Cotter & Smokowski, Reference Cotter and Smokowski2016; Crews et al., Reference Crews, Bender, Cook, Gresham, Kern and Vanderwood2007; Vitaro et al., Reference Vitaro, Brendgen and Wanner2005; Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021). We expected this pattern in our study.
Finally, general and specific factors have been linked concurrently and longitudinally to other psychopathology constructs to establish their predictive validity. For example, only general and externalizing factors derived in middle childhood have been prospectively linked to alcohol use/abuse diagnoses, antisocial behavior, daily cigarette smoking, and cannabis use in young adulthood (Petersen et al., 2018; Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013; Scalco et al., Reference Scalco, Colder, Read, Lengua, Wieczorek and Hawk2021). Our work suggests that the general and externalizing specific factors, but not the specific internalizing factor, predict initiation and escalation of alcohol, tobacco and cannabis use across adolescence as well as onset of alcohol abuse in early and late adolescence (Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013; Scalco et al., Reference Scalco, Colder, Read, Lengua, Wieczorek and Hawk2021). As a result, we expected positive associations between the general and externalizing factors of psychopathology with concurrent alcohol, tobacco, and cannabis use. Supplemental Materials contain additional literature review on the composition as well as correlates of psychopathology factors in adults for the purpose of highlighting patterns across the lifespan.
Limitations of extant research on bifactor models of psychopathology
While existing literature allows us to formulate some hypotheses regarding interpretation of the general and specific factors of psychopathology in youth based on their composition and correlates, important methodological limitations must be considered. In particular, over-reliance on single-informant, cross-sectional studies of youth psychopathology has been notable (e.g., only 15 out of 41 studies reviewed by Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021 were longitudinal). Even if longitudinal data are collected, comparisons of the structure of psychopathology across time points are sometimes not conducted (e.g., Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003; Laceulle et al., Reference Laceulle, Vollebergh and Ormel2015). Inconsistency of information entered into the model and indicator choice (Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015 used item-level data, while many other studies use ASEBA or DSM-based scales) are significant contributors to discrepancies in composition of the general factor and different numbers and composition of specific factors extracted across studies (Levin-Aspenson et al., Reference Levin-Aspenson, Watson, Clark and Zimmerman2021; Lynch et al., Reference Lynch, Sunderland, Newton and Chapman2021), suggesting that choices of indicators affect complexity of the model as well as the number and composition of the factors extracted. Inconsistency in model specification further detracts from comparability of results across multiple studies. For example, some studies do not allow for specific internalizing and externalizing factors to correlate, but some do (e.g., Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod, Barrett, Champion, Nair and Slade2016; Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011). Some studies constrain certain indicators to load only on a certain factor, thus artificially influencing their composition (e.g., Lahey et al., Reference Lahey, Van Hulle, Singh, Waldman and Rathouz2011 constrained GAD and MDD to load on the general factor only; Hamlat et al., Reference Hamlat, Snyder, Young and Hankin2019 constrained inattention and hyperactivity to load on the general factor only).
Further, although bifactor models have gained popularity in studies of psychopathology over the last 20 years, they are vulnerable to overfitting (Bonifay et al., Reference Bonifay, Lane and Reise2017; Greene et al., Reference Greene, Eaton, Li, Forbes, Krueger, Markon, Waldman, Cicero, Conway, Docherty, Fried, Ivanova, Jonas, Latzman, Patrick, Reininghaus, Tackett, Wright and Kotov2019; Reise, Reference Reise2012). Second order factor models impose “proportionality” on the loadings for the higher and lower order factor, while bifactors do not (Gignac, Reference Gignac2016). When the proportionality assumption holds bifactor models will necessarily have equivalent fit to second order factor models (Bader & Moshagen, Reference Bader and Moshagen2022; Reise, Reference Reise2012). Each of these difficulties make fit-based comparisons insufficient when comparing alternative models (Smith et al., Reference Smith, Atkinson, Davis, Riley and Oltmanns2020). Further, despite the superior statistical fit of bifactor models, interpretability and utility of the general and specific factors remains debated (e.g., Fried et al., Reference Fried, Greene and Eaton2021; Levin-Aspenson et al., Reference Levin-Aspenson, Watson, Clark and Zimmerman2021; Smith et al., Reference Smith, Atkinson, Davis, Riley and Oltmanns2020). For instance, Fried and colleagues have proposed that the general factor of psychopathology in bifactor models is equivalent to the sum of underlying observed indicators of diagnoses, indicating that p represents severity or comorbidity of psychopathology. However, the use of dichotomous diagnoses, their limited number, and the particularly large size of their data set derived from the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC) could have influenced the extent of associations between the latent factors and sums scores of diagnoses. Other interpretations of the general factor of psychopathology include overall impairment resulting from underlying psychopathology (Smith et al., Reference Smith, Atkinson, Davis, Riley and Oltmanns2020) and general liability for psychopathology through non-specific processes such as cognitive distortions and temperament/personality traits (e.g., negative emotionality; Lahey et al., Reference Lahey, Moore, Kaczkurkin and Zald2021). As explored in detail above, interpretability of the general and specific factors is also limited by the extent of their replicability across samples, with their nature and size showing significant differences and specific factors being characterized by particularly low level of consistency (Levin-Aspenson et al., Reference Levin-Aspenson, Watson, Clark and Zimmerman2021), making convergent and discriminant validity particularly important to assess.
In sum, there are several critical gaps in the literature. First, there is a paucity of studies that report longitudinal tests of psychopathology structures in youth when the literature suggests substantial changes in different forms of psychopathology across development (Scalco et al., Reference Scalco, Colder, Read, Lengua, Wieczorek and Hawk2021; Sung et al., Reference Sung, Erkanli, Angold and Costello2004; van der Ende et al., Reference van der Ende, Verhulst and Tiemeier2020). Not testing different models within timepoint and assessing validity and reliability across development increases the potential for inaccurate conclusions about the “p” factor and specific factors. We test a bifactor model at each time point and test measurement invariance across time in our study. This allows for evaluation of consistency and reliability of the “p” factor and specific factors over time. Second, it is critical to compare validity and reliability across different structures of psychopathology to help distinguish a preferred model of psychopathology. Comparing different structures of psychopathology within an extensive network of external validators is essential to providing evidence for or against different factor structures. The current study includes a large variety of external validators to help to distinguish a preferred factor model of psychopathology. No study to our knowledge has compared different psychopathology factor structures in youth over time using multiple informants nor included an external validity analysis of alternative structures at each time point.
The current study
Given that bifactor models have been argued to be more theoretically meaningful, the present study focuses on bifactor models and does not consider second-order factor models. Bifactor models are expected to fit better than correlated two factor models for both reporters, but a key question is whether the derived factors from a bifactor model provide adequate or added validity to justify the added complexity. To address this question, we ran a series of analyses, which addressed two aims. First, we compared a two-factor and bifactor model of psychopathology (see Figure 1) over time using parent and child report to conduct a test of convergent and discriminant validity of each structure (Campbell & Fiske, Reference Campbell and Fiske1959). In this context, convergent validity was defined as significant positive associations between the same factors within model but across reporter and time, while discriminant validity was defined as non-significant or significant negative associations between different factors of psychopathology within and across reporters and time. Second, we tested the external validity of the factors from each model and reporter using 25-28 developmental correlates at each assessment (see Table 1 for the name, citation, and descriptive statistics for all validators with the exception of demographics; see Table 2 for hypotheses). The second stage of the analysis is consistent with establishing a nomological network around psychopathology (Cronbach & Meehl, Reference Cronbach and Meehl1955) as has been done in much of the literature (e.g., Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020). In this context, convergent validity was defined as significant associations with external validators consistent with Table 2, while discriminate validity was defined as non-significant or significant negative associations consistent with Table 2. Importantly, our external validators represent a more extensive network for a study in this age range than has been done previously (e.g., Clark et al., Reference Clark, Hicks, Angstadt, Rutherford, Taxali, Hyde, Weigard, Heitzeg and Sripada2021; Laceulle et al., Reference Laceulle, Chung, Vollebergh and Ormel2019; Moore et al., Reference Moore, Kaczkurkin, Durham, Jeong, McDowell, Dupont, Applegate, Tackett, Cardenas-Iniguez, Kardan, Akcelik, Stier, Rosenberg, Hedeker, Berman and Lahey2020; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015).
Parent bifactor and two-factor structural models across three waves of longitudinal data. W = wave, intern = internalizing; extern = externalizing; COOCCUR = co-occurring internalizing and externalizing symptoms, ADHH = attention deficit hyperactivity disorder hyperactivity scale, ADHI = attention deficit hyperactivity disorder inattention scale, ODD = oppositional defiant disorder, CD = conduct disorder, DEP = major depression, SOM = somatic complaints, GA = generalized anxiety, FNE = fear of negative evaluation, SAN = social anxiety in novel contexts, SAG = social anxiety general, OCD = anxious obsessive compulsive disorder, PSYC = psychotic symptoms. Note that both reporters (child and parent) were included in each final model. ADHH, ADHI, ODD2, CD2, and PSYC were not measured with child report. As such, child report latent factors had 5 fewer indicators than the parent models depicted above.
Measure, reporter, citation, and descriptive statistics for validators
Note: NA in table = only 1 item so α could not be calculated; A = ASEBA measures or YSR and CBCL, S = SASC or social anxiety scale for children; EATQ = early adolescent temperament questionnaire revised, BAS/BIS = behavioral activation/inhibition system, FFFS = fight/flight/freeze system, HI pleasure = High intensity pleasure, Imp/fun = impulsivity/fun seeking, DBD = disruptive behavior scale, ADHD = attention–deficit hyperactivity disorder, def = defiant, L = lifetime, F = frequency, con = control.
Hypotheses for ideal nomological net across models
Note. IN-Bi = internalizing bifactor model; EX-Bi = externalizing bifactor model; CO-Bi = co–occurring bifactor model; IN-2FM= internalizing two–factor model; EX-2FM= externalizing two-factor model; 0, +, – indicate our study hypotheses; 0 = non-significant relationship is ideal; + = positive relationship is ideal; = a negative relationship is ideal. BAS/BIS = behavioral activation/inhibition system, FFFS = fight/flight/freeze system.
Due to the age of our sample youth had not initiated cannabis use by W1 and we had quantity and frequency measures for alcohol and tobacco use at W2 and W3. As such, we had 25 validators at W1 and 28 at W2 and W3. Assessments occurred annually for three years starting at mean age 12 years (range: 11–13). This developmental period includes the transition from early to middle adolescence (11–15), a period characterized by increased psycho-social challenges and an increase in all forms of psychopathology (American Psychiatric Association, 2013). Each model was tested at each time-point of assessment to ascertain the robustness of the structures, fit, reliability, convergent and discriminate validity over time, as well as measurement invariance. Next, we examined associations with external validators for both the two factor and bifactor models and then compared the correlations statistically to explicitly test for stronger or weaker external convergent and discriminant validity in the nomological network across the two models. Each of these design features allowed for a rigorous evaluation of discriminant and convergent validity.
Considering an additive model of comorbidity implied by the DSM (Keeley & Blashfield, Reference Keeley and Blashfield2010) and the importance of discriminant validity (Cronbach & Meehl, Reference Cronbach and Meehl1955; Campbell & Fiske, Reference Campbell and Fiske1959), we made explicit predictions concerning the ideal nomological network between a matrix of validators and derived psychopathology factors for each model (see Table 2). A model with strong convergent and discriminant validity will demonstrate a pattern of associations with external validators that is consistent with those outlined in Table 2 and is consistent across reporters and waves.
The DSM classification system implicitly follows the additive model of comorbidity, such that to obtain a comprehensive clinical picture, the symptoms of one disorder are added to the symptoms of the second, third, etc. disorder. Conceptually, this model suggests that more psychopathology is worse in linear fashion. In contrast, non-additive comorbidity employs conceptual combinations that may change one’s understanding of combined pathology. Non-additive comorbidity between two disorders or qualitative difference may result in symptoms that are not considered to represent either individual disorder (Keeley & Blashfield, Reference Keeley and Blashfield2010). In the ideal matrix presented in Table 2, zero indicates a non-significant association (or small effect size < .10; Cohen, Reference Cohen1988), a plus indicates a positive relationship, while a minus indicates a negative relationship (or small to strong effect size > .10; Cohen, Reference Cohen1988). Predictions for co-occurring symptoms are the average of the predicted effects for internalizing and externalizing symptoms in the two-factor model, given the prior literature and assuming the additive comorbidity model. For example, if a validator has a positive effect on both domains in the two-factor model, additive comorbidity would suggest a stronger effect on the general factor in the bifactor model, but that would not be the case if the effect of the validators on internalizing and externalizing symptoms was differentiated by direction of the effect (e.g., if BAS components had a positive effect on the externalizing factor and a negative effect on the internalizing factor, we would predict a null relationship with the general factor). Such a pattern suggests that the comorbidity is just a simple function of the two factors and is consistent with additive comorbidity.
As reviewed above, the predictions outlined in Table 2 are based on the extant literature. Assuming effect sizes were similar and the pattern in Table 2 was observed, it would suggest equal convergent and discriminant validity and equal external convergent and discriminant validity for the two factor and the bifactor model and that one would prefer the simpler two factor model. To the extent that the patterns deviate from Table 2 or effect sizes deviate, it may suggest differing levels of convergent and discriminate validity as well as differing levels of external convergent and discriminant validity across the two factor models and/or that additive comorbidity is not supported. If comorbidity is not additive, then co-occurring symptoms would be essential to consider in developmental antecedent and consequent studies as effects to and from psychopathology would be different depending on whether a bifactor or two factor model of psychopathology was utilized. In this case, the bifactor model would be preferred.
Method
Participants
The current study is part of a longitudinal study of risk and resilience factors for the development of psychopathology involving 387 families. Sample size was determined using a power analysis to detect small effects between psychopathology and substance use. Participants were recruited utilizing random digit dialing; children and caregivers with a physical or mental disability interfering with assessment were excluded from participation. Youth in the current study were 12.1 years old (SD = .88) at the first assessment (female = 55%), and the second and third assessments occurred on the annual anniversary of the first. Racial demographics for the sample were: 83 % White, 9 % Black or African American, 2 % Hispanic, 1 % Asian and 5 % had another race or ethnicity. Median income for the sample was $70,000 and 6 % received public assistance. Youth also nominated four close friends and one was assessed to obtain close friend report of peer delinquency and peer aggression.
Retention of the target families was high as only 7.5 % were missing from wave 1 (W1) to wave 3 (W3). There were no significant demographic (e.g., adolescent gender, caregiver marital status, race, level of parental education, and family income) or psychopathology differences between the families who participated at all three waves and families with missing data at any wave. Missing data also occurred when it was not possible to recruit one of the four nominated close friends. Proportion of missing peer data were 8.3%, 13.9%, and 18.6% for W1–W3, respectively. In prior papers, missing peer data was associated with lower socioeconomic status (SES), being from single-parent family, minority status, and with increased likelihood of cigarette use, peer cigarette use, and perceived peer cigarette use (p < 0.05), however these effects did not meet Cohen’s lower bound for a small effect (R 2 = .4%; phi range 0.07–0.13, see Scalco et al., Reference Scalco, Meisel and Colder2016) suggesting they may be significant due to the large sample size. Missing peer data was not related to 24 other developmental constructs. These findings suggest that missing data likely had a limited impact on study findings. Nonetheless, the current study used full information maximum likelihood estimation (FIMLE) to account for missing data.
Procedure
Interviews were conducted annually in university research offices. Parents provided consent and adolescents provided assent to participate at the start of each visit. To ensure privacy, the adolescent and caregiver filled out questionnaires in separate rooms with a sound machine in between rooms. Interviews took approximately 2 ½ hours to complete, and the procedure was the same at each assessment. The study was approved by an IRB. Families were compensated $75 for their time at Wave 1, $85 at Wave 2, and $125 at Wave 3 (see Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013 for additional details on the sample and methods).
Measures
Child/adolescent problem behavior, adolescent report
Target adolescent internalizing and externalizing symptoms were assessed using the Youth Self Report form (YSR; Achenbach & Rescorla, Reference Achenbach and Rescorla2001), which was designed for 11–18-year-olds. Items were rated on a 3-point scale (0 = “not true” to 2 = “very true or often true”). The DSM-based scales for the YSR (Achenbach & Rescorla, Reference Achenbach and Rescorla2001) were calculated based on Lengua and colleagues’ (Reference Lengua, Sadowski, Friedrich and Fisher2001) evidence-based approach which utilized parent report. Of note, psychotic symptoms and ADHD were not measured by child report. Furthermore, we only had one item for child somatic complaints. Descriptive statistics and reliability can be found in Table 1. With the exception of anxious OCD, all reliabilities were in the acceptable range. The YSR is a widely used valid and reliable measure of adolescent psychopathology (Achenbach & Rescorla, Reference Achenbach and Rescorla2001). The scores of the present sample were consistent with those reported previously in community samples (e.g., average T-scores 51-55 in the current sample vs. average T-score 54 for anxious depressed, withdrawn depressed, rule-breaking, and aggressive behavior scales reported in 11-18-year-olds; Achenbach and Rescorla [Reference Achenbach and Rescorla2001]).
Child/adolescent problem behavior, parent report
The Child Behavioral Checklist, (CBCL, Achenbach & Rescorla, Reference Achenbach and Rescorla2001) designed for 6-18-year-olds was used for parent reports of adolescent psychopathology. The items were rated on a three-point scale (0= “not true” to 2= “very true or often true”). Descriptive statistics and reliability can be found in Table 1. With the exception of anxious OCD, all reliabilities were in the acceptable range. The CBCL is a widely used valid and reliable measure of adolescent psychopathology (Achenbach & Rescorla, Reference Achenbach and Rescorla2001). The scores in the present sample were consistent with those reported previously in community samples (e.g., average T-scores 50.0–52.2 in the current sample vs. average T-score 50.0–54.4 for anxious depressed, withdrawn depressed, rule-breaking, and aggressive behavior reported in 11–18-year-olds; Achenbach and Rescorla [Reference Achenbach and Rescorla2001]).
Additional measures of psychopathology
The Social Anxiety Scale for Children and Adolescents (La Greca & Lopez, Reference La Greca and Lopez1998) was used to assess fear of negative evaluation, social anxiety general, and social anxiety in novel situations. Parents and children reported on the child’s symptoms using a five-point Likert scale (1=“not at all” to 5 “all the time”). Descriptive statistics and reliability can be found in Table 1. Reliability was strong across reporters. The Disruptive Behavior Disorder Scale (DBD; Pillow et al., Reference Pillow, Pelham, Hoza, Molina and Stultz1998) was used to assess child attention deficit hyperactivity disorder (ADHD) symptoms and provided additional DSM based scales for conduct disorder (CD) and oppositional defiant disorder (ODD). Parents rated child symptoms on a four point Likert scale (1= “not at all” to 4=“very much”). Descriptive statistics and reliability can be found in Table 1. With the exception of CD, which was acceptable, the remaining DBD scales had strong reliability.
Validators
As described in the introduction, validators with established associations to psychopathology were selected, including demographic and maturation characteristics, dimensions of temperament (fear/sadness, anger/aggression, positive emotionality/surgency, and effortful control), environmental factors (peer behavior), and maladaptive outcomes (substance use). For more details on the measures including name, reporter, M, SD, Cronbach’s α, and citation see Table 1. Except for the fear and inhibitory control subscales, all constructs met minimal cutoffs for reliability. We retained these EATQ scales given their widespread use in the literature; however, it is important to note structural problems with the Rothbart battery of temperament measures (e.g., Kotelnikova et al., Reference Kotelnikova, Olino, Klein, Kryski and Hayden2016, Reference Kotelnikova, Olino, Klein, Mackrell and Hayden2017) and the scales most affected are fear and inhibitory control. Additional information on the validators can be found in Supplemental Materials, including example items and response format.
Data analysis
Confirmatory factor analysis (CFA) with maximum likelihood with robust standard errors was used to estimate all measurement models in Mplus 8.1 (Muthén & Muthén, Reference Muthén and Muthén2017). To compare the two-factor model of internalizing and externalizing symptoms to the bifactor model, a series of CFA’s were estimated from W1 to W3, tested for measurement invariance, and then assessed for fit. Given the large number of parameters involved with having both parent and child report latent factors in one model, we first tested for measurement variance within reporters and then combined parent and child report models into two final models: (1) two factor model for the child and parent (child-2 FM and parent-2 FM, respectively) and (2) bifactor model for the child and parent (child-bi and parent-bi, respectively).
In the two-factor CFA, symptoms of generalized anxiety, anxious obsessive-compulsive disorder, major depression, somatic complaints, fear of negative evaluation, social anxiety general, social anxiety in novel contexts, and psychotic symptomsFootnote 2 were indicators of internalizing latent factors and conduct disorder, oppositional defiant disorder, and attention deficit hyperactivity (ADHD) symptoms were indicators of externalizing latent factors at each of three waves for a total of six latent factors. Of note, we had fewer indicators for child report than for parent report. ADHD and psychotic symptoms were not measured by child report, and we only had one item from the somatic symptom scale for youth. Furthermore, we had two measures of ODD and CD for parents and both were included in structural models for parents. Since we had multiple indicators of some disorders (ADHD, social anxiety, ODD and CD) we also assessed whether autocorrelated errors improved model fit within the lower order disorder. Autocorrelated errors across waves were estimated given large χ 2 difference tests (all χ 2 difference tests > 280). Santora-Bentler nested χ 2 difference tests were used to assess whether weak (constraining factor loadings), strong (constraining intercepts), strict (constraining errors), and factor variance measurement invariance were observed across all three measurement waves.
In the bifactor CFA, the indicators for internalizing and the indicators for externalizing were allowed to load on internalizing and externalizing factors, respectively, and also on a co-occurring factor at each wave. Covariances between latent factors were constrained to be zero within wave but were estimated across waves within construct (see Figure 1). Here again autocorrelated errors were estimated (all χ 2 difference tests > 280). Santora-Bentler nested χ 2 difference tests were used to assess whether weak, strong, strict, and latent factor variance measurement invariance were observed across waves. Factor ρ, a latent measure of reliability similar to Cronbach’s α, and factor score determinacy, an assessment of how well factor scores correlate with latent constructs, were calculated for all factors in all models. For bifactor models, additional statistics were calculated. Explained common variance (ECV) is the proportion of common variance explained by each factor. For the bifactor ω H reflects the proportion of total variance in raw scores accounted for by the general factor, while for specific factors ω Hs reflects the proportion of reliable variance accounted for by the specific factors after partitioning the general factor. Finally, H reflects the association between a factor and an optimally weighted item composite with values > .8 reflecting well defined latent factors (see Rodriguez et al., Reference Rodriguez, Reise and Haviland2016 for details and Dueber, Reference Dueber2017 for a calculator).
After the final model was determined for each model, fit was assessed. Cutoffs for fit indices and residual correlations were used to assess model fit (Hu & Bentler, Reference Hu and Bentler1999). Next, factor scores were estimated and used to examine associations with external validators. Spearman’s ρ, standard errors, the associated 95% asymmetrical confidence intervals (CIs), and p-values were estimated in R 4.3. Bonferroni correction (W1: .05/(25*5) = .0004; W2 and W3: .05/(28*5) = .00037) was used within wave and reporter to describe associations between factor scores and validators. Heat maps were created to provide a visual representation of the matrix of correlations. Empty cells in the matrix reflect non-significant effects (p > .05), while bold and italics reflects p < .05 but greater than the Bonferroni cutoff. The remaining correlations are significant despite the Bonferroni adjustment. Cuts offs were used for small (r > .10 and < .30), medium (r > .30 and < .50), and large (r > .50) effects (Cohen, Reference Cohen1988). The use of Spearman’s ρ is important given that psychopathology, substance use, and social contextual risk factors tend to be skewed and kurtotic. Finally, differences in Spearman’s ρ across the factor scores from the bifactor and two factor models were tested for significance using R 4.3. One thousand bootstraps were estimated using the boot library (Canty & Ripley, Reference Canty and Ripley2022) to provide 95% asymmetrical CIs for the difference between Spearman’s ρ across models.
Results
Measurement models
Table 3 contains scaled χ 2 difference tests for weak, strong, strict, and factor variance measurement invariance as well as fit indices for the best fitting models. The child models demonstrated weak and strong measurement invariance, but not strict measurement invariance over time. The parent models demonstrated weak, but not strong or strict measurement invariance over time. Factor variances over time were measurement invariant for both reporters. In the combined two factor model (2FM) and bifactor models across reporter, constraints were retained and exact fit to the data was rejected. Approximate fit varied from acceptable to excellent depending on the model and the fit index. All fit indices suggested that the bifactor model fit better than the two-factor model. Furthermore, the bifactor models had lower and non-overlapping 90% CI’s for RMSEA when compared to the two-factor model. Interestingly, despite added complexity, AIC and BIC strongly preferred the bifactor model over the 2 FM.
Santora-Bentler nested χ2 tests for measurement invariance over time and fit of final models
Note: C = child/adolescent, P = parent, 2 FM = two-factor model, BI = bifactor model, MI = measurement invariance, FactorV = factor variance, RMSEA = root mean square error of approximation, RMS CI = RMSEA 90% confidence interval, CFI = comparative fit index, SRMR = Standardized Root Mean Squared Residual, AIC = Akaike information criterion, and BIC = Bayesian information criterion.
Factor loadings from both models that combined parent and child reports can be found in Table 4. A detailed description of the factor loadings and psychometrics can be found in the Supplemental Material. Briefly, all indicators of psychopathology loaded on internalizing and externalizing symptoms in the 2 FM model. The largest loadings for internalizing symptoms were generalized anxiety and depression, while the largest loadings for externalizing symptoms were CD and ODD. In the bifactor model, social anxiety and general anxiety had the largest loadings on the specific internalizing factors, while the remaining internalizing indicators did not load on specific internalizing symptoms but did load on co-occurring symptoms. Depression had the largest loading on the co-occurring factors. Social anxiety indicators did not load on co-occurring symptoms. CD had the largest loading for specific externalizing symptoms. Although coefficients varied slightly, the pattern replicated across reporters. In parent reported models, ADHD indicators had comparable loadings with specific externalizing symptoms and co-occurring symptoms. The composition of the general and specific factors in the bifactor model of psychopathology is consistent with the predictions that were outlined in the Introduction.
Factor loadings across models and reporter
Note: C = child; IN = internalizing; EX = externalizing; CO = co-occurring; B = bifactor, * = p < .05, GA = generalized anxiety, FNE = fear of negative evaluation, SAN = social anxiety in novel contexts, SAG = social anxiety in general, OCD = anxious obsessive compulsive disorder, MD = major depression, SOM = somatic complaints, CD1 = conduct disorder from the ASEBA, ODD1 = oppositional defiant disorder from the ASEBA, ODD2 = oppositional defiant disorder from the DBD, CD2 = Conduct Disorder form the DBD, ADHH = attention deficit hyperactivity disorder hyperactivity scale, ADHI = attention deficit hyperactivity disorder inattention scale, PSYC = psychotic symptoms.
Factor ρ, factor score determinacy, ECV, ω H, and H can be found in Table 5. All reliabilities were in acceptable ranges. For child report, the specific factors had higher reliabilities than the co-occurring factor, while internalizing and externalizing symptoms had lower reliabilities than the bifactor model. For parent report, internalizing in the 2 FM and co-occurring factors had the largest reliabilities, while specific externalizing symptoms had lower reliability. In the two-factor model for parents, externalizing symptoms had the highest reliability of any factor, while internalizing symptoms had a smaller reliability than child report. Other bifactor metrics suggested that co-occurring symptoms explained the same amount of common variance across reporters (53%), while the child specific factors explained less variance than both bifactors and the parent specific factors. For parents, specific externalizing symptoms explained 1% more variance than the general factor, while specific internalizing explained less (42%). Results do not support that the bifactor or 2 FM performed better across reporter. For child report, the bifactor model outperformed the 2 FM, while the opposite was the case for parent report.
Reliability and validity coefficients for the latent factors
Note: C = child; P = parent; IN = internalizing; EX = externalizing; B = bifactor, ρ = factor reliability, FD = factor score determinacy, ECV = Explained Common Variance, ω H = proportion of total variance in raw scores accounted for by the general factor, while for specific factors ωH = proportion of reliable variance accounted for by the specific factors after partioning the general factor, H = association between a factor and an optimally weighted item composite.
Cross-reporter psychopathology associations: convergent and discriminative validity
Table 6 contains correlations from each final model for all latent factors across parent and child report and within waves. Results suggested an ordered and consistent pattern of relationships across waves. Considering convergent validity, the largest cross-reporter correlations were for externalizing symptoms in both models. The 2-factor model had slightly larger (nearing a large effect) cross-reporter correlations for externalizing symptoms than the specific externalizing factor in the bifactor model. Internalizing symptoms in the 2 FM had the smallest cross reporter associations while the specific internalizing and co-occurring factors in the bifactor model had moderate correlations across reporters. Considering discriminant validity, the 2 FM had slightly worse discriminant validity than the bifactor model. For instance, child internalizing and externalizing symptoms in the 2-factor model had small correlations with parent externalizing and internalizing, respectively, while this was not observed in the bifactor model. The only factors to correlate across reporter in the bifactor model were parent report of specific internalizing and child report of co-occurring symptoms and the correlations were negative. Results suggest that convergent validity preferred the 2 FM, while discriminant validity preferred the bifactor model.
Correlations between latent factors across reporter for the two factor and bifactor model
Note: C = child; P = parent; IN = internalizing; EX = externalizing; CO = co-occurring; B = bifactor; Factor ρ = reliability coefficient; associations ≥ .10 were statistically significant at p < .05.
Validity analyses within a nomological network of validators
Child 2FM
Figure 2 contains heat maps of the Spearman’s ρ between the 25-28 validators and the child and parent reported factor scores across all three waves. Blank cells indicate lack of conventional statistical significance (p > .05), italicized and bolded cells reflect associations that were p < .05 but above the Bonferroni corrected p-value, while the remaining cells reflect correlations that were significant after the Bonferroni correction. The Supplemental Material contains asymmetric 95% CIs for all the associations in Figure 2 (see tables S2 through S7). For child-2 FM, temperament and psychopathology constructs had expected relationships with the primary domain. For instance, behavioral inhibition anxiety and depressed mood had small correlations with internalizing symptoms, while activational control, attentional control, inhibitory control, frustration, aggression, impulsivity fun seeking, drive and low shyness had small correlations with externalizing symptoms. The same pattern was observed for social contextual risk factors with social exclusion and peer victimization having a moderate correlation with internalizing, while peer delinquency across reporters, peer report of peer aggression, and early substance use had positive correlations with externalizing symptoms. Reporting female gender was associated with lower likelihood of internalizing and externalizing symptoms, while low SES was associated with externalizing symptoms. These results indicate strong convergent validity for the two-factor model. Results are consistent with the hypotheses outlined in Table 2 with several notable exceptions, including male sex having higher likelihood of internalizing symptoms and our maturational variables (age and puberty) did not correlate with psychopathology. Regarding discriminant validity, activational, attentional, and inhibitory control also correlated negatively with internalizing symptoms, while impulsivity fun seeking, frustration, aggression, substance use, and peer report of peer aggression also correlated positively with internalizing symptoms. The same pattern was not as extreme for externalizing symptoms in that only depressed mood was positively related to externalizing symptoms at W1. Social contextual risk factors displayed a similar pattern of associations in the two-factor model across internalizing and externalizing symptoms with some differences in effect size. These patterns support poor discriminant validity for internalizing symptoms in the two-factor model, but excellent discriminant validity for externalizing symptoms. There were some temperament dimensions and social contextual factors which did discriminate the two domains in the 2 FM. For instance, reward responsiveness, drive, and peer report of peer delinquency correlated with externalizing symptoms but not internalizing symptoms, while BIS anxiety correlated with internalizing but not externalizing symptoms. Patterns suggest strong convergent validity for the 2 FM model, but weaker discriminant validity for internalizing symptoms than the ideal matrices presented in Table 2. Externalizing symptoms, however, had associations comparable to the ideal matrices presented in Table 2 apart from depressed mood and the maturational constructs.
Heat maps of convergent and discriminant validity analysis across models, three waves of longitudinal data, and parent and child report. Blank cells are non-significant (p > .05); spearman’s ρ not italicized have p < .05 but p > .0004 (W1) or p > .00036 (W2 and W3), while spearman’s ρ italicized and bolded reflect p < .0004 (W1) or .00036 (W2 and W3). C = child; P = parent; IN = internalizing; EX = externalizing; CO = co-occurring; B = bifactor; FFS = fight/flight/freeze; depmood = depressed mood; surge = surgency or high intensity pleasure; rewres = reward responsiveness; frust = frustration; aggres = aggression; ppd = perceived peer delinquency; rule = rule breaking; aggr = aggression; atten = attention; inhib = inhibitory control; actct = activational control; excl = exclusion; victim = victimization.
Parent 2FM
Models comprised of indicators based on parent reports on psychopathology replicated some but not all the child external validity findings. Furthermore, effect sizes were generally larger for temperament associations in the parent model because parents reported on both temperament and psychopathology. Here, we focus on results that did not replicate across reporters. High intensity pleasure and reward responsiveness correlated negatively and positively with parent report of internalizing, respectively, which was not found in the child reported models. Reporting female gender, frustration, substance use, and perceived peer delinquency did not correlate with parent report of internalizing but did with child report. Additionally, fear and behavioral inhibition anxiety correlated positively, and affiliation correlated negatively with parent report of externalizing symptoms which was not found in the child reported models. These results suggest that parent report of internalizing symptoms had slightly better convergent and slightly weaker discriminant validity than child report of internalizing symptoms, while parent report of externalizing symptoms had very similar convergent validity as the child reported model, but slightly worse discriminant validity. Of note, the increase in convergent validity for parent report is not as impressive given that parents reported on both temperament and psychopathology.
Child-Bi
Given the number of correlations and differences, our analysis focuses on general patterns in the direction of the associations and then general patterns in the direction of the differences between associations. The latter directly tests whether convergent or discriminant validity were significantly different across the models. Temperament and social contextual risk constructs had mostly expected relationships for convergent and discriminant validity consistent with Table 2 with some exceptions. For instance, fear, FFFS, and BIS anxiety had small positive correlations and low shyness and high intensity pleasure had negative correlations with specific internalizing symptoms, while none of these constructs correlated with specific externalizing symptoms. Furthermore, low shyness correlated positively with co-occurring symptoms. Attentional, activational, and inhibitory control, impulsivity fun seeking, frustration, aggression, substance use, and male gender had small to moderate correlations with specific externalizing and co-occurring symptoms, but not with specific internalizing symptoms. In fact, reward responsiveness and drive correlated negatively with specific internalizing symptoms. A similar pattern was observed for social contextual risk factors: social exclusion and peer victimization had moderate to large positive correlation with specific internalizing and moderate positive correlations with co-occurring symptoms but not specific externalizing symptoms. Similarly, perceived peer delinquency had moderate positive correlations with specific externalizing symptoms and co-occurring symptoms, but not specific internalizing symptoms. There were also a set of correlates that discriminated specific internalizing and specific externalizing symptoms from each other and co-occurring symptoms. For instance, the FFFS correlated positively with specific internalizing symptoms, while high intensity pleasure, low shyness, reward responsiveness and drive correlated negatively with specific internalizing. Affiliation correlated negatively and reward responsiveness correlated positively with specific externalizing symptoms but did not correlate with co-occurring symptoms and either did not correlate with specific internalizing symptoms or correlated negatively with specific internalizing symptoms. Co-occurring symptoms had positive associations with depressed mood, low shyness, and peer report of peer aggression. The fact that many validators did not have an additive effect with co-occurring symptoms (e.g., traits and substance use) suggests that co-occurring symptoms are not always associated with simple additive risk of the specific domains when considering other developmental constructs. Furthermore, the bifactor model had comparable convergent validity to the 2 FM, but stronger discriminant validity than the 2 FM.
Parent-Bi
The parent-bi model also had many, but not all effects replicate associations from the child-bi model. Here we focus on what did not replicate. Specific internalizing symptoms correlated negatively with substance use and affiliation which was not found in the child model. Parent report of specific internalizing symptoms had an overall decrease in convergent validity relative to child report of specific internalizing symptoms as child specific internalizing symptoms additionally correlated positively with FFFS, social exclusion, and peer victimization, while parent report of specific internalizing did not; however, one exception to this pattern was that parent report of specific internalizing had an association with depressed mood while child specific internalizing symptoms did not. Specific parent-report of internalizing symptoms had increased discriminant validity relative to child report in that specific internalizing symptoms correlated positively with female gender and inhibitory control, while correlating negatively with substance use and affiliation. Attentional control, inhibitory control, activational control, and affiliation correlated negatively, while low shyness, high intensity pleasure, and peer victimization correlated positively with parent report of specific externalizing symptoms which was not found with child report of specific externalizing symptoms. Specific externalizing symptoms had an increase in convergent and discriminant validity relative to child report of specific externalizing symptoms. FFFS, fear, behavioral inhibition anxiety, high intensity pleasure, and affiliation had negative associations with co-occurring symptoms which was not found with child report of co-occurring symptoms. Furthermore, peer report of peer delinquency and reward responsiveness correlated positively with parent report of co-occurring symptoms but did not correlate in child reported models. In contrast, male gender, substance use, drive, lifetime tobacco use, perceived peer delinquency and social exclusion correlated with child report of co-occurring symptoms, while these validators did not correlate with parent report of co-occurring symptoms. This suggests that parent and child report of co-occurring symptoms have similar and poorer convergent and discriminant validity than other factors.
Of note, parents reported on temperament and children or peers reported on substance use and social factors. Given that the differences between the parent and the child model were mostly temperament constructs, there are several conclusions that can be drawn from these patterns: (1) child report had better overall convergent and discriminant validity given more expected associations across reporter, (2) co-occurring factors had more variance unique to reporter than the specific factors and (3) co-occurring factors were associated with a larger number of constructs but had consistently smaller correlations than the factors in the 2 FM.
Child-BI compared to child 2FM
Differences in Spearman’s ρ and associated asymmetrical 95% CI’s between the bifactor and two factor models can be found in Table 7. It is important to note that the 2 FM associations were subtracted from the bifactor model associations. As such, if a validator correlates positively with externalizing and specific externalizing symptoms in the 2 FM and bifactor models, respectively, a negative difference between the associations would mean the correlation with externalizing symptoms in the 2 FM was stronger than the association with specific externalizing symptoms in the bifactor model, while a positive difference in association would mean that specific externalizing symptoms in the bifactor model had a stronger association than externalizing symptoms in the 2 FM.
Differences in Spearman’s ρ and associated 95% CI’s between the bifactor and two factor model of psychopathology for child and parent report
Note: W = wave of data collection; C = child; P = parent; IN = internalizing; EX = externalizing; B = bifactor; EATQ = Early Adolescent Temperament Questionnaire; BAS/BIS = behavioral activation/inhibition system, FFFS = fight/flight/freeze system; EATQ DepMood =EATQ Depressed Mood; EATQHPL = EATQ High Intensity Pleasure; BASImpFun = Impulsivity/Fun Seek (BAS); BASRewRes = Reward Responsive (BAS); BASDrive = Drive (BAS); EATQLowShy = EATQ Low Shyness; EATQFrust = EATQ Frustration; EATQAggres = EATQ Aggression; PercPeerDelinq = Perceived Peer Delinquency; PeerDelinq = Peer Delinquency–Peer Report; PeerAggres = Peer Aggression–Peer Report; EATQAttenCntrl = EATQ Attentional Control; EATQInhibCntrl = EATQ Attentional Control; EATQActCntrl = EATQ Activational Control; EATQAffiliation = EATQ Affiliation; SocialExcl = Social Exclusion; PeerVictim = Peer Victimization.
The pattern of differences was similar across waves. As such, we focus on W1 in Table 7 to simplify the presentation (see supplemental material for full presentation across waves). Table 7 shows that discriminant and convergent validity for specific internalizing symptoms in the bifactor model is significantly stronger than the discriminant and convergent validity for internalizing symptoms in the two-factor model. For instance, correlations of all BAS (drive, reward responsiveness, and impulsivity/fun seeking) and surgency variables were different across the two models. Correlations were negative for the internalizing factor in the bifactor model as expected, whereas these correlations with the internalizing factor from the 2 FM model were either positive or non-significant. A similar pattern was observed for lifetime alcohol use and perceived peer delinquency. Furthermore, associations of effortful control variables (attention, inhibitory, and activational control) were different. Effortful control variables were unrelated to the specific internalizing factor in the bifactor model as expected, but these variables were negatively associated with the internalizing factor from the 2 FM. FFFS had a significantly stronger positive association with specific internalizing symptoms from the bifactor model than with internalizing symptoms in the 2 FM.
Specific externalizing symptoms from the bifactor model had significantly stronger discriminant validity but significantly weaker convergent validity than externalizing symptoms in the 2 FM. For instance, specific externalizing symptoms from the bifactor model was not associated with depressed mood, and this was a significantly smaller correlation than the significant positive association between externalizing symptoms and depressed mood in the 2 FM. Impulsivity/fun seeking, perceived peer delinquency and peer reported peer aggression had significantly smaller associations with specific externalizing symptoms from the bifactor model than with externalizing symptoms in the 2 FM. A similar pattern was observed for social exclusion and peer victimization.
Co-occurring symptoms from the bifactor model had significantly smaller associations with constructs related to both internalizing and externalizing domains. Co-occurring internalizing and externalizing symptoms from the bifactor model had significantly smaller associations with all negative affect, surgency, and BAS indicators than internalizing symptoms in the 2 FM. Co-occurring internalizing and externalizing symptoms also had significantly smaller associations with impulsivity/fun seeking, aggression, alcohol use, tobacco use, perceived peer delinquency, and peer victimization than externalizing symptoms in the 2 FM. Results suggest that co-occurring internalizing and externalizing symptoms had less convergent validity than the factors in the 2 FM. However, there were a larger number of associations with the validators for co-occurring symptoms when compared to each factor in the 2 FM.
Parent-Bi compared to parent 2FM
The overall pattern of differences between Spearman’s ρ across the bifactor and two-factor model for parents largely replicated the patterns in the child reported model with some notable exceptions. There were more and larger differences for temperament for parents given that parents reported on psychopathology and temperament in the parent models. That is, most of the temperament variables had significantly different associations between the bifactor model and the 2 FM. Specific internalizing symptoms from the bifactor model and internalizing symptoms in the 2 FM differed on all temperament constructs and the direction of differences were consistent with the pattern in child reported models. Moreover, co-occurring symptoms also had significantly stronger associations with alcohol, perceived peer delinquency, and peer report of aggression and rule breaking than internalizing symptoms in the 2 FM suggesting some cross reporter convergent validity for parent co-occurring symptoms. The overall pattern of results suggested that while discriminant validity was better in the bifactor model, convergent validity was stronger in the 2 FM. Of note, the larger number of differences in the parent model is likely due to common reporter variance.
Consistency across the waves
Spearman’s ρ across the waves
Replication was defined as similarity in Spearman’s ρ across the waves (within .02 across the waves) and the same inference regarding significance. There was a large degree of consistency across the waves for both models and reporters. Child-2 FM was associated with 90% (parent = 96%) replication while child-bi was associated with 87% (parent = 83%) from W1 to W2. From W2 to W3, child-2 FM was associated with 83% (parent = 88%) replication while child-bi was associated with 89% (parent = 83%) replication. The individual factors with the highest percentage of replication were child report of specific internalizing symptoms in which 98% of the associations replicated at all three waves and parent specific externalizing in which 93% of the associations replicated at all three waves. Parent specific internalizing and parent co-occurring symptoms were tied for the lowest percentage of replication with both having 80% replication across all three waves. For further explanation of the pattern of effects that did and did not replicate across waves see the Supplemental Material.
Summary
Results suggest a complicated pattern of differences across models and reporters. For child report, specific internalizing symptoms had stronger convergent and discriminant validity and higher reliability and replicability than internalizing in the 2 FM. Differences between the two models got larger across the waves. In contrast, specific externalizing symptoms had worse convergent validity, comparable discriminant validity, but markedly stronger reliability than externalizing symptoms in the 2 FM. Specific externalizing symptoms however, had an increase in discriminant validity over time, while externalizing symptoms in the 2 FM had an increase in convergent validity. Co-occurring symptoms had worse convergent validity than both factors in the 2 FM but was associated with a higher number of constructs and comparable reliability to both factors in the 2 FM. Furthermore, co-occurring symptoms had an increase in convergent and discriminant validity across the waves.
Parents reported on both temperament and psychopathology resulting in larger effect sizes, more effects, and larger differences across the bifactor and 2 FM; however, the patterns are remarkably similar to child report in that the bifactor model had superior discriminant validity than the 2 FM and the 2 FM had stronger convergent validity, particularly for externalizing symptoms. However, unlike child report, parent report of co-occurring symptoms in the bifactor model and externalizing in the 2 FM had the largest reliabilities, while all other factors were near .70. Replicability was also consistently stronger in the 2 FM than the bifactor model for parents.
While replicability over time was typically high, effects that did not replicate may speak to developmental changes over time. Here we describe general patterns that replicated across reporter. Social exclusion and peer victimization had associations at earlier waves but not later waves, while substance use tended to have effects at later waves. While adolescent perception of peer delinquency consistently correlated with externalizing and co-occurring constructs, peer report of rule breaking, and aggression discriminated factors at later waves. SES discriminated many of the factors at later waves when our sample was in middle adolescence.
Robustness to bonferroni correction
In Figure 2, cells that are italicized and bolded met the Bonferroni cutoff for significance (p < .0004 at W1 or p < .00036 at W2 and W3). Cells not italicized and bolded were associated with p < .05 but did not meet the more conservative Bonferroni cutoff for significance. At W1, 26% of associations (child = 25%; parent = 28%) followed this pattern, while at W2 and W3, 19% (child = 18%; parent = 20%) and 26% of associations (child = 26%; parent = 27%) followed this pattern. Most associations that changed were cross reporter associations or validators that had lower reliabilities although each domain of temperament was still associated with the psychopathology factors. The remaining variables that dropped out either retained significance across reporter or across waves suggesting some robustness to correction. Of note, Bonferroni correction is the most conservative correction and p < .05 is the least stringent. As such, Figure 2 reflects a range of possible determinations of statistical significance.
Discussion
Composition of the two-factor model was consistent with prior literature and replicated across reporters and waves. Primary markers for internalizing were major depression and generalized anxiety, while primary markers of externalizing were CD and ODD. Composition of the bifactor model was also consistent with a portion of the prior literature and replicated across reporters and waves. The general factor (“p”) was a mix of major depression, generalized anxiety, ODD and CD, while specific internalizing symptoms were composed of more anxiety than depression, although parent specific internalizing factors did correlate with depression mood, while child specific internalizing symptoms correlated with depressed mood at later waves but not earlier waves. It is possible that the composition of the general and specific externalizing factors would have been influenced more by symptoms of inattention and hyperactivity if we had more of such indicators across both informants. Specific externalizing symptoms were primarily indexed by ODD and CD. Interestingly, the patterns of loadings and convergent and discriminant validity support the developmental literature in that anxiety tended to define the specific internalizing spectra at earlier waves, while depression became associated at later waves.
In addition to composition, we examined construct validity for correlated two-factor representations of internalizing and externalizing symptoms and compared that to a model with a co-occurring internalizing and externalizing, and specific internalizing and externalizing dimensions across reporters and across the important developmental period from late childhood to middle adolescence. Relationships were compared to a-prior hypotheses for convergent and discriminant validity outlined in Table 2. While the majority of our hypotheses were supported for the temperament, social constructs, and substance use constructs associations with demographic variables and maturational variables were either not supported at all (puberty and age) or had inconsistent associations (sex).
There was clear evidence for construct validity for each model and factor. Many correlations with psychopathology in our study were consistent with prior research (see Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel, Meier, Ramrakha, Shalev, Poulton and Moffitt2014; Keiley et al., Reference Keiley, Lofthouse, Bates, Dodge and Pettit2003) and many results replicated across measurement waves. We also showed that there are strengths and weaknesses of both correlated two-factor and bifactor models of psychopathology and that validity and reliability vary by reporter and model. For parents, stronger reliability was observed for externalizing symptoms in the two-factor model and co-occurring symptoms in the bifactor model, while other factors hovered around the cutoff of .70. For child report, co-occurring symptoms and specific internalizing symptoms had the highest reliabilities with the rest of the factors hovering around .70. This is not surprising given prior literature suggesting that parents do not report as well on internalizing as externalizing symptoms. Across reporters, the bifactor model had stronger discriminant validity but weaker convergent validity than the two-factor model. Although there were larger effects for temperament and psychopathology in the parent model, this does not mean that the parent model outperformed the child model given that parents reported on both temperament and psychopathology in the parent model.
Construct validity of psychopathology
Two-factor model of internalizing and externalizing symptoms
Strong convergent validity for internalizing and externalizing dimensions in the correlated two-factor model was observed. Negative affect and avoidance (e.g., BIS anxiety) and psychosocial risk (social exclusion and peer victimization) correlated positively with the internalizing factor. Effortful control, impulsivity and psychosocial risk (peer delinquency, peer reported aggression, substance use) correlated positively with the externalizing factor. For child and adolescent self-report of psychopathology, these effects ranged from small to moderate. When parents reported on child and adolescent psychopathology, these effects ranged from moderate-to-large for temperament, but small when children or peers reported on the social context. While this demonstrates good convergent validity, there were also comparable and smaller effects across domains. That is, externalizing latent spectra correlated positively with depressed mood and BIS anxiety while internalizing correlated negatively with effortful control facets, and positively with substance use. Importantly, associations assessing convergent validity in the two-factor model were typically significantly stronger than the associations with specific internalizing symptoms and co-occurring symptoms in the bifactor model. These patterns were observed across reporters and time. Results indicate substantial common variance across internalizing and externalizing symptoms in the two-factor model which systematically relates to validators across domains. Whether this should be called “p” is an empirical question which should be based on tests of causal hypotheses and not structural tests or construct validity (Bonifay et al., Reference Bonifay, Lane and Reise2017).
Bifactor model of internalizing and externalizing symptoms and “p”
In the bifactor model, specific internalizing symptoms were consistently positively related to fear and anxiety, negatively related to BAS constructs, positively related to social exclusion and victimization, and not related to peer delinquency or substance use. Specific externalizing symptoms were positively related to frustration, surgency facets, BAS scales, peer victimization, peer delinquency and substance use and either not associated with fear, anxiety, or avoidance or associated in the opposite direction than specific internalizing. In contrast, co-occurring symptoms typically had smaller associations than either specific internalizing or specific externalizing symptoms in the bifactor model as well as internalizing and externalizing symptoms in the two-factor model, although this differed slightly by reporter. For instance, in the child reported bifactor model, co-occurring symptoms were associated with a range of risk from both internalizing and externalizing domains. The only constructs that were not related to co-occurring symptoms but were related to one of the primary domains in the two-factor were behavioral inhibition anxiety and peer report of peer delinquency. For parent report, high intensity pleasure, low shyness, lifetime tobacco use, and perceived peer delinquency did not correlate with co-occurring symptoms but did with at least one domain in the two-factor model. Furthermore, some of the traits representing internalizing symptoms were negatively related to co-occurring symptoms when parents reported on psychopathology. Put simply, co-occurring symptoms resembled specific externalizing symptoms for parents. Furthermore, specific internalizing symptom spectra for parents had lower reliability and worse replicability across reporters and time than child report of specific internalizing symptoms, which had the strongest replicability of any factor.
Worse validity of parent-reported co-occurring internalizing and externalizing symptoms may be related to lower validity, reliability, and replicability of parent report of internalizing symptoms. That is, if parents cannot report reliably on children and adolescents’ internal states (e.g., Dwyer et al., Reference Dwyer, Nicholson and Battistutta2006) it would produce bias in both specific internalizing symptoms and co-occurring internalizing and externalizing symptoms in the bifactor model. As a result, studies using bifactor models of parent-reported youth psychopathology may not capture the full complement of internalizing symptoms in the “p” factor and produce inaccurate results. This may be especially pronounced during adolescence when certain youth tend to share less with parents (Darling et al., Reference Darling, Cumsille, Caldwell and Dowdy2006). The consequences of these findings are that studies which focus on parent report of child and adolescent psychopathology may find that specific internalizing symptoms do not contribute to impairment or relate meaningfully to important risk factors or maladaptive outcomes due to poorer measurement as opposed to substantive differences (e.g., Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel, Meier, Ramrakha, Shalev, Poulton and Moffitt2014; Clark et al., Reference Clark, Hicks, Angstadt, Rutherford, Taxali, Hyde, Weigard, Heitzeg and Sripada2021).
Rarely were relationships between validators and co-occurring internalizing and externalizing symptoms the average of associations with internalizing and externalizing symptoms in the two-factor model, but it did occur on some occasions. For instance, co-occurring internalizing and externalizing symptoms were not related to high intensity pleasure, low shyness, and perceived peer delinquency but were negatively related to facets of effortful control and positively related to impulsivity and peer report of peer delinquency like specific externalizing symptoms. Furthermore, co-occurring internalizing and externalizing symptoms were positively related to social exclusion and peer victimization similarly to specific internalizing and specific externalizing symptoms, respectively. Substance use was positively associated with co-occurring symptoms and specific externalizing symptoms, especially at later waves, but was not related to specific internalizing symptoms. These patterns suggest non-additive associations between co-occurring internalizing and externalizing symptoms and certain developmental risk factors and maladaptive outcomes. Non-additive associations suggest qualitative differences in co-occurring, internalizing and externalizing symptoms. Qualitative differences among youth with co-occurring symptoms suggests that co-occurring symptoms may have a unique developmental trajectory, especially given the debates about the interpretation of the general factor.
Some researchers argue that “p” is a composite of all internalizing and externalizing symptoms (Fried et al., Reference Fried, Greene and Eaton2021), while others consider “p” an indicator of overall impairment resulting from underlying psychopathology (Smith et al., Reference Smith, Atkinson, Davis, Riley and Oltmanns2020). Still others argue that “p” is a general liability for psychopathology through non-specific processes such as cognitive distortions and temperament/personality traits (e.g., negative emotionality; Lahey et al., Reference Lahey, Moore, Kaczkurkin and Zald2021). While our results provide some support for each conceptualization, they do not support any one entirely. In our study, “p” was frequently associated with the same constructs as internalizing and externalizing symptoms in the two-factor model. Since our validators spanned temperament, psychosocial risk, and maladaptive behavior, “p” has components of variance of general liability (e.g., temperament), psychosocial impairment (e.g., substance use, peer delinquency and aggression), and maladaptive behavior (e.g., aggression, sad mood). While it is tempting to conclude that the overall pattern suggests that “p” behaves like a composite, there were also instances in which “p” was not associated with constructs that were associated with internalizing and externalizing symptoms in the two-factor model (child report: behavioral inhibition anxiety and SES; parent report: gender, SES, low shyness, tobacco use, and perceived peer delinquency). Furthermore, some of these associations changed over time and two common negative life events (e.g., social exclusion and peer victimization) also correlated with “p” consistently, earlier in development. As such, “p” may also be related to developmental change in risk factors for psychopathology and its maladaptive outcomes.
Parent report of child psychopathology provided less divergence of co-occurring internalizing and externalizing symptoms from specific externalizing symptoms. Nonetheless, there were a few differences. Frustration and peer report of peer aggression had significantly larger associations with the general factor than specific externalizing symptoms, while lifetime tobacco use correlated with specific externalizing symptoms and not with the general factor.
As we noted in the introduction, specific factors of bifactor models of psychopathology tend to be less replicable and less stable than the general factor across studies and age groups. However, in our study, we have demonstrated that the specific internalizing and externalizing factors were quite stable and replicable over time. They had strong reliability and replicability for child report as well as stronger test-retest correlations than the two-factor model. While reliability and replicability were slightly lower for parent report in the bifactor model relative to the two-factor model, parent report was also associated with stronger test-retest correlations in the bifactor than the two-factor model. As such, the specific factors are psychologically meaningful constructs for both reporters and the instability of the specific factors observed in the extant literature could be stemming from inconsistency in the indicators used and modeling specifications.
Implications for the literature
Limited convergent or discriminant validity in any given study could have a range of effects on the inferences made in the extant literature depending on specific design features of individual studies. Here we consider several common designs in the clinical and developmental literature for illustrative purposes. Design 1 involves using only child or only parent report to measure psychopathology. Design 2 involves modeling psychopathology as internalizing and externalizing factors and including both as outcomes or predictors in developmental prediction models. Design 3 involves separating internalizing and externalizing symptoms in different statistical models due to large overlap. Finally, Design 4 involves using bifactor modeling over time. Following our results, utilizing Design 1 could result in different conclusions regarding temperament-based and psychosocial risk, especially in studies (1) which only include parent report and (2) utilize internalizing symptoms as an outcome. Design 2 would remove large portions of variance which overlap with temperament and psychosocial risk, especially in studies which adjust for concurrent psychopathology. Design 3 would reduce likelihood of detecting unique associations with internalizing and externalizing symptoms given that a large portion of the variance in each model overlaps with the other form of psychopathology as well as temperament, psychosocial risk, and maladaptive outcomes. Furthermore, Design 2 makes effects harder to detect and Design 3 could result in the two forms of psychopathology appearing more similar than they are. Design 4 may increase likelihood of discriminant relationships that may vary depending on whether child or parent report of psychopathology is utilized. Design 4 could also result in different forms of psychopathology appearing more similar than they are if only parent report is utilized. For instance, using parent report could result in specific externalizing symptoms and co-occurring internalizing and externalizing symptoms appearing more similar if frustration, peer report of peer aggression, or tobacco use is not included. The effect of each of these designs on the extant literature may be more extreme in studies which observe large stability of psychopathology and a larger correlation across domains within each measurement wave. When considering the sampling distribution of all studies, the magnitude of the correlation between internalizing and externalizing symptoms across and within reporters will vary. Lack of convergent or discriminant validity combined with the above design features and natural variations across studies likely adds noise to the literature on the structure of psychopathology and the literature on temperament-based and psychosocial risk factors as well as maladaptive outcomes.
Limitations
Our study should be understood within the context of certain limitations. First, to the extent that diagnostic interviews increase reliability and decrease covariance across different forms of psychopathology, our results may not replicate using diagnostic interviews. However, the extant literature has been consistent in supporting dimensional models of psychopathology (e.g., Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby, Brown, Carpenter, Caspi, Clark, Eaton, Forbes, Forbush, Goldberg, Hasin, Hyman, Ivanova, Lynam, Markon, Miller, Moffitt, Morey, Mullins-Sweatt, Ormel, Patrick, Regier, Rescorla, Ruggero, Samuel, Sellbom, Simms, Skodol, Slade, South, Tackett, Waldman, Waszczuk, Widiger, Wright and Zimmerman2017). Second, we did not have as many items for child report of somatic symptoms, while we did not have child report measures of ADHD and psychotic symptoms. Including more externalizing and psychotic symptoms for youth could change the composition of the latent factors we estimated and change our results. Third, our analyses included youth from late childhood to middle adolescence. To the extent that psychopathology continues to change into young adulthood, our results may not generalize to older populations and should be the target of future studies. Fourth, we estimated bivariate correlations between validators and different reporters of psychopathology; as such, reported effects do not reflect causal relationships. Fifth, our analyses did not include different methods of measurement for psychopathology (e.g., structured interviews or observation), just different reporters. While multiple reporters is an improvement, not having more than one method does not allow for a more structured analysis of convergent and discriminant validity such as a multimethod multi-trait matrix.
Clinical implications
The clinical implications of a general factor are far-reaching, especially in the context of non-additive associations with certain risk factors and consequences of psychopathology. The general factor reflects overlap between internalizing and externalizing symptoms and this overlap had non-linear effects with etiological factors, psychosocial risk, and maladaptive outcomes of psychopathology suggesting that co-occurring internalizing and externalizing symptoms could be a unique developmental trajectory. Work on the etiology of psychopathology may benefit from focusing on differentiating youth who stay within one domain of psychopathology from those who develop co-occurrence between internalizing and externalizing domains. Similarly, work on the maladaptive outcomes of psychopathology may benefit from considering co-occurrence of internalizing and externalizing symptoms as well as specific domains. Understanding these trajectories has the potential to lead to more targeted intervention and prevention at specific ages and speaks to the importance of transdiagnostic treatments within and across internalizing and externalizing symptoms. Our results also speak to the importance of using multiple informants in clinical settings, (self, parent, peer) especially for adolescents.
Conclusions and future directions
There are strengths and weaknesses of using a traditional two factor model of psychopathology. Strengths of the traditional two-factor model are consistent and high reliability and replicability for parent report and high convergent validity with a range of developmental validators thought to be involved in the etiology and maladaptive outcomes of different forms of psychopathology. The weakness of the two-factor model is a marked decrease in discriminant validity. This disadvantage will be especially pronounced when internalizing and externalizing symptoms are modeled alone or together as correlated independent variables given that systematic variance which is differentially related to developmental outcomes will be retained or partitioned out of the model, respectively. Research focusing on one outcome or the other will necessarily be biased and this bias will increase when only parent report is used. An example from the literature is the positive association between internalizing symptoms and substance use, which becomes non-significant or negative when externalizing symptoms are included (see Colder et al., Reference Colder, Scalco, Trucco, Read, Lengua, Wieczorek and Hawk2013; Foster et al., Reference Foster, Hicks and Zucker2018). Researchers should pay close attention to variance explained in addition to unique prediction when assessing etiological factors as predictors of psychopathology or consequences of psychopathology.
Bifactor models which include a “p” factor for psychopathology also have advantages and disadvantages. Reliability was lower for parent report, suggesting that etiological prediction may require larger sample sizes. However, reliability was higher in the bifactor model when youth reported on psychopathology and was associated with higher replicability relative to the two-factor model. While convergent validity was comparable for the bifactor model, discriminant validity was much stronger, especially for child report of psychopathology. Taken together, findings suggest that for child report, a bifactor model is superior in many ways to the two factor model across reporters, especially when trying to differentiate different patterns of psychopathology; however, the two-factor model may be better for parent report as parents may struggle to differentiate internal states of youth and bias two factors in the bifactor model (specific internalizing symptoms and co-occurring internalizing and externalizing symptoms).
Our broader theoretical contention is that “p” is likely related to temperament, psychosocial risk, and maladaptive outcomes due to potentially complex mediation and moderation sequences across development which are nested in historical time. Much of the work on “p” has focused on structural analyses that have been collapsed across time. Furthermore, cross sectional associations are typical for establishing convergent and discriminant validity as we have done. In contrast, most developmental work on internalizing and externalizing symptoms in the two-factor model, assesses longitudinal relationships and utilizes diverse methods (e.g., cross lag panel design vs. growth curve model) to assess change including within and between person change. If changes in “p” are systematically related to prior temperament or prior negative life experiences and then related to later impairment, it would support several interpretations of “p” just at different points in development.
Considering patterns of psychopathology over time in the context of an extensive nomological network may contribute to our understanding of unique developmental trajectories. Future work may benefit from considering prospective prediction across development to assess whether constructs can explain the development of different patterns of psychopathology within a crucial developmental epoch for the onset and escalation of psychopathology.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S095457942400107X.
Funding statement
This research was supported by a grant from NIDA (R01 DA020171) awarded to Dr Craig Colder.
Competing interests
None.
