Introduction
Externalizing problems encompass a range of maladaptive behavioral and conduct-related problems, typically directed toward the social environment (Abad & Forns, Reference Abad and Forns2008). They include a variety of disruptive behaviors, falling into two primary categories (Frick & Kimonis, Reference Frick, Kimonis, Maddux and Winstead2005). The first involves symptoms related to attention, impulsivity, and hyperactivity, which are commonly associated with attention deficit hyperactivity disorder (ADHD). The second includes conduct problems and aggressive behavior linked to the diagnostic categories of oppositional defiant disorder (ODD) and conduct disorder (CD). Externalizing problems are one of the most prevalent mental health problems in children and adolescents, posing significant challenges to mental health systems globally (Zwaanswijk et al., Reference Zwaanswijk, Van Der Ende, Verhaak, Bensing and Verhulst2003). They have serious adverse personal and social effects on youth, including alcohol use (King & Chassin, Reference King and Chassin2007; Moss & Lynch, Reference Moss and Lynch2001), bullying (Arslan, Reference Arslan2022), school discrimination (Bennett et al., Reference Bennett, Roche, Huebner and Lambert2020), early school dropout (Esch et al., Reference Esch, Bocquet, Pull, Couffignal, Lehnert, Graas, Fond-Harmant and Ansseau2014), and poorer academic performance (e.g., Van der Ende et al., Reference Van der Ende, Verhulst and Tiemeier2016).
The worldwide understanding of externalizing problems has improved significantly over the past several decades, leading to a growing interest in the cross-cultural assessment of externalizing problems (e.g., Stevanovic et al., Reference Stevanovic, Jafari, Knez, Franic, Atilola, Davidovic, Bagheri and Lakic2017) and the influence of cultural factors on the development, perception, expression, and reporting of these problems in children and adolescents (e.g., Lansford et al., Reference Lansford, Godwin, Bornstein, Chang, Deater-Deckard, Di Giunta, Dodge, Malone, Oburu, Pastorelli, Skinner, Sorbring, Steinberg, Tapanya, Uribe Tirado, Alampay, Al-Hassan and Bacchini2018; Watabe et al., Reference Watabe, Lee and Matsuhashi2024). Cross-cultural studies consistently show that cultural norms and values play a critical role in shaping how externalizing behaviors are recognized and evaluated by parents, teachers, and clinicians (e.g., Pelham et al., Reference Pelham, Fabiano and Massetti2005; Rescorla et al., Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Wei Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson and Verhulst2007; Zwirs et al., Reference Zwirs, Burger, Buitelaar and Schulpen2006). However, there are mixed findings on the cross-cultural differences in the relationships between externalizing symptoms and the well-known predictors of child and adolescent psychopathology, such as harsh parenting (Pinquart, Reference Pinquart2021), parental warmth and control (Rothenberg et al., Reference Rothenberg, Lansford, Alampay, Al-Hassan, Bacchini, Bornstein, Chang, Deater-Deckard, Di Giunta, Dodge, Malone, Oburu, Pastorelli, Skinner, Sorbring, Steinberg, Tapanya, Tirado and Yotanyamaneewong2020), and parent–adolescent communication (Kapetanovic et al., Reference Kapetanovic, Rothenberg, Lansford, Bornstein, Chang, Deater-Deckard, Di Giunta, Dodge, Gurdal, Malone, Oburu, Pastorelli, Skinner, Sorbring, Steinberg, Tapanya, Uribe Tirado, Yotanyamaneewong and Peña Alampay2020). Despite the increase in cross-cultural studies on adolescent psychopathology, evidence remains limited regarding the cross-national measurement invariance of instruments commonly used to assess externalizing problems (e.g., Petot et al., Reference Petot, Petot and Chahed2023; Stevanovic et al., Reference Stevanovic, Urbán, Atilola, Vostanis, Singh Balhara, Avicenna, Kandemir, Knez, Franic and Petrov2015; Sourander et al., Reference Sourander, Westerlund, Kaneko, Heinonen, Klomek, How Ong, Fossum, Kolaitis, Lesinskiene, Li, Nguyen, Kumar Praharaj, Wiguna, Zamani and Gilbert2024). Furthermore, a lack of measurement invariance testing is surprisingly common in studies focusing on the cross-national comparisons in levels and correlates of externalizing problems (e.g., Ruchkin et al., Reference Ruchkin, Sukhodolsky, Vermeiren, Koposov and Schwab-Stone2006; Tsitsika et al., Reference Tsitsika, Janikian, Wójcik, Makaruk, Tzavela, Tzavara, Greydanus, Merrick and Richardson2015; van Oort et al., Reference van Oort, van der Ende, Wadsworth, Verhulst and Achenbach2011). Without establishing measurement invariance, it is difficult to determine whether observed differences across cultural groups reflect true differences in behavior or are artifacts of measurement bias. Large-scale cross-national studies, including those on adolescent mental health, are time- and resource-consuming and challenging to execute, which encourages researchers to assess a wide range of factors. Therefore, brief, cost-effective, and easily administered self-report tools have great potential for use in cross-cultural studies involving participants from diverse countries. In addition, recent global initiatives, such as those led by UNICEF and the WHO (e.g., Carvajal-Velez et al., Reference Carvajal-Velez, Ahs, Requejo, Kieling, Lundin, Kumar, Luitel, Marlow, Skeen, Tomlinson and Kohrt2023; Guthold et al., Reference Guthold, Carvajal-Velez, Adebayo, Azzopardi, Baltag, Dastgiri, Dua, Fagan, Ferguson, Inchley, Mekuria, Moller, Servili and Requejo2023) focusing on mental health among adolescents and young people at the population level, also call for such brief, cross-culturally validated tools. With the aim of contributing to the field of cross-cultural assessment of adolescent mental health, the present study evaluated the cross-national measurement invariance of a recently developed Youth Externalizing Problems Screener (YEPS; Renshaw & Cook, Reference Renshaw and Cook2019), using adolescent samples recruited across 32 countries. We evaluated the degree to which the YEPS ensures valid cross-cultural comparisons of externalizing problems and investigated its potential as a culturally sensitive and psychometrically robust tool.
Youth externalizing problems screener
Researchers have shown increasing interest in universal school-based screening as an approach to detecting emotional and behavioral problems in adolescents (Cook et al., Reference Cook, Williams, Guerra, Kim and Sadek2010). For at-risk adolescents, self-report assessments are considered the most effective method for identifying these problems and developing appropriate school-based prevention or intervention strategies (Moore et al., Reference Moore, Littlecott, Turley, Waters and Murphy2015). Despite this recognized need, there is a notable lack of brief and cost-effective self-report measures for assessing externalizing problems. Existing assessment tools are either lengthy (Child Behavior Checklist, Achenbach & Rescorla, Reference Achenbach and Rescorla2001; Burks Behavior Rating Scales-2, Burks & Gruber, Reference Burks and Gruber2006), designed for parent or teacher rather than adolescent self-report (Conners 3 Global Index, Conners, Reference Conners2008), or have low sensitivity (Strengths and Difficulties Questionnaire, Goodman, Reference Goodman2001). These limitations have been outlined by Renshaw and Cook (Reference Renshaw and Cook2019), who identified three key reasons why a new measure of externalizing problems in school-aged children is needed: (1) current tools often fail to adequately cover externalizing problems, raising concerns about construct validity; (2) existing screeners have limited diagnostic accuracy and classification utility; and (3) there is a shortage of free, publicly available self-report screeners for adolescents, making regular mental health screening in secondary schools cost-prohibitive. In response to these challenges, several new brief scales for assessing externalizing problems in adolescents have been developed, including the YEPS. However, these instruments still lack extensive validation across different cultural contexts.
The YEPS is a 10-item self-report questionnaire designed to assess global externalizing behavior problems in children and adolescents (Renshaw & Cook, Reference Renshaw and Cook2019). It was developed as a brief tool for assessing mental health in educational settings, focusing on general externalizing behaviors rather than specific diagnostic criteria (Weeks et al., Reference Weeks, Renshaw, Rainey and Hiatt2024). Based on broader psychopathology theory (Achenbach, Reference Achenbach1985) and previous research on similar brief behavior rating scales, the scale’s authors hypothesized that the YEPS would demonstrate a robust, one-factor structure, aligning with its intended use as a unidimensional screener for general externalizing problems (Jovanović, Reference Jovanović2024; Renshaw & Cook, Reference Renshaw and Cook2019).
To date, only a few studies have investigated the psychometric properties of the YEPS. In their preliminary validation of the YEPS, Renshaw and Cook (Reference Renshaw and Cook2019) initially established a one-factor structure by removing four low-loading items (from the 14-item version) and introducing covariance between conceptually overlapping items. A modified one-factor model of the YEPS with correlated error terms between two pairs of similarly worded items provided the best fit in Serbian samples, demonstrating full scalar invariance across gender and over time (Jovanović, Reference Jovanović2024). Using a sample of Korean primary school students, Kim and Choe (Reference Kim and Choe2022) examined the factor structure and predictive utility of several screening measures, including the YEPS. They found that a two-factor model of the YEPS provided a better fit to the data than the one-factor model previously supported in US samples. Similarly, Weeks et al. (Reference Weeks, Renshaw, Rainey and Hiatt2024) found that three YEPS items (item 2: “I have a hard time sitting still when other people want me to”; item 7: “I have a hard time focusing on things that are important”; and item 9: “I get distracted by the little things happening around me”) showed low or inverse loadings onto their expected factor. Further exploration showed that these items – related to difficulties with sitting still, focusing, and getting distracted – formed a new Attention Problems factor, implying that a more complex factor structure might be more appropriate. These studies highlight the importance of cross-cultural validation of mental health screening tools, as they demonstrate that the YEPS may have a different factor structure in different cultural contexts.
It is important to emphasize that differences in the factor structure of the YEPS across cultural contexts may reflect cultural variations in how externalizing behaviors are conceptualized or expressed. Different cultures might prioritize and interpret behavioral symptoms in distinct ways, potentially leading to variation in symptom clustering (Watabe et al., Reference Watabe, Lee and Matsuhashi2024). Moreover, the conceptual boundaries of externalizing problems may differ across cultures. Some research suggests that there is a relatively high degree of overlap between these constructs in the United States (Waschbusch, Reference Waschbusch2002), resulting in a one-factor model fitting best, whereas, in other countries, comorbidity may be far less common (e.g., Kim & Choe, Reference Kim and Choe2022). Although the intricate interplay between sociocultural contexts and mental health is well recognized, culture remains disconnected from assessment in adolescent psychopathology (Causadias & Cicchetti, Reference Causadias and Cicchetti2018), necessitating further cross-cultural research in the field of adolescent mental health.
The present study
The cross-cultural utility of the YEPS is still largely unknown. To date, its factor structure has been examined only in American (Renshaw & Cook, Reference Renshaw and Cook2019; Weeks et al., Reference Weeks, Renshaw, Rainey and Hiatt2024), South Korean (Kim & Choe, Reference Kim and Choe2022), and Serbian (Jovanović, Reference Jovanović2024) samples. Despite this limited number of validation studies, the existing research reveals inconsistencies in the YEPS’s factor structure, with empirical evidence supporting both one-factor and two-factor solutions (e.g., Kim & Choe, Reference Kim and Choe2022; Renshaw & Cook, Reference Renshaw and Cook2019). Furthermore, Jovanović (Reference Jovanović2024) conducted the only study to date examining cross-national measurement invariance between Serbian and US samples, highlighting the need for caution when comparing YEPS scores across these countries. This emphasizes the importance of validating the scale across a broader range of countries and languages to ensure its reliability and validity for cross-national use.
Cross-cultural comparisons in psychological research often face challenges related to bias, including differential item functioning, cultural norms, and linguistic nuances. These issues can distort findings and lead to erroneous conclusions about group differences (Milfont & Fischer, Reference Milfont and Fischer2010). Measurement invariance testing plays a pivotal role in addressing these challenges by ensuring that the instrument measures the intended construct (externalizing problems) equivalently across diverse populations. Without such validation, observed differences in externalizing behavior across countries may reflect artifacts of cultural biases or measurement errors rather than genuine variations. To address these limitations, this study evaluated the factor structure and cross-national measurement invariance of the YEPS using adolescent samples from 32 countries. Establishing cross-national measurement invariance not only ensures the robustness of the YEPS as a tool for international research but also facilitates meaningful cross-cultural comparisons. Such comparisons offer invaluable insights into how externalizing behaviors manifest across diverse sociocultural contexts, enabling researchers and practitioners to develop more culturally sensitive interventions and policies.
Methods
Sample and procedure
The data used in this study were collected via an international research collaboration, which is a part of a larger research project titled “Identity orientations and mental health in adolescence: Examining the roles of basic psychological needs and culture (IDIOMATIC)”. The sample included 17,489 adolescents (age range = 14 – 19 years, 58.7% female) from 32 countries across five continents. Sample sizes varied across countries (e.g., n = 200 in Austria, n = 1,150 in Hungary; M = 546, Mdn = 507). Detailed information about each country’s sample size, language used, method of administration (online or paper-and-pencil), and participants’ demographic characteristics can be found in Table 1.
Sample characteristics broken down by country
In most countries, researchers used convenience sampling to recruit participants from secondary schools. Before data collection began, each research group obtained approval from its institutional review board or ethics committee. Informed consent was obtained from all participants, and participation in the study was voluntary.
Instruments
The YEPS (YEPS; Renshaw & Cook, Reference Renshaw and Cook2019) is a brief 10-item instrument designed to assess externalizing problems in adolescents. It consists of six items for the conduct problems/oppositional defiance (C/O) domain (e.g., I break rules whenever I feel like it) and four items for the hyperactivity–impulsivity/inattention (H/I) domain (e.g., I have a hard time focusing on things that are important). Items are rated on a 4-point Likert-type scale ranging from 1 (almost never) to 4 (almost always). Table S1 in the Supplementary Materials provides the item contents and descriptive statistics.
The YEPS was administered in different languages across various countries and regions. In Canada and South Africa, researchers used the original English version. For other countries, the scale was translated into the local language by native speakers. The translation process utilized either the most popular cross-cultural social research back-translation method (Kowal, Reference Kowal2024) or a committee approach (van de Vijver & Leung, Reference Van de Vijver and Leung2021).
Data analysis plan
All analyses were conducted in Mplus version 8.11 (Muthén & Muthén, 1998–Reference Muthén and Muthén2017). The study’s data and code can be accessed through the Open Science Framework website at: https://osf.io/y5mj6/. To address missing data, we employed the default algorithms provided by Mplus for the Weighted Least Square Mean and Variance Adjusted (WLSMV) estimator. WLSMV accommodates missing data under the missing at random assumption by using pairwise available information, thus retaining partially missing cases without requiring imputation (Asparouhov & Muthén, Reference Asparouhov and Muthén2010). In addition, this estimator performs well with ordinal indicators and produces robust estimates without the added complexity of pooling results across imputed data sets (Rhemtulla et al., Reference Rhemtulla, Brosseau-Liard and Savalei2012).
The factor structure of the YEPS was tested using the CFA. Two competing models were evaluated: (1) a one-factor model with all 10 items loading on a single factor, and (2) a two-factor model with two correlated factors of H/I (items: 2, 5, 7, 9) and C/O (items: 1, 3, 4, 6, 8, 10). The YEPS response scale was treated as ordinal, leading to the use of the WLSMV estimator, which is recommended for ordinal categorical data with five or fewer response options (Brauer et al., Reference Brauer, Ranger and Ziegler2023). The following fit indices were used to evaluate the model fit: the RMSEA, the CFI, and the TLI. Criteria for determining model fit (Wang & Wang, Reference Wang and Wang2020) were: CFI and TLI ≥ .95 (good) and ≥ .90 (acceptable), and RMSEA 90% CI upper values ≤ .05 (good) and ≤ .10 (acceptable).
To examine measurement invariance across countries, two complementary procedures were employed: (1) a traditional, multi-group CFA approach for testing exact invariance (Bowen & Masa, Reference Bowen and Masa2015) and (2) alignment optimization to assess approximate invariance (Asparouhov & Muthén, Reference Asparouhov and Muthén2023). Multi-group CFA approach started with testing a configural invariance model that assumed equal factor structure across groups. In the next step, scalar invariance was tested, which held factor loadings and thresholds equal across groups. We omitted the metric invariance step because factor loadings and thresholds should be tested in tandem when items are considered ordinal (e.g., Sass et al., Reference Sass, Schmitt and Marsh2014). If the full scalar invariance was not supported, we proceeded with testing the partial scalar invariance by relaxing constraints on parameters that showed the greatest level of non-invariance as evidenced by modification indices. Since no clear guidelines have been offered for invariance testing of the CFA models with ordinal data across a large number of groups, we decided to compare the models using the traditional cut-off values of .01 for ΔCFI and .015 for ΔRMSEA (Chen, Reference Chen2007; Cheung & Rensvold, Reference Cheung and Rensvold2002). Given the challenges of achieving scalar invariance in large cross-cultural studies involving many countries (Marsh et al., Reference Marsh, Guo, Parker, Nagengast, Asparouhov, Muthén and Dicke2018), we also evaluated measurement invariance within groups of countries within the same regions (Asia, Europe, Latin America and the Caribbean, and the Americas).
Given the large number of groups involved in our study, which typically makes scalar invariance untenable (Marsh et al., Reference Marsh, Guo, Parker, Nagengast, Asparouhov, Muthén and Dicke2018), we also tested approximate invariance using the alignment procedure (see Asparouhov & Muthén, Reference Asparouhov and Muthén2014); this estimates group-specific factor means and variances without requiring identical factor loadings and intercepts (thresholds for ordinal data) across all groups. The procedure starts from a baseline configural invariance model (allowing all parameters to differ across groups) and then uses an optimization algorithm to find the factor mean and variance values for each group that minimize the total amount of non-invariance across the groups’ loading and intercept parameters. This procedure enables meaningful comparisons of factor means even when some measurement parameters differ between countries. In our analysis, we used the “free” alignment method, which allows both the factor means and factor variances for all groups to be freely estimated. This optimization process prefers solutions that are simpler to interpret, meaning it favors patterns where only a small number of measurement parameters vary significantly across countries. We used the 25% cutpoint for evaluating the acceptable proportion of non-invariant parameters (Muthén & Asparouhov, Reference Muthén and Asparouhov2014).
Internal consistency was examined using McDonald’s omega (ω) coefficient, with values above .70 typically considered acceptable (Flora, Reference Flora2020).
Results
Factor structure of the YEPS in each country separately and the pooled sample
Table 2 presents the fit indices for the original one-factor and two-factor models in each of the 32 countries as well as in the pooled sample. Both one- and two-factor models provided a poor fit to the data based on the CFI, TLI, or upper CI of RMSEA values in 28 and 27 countries, respectively. An acceptable or good fit to the data was achieved in four countries (Bangladesh, India, Indonesia, Japan) for the one-factor model and in five countries (Bangladesh, India, Indonesia, Japan, Turkey) for the two-factor model.
Factor structure of the YEPS in each country and in the pooled sample
Note. CFI = Comparative Fit Index; TLI = Tucker–Lewis Index; RMSEA = Root Mean Square Error of Approximation; CI = confidence interval.
Due to the high correlations between factors in the two-factor model across all countries (average r = .775), we proceeded with modifying the one-factor model to yield a well-fitting model. After inspecting modification indices, we included correlated error terms between theoretically meaningful and conceptually related pairs of items that are similar in meaning (Bandalos, Reference Bandalos2021).
A total of 22 countries had an acceptable fit for the modified one-factor solution after introducing correlated residuals into the model (Table 3), whereas six countries (Estonia, Iran, Lithuania, Macau, North Macedonia, and South Africa) failed to reach an acceptable fit even with model modifications. The number of correlated item pairs needed to achieve an adequate model fit ranged between one (Cuba, Germany, and Mexico) and five (Canada) (see Table 3).
Modified one-factor CFA model
Note. CFI = Comparative Fit Index; TLI = Tucker–Lewis Index; RMSEA = Root Mean Square Error of Approximation; CI = confidence interval.
In all countries except Cuba, there was evidence of correlated residuals between item 7 (“I have a hard time focusing on things that are important”) and item 9 (“I get distracted by the little things happening around me”), which measure symptoms of ADHD. Both items assess attention regulation difficulties, with the former tapping into poor concentration and the latter tapping into susceptibility to external distractions. Similarly, correlated residuals between item 1 (“I lose my temper and get angry with other people”) and item 3 (“I fight and argue with other people”) were found in 16 out of 22 countries. These items also show content overlap, as both measure conduct problems. Given this substantial content overlap, the scale may benefit from additional revision by removing one item from each pair. To maintain scale brevity while preserving clarity, we removed two items with more complex wording (item 1: “I lose my temper and get angry with other people”; and item 9: “I get distracted by the little things happening around me”) and re-ran the analyses on the abbreviated, 8-item version of the YEPS (Youth Externalizing Problems Screener-Short Form; YEPS-SF).
Factor structure of the 8-item youth externalizing problems screener (YEPS-SF)
Table 4 presents fit indices for the one-factor model of the YEPS-SF in each country separately. The original one-factor model of the YEPS-SF yielded an acceptable or good fit to the data in 17 countries. In all other countries except Macau, acceptable fit was achieved after allowing correlated residuals between items with similar content (see Table 4 for details). These were item 2 (“I have a hard time sitting still when other people want me to”) and item 7 (“I have a hard time focusing on things that are important”); item 6 (“I say or do mean things to hurt other people”) and item 8 (“I like to annoy people or make them upset”); and item 4 (“I break rules whenever I feel like it”) and item 10 (“I choose not to follow directions and don’t listen to adults”). These correlations reflect meaningful content overlap: the first pair of items measures different aspects of self-regulation deficits, i.e., physical restlessness versus attentional control; the second pair reflects intentional harm through either direct cruelty or provocative annoyance; and the third pair taps into defiance toward authority through situational rule-breaking and rejection of adult guidance. The model fit was not acceptable in Macau even after model modifications.
Factor structure of the YEPS-SF in each country and in the pooled sample
Note. CFI = Comparative Fit Index; TLI = Tucker–Lewis Index; RMSEA = Root Mean Square Error of Approximation; CI = confidence interval.
Overall, the one-factor solution resulted in well-defined and reliable factors in the vast majority of countries, following the addition of residual covariances in some cases (see Table S2 for standardized factor loadings and Table 5 for omega coefficients and mean loadings in each country). The only exceptions were Turkey, with somewhat lower reliability (ω = .619), and South Africa, which had two (out of eight) zero loadings. Average factor loadings ranged from .427 (Turkey) to .816 (Japan), and omegas (excluding Turkey) ranged from .771 (Indonesia) to .903 (Japan).
Mean loadings and omegas for full and short YEPS versions
Note. λ = standardized factor loading; ω = Omega coefficient of reliability.
In sum, the original or modified one-factor solutions of the YEPS-SF were supported in 31 out of 32 countries, so these models were retained for the measurement invariance analyses. Macau was excluded from the measurement invariance testing due to the poor fit of all models.
Cross-national measurement invariance of the YEPS-SF
The fit indices for the measurement invariance models are shown in Table 6. Configural invariance models demonstrated acceptable or good fit across all countries and within continents in each analysis. In the configural model, specific structural modifications for different groups are permitted, including error covariances and cross-loadings, as long as the basic pattern of factor loadings is maintained (Byrne & van de Vijver, Reference Byrne and Van de Vijver2017). The scalar model showed a significant decline in fit compared to the configural model, both across all 31 countries and within continents. Scalar invariance was supported only when the analysis was restricted to Northern European countries (Estonia and Lithuania).
Exact measurement invariance testing of the YEPS-SF across countries and continents
Note. CFI = Comparative Fit Index; RMSEA = Root Mean Square Error of Approximation; CI = confidence interval; Δ = change.
Partial scalar measurement invariance across 31 countries was supported only after six out of eight items (i.e., loadings and thresholds) were allowed to vary across countries. The most invariant items were items 5 (“I talk a lot and interrupt others when they are talking”) and 10 (“I choose not to follow directions and don’t listen to adults”), whereas the remaining six items showed strong evidence of non-invariance. Partial scalar invariance testing within continents further supported non-invariance of many of the YEPS items. A total of seven, four, three, and four item loadings and thresholds had to be freely estimated in Europe, Asia, Latin America and the Caribbean, and the Americas, respectively, to achieve partial scalar invariance (see Table 6 for details).
Given that achieving partial scalar invariance was most problematic for Europe, measurement invariance was separately conducted across European UN sub-regions. In order to provide support for partial scalar invariance, it was necessary to relax the constraints of five (Eastern Europe), three (Southern Europe), and two (Western Europe) item loadings and thresholds (see Table 6 for details).
Alignment method results for the YEPS-SF
We used the original one-factor model as a baseline model for the alignment procedure of the YEPS-SF, as it fitted well to the data in most countries and resulted in an acceptable fit of the configural model.
The alignment method (see Table S3) revealed that approximately one-fifth of the parameters (216 out of 992, 22%) were non-invariant. The number of non-invariant thresholds and loadings was 194 and 22, respectively. Parameters that were non-invariant in the largest number of countries were: (a) threshold 1 (marking the transition from “almost never” to “sometimes”), which was non-invariant in at least five countries for each item (mean = 11.5), with the threshold 1 for items 3 (“I fight and argue with other people”), 8 (“I like to annoy people or make them upset”), and 6 (“I say or do mean things to hurt other people”) being non-invariant in 16, 16, and 15 countries, respectively; and (b) threshold 2 (“sometimes” to “often”), which was non-invariant on average in eight countries, with threshold 2 for item 7 (“I have a hard time focusing on things that are important”) being most non-invariant (in 14 countries). Item loadings generally showed low levels of non-invariance, with most item loadings being non-invariant in four countries of fewer.
The following countries had the largest number of non-invariant parameters: South Africa (18), Hungary (15), India (15), Japan (13), Bangladesh (12), Iran (12), and Montenegro (12), whereas 11 (mostly European) countries had fewer than five non-invariant parameters (see Table S4).
Discussion
Cross-cultural research on adolescent externalizing problems remains sparse, hindered by the scarcity of cross-culturally validated, brief, and cost-effective self-report instruments designed to assess externalizing problems. Hence, the main goal of the present study was to evaluate the factor structure and cross-national measurement invariance of the YEPS – a recently developed scale aimed at measuring global externalizing behaviors – using adolescent samples from 32 countries.
Several key findings of the present study should be highlighted. First, the analysis revealed consistently high inter-factor correlations in the two-factor model across all samples, supporting a unidimensional structure consistent with prior findings in American and Serbian samples (Jovanović, Reference Jovanović2024; Renshaw & Cook, Reference Renshaw and Cook2019). This pattern of high inter-factor correlations aligns with the well-documented overlap between ADHD and conduct problems among children and adolescents (Frick & Kimonis, Reference Frick, Kimonis, Maddux and Winstead2005). Research consistently demonstrates that approximately 40–60% of youth diagnosed with ADHD exhibit comorbid conduct problems (Hinshaw, Reference Hinshaw1987; Waschbusch, Reference Waschbusch2002), while the prevalence of comorbid ADHD among young individuals diagnosed with ODD and CD is even higher, ranging from 65 to 90% (Abikoff & Klein, Reference Abikoff and Klein1992). Two theoretical frameworks help to explain this substantial overlap. The correlated risk factors model (Rhee et al., Reference Rhee, Willcutt, Hartman, Pennington and DeFries2008) suggests that these disorders co-occur due to common risk factors that influence both conditions simultaneously (e.g., shared genetic or environmental factors). In contrast, the developmental precursor model (Barkley, Reference Barkley2006) proposes a sequential relationship in which ADHD symptoms contribute to and potentially trigger the subsequent development of oppositional defiant behaviors.
Second, model modifications through correlated residuals between semantically similar items were necessary to achieve acceptable fit. These modifications consistently involved two H/I items (7 and 9), which assess attention regulation difficulties, and two items (1 and 3) tapping O/C symptoms. This pattern is not unique to our study, as previous research has also reported consistent residual correlations. Renshaw and Cook (Reference Renshaw and Cook2019) allowed correlated residuals between items 1 and 3, while Jovanović (Reference Jovanović2024) introduced correlations between both item pairs (1 – 3 and 7 – 9) across three of four studies. Given this consistent pattern of problematic items and the poor fit of the original model, the YEPS would benefit from eliminating redundant items. Accordingly, we excluded two items with more complex and vague meaning (items 1 and 9) from further analyses and evaluated the factor structure and cross-national measurement invariance of the shortened version (YEPS-SF).
Third, the YEPS-SF’s one-factor model demonstrated acceptable to good fit in just over half of the studied countries (17 of 32). With minimal modifications (adding one to three correlated residuals), this increased to nearly all countries (31 of 32), the exception being Macau. The reasons for the poor fit of both the original and modified models of the YEPS data in Macau are unknown, but we speculate that the issue may be partially related to the diverse cultural backgrounds among high schoolers comprising a Macau sample. While most students communicate with peers in Chinese, Portuguese, or English during their childhood in Macau, there may be substantial differences in the socialization environments among different schools, which may be influenced by the composition of students in different classes at different schools. In addition, due to Macau’s relatively small geographical area and economy, a substantial number of Macanese students are socialized and raised outside Macau before returning to its high school system in order to pursue higher education opportunities overseas. Therefore, the results might have been influenced by the heterogeneity of the Macau sample, an issue that warrants thorough investigation in future studies.
Fourth, cross-national measurement invariance testing supported configural invariance of the (modified) one-factor model, suggesting that the overall factor structure of the YEPS-SF holds across countries (e.g., Putnick & Bornstein, Reference Putnick and Bornstein2016). However, the scalar model showed a significant deterioration in model fit when compared to the configural model. Scalar invariance remained unsupported even when countries were grouped by geographic regions, with Northern European (Estonia and Lithuania) countries being the sole exception. Items 5 (“I talk a lot and interrupt others when they are talking”) and 10 (“I choose not to follow directions and don’t listen to adults”) showed the highest level of invariance across countries. It is important to note that both items relate to respecting authority and social rules, behaviors that are relatively easily identified and assessed, regardless of cultural norms. The behaviors covered by these items are concrete and observable and are, therefore, less subject to interpretation.
The practical significance of the observed cross-national measurement invariance violations in this study is unclear, necessitating further investigation of both the YEPS’s cross-national invariance and the underlying causes of scalar non-invariance. However, the large number of non-invariant parameters across countries, as well as the variation in items showing the greatest evidence of non-invariance across regions, suggest that the YEPS may not be an appropriate tool for assessing externalizing problems in large-scale cross-cultural studies. The considerable cross-national variation in factor loadings and item thresholds prevents the comparison of latent mean differences in externalizing problems between countries (Leitgöb et al., Reference Leitgöb, Seddig, Asparouhov, Behr, Davidov, De Roover, Jak, Meitinger, Menold, Muthén, Rudnev, Schmidt and van de Schoot2023). When measures developed in Western cultures are applied to non-Western contexts, construct bias emerges as a major concern since the concept being measured must have equivalent meaning and structure across cultures (van de Vijver & Hambleton, Reference Van de Vijver and Hambleton1996). A systematic review conducted by Stevanovic et al. (Reference Stevanovic, Jafari, Knez, Franic, Atilola, Davidovic, Bagheri and Lakic2017) found that, among more than 100 published scales used to assess childhood psychopathology, only 26 had been tested for cross-cultural invariance, with just six showing moderate evidence of scalar invariance.
The evaluation of approximate invariance using alignment optimization indicates that the percentage of non-invariant parameters is marginal (22%), that is, slightly below the 25% threshold, suggesting that caution is needed when comparing the latent means on the YEPS across countries. The largest number of non-invariant parameters occurred in several Asian countries (Iran, India, Japan, and Bangladesh), South Africa, and two European countries (Hungary and Montenegro). This finding is not surprising given that these countries differ substantially in cultural characteristics from the USA, where the scale was originally developed. With the exception of Hungary, these are predominantly collectivistic cultures, where understanding of ADHD is conceptualized differently. As Sethia (Reference Sethia2024) notes, in many collectivistic cultures, behaviors typically associated with ADHD (e.g., impulsivity or inattention) may not be immediately pathologized. Instead, they might be seen within the broader context of an individual’s social and familial environment. The smallest number of non-invariant parameters was detected in European individualistic countries, which share certain Western values (e.g., Vignoles et al., Reference Vignoles, Smith, Becker and Easterbrook2018).
It is important to emphasize that non-invariance can primarily be attributed to thresholds (rather than factor loadings), which represent transition points between response categories. These results suggest that, although most parameters are invariant, there are significant differences in how response options are interpreted across different countries, especially when answering questions about aggressive behavior and attention. These findings point to well-known cross-cultural differences in response styles (e.g., Johnson et al., Reference Johnson, Shavitt, Holbrook, Matsumoto and van de Vijver2011), such as acquiescent and extreme responding (e.g., Kemmelmeier, Reference Kemmelmeier2016). Kemmelmeier (Reference Kemmelmeier2016) highlights that response styles should not be treated merely as measurement error but as cultural behaviors in themselves, reflecting different communication norms and values across cultures. Research indicates that individuals from collectivistic cultures tend to exhibit higher acquiescent responses due to their emphasis on social harmony and conformity, while those from individualistic societies may demonstrate more extreme response patterns reflecting their value on self-expression (He & van de Vijver, Reference He and Van de Vijver2016).
According to van de Vijver and Poortinga (Reference Van de Vijver and Poortinga1997), cross-cultural research faces validity challenges at both content and construct levels. Content validity can be compromised by cultural differences in societal structures, values, and socialization practices, which affect both the meaning of constructs and how item content is perceived. Similarly, construct validity, referred to as the extent to which an assessment accurately reflects the intended construct, may be compromised when items are interpreted differently across countries.
To address these challenges, researchers employ culturally sensitive methods that go beyond literal translation, carefully evaluating whether certain aspects of the measured construct may be missing from or irrelevant to specific cultural contexts (Solano-Flores & Nelson-Barber, Reference Solano-Flores and Nelson-Barber2001; van de Vijver & Hambleton, Reference Van de Vijver and Hambleton1996). The complexity of these cultural factors is well illustrated by the case of Estonia and Lithuania, where scalar invariance was achieved despite significant linguistic differences. While both countries share geographical proximity and historical experiences, their languages belong to entirely different families: Lithuanian belongs to the Indo-European language group, while Estonian is part of the Baltic-Finnic subgroup of the Finno-Ugric languages, closely related to Finnish in both language and cultural heritage (Hein et al., Reference Hein, Emeljanovas and Mieziene2018). This finding suggests that shared history and heritage rather than language similarity might drive the consistent interpretation of externalizing behaviors in these countries, highlighting the importance of considering broader cultural factors beyond linguistic similarities in cross-cultural validation studies.
Limitations and future directions
Several important limitations should be considered when interpreting the results of this study. First, the reliance on convenience sampling from secondary schools raises concerns about the generalizability of findings. Sample characteristics varied considerably across countries; sample sizes ranged between 200 and 1,150, and gender distribution showed substantial variation (between 43.8 and 79% female). To strengthen the validity of our findings, future research should employ more diverse sampling strategies. Specifically, efforts should be made to obtain representative samples and to include younger adolescents. Second, both the varying administration modes (online or paper-and-pencil) and the different translation processes (back-translation or committee approaches) across countries may have introduced method bias and affected measurement invariance. Finally, the original scale was developed in a Western context, potentially reflecting Western conceptualizations of externalizing behaviors rather than universal constructs. The study did not explicitly address cultural variations in the interpretation and expression of externalizing problems, nor did it systematically evaluate potential cultural bias in item content and wording. Future studies could adopt qualitative approaches to explore culturally specific meanings and manifestations of externalizing behaviors, contributing to the development of a more globally applicable framework for understanding and assessing these behaviors.
Conclusions
The present study provided evidence for a YEPS-SF as a unidimensional measure across 31 of 32 countries. However, the lack of scalar invariance across countries indicates that mean score comparisons may not be valid, highlighting difficulties in measuring externalizing problems across cultures. While the evaluation of approximate invariance using alignment optimization showed only a marginal percentage of non-invariant parameters, this still suggests caution when comparing latent means across countries and indicates the need for further research on the cross-cultural validity of the YEPS-SF. Our findings highlight the potential utility of the YEPS-SF for research within individual countries, while also underscoring the ongoing challenge of developing cross-culturally comparable measures of adolescent externalizing problems. To conclude, cultural adaptations may be necessary for the effective implementation of the YEPS-SF in diverse cultural contexts.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0954579425100473
Data availability statement
All materials, including data sets and analysis syntaxes can be found at: https://osf.io/y5mj6/
Acknowledgment
We would like to thank all the participants for taking part in the study.
Author contribution statement
Milica Lazić contributed to Conceptualization, Data Curation, Formal Analysis, Funding Acquisition, Investigation, Methodology, Resources, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing; Vesna Gavrilov-Jerković, Vojana Obradović, Dušana Šakan, Aleksandar Tomašević, Marija Zotović-Kostić, and Veljko Jovanović contributed to Funding Acquisition, Investigation, and Writing – Review & Editing; all remaining authors contributed to Investigation and Writing – Review & Editing.
All authors have agreed to the final submitted version.
Funding
The study is financially supported by the Science Fund of the Republic of Serbia, GRANT No. 1520 (Project title: Identity orientations and mental health in adolescence: Examining the roles of basic psychological needs and culture – IDIOMATIC).
The Slovenian Research and Innovation Agency supported the work of Andreja Avsec, Tina Kavčič, and Gaja Zager Kocjan (research core funding numbers P5-0110 and P5-0062).
Marta Kowal was supported by the Foundation for Polish Science (FNP) START scholarship.
Ethics approval
The study was approved by the Ethics Committee of the Faculty of Philosophy, University of Novi Sad (Approval Code: 02-313/4-1), and by the institutional ethics committee or review board in each country involved in the present study.
Disclosure statement
The authors report there are no competing interests to declare.
Patient consent statement
The parent/guardian consent was obtained in countries where parental consent was required for a certain age. In other countries, informed consent was obtained and the sample included only adolescents who are free to consent to research participation according to domestic legislation (for example, in the Netherlands and Serbia, adolescents are free to consent to research participation with 16 and 15 years, respectively).
