Introduction
To meet the mental health needs of children in a society, it is necessary to identify those children who have behavioural/emotional problems severe enough to warrant intervention. Identification of children in need of mental health referral thus requires a reliable and valid assessment instrument that is able to discriminate between typically developing children and children with high-enough levels of problems to need mental health services. Epidemiological data are needed to determine what level of behavioural/emotional problems are typical in children in the general population and what level of problems necessitates referral for intervention or prevention efforts (Zwirs et al. Reference Zwirs, Burger, Schulpen, Wiznitzer, Fedder and Buitelaar2007). Professionals in developing countries face major challenges in collecting such epidemiological data. These challenges may include lack of resources, lack of valid, reliable and cost-effective instruments, and lack of agreement about how to define impairment (Belfer, Reference Belfer2008). For these reasons, epidemiological research on child mental health in developing countries is most feasible when an instrument is available that can be administered by non-professionals or is self-administered, can be understood by people with varying levels of education, is inexpensive and simple to use, can be easily scored and interpreted, and has been shown to work well in many different societies.
The Child Behavior Checklist (CBCL/6–18; Achenbach & Rescorla, Reference Achenbach and Rescorla2001), developed in the US and translated into more than 85 languages, has been successfully used to assess behavioural/emotional problems in many societies, including many developing countries. Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) conducted multicultural comparisons of CBCL scores obtained for >50 000 children from general population samples in 31 societies. Both studies took an etic approach to research (Pike, Reference Pike1967), whereby the same instrument was used to obtain data in different societies so that the findings could be compared. Etic research is often contrasted with emic research, whereby the meaning of constructs is examined within each society.
When Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) used confirmatory factor analysis (CFA) to test the fit of CBCL data obtained from the 30 non-US societies to the CBCL's 8-syndrome factor model, fit indices strongly supported the correlated 8-syndrome structure in each of the 30 societies. Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) reported that internal consistencies of scales, mean item ratings, and age and gender patterns were very similar across the 31 societies. Although mean total problems scores for most of the societies fell within one standard deviation (s.d.) of the omnicultural mean, 12 societies had mean scores outside this range (six above and six below). This led Achenbach & Rescorla (Reference Achenbach and Rescorla2007) to establish low-, middle-, and high-scoring norm groups for multicultural scoring of the CBCL.
Although societies from Asia, Africa, Australia, the Caribbean, Europe, the Middle East, and North America were included in the Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) studies, no South American societies were included. More recently, Viola et al. (Reference Viola, Garrido and Rescorla2011), who used a Spanish translation of the CBCL, reported CBCL findings for 1364 6- to 11-year olds recruited through 65 schools nationwide in Uruguay. Mean item ratings, factor structure, and scale internal consistencies were very similar to findings reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) for 31 societies. Uruguay's mean total problems score was significantly higher than the US mean, but Uruguay's score still fell in the middle-scoring group of the Achenbach & Rescorla (Reference Achenbach and Rescorla2007) multicultural norms when children with diagnosed disabilities and/or documented clinical or special education services were excluded, following the practice of Achenbach & Rescorla (Reference Achenbach and Rescorla2001) in the US. The present study adds to the growing international CBCL literature by reporting epidemiological findings from Brazil, a much larger and more diverse South American country than Uruguay and one in which Portuguese, rather than Spanish, is spoken.
Brazil was not included in the Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) studies because data from a sufficiently broad general population sample were not available at that time. However, numerous CBCL studies have been conducted in Brazil, starting with the work of Bordin and co-workers, who conducted validation studies using the CBCL/4–18 (Achenbach, Reference Achenbach1991) with clinic sample. For example, Bordin et al. (Reference Bordin, Mari and Caeiro1995) reported that the CBCL had high sensitivity for predicting ICD-10 psychiatric diagnoses for 49 low-income pediatric outpatients, and Brasil & Bordin (Reference Brasil and Bordin2010) reported high sensitivity in a sample of 78 children seen for intake at a mental health outpatient clinic. The Portuguese CBCL/4–18 has also been widely used in Brazilian studies conducted by Rohde and co-workers (e.g. Lampert et al. Reference Lampert, Polanczyk, Tramontina, Mardini and Rohde2004; Roessner et al. Reference Roessner, Becker, Rothenberger, Rohde and Banaschewski2007; Petresco et al. Reference Petresco, Gutt, Krelling, Neto, Rohde and Moreno2009), primarily to examine children with Attention deficit hyperactivity disorder (ADHD). Additionally, the CBCL has been used by several Brazilian researchers to screen children for behavioural and emotional problems (e.g. Alvarenga & Piccinini, Reference Alvarenga and Piccinini2001; Silvares et al. Reference Silvares, Meyer, Santos, Gerencer and Silvares2006; Tanaka & Lauridsen-Ribeiro, Reference Tanaka and Lauridsen-Ribeiro2006; Schneider & Ramires, Reference Schneider and Ramires2007; Moraes & Enumo, Reference Moraes and Enumo2008; Garzuzi et al. Reference Garzuzi, Carreiro, Schwartzman, Mesquita, Palma, Ancona Lopez, Moraes, Macedo and Teixeira2009; Mota et al. Reference Mota, Bertola, Kim and Teixeira2010), and to assess treatment outcomes (e.g. Bolsoni-Silva et al. Reference Bolsoni-Silva, Silveira and Marturano2008; Pereira et al. Reference Pereira, Costa, Rocha, Arantes and Silvares2009).
Some larger scale studies have also used the CBCL in Brazil. Benvegnú et al. (2005) assessed 3139 children and adolescents from the Southern area of the country and reported that 13.5% of the sample achieved scores in the clinical range based on the American norms. Assis et al. (Reference Assis, Avanci, Pesce, Oliveira and Furtado2007), in a similar study done with 500 students from Sao Gonçalo/Rio de Janeiro/Brazil, found that 15.7% scored in the clinical range, also based on the American norms. Paula et al. (Reference Paula, Duarte and Bordin2007) reported that 14% did so in a sample from Sao Paulo, another state from Brazil. These studies highlighted the need for mental health services for Brazil's children.
Although all of these studies constituted important steps in developing CBCL research in Brazil, the existing studies had various limitations. Many of these studies used the 1991 rather than the 2001 version of the CBCL, many had small, clinical, and/or non-representative samples, and all were limited to a particular region of the country. Most importantly, none reported statistical comparisons between CBCL findings for Brazil and those for other societies. We addressed this limitation by comparing 2001 CBCL findings from a large Brazilian sample with findings from the US normative sample. Furthermore, because we used exactly the same methods employed by Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) and Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) in their international comparisons of 31 societies and by and Viola et al. (Reference Viola, Garrido and Rescorla2011) in comparing Uruguayan and US data, we were able to examine the degree to which Brazilian findings replicated those reported for other societies in the world as well as those reported for another South American society.
The present study had several aims, each of which involved comparing Brazilian data with data from the US: (a) to compare mean problem item ratings for Brazil and the US (b); to use CFA to test the fit of the Brazilian data to the CBCL 8-syndrome model derived in the US; (c) to compare Cronbach's alphas for the CBCL's scales with US alphas reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007); (d) to test effects of society (Brazil v. US), age, gender on CBCL problem scores for the non-referred and referred samples separately; and (e) to test the ability of the Brazilian CBCL to discriminate referred from non-referred children.
Method
Participants
Since resources did not permit national probability sampling for this study, we opted to use a convenience sample recruited from different areas of Brazil via colleagues who had purchased the CBCL's computer scoring software from the Brazilian distributor during the past 5 years. Of the 47 colleagues contacted via email, 32 responded and 16 had samples with more than 90 participants. Of these 16 invited to collaborate on the research, 11 colleagues from nine Brazilian cities agreed to share their data, for a total of 2369 potential participants. Each investigator who provided a school sample had recruited it for one's own research project. For example, the sample for Belo Horizonte/MG was obtained in order to analyse if families who lived in less developed areas of the city reported more behavioural/emotional problems than families from more developed areas, whereas the Sao Gonçalo/RJ sample was obtained to analyse the prevalence of behaviour problems among students exposed to violence.
From the initial pool of 2369 children, we excluded children identified with any genetic disease, neurological damage, or diagnosed psychiatric disorder, following the procedure of Achenbach & Rescorla (Reference Achenbach and Rescorla2001) used of excluding from their national survey sample children who had received mental health or special education services in the past year. After we excluded children with an identified condition, we had 1891 participants. This exclusion process resulted in some subsamples being <90 children. These subsamples were then excluded, leaving 1757 participants.
Since the majority (87.9%) of the 1757 participants were in the age group of 6–11, children >11 years were excluded, resulting in 1494 participants. Finally, following Achenbach & Rescorla's (Reference Achenbach and Rescorla2001) procedure, cases with more than eight blank items on the CBCL were excluded. This left a final sample of 1475 children, aged 6–11. Within this sample of 1475, there was a group of 247 children referred for psychological treatment at a clinic in Porto Alegre/RS. These 247 children were retained as a separate ‘referred’ sample. The remaining 1228 children were recruited through schools in six cities by researchers studying children's behaviour. These 1228 children comprised our ‘non-referred’ general population sample. Because information about receipt of mental health and special educational services was incomplete for the non-referred sample, it is possible that some of the children were receiving mental health or special educational treatment. Table 1 presents details about each of the six sub-samples comprising the non-referred sample of 1228 children, as well as about the referred sample comprising 247 children.
Sample characteristics
Note. NI = No information.
The non-referred sample comprised 608 girls (49.5%) and 620 boys (50.5%), with a mean age of 8.1 years (s.d. = 1.3). The referred sample comprised 38.5% girls and 61.5% boys, with a mean age of 8.8 (s.d. = 1.6). The non-referred sample had children from three different areas of Brazil: 66.2% from the Southeast, 26.9% from the Northeast and 6.9% from the South, while the referred sample was composed only of children from Porto Alegre/RS, a big city in the southern region of the country.
To compare Brazilian children with US children, we selected all children with ages of 6–11 from the US referred and non-referred samples. These two US samples had previously been matched on age, sex, ethnicity, and Socioeconomic status (SES) (Achenbach & Rescorla, Reference Achenbach and Rescorla2001). As noted above, the US non-referred sample excluded children with special needs. The US referred group was composed of children from the same age group who had been seen in mental health clinics. The US referred and non-referred samples each had 733 children, 52% boys and 48% girls.
Measure
The 2001 version of the CBCL/6–18 (Achenbach & Rescorla, Reference Achenbach and Rescorla2001) comprises 118 problem items that parents rate 0 = not true, 1 – somewhat or sometimes true, or 2 = very true or often true, based on the past 6 months. The competence items of the CBCL were not used in the present study, as too many children had missing data. The 2001 scales for the CBCL (Achenbach & Rescorla, Reference Achenbach and Rescorla2001) were computed from these 118 problem items, following the methodology of Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007). These included eight syndromes, the three broadband scales (Internalizing, Externalizing, and Total Problems), and six DSM-oriented scales. High test–retest reliability (e.g., mean r = 0.90 for empirically based scales) and strong internal consistency (e.g., alpha = 0.97 for total problems score) have been reported by Achenbach & Rescorla (Reference Achenbach and Rescorla2001). The CBCL also contains some open-ended items on which respondents can choose to write in physical problems, concerns, and strengths of the child. Although qualitative data provided by these items can be very useful in the clinical assessment context, they were not analysed in the present study because written comments were not included in the data sets we received due to the fact that they are not amenable to quantitative analysis.
The Brazilian Portuguese version of the 2001 CBCL/6–18 was developed in a series of steps. Initially, it was translated into Portuguese by Silvares et al. (Reference Silvares and Rocha2007). To make it similar to the US version, the Portuguese version was written at about a fifth grade reading level. This Portuguese version was then back-translated to English by a professional bilingual translator originally from the US and blind to the American version of the instrument. The back-translation was reviewed by the original authors, some minor changes in item wording were made, and the translation was approved for use.
Data analysis
MPlus 5.0 (Muthén & Muthén, Reference Muthén and Muthén2007) was used for the CFA, whereas PASW Statistics 18 was used for all other data analyses. First, mean item ratings were correlated for the non-referred Brazilian and US samples, in order to verify the comparability between the items scores in the two societies. Second, to test the CBCL 8-syndrome model in Brazil, CFA was implemented using the same procedure employed by Achenbach & Rescorla (Reference Achenbach and Rescorla2001) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007). Third, the internal consistency of the CBCL scales was calculated using Cronbach's alphas. Fourth, the effects of society, age, and gender on CBCL scores were tested using analyses of variance (ANOVAs and MANOVAs). Fifth, decision statistics were used to test the ability of CBCL scales to differentiate referred and non-referred children in the Brazilian sample. For this last analysis, 247 children were chosen from the 1228 children in the Brazilian non-referred sample to demographically match the referred sample in age and gender, following the procedure of Achenbach & Rescorla (Reference Achenbach and Rescorla2001). For all analyses, p < 0.001 was used to determine statistical significance, due to the large sample sizes and the number of tests in each set of analyses (17, one for each problem scale). Effect sizes for ANOVAs and MANOVAs are represented by η2.
Results
Mean item ratings
In order to verify the comparability between Brazil and the US with respect to which items tended to receive high, medium, or low ratings, a correlation was computed between the 118 mean item ratings for the 1228 Brazilian non-referred children and for the 733 US non-referred children. The correlation coefficient was 0.84, higher than the mean correlation of 0.79 for the US and 30 other societies reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007). This very high correlation indicates strong comparability between Brazil and the US regarding which items received high, medium, or low ratings.
CFA results
The fit of the Brazilian data to the 2001 CBCL 8-syndrome model was tested using the robust weighted least squares (WLSMV) estimator on tetrachoric correlations (ratings of 0 v. 1 and 2) for the 102 items comprising those syndromes, following Ivanova et al.'s (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) procedures. The root-mean-squared error of approximation (RMSEA) was chosen as the primary index of the model's fit (values ≤ 0.06 indicate good fit), and the Tucker–Lewis index (TLI) and the comparative fit index (CFI) were used as additional measures of model fit (values ≥ 0.90 indicate good fit).
The RMSEA index was 0.023 for the Brazilian data, indicating very good fit. This index was even below the range reported by Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) for 30 societies (0.026–0.055). The TLI and CFI also indicated that the Brazilian data fit the US 8-syndrome model quite well (TLI = 0.900 and CFI = 0.903). All Brazilian items loaded significantly on their predicted factor, with the following mean item loadings: anxious/depressed = 0.51, withdrawn/depressed = 0.59, somatic complaints = 0.62, social problems = 0.51, thought problems = 0.57, attention problems = 0.63, rule-breaking behaviour = 0.56, and aggressive behaviour = 0.65. To verify the consistency between the Brazilian loadings and the average of the loadings for 30 other societies, the differences between the Brazilian mean factor loadings and the mean loadings reported by Ivanova for 30 societies were calculated. They ranged from 0.02 to 0.08 (mean of 0.05), indicating great consistency.
Internal consistency of CBCL scales
Table 2 presents the Cronbach's alphas of the CBCL scales for both referred and non-referred samples in Brazil and the US. The highest alphas in both countries were found for the three broadband scales (Internalizing, Externalizing, and Total Problems), with all alphas ≥ 0.80. Alphas for syndromes and DSM-oriented scales ranged from 0.56 (anxiety problems) to 0.81 (conduct problems), very similar to what was found for the US sample. The correlation between alphas for the 17 problem scales obtained for the referred v. non-referred samples were 0.92 in Brazil and 0.89 in the US. The Brazil–US correlation was 0.93 for non-referred samples and 0.88 for referred samples. Both correlations are close to the mean bi-society correlation of 0.88 reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007).
Demographic characteristics of non-referred and referred samples
Effects of gender, age, and country on CBCL scores
Since the Brazilian referred sample was only 20% as large as the non-referred sample, whereas the US referred and non-referred groups were the same size, effects of gender, age, and society were analysed separately for referral status groups. Table 3 shows the mean scores of the Brazilian and US referred and non-referred samples for all CBCL problem scales. For non-referred children, the gender × age (6–8, 9–11 years) ×society ANOVA on Total Problems indicated main effects for society (Brazil > US, ES = 16%) and gender (boys > girls, ES = 1%), but not for age. For referred children, the ANOVA indicated main effects for gender (boys > girls, ES = 1%) and age (older > younger, ES = 1%) but not for society. As shown in Table 4, the mean total problems score for the non-referred sample was much higher in Brazil (43.34) than in the US (23.12), and sufficiently high to place Brazil in the high-scoring norm group for the CBCL (Achenbach & Rescorla, Reference Achenbach and Rescorla2007). In contrast, scores for the referred groups were very similar in the two societies (62.88 in Brazil and 63.22 in the US).
Cronbach's Alpha for CBCL for Brazil and US samples for ages 6–11
Mean CBCL scores for Brazil and US samples for ages 6–11
aIndicates that Brazilian mean was significantly higher than US mean (p < 0.001) by simple effects analysis.
bIndicates that US mean was significantly higher than Brazilian mean (p < 0.001) by simple effects analysis.
To test the effects of society, gender, and age on internalizing and externalizing scores, 2 × 2 × 2 ANOVAs were conducted separately by referral status groups. For internalizing, the society effect was significant in both samples but much larger in the non-referred sample (ES = 16%) than the referred sample (ES = 1%). Older children scored significantly higher than younger children in the referred sample (ES = 2%) but not the non-referred sample. The gender effect was not significant in either sample. For externalizing, Brazilian children had significantly higher scores than US children in the non-referred sample (ES = 12%), whereas Brazilian children had significantly lower scores than US children in the referred sample (ES = 2%). In both referred and non-referred samples, boys scored significantly higher than girls on externalizing (both ESs = 1%). No significant age effect was found for externalizing problems in either sample.
A 2 (gender) × 2 (age) × 2 (society) MANOVA was conducted on the eight CBCL syndromes for the non-referred and referred samples separately. For the non-referred sample, the significant ESs for society ranged from 4% for thought problems to 15% for anxious/depressed, with Brazilian scores higher on all scales. Boys had significantly higher scores on two syndromes: Attention problem and rule-breaking behaviour (both ESs = 1%). No significant age or interaction effects were found. For the referred sample, US children obtained significantly higher scores on rule-breaking behaviour (ES = 2%) and aggressive behaviour (ES = 1%), whereas Brazilian children obtained higher scores on Somatic complaints (ES = 2%). Boys obtained significantly higher scores on attention problems and rule-breaking behaviour (ES = 2%). Older children had significantly higher scores than younger children on withdrawn/depressed, and rule-reaking behaviour (ESs = 2 and 1%, respectively).
The 2 × 2 × 2 MANOVA on the six DSM-oriented scales for the non-referred sample yielded significant society effects for all the scales (ESs from 2 to 16%), with Brazilian scores higher than US scores. Boys obtained higher scores than girls on DSM-attention deficit hyperactivity problems, DSM-oppositional defiant problems, and DSM-conduct problems (all ESs = 1%). For the referred sample, Brazilian children obtained higher scores on affective problems (ES = 3%), whereas US children obtained higher scores for conduct problems (ES = 6%). Boys obtained higher scores than girls on DSM-attention deficit hyperactivity problems, DSM-oppositional defiant problems, and DSM-conduct problems (all ESs = 2%).
Decision statistics analysis
In order to verify the ability of the Brazilian CBCL to discriminate between referred and non-referred children, a random sample of 247 non-referred children, matched by age and gender with the referred sample, was selected. Both samples were composed by 61.5% of boys, and were equivalent in age (mean of 8.7 in the non-referred and 8.8 in the referred group). Prior to the decision statistics analyses, group differences in mean CBCL problem scores were tested using ANOVAs and MANOVAs. Referred children had significantly higher scores on all scales, as would be expected. For example, mean Total problems scores were 44.28 in the non-referred group and 62.88 in the referred group. Effect sizes ranged from to 0.02 for Anxious/depressed to 0.08 for Total problems scale.
To determine deviance in the Brazilian sample, we used a 90th percentile cutpoint (by gender) on Total Problems score, which is the threshold for the clinical range. We identified the raw score corresponding to the 90th percentile using Achenbach & Rescorla's (Reference Achenbach and Rescorla2007) multicultural norms. Specifically, we used the cutpoints for Achenbach & Rescorla's (Reference Achenbach and Rescorla2007) high-scoring norm group (77 for boys and 78 for girls), because Brazil's Total roblems score placed it in the high-scoring group of societies. For our decision statistics analysis, we classified children scoring > 90th percentile as ‘deviant’ and children scoring ≤ 90th percentile as ‘non-deviant’. Following Achenbach & Rescorla (Reference Achenbach and Rescorla2001), we then looked at the cross-tabulation of deviance and referral status.
Within the non-referred sample, 23% obtained scores in the deviant range, more than twice as many as would be expected using a 90th percentile cutpoint. This indicates a negative predictive value of only 77% (i.e., the percentage of non-referred children who were not deviant). Additionally, 67% of the children scoring in the deviant range were from the referred group, but 33% were from the non-referred group, further confirming the high scores in the non-referred sample. This corresponds to a sensitivity of 67% (i.e., the percentage of deviant children who were from the referred sample). Although 46% of the referred group scored in the deviant range, 54% did not, indicating that the referred group did not have exceedingly high scores. This indicates a positive predictive value of 46% (i.e., the percentage of referred children who scored in the deviant range). Finally, only 59% of the children scoring below the cutpoint were from the non-referred group, not that much higher than the 41% who were from the referred group. This indicates a specificity of 59% (i.e., the percentage of non-deviant children who were from the non-referred group).
Receiver operating curve (ROC) analysis (Swets, Reference Swets1996) results were consistent with these cross-tabulation findings. The area under the curve (AUC) was 62%, indicating only moderately good prediction. Discrimination between the referred and non-referred samples based on Total Problems score was much stronger in the US sample than in the Brazilian sample. Nevertheless, the odds ratio for a deviant Total Problems score being from a referred child was 2.86 (CI 1.94–4.21), indicating that children who scored in the deviant range on the CBCL Total problems scale were almost three times more likely to be in the referred group than the non-referred group.
Discussion
The goal of the present study was to examine the psychometric properties of the Brazilian CBCL for ages 6–11 by conducting statistical comparisons between Brazilian findings and those from the US. In addition to discussing the findings of our Brazil–US comparisons, we also discuss our findings with respect to findings reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) for 31 societies, as well as with respect to findings reported by Viola et al. (Reference Viola, Garrido and Rescorla2011) for Uruguay.
Although there are many cultural differences between Brazil and the US, our analysis revealed many similarities between parents' reports in these two societies. It is interesting to note that these similarities are the same as those found by Viola et al. (Reference Viola, Garrido and Rescorla2011) in a Uruguayan sample. For example, the correlation between the 118 mean item ratings for the 1228 Brazilian non-referred children and the 733 US non-referred children was 0.84, higher than the mean correlation of 0.79 for the US and 30 other societies reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and comparable with the Uruguay–US correlation of 0.82. Thus, Brazilian and US parents were very similar with respect to which items, on average, received high, medium, or low ratings. Additionally, CFA confirmed that the Brazilian data showed excellent fit to the 8-syndrome model derived in the US structure. The RMSEA of 0.023 indicates better fit than found in any of the 30 societies compared by Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) (range 026–0.055) or than found in Uruguay (RMSEA of 0.037) (Viola et al. Reference Viola, Garrido and Rescorla2011). Furthermore, Cronbach's alphas were very similar to US values. Correlations between alphas for the 17 problem scales obtained in Brazil and the US were very high (0.93 for non-referred and 0.88 for referred children), close to the mean bi-society correlation of 0.88 reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and comparable with those reported for the Uruguay–US comparison (0.92 for non-referred and 0.93 for referred).
In both referred and non-referred groups, boys scored higher than girls on Externalizing, consistent with Viola et al. (Reference Viola, Garrido and Rescorla2011) in Uruguay and with Rescorla et al.'s (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) report that boys had significantly higher Externalizing scores than girls for 19 of 28 societies with samples ages 6–11. Higher scores on Attention Problem, Rule-Breaking Behaviour, DSM-attention deficit hyperactivity problems, DSM-oppositional defiant problems, and DSM-conduct problems found for boys in many societies help explain the higher rate of boys referred for mental health care. Brazilian girls did not have significantly higher Internalizing scores than boys, consistent with the Uruguayan findings and with Rescorla et al.'s (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) report that none of the 28 societies with samples of children ages 6–11 showed a significant gender difference on Internalizing.
Since we had a referred sample of 247 children, we were able to examine the CBCL's ability to discriminate referred from non-referred children in Brazil. We used the procedure employed by Achenbach & Rescorla (Reference Achenbach and Rescorla2001) of comparing equal-sized groups matched on age and gender, although we could not also match on SES because we lacked SES data. As noted above, the Brazilian non-referred group had much higher scores than the US non-referred group. Furthermore, given the high scores in the Brazilian non-referred group, the scores in the Brazilian referred group were not sufficiently high to obtain good discrimination. This pattern most likely explains why Brazilian decision statistics results were so much weaker than US results, where referred and non-referred group mean scores were very different. A similar pattern of poor discrimination was found for Uruguay, where Viola et al. (Reference Viola, Garrido and Rescorla2011) did not have a clinic sample as their ‘referred’ group but rather used children from their school-based sample who had documented disabilities or confirmed mental health treatment/special education status.
Limitations
Our sample was the largest and most nationally representative sample obtained to date in Brazil. Nevertheless, it presented numerous limitations that must be considered in interpreting our findings. Ours was a convenience school-based sample recruited in several regions of Brazil, rather than a sample recruited using a nationally representative general population survey. Our data were collected by researchers in different regions of Brazil who had conducted individual projects to assess behavioural problems in school settings and then shared their data with us for this study. Since the data were not collected expressly for this study, information was incomplete with respect to whether participants were receiving mental health or special education services, as well as with respect to SES level of each child. Furthermore, we had to restrict our study to ages 6–11. Additionally, the referred sample was rather small, was drawn from only one clinic, and was most likely not very representative of children attending mental health clinics in Brazil. A final limitation is that no other measure of children's emotional/behavioural difficulties was available for the children, thus preventing a cross-validation analysis for this study. These limitations must be considered when interpreting the findings of our study.
Conclusions
Despite these limitations, our findings provide an important addition to the literature on the use of the CBCL in South America. Our psychometric findings on mean item ratings, factor structure, Cronbach's alphas, and gender effects replicated findings reported by Viola et al. (Reference Viola, Garrido and Rescorla2011) for Uruguay and were comparable with those reported by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007) and Ivanova et al. (Reference Ivanova, Dobrean, Dopfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Achenbach, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Dumenci, Steinhausen, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene, Verhulst, Rescorla, Almqvist, Weintraub, Bilenberg, Bird and Chen2007) for 31 societies in Europe, Asia, and other parts of the world. Therefore, with respect to mean item rating, factor structure, scale internal consistency, and gender patterns, few differences were found between Brazil and the many other societies that have been studied.
In Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007), the omnicultural mean for Total Problems score for the 31 societies was 22.5 (s.d. = 5.7.) The six societies scoring > 1 s.d. above the omnicultural mean were Puerto Rico, Portugal, Ethiopia, Greece, Lithuania, and Hong Kong, with Puerto Rico having the highest score (34.7). The mean of 43.34 found for Brazil in the current study is therefore higher than the means of all 31 societies as compared by Rescorla et al. (Reference Rescorla, Achenbach, Ivanova, Dumenci, Almqvist, Bilenberg, Bird, Chen, Dobrean, Döpfner, Erol, Fombonne, Fonseca, Frigerio, Grietens, Hannesdottir, Kanbayashi, Lambert, Larsson, Leung, Liu, Minaei, Mulatu, Novik, Oh, Roussos, Sawyer, Simsek, Steinhausen, Weintraub, Weisz, Metzke, Wolanczyk, Yang, Zilber, Zukauskiene and Verhulst2007). Brazil's mean Ttotal problems score placed it within the range for the high-scoring norm group specified by Achenbach & Rescorla (Reference Achenbach and Rescorla2007), but it fell at the high end of that range.
Although we cannot say for certain why Brazilian non-referred children scored higher than American non-referred children on virtually all CBCL scales, it is likely that several factors played a role. First, the fact that ours was a convenience sample may have resulted in over-representing children with relatively high scores. It is possible that a sample obtained through probability sampling might have yielded lower scores. Another factor may be ‘cultural,’ in that our CBCL findings for this Brazilian sample are consistent with pre-2001 CBCL findings for Argentina (Samaniego, Reference Samaniego2008) and Chile (Bralio et al. Reference Bralio, Seguel and Montenegro1987), as well as with the Viola et al. (Reference Viola, Garrido and Rescorla2011) findings based on use of the 2001 CBCL in Uruguay. In all these studies, parents of South American children, on average, reported more problems on the CBCL than parents of US children, which might reflect a cultural tendency of having a lower threshold for reporting problems. However, it should be noted that Brazil's mean Total problems score placed it in the high-scoring norm group for multicultural scoring of the CBCL (Achenbach & Rescorla, Reference Achenbach and Rescorla2007), whereas Uruguay's mean placed it in the middle-scoring group (when children with known diagnoses, mental health services, and developmental disabilities were removed from the school-based sample). A third factor that may have contributed to higher scores in Brazil might be the SES of the children. Given that lower SES is associated with somewhat higher CBCL problem scores in all societies where this has been measured to date (Achenbach & Rescorla, Reference Achenbach and Rescorla2007), it may be that a greater preponderance of low SES children in the Brazilian sample relative to the American sample contributed to the score differences. Although we did not have SES levels for individual children, many of the schools from which the children were recruited served low-income families, making it likely that the sample had a large low SES component. A final possible reason is that scores in the Brazilian non-referred group may have been high because the group actually contained children who had been referred for problems, or would have had services been available. Because referral information was not consistently available, it cannot be known how significant this factor was in accounting for US–Brazil score differences.
Our study yielded poorer discrimination between referred and non-referred groups for Brazil than was found in the US, but our findings were comparable with those from Uruguay. Compared with the US, scores for referred children were rather low, whereas scores for non-referred children were rather high, a pattern also found in Uruguay. The non-referred sample may have had high scores because it contained children who had received mental health or special education services of which we were unaware, or who would have been good candidates for services had they been available. The referred children probably had relatively low scores because they were drawn from a single clinic and are probably not that representative of children receiving mental health treatment in Brazil more generally. Since previous Brazilian studies using clinical samples (Bordin et al. Reference Bordin, Mari and Caeiro1995; Silvares et al. Reference Silvares, Meyer, Santos, Gerencer and Silvares2006; Brazil & Bordin, Reference Brasil and Bordin2010) did not report raw CBCL scores, it is difficult to determine if their referred groups obtained higher scores than the referred group in this study. However, these previous studies generally reported higher sensitivity than we obtained and demonstrated good discriminant validity of the CBCL when cross-validated with diagnostic measures. In sum, while it is possible that cultural factors contributed to the weak discrimination between referred and non-referred children in our Brazilian study, we think that these sampling issues are more likely to be the main explanation for our findings. It is therefore possible that our study may have underestimated the ability of the CBCL to discriminate between referred and non-referred children in Brazil.
Future Brazilian studies should address differences on CBCL scores in different regions of Brazil based on a nationally representative general population survey. Additionally, future Brazilian studies should obtain information on referral status and SES, obtain CBCL data for adolescents, and obtain CBCL data for a larger sample of referred children drawn from a variety of mental health facilities in the country. Furthermore, additional studies are needed from other South American societies to test the generalizability of the findings reported here for Brazil and by Viola et al. (Reference Viola, Garrido and Rescorla2011) for Uruguay. Nonetheless, the present study represents an important advance in the use of standardized assessment of behavioural/emotional problems in Brazil.
Declaration of Interest
Dr Rocha and Ms Emerich provide training workshops on the Achenbach System of Empirically Based Assessment (ASEBA) in Brazil, from which they receive financial remuneration. Dr Rescorla is remunerated by the Research Center for Children, Youth, and Families (RCCYF), which publishes the ASEBA. Dr Silvares is the ASEBA distributor in Brazil. The other authors have no conflict of interest to declare.
