Social and emotional factors have an important role in the learning process because schools are social places, learning is a social process, and emotions can facilitate or hamper learning (Weissberg, Durlak, Domitrovich, & Gullotta, Reference Weissberg, Durlak, Domitrovich, Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2015). Given the potential contribution of social and emotional competencies to school success, schools should attend to this aspect of the educational process for the benefit of all students. Thus, the teaching of social and emotional competencies should be considered a core task of school systems, in addition to cognitive-academic skills (Sklad, Diekstra, deRitter, Ben, & Gravesteijn, Reference Sklad, Diekstra, DeRitter, Ben and Gravesteijn2012), so that children and youngsters may learn to interact in socially skilled and respectful ways with their peers, family, and school staff, as well as to practise sage and healthy behaviours (Weissberg et al., Reference Weissberg, Durlak, Domitrovich, Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2015).
Among school-based interventions oriented towards children and adolescents, the teaching of social and emotional competencies is an area that has been receiving increased attention in several countries (Coelho, Marchante, & Sousa, Reference Coelho, Marchante and Sousa2015b; Bowles et al., Reference Bowles, Jimerson, Haddock, Nolan, Jablonski, Czub and Coelho2017; Castillo, Salguero, Fernández-Berrocal, & Balluerka, Reference Castillo, Salguero, Fernández-Berrocal and Balluerka2013; Kimber & Sandell, Reference Kimber and Sandell2009), supported by economic reports that classify these programs as having a good cost-benefit ratio (Belfield et al., Reference Belfield, Bowden, Klapp, Levin, Shand and Zander2015). Presently, the focus for research in social and emotional has shifted from effectiveness to differential effectiveness; namely, how and for whom these interventions work (Wigelsworth et al., Reference Wigelsworth, Lendrum, Oldfield, Scott, tenBokkel, Tate and Emery2016).
Social and Emotional Learning (SEL)
The SEL conceptual framework (Collaborative for Academic, Social and Emotional Learning [CASEL], 2012) targets a combination of behaviours, emotions, and cognitions. CASEL (2012) defines social and emotional learning as that which ‘involves the processes through which children and adults acquire and effectively apply the knowledge, attitudes, and skills to understand and manage emotions, set and achieve positive goals, feel and show empathy for others, establish and maintain positive relationships, and make responsible decisions’ (p. 6).
Most SEL programs are focused on universal prevention, that is, the prevention of behavioural problems through the development of social and emotional competence (Taylor, Oberle, Durlak, & Weissberg, Reference Taylor, Oberle, Durlak and Weissberg2017). There is extensive evidence on the positive impacts of SEL interventions, and SEL programs have been shown to lead to gains in prosocial behaviour and academic performance (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Sklad, Diekstra, DeRitter, Ben, & Gravesteijn, Reference Sklad, Diekstra, DeRitter, Ben and Gravesteijn2012; Taylor et al., Reference Taylor, Oberle, Durlak and Weissberg2017), improvements in classroom and school climate (Hahn et al., Reference Hahn, Fuqua-Whitley, Wethington, Lowy, Crosby, Fullilove and Dahlberg2007; Rivers, Brackett, Reyes, Elbertson, & Salovey, Reference Rivers, Brackett, Reyes, Elbertson and Salovey2012), and reductions in aggressiveness (Castillo et al., Reference Castillo, Salguero, Fernández-Berrocal and Balluerka2013; Hahn et al., Reference Hahn, Fuqua-Whitley, Wethington, Lowy, Crosby, Fullilove and Dahlberg2007; Schonert-Reichl, Smith, Zaidman-Zait & Hertzman, Reference Schonert-Reichl, Smith, Zaidman-Zait and Hertzman2012), mental health issues (Castillo et al., Reference Castillo, Salguero, Fernández-Berrocal and Balluerka2013; Jones, Brown, & Aber, Reference Jones, Brown and Aber2011), and substance use (Dodge et al., Reference Dodge, Bierman, Coie, Greenberg, Lochman, McMahon and Pinderhughes2014; Kimber & Sandell, Reference Kimber and Sandell2009).
Ongoing Issues With SEL Programs
Given the broad definition of SEL, it is difficult to differentiate or identify what makes a SEL program successful (Wigelsworth et al., Reference Wigelsworth, Lendrum, Oldfield, Scott, tenBokkel, Tate and Emery2016), leaving some important issues still to be addressed in SEL research. First, although CASEL (2012) recommended that SEL programming be carried out from preschool to high school to enhance the effectiveness of SEL programs, most recent meta-analysis mainly analysed programs applied in elementary schools (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Taylor et al., Reference Taylor, Oberle, Durlak and Weissberg2017). Intervention during middle school is crucial because students become more disengaged from school as they progress from elementary to middle and high school (Klem & Connell, Reference Klem and Connell2004). This is a pressing issue in Portugal, given that there is still a school dropout rate of 19.2% (compared to an average of 11.9% in the European Union; Eurostat, 2012). Second, the use of more than one informant to report program results is often recommended (Achenbach et al., Reference Achenbach, Becker, Dopfner, Heiervang, Roessner, Steinhausen and Rothenberger2008; Wigelsworth, Humphrey, Lendrum, & Kalambouka, Reference Wigelsworth, Humphrey, Lendrum and Kalambouka2010), but seldom used. Third, Taylor et al. (Reference Taylor, Oberle, Durlak and Weissberg2017) recommended that more follow-up investigations were needed to confirm the durability of a program's impact. Finally, recent meta-analyses (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Sklad et al., Reference Sklad, Diekstra, DeRitter, Ben and Gravesteijn2012) also concluded that the lack of studies analysing the impact of variables such as school location may influence the effectiveness of the programs. In the literature, some of these issues have been more discussed than others, as detailed below.
Multiple informants
The usefulness of using different informants has been widely accepted (Achenbach et al., Reference Achenbach, Becker, Dopfner, Heiervang, Roessner, Steinhausen and Rothenberger2008; Coelho, Marchante, & Sousa, Reference Coelho, Marchante and Sousa2015a; Humphrey et al., Reference Humphrey, Kalambouka, Lendrum, Wigelsworth, Lennie and Farrell2010; Shapiro, Burgoon, Welker, & Clough, Reference Shapiro, Burgoon, Welker and Clough2002) because different informants see children in different contexts, interact differently with them, and have different mind-sets for judging and reporting. Agreement between different informants may be limited, with several studies reporting different results between self- and teacher reports (Coelho et al., Reference Coelho, Marchante and Sousa2015b; Humphrey et al., Reference Humphrey, Kalambouka, Lendrum, Wigelsworth, Lennie and Farrell2010; Shapiro et al., Reference Shapiro, Burgoon, Welker and Clough2002) — some with teachers reporting stronger program effects (Shapiro et al., Reference Shapiro, Burgoon, Welker and Clough2002), while in others, teachers do not support positive results indicated from the self-reports (Humphrey et al., Reference Humphrey, Kalambouka, Lendrum, Wigelsworth, Lennie and Farrell2010). Additionally, Domitrovich, Cortes, and Greenberg (Reference Domitrovich, Cortes and Greenberg2007) suggested that teacher reports should be included only if the same teachers were not applying the program themselves. However, studies that report teacher results when these are not the program developers are scarce. Therefore, even though there is some consensus (Achenbach et al., Reference Achenbach, Becker, Dopfner, Heiervang, Roessner, Steinhausen and Rothenberger2008; Sklad et al., Reference Sklad, Diekstra, DeRitter, Ben and Gravesteijn2012; Wigelsworth et al., Reference Wigelsworth, Humphrey, Lendrum and Kalambouka2010) concerning the importance of use of multiple informants to report intervention results, because each informant may reliably and validly report different aspects of children's functioning (Achenbach et al., Reference Achenbach, Becker, Dopfner, Heiervang, Roessner, Steinhausen and Rothenberger2008), there are very few studies that present the perspective of multiple informants concerning SEL programs’ effectiveness (Coelho et al., Reference Coelho, Marchante and Sousa2015a).
School grade
There are few studies that have analysed the differences in social and emotional competencies according to the school grade. Fifth-graders were found to be lower in self-esteem than 4th-graders (Arens, Yeung, Craven, Watermann, & Hasselhorn, Reference Arens, Yeung, Craven, Watermann and Hasselhorn2013), but this result followed a school transition at the end of 4th grade. Diseth, Meland, and Breidablik (Reference Diseth, Meland and Breidablik2014) concluded that self-esteem levels were lower in 8th-graders than in 6th-graders, but that differences were small. There are also studies that did not find differences in social and emotional competencies due to grades, Coelho et al. (Reference Coelho, Marchante and Sousa2015a) did not find differences between 7th-, 8th- and 9th-graders in social awareness, self-control, social isolation, and social anxiety. Even scarcer in the literature are studies on the differential effectiveness of SEL programs in different school grades. Coelho, Marchante, and Sousa (Reference Coelho, Marchante and Sousa2016) reported that a SEL program yielded larger gains in several dimensions of self-concept for 4th-graders than for 7th-graders.
Class size
Olweus (Reference Olweus1993) suggested that in larger classes, teachers could have difficulties in developing and maintaining meaningful relationships with students or in being able to monitor their students’ behaviour effectively, so problems with discipline may increase. Class size has also been shown to have a positive impact on students’ social behaviour, with antisocial behaviour decreasing when class size is reduced (Finn, Panozzo, & Achilles, Reference Finn, Panozzo and Achilles2003). These authors also concluded that smaller classes promoted an atmosphere in which students are more supportive and caring about each other. Therefore, it is likely that class size might influence the effectiveness of SEL programs.
School location
A general issue reported in educational research is the focus on urban youth, largely ignoring rural students (Hardré, Sullivan, & Crowson, Reference Hardré, Sullivan and Crowson2009). This is an important issue given differential results in educational outcomes reported by several studies (McCoy, Morris, Connors, Gomez, & Yoshikawa, Reference McCoy, Morris, Connors, Gomez and Yoshikawa2016; Nansel et al., Reference Nansel, Overpeck, Pilla, Ruan, Simons-Morton and Scheidt2001). Students in rural areas were more likely to report being a bully as compared to youth in urban and suburban schools (Nansel et al., Reference Nansel, Overpeck, Pilla, Ruan, Simons-Morton and Scheidt2001), and also more likely to be asked to repeat the same grade instead of progressing to the next grade at the end of the school year (U.S. Department of Health and Human Services, Reference Weissberg, Durlak, Domitrovich, Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2005). However, Coelho, Marchante, and Sousa (Reference Coelho, Sousa and Figueira2014) did not find any differences in social and emotional competences according to school location. Even so, school location may have an impact on the effectiveness of SEL programs, given that size and urbanicity are school variables that should be related to a student's sense of belonging (Anderman, Reference Anderman2002), as rural schools tend to be smaller in size and, therefore, students are more likely to get to know their teachers and their classmates on a more interpersonal level (Anderman, Reference Anderman2002). In SEL research, there is an underrepresentation of SEL programs carried out in rural schools; for example, they comprised only 15% of schools in Durlak et al. (Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011). Additionally, the response to intervention programs may be influenced by school location. McCoy et al. (Reference McCoy, Morris, Connors, Gomez and Yoshikawa2016) concluded that the Head Start program was more effective in increasing children's receptive vocabulary in urban areas, but it was more effective in increasing their oral comprehension in rural areas.
Present Study
The present study analysed the impact of school grade and class size (class-level variables), and school location (a school-level variable) on the effectiveness of a SEL program applied among Portuguese high middle school students in two consecutive cohorts. It was hypothesised that students who participated in the program (intervention groups) would improve their levels of social and emotional competencies when compared with students in the control group (Hypothesis 1); that both informants (students and teacher) would report on it (Hypothesis 2); and that the program's effectiveness would be similar for students from all the three grades analysed (Hypothesis 3) and different class sizes (Hypothesis 4). Given the lack of conclusive previous inputs, two research questions were created: Does school location influence the initial levels of social and emotional competencies (1)? Does school location influence the effectiveness of a SEL program (2)?
Method
Design
This study employed a quasi-experimental, pretest, post-test and follow-up design with a control group. Both control and intervention groups were chosen, upon approval by the school boards, among those classes where students (on average) presented lower levels of social and emotional competencies, matching as much as possible classes with similar characteristics (rural or urban school location, class size, and gender distribution). Self- and teacher reports were gathered at baseline and post-test, whereas self-reports were also collected 7 months after the program ended for the follow-up, while demographic data was recorded at pretest. The teacher reports were completed by the class directors, who are teachers who coordinate the group of teachers for each specific class and therefore have a better knowledge of their students. The assessment of both groups was carried out under the same conditions. In the intervention group, questionnaires were administered in the first and last sessions of the program; control groups were assessed in the same period, but did not received any SEL content.
Participants
Seven hundred and forty-six middle school students (7th to 9th grade, 9th grade being the last year of middle school in Portugal), of whom 345 were girls and 401 boys aged from 11 to 17 years (M age = 13.38; SD = 1.36), from six Portuguese public middle schools belonging to the municipality of Torres Vedras participated in this study. The total number of students per class ranged between 12 and 26 (M = 17.65). The composition of the intervention and control groups were homogeneous in terms of gender, grade level, school location, and also for class dimension, t(39) = −1.03, p > .05, and age, t(744) = 1.37, p > .05, and were extremely homogeneous regarding ethnicity. Additional information about participant characteristics is displayed in Table 1.
Student Characteristics across Groups and Years of Implementation
Note: N = 746.
Seven hundred and forty-three students (99.6%) completed the post-test measurement and 447 students (59.9%), the follow-up. The attrition rate was homogeneous for intervention and control groups, χ2(1) = 0.02, p > .05, and for gender, χ2(1) = 0.71, p > .05. Intervention group students who completed the follow-up measurement did not differ significantly from those who did not complete it in any of the baseline scores.
Instruments
Social and emotional competencies—Students
The evaluation of social and emotional competencies was conducted with the Portuguese version (Ferreira & Rocha, Reference Ferreira and Rocha2004) of the Batería de Socialización 3 (BAS-3, Silva & Martorell, 1995). This self-report instrument for adolescents (aged between 11 and 18 years old) is composed of 75 items rated dichotomously (yes or no). The questionnaire consists of five dimensions of socialisation, two of which were used in the present study: social awareness (14 items; Cronbach's α = .82 for the original instrument and .73 in the Portuguese adaptation allows for the detection of the degree of sensitivity or concern for others) and self-control (14 items; α = .78 for the original instrument and .73 in the Portuguese adaptation allows for the assessment of the acceptance of rules and social norms, as well as aggressive behaviours). These dimensions were used because they are directly in alignment with the chosen theoretical framework (SEL); due to space constraints, the other dimensions of socialisation (social isolation, social anxiety and leadership) were not analysed in the present study.
Self-esteem
Self-esteem was assessed through the Global Self-Esteem scale (Portuguese version; Fontaine, Reference Fontaine1991) of the Self-Description Questionnaire II (Marsh, Relich, & Smith, Reference Marsh, Relich and Smith1983). The scale is composed by 10 items (e.g.: ‘In general I am happy to be the way I am’) rated in a 5-point scale (1 = false, 2 = mostly false, 3 = not true or false, 4 = mostly true and 5 = true); five of the items are presented in negative statements. The scale's internal consistency was adequate for both versions (original and Portuguese adaptation) with Cronbach's alphas of .88 and .82 respectively.
Social and emotional competencies—Teacher reports
The Social and Emotional Competencies Evaluation Questionnaire — Teacher's version (QACSE-P; Coelho, Marchante, & Sousa, Reference Coelho, Sousa and Figueira2014) was used. This instrument evaluates the perceptions of teachers regarding the social behaviour of their students and is composed by 56 items over five scales. The items are presented as statements to be rated in a 4-point scale (A = never, B = sometimes, C = frequently, and D = always). Only two subscales were used in the present study: social awareness (10 items; e.g.: ‘He/she is concerned when someone has problems’; α = .91); and self-control (12 items; e.g.: ‘When he/she wants to talk, he/she waits for its turn’; α = .92). The subscales pertaining to relationship skills, social isolation, and social anxiety were not used.
Procedure
Originally, Project Positive Attitude was conceived as part of the municipal drug prevention plan, in partnership with the municipality, the Drug Addiction Prevention Institute, and three public school groupings. Since the second year of implementation, all six public school groupings elected to have the middle school SEL program and none were committed to other SEL programs. Previous results (reported in Coelho et al., Reference Coelho, Sousa and Figueira2014, Reference Coelho, Marchante and Sousa2015b, Reference Coelho and Sousa2017) have made school groupings pressure for a higher number of experimental groups over control groups. The present study describes results from the fourth and fifth year of program implementation.
The Positive Attitude program for high middle school is a universal program composed of 13 weekly one-hour sessions, delivered by one of the four educational psychologists (with at least three years of experience in the Project), in the presence of the class director, following a program manual that contains detailed plans for each session. Program contents and activities (described in detail in Coelho et al., Reference Coelho and Figueira2011) are based on CASEL's (2012) theoretical framework and were developed during the first two years of implementation, following meetings with school grouping directors and teachers conducted to better understand the behavioural and social characteristics of each school. The program is classroom based (as part of the curriculum of a school subject named Civic Training), includes all students in each class, and is infused into the school curriculum and integrated into a multiannual project (Project Positive Attitude).
The psychologists who implemented the program were present at the meetings at the beginning of the school year (mandatory for parents) to explain the program and answer questions. All schools used passive informed consent, following national legislation when a program is integrated into the school curriculum. A preliminary session where preliminary evaluation questionnaires were completed and the psychologists presented the project and themselves was implemented in all classes in the schools. Questionnaire instructions were read out aloud to the students. If a student was not present in the class during the assessment, questionnaires were administered in another class within two weeks (n = 8). Teachers were asked to deliver their post-program teacher report questionnaires up to two weeks after the program ended.
Data Analysis
T tests were used to compare initial levels of competence between conditions and school locations. A one-way ANOVA was conducted in order to analyse the existence of differences in the initial levels of competence by grade. Little's Missing Completely at Random (MCAR) test was used to analyse the patterns of missing data, and the value indicated that the missing values were MCAR, χ2(2) = 2.34; p > .05. Multiple imputation was used to deal with missing values, the imputed data set, composed by 5 imputations, was created and pooled results were used.
Multilevel linear modelling (MLM) with a repeated measures design was used (SPSS, mixed models) to evaluate the effectiveness of the program and the influence of school location and grade level on its effectiveness. There are several advantages to MLM when dealing with repeated measures compared with other analyses, such as the lack of requirement for complete data across time points (Heck, Thomas, & Tabata, Reference Heck, Thomas and Tabata2013) and the possibility of adjusting the models for the effects of numerous factors that influence the interaction between condition, time, and allowing for the analysis of potential moderators such as school location and grade. In this repeated measures study design, individual scores are nested within individuals and these in turn are nested within classes. Nested data are more likely to correlate within the group, so student evaluations from the same class are more likely to be highly correlated. The assumptions for linearity, multivariate normality, and multicollinearity were met; multivariate outliers (n = 3) were removed from the analysis.
After creating a null model, there was not sufficient variance explained at the fourth level (school grouping; intraclass correlations were between .001 for Self-Control — Teachers and .013 for Self-Control — Self-Reports), and therefore that level was not introduced into the models, following Heck at al. (Reference Heck, Thomas and Tabata2013). Therefore, a three-level model was run to account for the three measurements nested within 746 students, and these students were nested in 41 classes. The best model fit was achieved by using auto-regressive as the covariance structure at level 1.
A growth rate indicator variable was constructed to manage the non-linear growth trajectories in outcome measures. There are different ways to code the growth rate indicator when the trajectory is found to be non-linear. After several analyses, the best fit was achieved by coding the first measurement occasion (baseline) as 0 and the third measurement occasion (follow-up) as 1. The growth that occurs over the entire trend is captured when using this approach. No time-varying covariates were included in the model. The intercept was used as a random effect in the models, and best model fit was achieved with scaled identity as the covariance structure for levels 2 and 3. These procedures were introduced in creating model 1 (time model).
A condition model (model 2) included an interaction between time and condition. Condition was dummy-coded as a level 3 variable since it was classes (not students) that were assigned to intervention or control groups. In MLM, the models can be adjusted simultaneously for the effects of numerous factors, so analyses were run to evaluate potential effects, including interactions between school location, class size, and grade level with condition and time. All interactions were included in a final model (model 3). A likelihood ratio test (-2*log likelihood) was used to analyse the differences in model variance between models. In all dimensions, the level of variance at the class level (level 3) was over .05, and therefore it was relevant to include this level into the analysis, as recommended by Heck et al. (Reference Heck, Thomas and Tabata2013). IBM SPSS Statistics for Windows, Version 20 (IBM Corp, Armonk, NY) was used, and the alpha level was set to <.05.
Results
Preliminary Analysis
The descriptive data for self- and teacher reports are summarised in Table 2. Intervention group students had lower levels of self-esteem at pretest, t(744) = 2.96, p < .005, according to self-reports, whereas the teachers did not report any initial differences between control and intervention groups. Regarding school location, self-reports indicated that students from urban schools showed higher levels of self-control, t(744) = 2.09, p > .05; while teacher reports showed no initial significant differences. Concerning grades, there were differences in the initial levels of social awareness, F(2, 743) = 6.16, p < .005, with 7th-graders reporting higher levels of social awareness than 8th- and 9th-graders. There were no significant differences reported due to year of implementation, either in self- or in teacher reports.
Self and Teacher Reports – Pre-Test, Post-Test and Follow-Up Means and Dtandard Deviations for Control and Intervention Group, Effect Sizes for Interaction Group*Time
Program Effectiveness
As can be seen in model 2 from Tables 3, 4, 5, 6, and 7, model fit improved significantly in all the dimensions (both self- and teacher reports) when condition and the interaction between time and condition were added. Program effects were reported by self- and teacher reports, with both showing that there was an interaction effect representing an increase in the intervention group across the entire measurement period in social awareness, β = −0.19, SE = 0.09, t = −2.07, p = .039; self-control, β = −0.30, SE = 0.13, t = −2.42, p = .016; and self-esteem, β = −1.10, SE = 0.21, t = −5.27, p < .001; and for teacher reports: social awareness, β = −1.96, SE = 0.33, t = −5.90, p < .001; and self-control, β = −1.20, SE = 0.34, t = −3.51, p < .001.
Self-Reports — Multilevel Model Analysis Models for Social Awareness
Note: *p < .05; **p < .01; ***p < .001.
Self-Reports—Multilevel Model Analysis Models for Self-Control
Note: *p < .05; **p < .01; ***p < .001.
Self-Reports—Multilevel Model Analysis Models for Self-Esteem
Teacher Reports—Multilevel Model Analysis Models for Social Awareness
Note: *p < .05; **p < .01; ***p < .001.
Teacher-Reports—Multilevel Model Analysis Models for Self-Control
Note: *p < .05; **p < .01; ***p < .001
Program effectiveness by grade
Given that there were significant differences in the initial levels of social awareness, for this dimension the predictor was inserted as a factor instead of a covariate. There were no significant interactions in self-reports between school grade, time and condition, indicating there were no differences in the impact of SEL program per grade across the whole measurement period: social awareness (7th- vs. 8th-graders), β = 0.17, SE = 0.13, t = 1.34, p = .204; social awareness (7th- vs. 9th-graders), β = 0.18, SE = 0.10, t = 1.79, p = .112; self-control, β = 0.06, SE = 0.09, t = 0.76, p = .458; self-esteem, β = 0.05, SE = 0.17, t = 0.27, p = .768. However, teacher reports show significant interactions between grade, time and condition for social awareness, indicating smaller increases for 9th-graders when compared to 7th- and 8th-graders: social awareness (9th- vs. 7th-graders), β = 2.48, SE = 0.58, t = 4.28, p < .001; social awareness (9th- vs. 8th-graders), β = 1.66, SE = 0.48, t = 3.46, p = .001; self-control, β = 0.54, SE = 0.61, t = 0.88, p = .377.
Program effectiveness by class size
There were no significant interactions in self-reports between class size, time and condition indicating there were no differential impacts of the SEL program due to class size across the whole measurement period: social awareness, β = −0.01, SE = 0.01, t = −0.64, p = .521; self-control, β = −0.01, SE = 0.01, t = 0.46, p = .650; self-esteem, β = 0.03, SE = 0.03, t = 1.13, p = .258. There were also no significant differential impacts of the SEL program in the teacher reports: social awareness, β = 0.63, SE = 1.34, t = −0.47, p = .641; self-control, β = 0.85, SE = 1.29, t = 0.66, p = .513.
Program effectiveness by school location
There was one significant interaction between school location, time and condition in self-reports for social awareness, β = −0.26, SE = 0.12, t = −2.19, p = .029, indicating that rural students gained more from participating in the SEL program than urban students across the whole measurement period (as displayed in Figure 1). There were no other significant interactions in self-reports between class size, time and condition for self-control, β = −0.12, SE = 0.10, t = −1.14, p = .254; or self-esteem, β = −0.11, SE = 0.23, t = −0.47, p = .642. Teacher reports (displayed in Figure 2), on the other hand, show a significant interaction between school location, time, and the control condition for social awareness, β = −2.21, SE = 0.57, t = −3.92, p < .001, indicating that without intervention, control group students from rural schools showed a much more pronounced decrease than students from urban schools. Teachers did not report a significant interaction between school location, time and condition for self-control, β = 0.82, SE = 0.48, t = 1.69, p = .092.
Social awareness (self-reports), per condition and school location.
Social Awareness (teacher reports) per condition and school location.
Discussion
The main purpose of the current study was to investigate whether previously hypothesised class-level factors could lead to differential effectiveness of a universal, high middle school-based SEL program, following recommendations by Wigelsworth et al. (Reference Wigelsworth, Lendrum, Oldfield, Scott, tenBokkel, Tate and Emery2016). After analysing self- and teacher reports, the results show that class-level variables should be taken into account when analysing the effectiveness of SEL programs, as the level of explained variance at this level warrants their inclusion into the analysis. The results also support the use of multiple informants, because different patterns in self- and teacher reports were found, which is consistent with Achenbach et al. (Reference Achenbach, Becker, Dopfner, Heiervang, Roessner, Steinhausen and Rothenberger2008), who concluded for the existence of a modest level of agreement between different informants, making it essential to obtain and compare reports when conducting research on groups of children.
Regarding the two research questions posed, it is possible to conclude that it is important to use a multilevel approach, as highlighted by the initial level of social and emotional competencies, with students from urban schools reporting higher initial levels of self-control. The lower level of social and emotional competencies found in students from rural schools is quite rare in the literature, which is probably of consequence for the underrepresentation of this setting in recent studies (e.g., Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Taylor et al., Reference Taylor, Oberle, Durlak and Weissberg2017); however, it is consistent with previous results in other similar areas (McCoy et al., Reference McCoy, Morris, Connors, Gomez and Yoshikawa2016; Nansel et al., Reference Nansel, Overpeck, Pilla, Ruan, Simons-Morton and Scheidt2001). As to the second research question, students from rural schools also benefitted more from participating in the program in social awareness; without intervention, control group students from rural schools experienced a much steeper decline in social awareness than students from urban schools. In this regard, the results from the present study are in accordance with those reported in the bulk of the literature (Castillo et al., Reference Castillo, Salguero, Fernández-Berrocal and Balluerka2013; Coelho et al., Reference Coelho, Marchante and Sousa2015b; Conduct Problems Prevention Research Group, 2011), which indicate that the longer the distance children have to travel, the longer the distance they tend to cover in socio-emotional adjustment, thus highlighting the need to implement SEL programs among rural school students. Therefore, the effectiveness of intervention programs may be influenced by school location, as suggested by McCoy et al. (Reference McCoy, Morris, Connors, Gomez and Yoshikawa2016).
The results supported Hypotheses 1 and 2, as both self- and teacher reports showed that the participation in the universal SEL Program Positive Attitude for high middle school students led to significant increases in social awareness and self-control, and self-reports also showed increases in self-esteem. The larger effect sizes (as measured by the reduction in unexplained variance) found in self-reports contradict results reported by several previous studies (Coelho et al., Reference Coelho, Marchante and Sousa2015b; Shapiro et al., Reference Shapiro, Burgoon, Welker and Clough2002), where it had been proposed that SEL program effects might be more noticeable by teachers than by the students themselves.
Conclusions about the relevance of contemplating grade level when analysing the differential effectiveness of this particular SEL program are harder to draw, as there were mixed findings in self- and teacher reports. There were no significant different impacts according to self-reports. However, for social awareness, teacher reports showed there were significant interactions between grade, time and condition, with 9th-graders benefitting less from the program than 7th- or 8th-graders. This pattern of results is not in line with Coelho et al. (Reference Coelho, Marchante and Sousa2015a) and highlights the importance of applying the program as early as possible. The most likely explanation for this is that 9th-graders are finishing high middle school and are probably more focused on the upcoming exams and their imminent transition to secondary school and therefore engage less in other activities. Therefore, due to the mixed results between the self- and teacher reports, the third hypothesis is only partially supported, whereas the fourth hypothesis was not supported given that class size did not influence the effectiveness of SEL program in any dimension.
The results showed that the universal SEL program Project Positive Attitude for high middle school significantly contributed to the improvement of several social and emotional competencies. However, more importantly, the findings from the current study also present a more complex picture, supporting the need for differential analyses of SEL programs’ effectiveness. The improvement was different for rural and urban students, with rural school students (who presented lower initial levels of competence) benefitting more from participation in the program. Additionally, teachers perceived the program to be more beneficial for students from lower grades, supporting the recommendation made by Weissberg et al. (Reference Weissberg, Durlak, Domitrovich, Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2015) regarding the advantages of having programs implemented during several years. As a whole, the study contributed to the comprehension of how group level characteristics such as school location and grade influence SEL program results and provide support for the need to analyse the differential effectiveness of SEL programs (Wigelsworth et al., Reference Wigelsworth, Lendrum, Oldfield, Scott, tenBokkel, Tate and Emery2016).
Limitations of the Study
Some of the major limitations of the study were related to program implementation, as it took place in a professional context and was not intended only for research purposes. As a consequence, teachers and school boards were pressured to have more intervention than control groups, in part because the intervention groups were perceived to have significant improvements in social and emotional competencies and, as such, the size of the control groups was smaller than intended. It is also worth mentioning that the program's length of 13 weekly sessions places it among the shorter interventions reported in the latest meta-analysis (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; Sklad et al., Reference Sklad, Diekstra, DeRitter, Ben and Gravesteijn2012). Therefore, a longer intervention could clarify the impact of grade upon the results.
Additionally, even after adding class-level predictors (age and gender), much of the within-class and between-class variance in most dimensions was left unexplained, so future studies should try to identify other class characteristics, such as gender ratio or socioeconomic status, as well as relevant individual predictors (age, gender, or retention rates) that may contribute to the differential effectiveness of SEL programs.
A better understanding of the results would be possible if it was able to further triangulate the assessments through the use of parent reports, as this would permit us to analyse whether the positive results detected by self- and teacher reports were noticeable in the home setting. However, the low level of qualifications for the parents of students in this sample prevented their involvement.
Future Directions
Further investigation into other potential group level variables such as class size and school socioeconomical status is important to further understand the role of group variables upon the effectiveness of this SEL program. Additionally, research about the potential effects of the cumulative application of the program over several grades is needed, following Weissberg et al. (Reference Weissberg, Durlak, Domitrovich, Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2015), who suggested that programs should be longitudinal.
