Socio-emotional skills are associated with children and adolescents’ greater well-being and better school performance, while the failure to develop these skills appropriately may result in personal, social, and academic difficulties (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011). Therefore, children and youths’ positive development may be enhanced through Socio-Emotional Learning (SEL) based school programs geared towards the promotion of social and emotional skills.
Although the positive impact of SEL programs has been widely researched, implementation quality is not yet sufficiently contemplated in the evaluation of SEL interventions. Developing an evidence-based SEL intervention does not guarantee success, since poor implementation quality may lead to inconsistent results (Evans, Murphy, & Scourfield, Reference Evans, Murphy and Scourfield2015; Freeman, Wertheim, & Trinder, Reference Freeman, Wertheim and Trinder2014). Therefore, SEL programs must also be well implemented. In this sense, the study of implementation quality has important implications for research and practice, since it enables an understanding of whether the programs have the potential to work if well implemented, and also of how and why they work (Pettigrew et al., Reference Pettigrew, Graham, Miller-Day, Hecht, Krieger and Shin2015).
In recent years, the body of evidence regarding the importance of program implementation quality has grown rapidly. Implementation quality is viewed as a multidimensional construct which has been more usually defined as the degree to which programs are implemented as intended by the program developers (Haataja et al., Reference Haataja, Voeten, Boulton, Ahtolaa, Poskiparta and Salmivalli2014; Schultes, Stefanek, van de Schoot, Strohmeier, & Spiel, Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014). Nevertheless, several definitions of implementation quality, also labeled as treatment integrity and fidelity (Domitrovich & Greenberg, Reference Domitrovich and Greenberg2000), may be found.
A review of some studies (Carroll et al., Reference Carroll, Patterson, Wood, Booth, Rick and Balain2007) led to the conclusion that implementation quality has been described in the literature as including five main dimensions, originally defined by Dane and Schneider (Reference Dane and Schneider1998): (a) adherence (more commonly referred to as fidelity), referring to the degree to which program components are delivered as intended, (b) dosage, regarding the frequency and duration of the program, (c) affective quality (generally referred to as quality of delivery), which refers to the qualitative aspects of the program delivery, (d) participant responsiveness, referring to how participants are engaged and involved in the program, and (e) program differentiation, which includes the attempts made by the program evaluators to verify the design conditions. Subsequent studies (Berkel, Mauricio, Schoenfelder, & Sandler, Reference Berkel, Mauricio, Schoenfelder and Sandler2011; Durlak & DuPre, Reference Durlak and DuPre2008) identified another two dimensions of implementation quality in addition to the afore-mentioned: (a) program reach, referring to the extent to which participants are representative of the target population, and (c) adaptation, which is related to possible changes made to the program. Despite the differences between approaches and conceptual models, the vast majority of studies consider participant responsiveness to be one of the core dimensions of implementation quality.
While most of the dimensions of implementation quality are determined by the program developers and/or program facilitators, participant responsiveness is determined by participants, and, thereby, is viewed as one of the potential sources of variability of the program which influences program outcomes (Berkel et al., Reference Berkel, Mauricio, Schoenfelder and Sandler2011). However, research has focused almost exclusively on adherence or fidelity as an important predictor of program outcomes, therefore, considerably less is known about how participant responsiveness, and other important aspects of implementation quality, actually influence outcomes (Pettigrew et al., Reference Pettigrew, Graham, Miller-Day, Hecht, Krieger and Shin2015).
Given the importance of participant responsiveness in programs’ efficacy, the present study focuses on the evaluation of this dimension of implementation quality in a Portuguese SEL after-school program, Experiencing Emotions (Pereira & Marques-Pinto, Reference Pereira, Marques-Pinto, Marques-Pinto and Raimundo2016). The program intends to promote middle-school pupils’ socio-emotional skills in the SEL areas of self-awareness, social awareness, self-management, relationship skills and responsible decision-making (Payton et al., Reference Payton, Weissberg, Durlak, Dymnicki, Taylor, Schellinger and Pachan2008). The program consists of 12 hourly sessions distributed by three units, and is implemented as an extracurricular activity in schools, through group dynamics using Educational Dance activities in the domain of Education through Art, and moments of reflection and group discussion on the program content. Each session of the program begins with an Educational Dance warm-up activity in order to prepare the body for the session, to boost group cohesion, and also to practice socio-emotional skills. Then, the main Educational Dance activity of the session takes place, which is always related to the program contents of that session. Finally, the pupils sit in a circle with the facilitator, and all sessions end with a group reflection on the contents. In this last part of the session, a homework activity is proposed to the pupils regarding the contents of each session. The outcomes of the program were assessed in a previous study (Pereira & Marques-Pinto, Reference Pereira, Marques-Pinto, Marques-Pinto and Raimundo2016) which indicated that the program had positive effects on pupils’ socio-emotional skills in the areas of self-management and relationship skills when compared with the control condition pupils.
According to several studies (Baydar, Reid, & Webster-Stratton, Reference Baydar, Reid and Webster-Stratton2003; Calear, Christensen, Mackinnon, & Griffiths, Reference Calear, Christensen, Mackinnon and Griffiths2013; Low, Ryzin, Brown, Smith, & Haggerty, Reference Low, Ryzin, Brown, Smith and Haggerty2014; Pettigrew et al., Reference Pettigrew, Graham, Miller-Day, Hecht, Krieger and Shin2015; Prado, Pantin, Schwartz, Lupei, & Szapocznik, Reference Prado, Pantin, Schwartz, Lupei and Szapocznik2006; Schultes et al., Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014; Watts, Witt, & King, Reference Watts, Witt and King2008), participant responsiveness seems to be positively associated with program outcomes. Attendance is the most frequently evaluated indicator of participant responsiveness which proves to be associated with stronger program effects (e.g., Prado et al., Reference Prado, Pantin, Schwartz, Lupei and Szapocznik2006; Schultes et al., Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014). Other frequently examined indicators of participant responsiveness include participant satisfaction and homework completion (i.e., activities completed at home by the participants as a part of the program), which also prove to be associated with program outcomes (e.g., Baydar et al., Reference Baydar, Reid and Webster-Stratton2003; Calear et al., Reference Calear, Christensen, Mackinnon and Griffiths2013; Pettigrew et al., Reference Pettigrew, Graham, Miller-Day, Hecht, Krieger and Shin2015; Watts et al., Reference Watts, Witt and King2008).
Interventions targeting children and youths are confronted with the difficulty of maintaining interest and motivation towards the programs, especially if they are perceived as an extension of the school day (Watts et al., Reference Watts, Witt and King2008). A recent study has suggested that in after-school interventions with youths, participant responsiveness is one of the dimensions positively associated with youth experiences towards the programs (Cross, Gottfredson, Wilson, Rorie, & Connell, Reference Cross, Gottfredson, Wilson, Rorie and Connell2010). Furthermore, a recent study focusing on positive development youth programs showed that the success of the interventions was significantly predicted by student participation and involvement (Shek & Liu, Reference Shek and Liu2013). Therefore, it is important to design interventions which take the potential interest and involvement of children and youths into consideration, by creating programs that are relevant and appealing to them.
In the case of SEL programs, they are usually delivered through classroom-based verbal instruction (Merrell & Gueldner, Reference Merrell and Gueldner2010), so it is important to think of different approaches that can be more appealing and interesting for children and youths, particularly with regard to after-school interventions which are mainly optional. For example, programs that use artistic activities seem to match children and adolescents’ interests and satisfaction (Hutzel, Russell, & Gross, Reference Hutzel, Russell and Gross2010; Wright, John, Alaggia, & Sheel, Reference Wright, John, Alaggia and Sheel2006). More recently, one of the main features associated with effective SEL programs has been the use of activities to teach socio-emotional skills which are based on movement, participation, manipulation, and practice (Gullotta, Reference Gullotta, Durlak, Domitrovich, Weissberg and Gullotta2015). It is on this level that Education through Art activities such as Educational Dance may constitute an appealing and motivating strategy to enhance children and youths’ responsiveness towards SEL programs while corresponding to the recommendations of SEL interventions on the use of active forms of learning (Collaborative for Academic, Social and Emotional Learning [CASEL], 2015).
The present study aims to analyze three of the most frequently evaluated dimensions of participant responsiveness – attendance, homework completion and satisfaction – with regard to the Experiencing Emotions SEL program, which uses Educational Dance activities in the domain of Education through Art. It also sets out to examine the influence of attendance and homework completion on program outcomes. Higher levels of pupils’ attendance and homework completion in the program were expected to significantly predict a greater increase in pupils’ socio-emotional skills, well-being and school engagement, in comparison with the control condition, since responsiveness seems to be positively related to program outcomes.
Method
Participants
Data was collected as part of a previous study (Pereira & Marques-Pinto, Reference Pereira, Marques-Pinto, Marques-Pinto and Raimundo2016) using a 2 (intervention and control) X 2 (pre and post-test) quasi-experimental design, focusing on the effects of the Experiencing Emotions SEL program. The initial sample consisted of 105 pupils however, the sample of the present study included only those who participated in both the pre and post-test sessions. The sample consisted of 98 pupils from three randomly chosen state schools in Lisbon, from low to medium socio-economic backgrounds, 53 of whom participated in the program (7 groups) and 45 in the control group (5 groups). The program sample consisted of 27 5th grade pupils (60%), 16 6th grade pupils (36%) and two 7th grade pupils (4%), aged between 10 and 12 years (M = 10.62; SD = .72); 39 girls (87%) and 6 boys (13%); 29 pupils (64%) of Portuguese nationality and 16 (36%) of other nationalities/ethnicities (e.g., Brazilian, Cape Verdean, Gypsy). The control sample consisted of 20 5th grade pupils (53%), 12 6th grade pupils (32%), 5 7th grade pupils (13%) and one 8th grade pupil (3%), aged between 9 and 13 years (M = 10.87; SD = 1.12); 27 girls (71%) and 11 boys (29%); 24 pupils (63%) of Portuguese nationality and 14 pupils (37%) of other nationalities/ethnicities.
Measures
Measures of program outcomes
The outcomes on pupils’ socio-emotional skills, well-being and school engagement were assessed through measures filled in by the teachers and self-report measures filled in by the pupils.
Socio-emotional skills measures
Measures were selected considering the five main SEL domains (CASEL, 2015). The self-awareness domain was assessed through the Perceiving and understanding emotion scale of the Portuguese version (Faria & Lima-Santos, Reference Faria, Lima-Santos and Takšić2005) of the Emotional Skills and Competence Questionnaire (ESCQ) developed by Taksic (Reference Taksic2000) which measured the ability to perceive and understand emotion (15 items, e.g., “When I don’t like something, I immediately let it show”; α = .84 in the Portuguese adaptation study, and α = .86 in the present study sample). The ESCQ is a self-report measure with a five point scale ranging from 1 (never) to 5 (frequently). The SEL social awareness domain was assessed through another scale of the ESCQ, the Expressing and labeling emotion scale (14 items, e.g., “I can tell when someone is trying to hide a bad mood”; α = .83 in the Portuguese adaptation study, and α = .84 in the present study sample) measuring the ability to express and label emotion. The measures selected to evaluate the self-management SEL domain were the Managing and Regulating Emotion scale of the ESCQ (16 items, e.g., “When someone praises me, I work more enthusiastically”; α = .64 in the Portuguese adaptation study, and Cronbach α = .75 in the present study sample) measuring the ability to manage and regulate emotion, and also the Self-management sub-scale (10 items, e.g., “Remains calm when problems arise”; α = .83 in the Portuguese adaptation study, and α = .82 in the present study sample) of scale A of the School Social Behavior Scales (SSBS-2) developed by Merrell (Reference Merrell2002) and adapted to the Portuguese population by Raimundo et al. (Reference Raimundo, Carapito, Pereira, Marques-Pinto, Lima and Ribeiro2012). This sub-scale, which measured self-restraint, cooperation and compliance with rules and expectations (Merrell, Reference Merrell2002), was filled in by the teachers using a five point scale ranging from 1 (never) to 5 (very often). The relationship skills SEL domain was evaluated through another sub-scale of the SSBS-2, the Interpersonal skills sub-scale, filled in by the teachers (14 items, e.g., “Understands other pupils’ problems and needs”; α = .91 in the Portuguese adaptation study, and α = .92 in the present sample study), which measured the ability to establish positive relationships and gain social acceptance (Merrell, Reference Merrell2002). Finally, an indicator of the responsible-decision making domain was measured through the Interpersonal negotiation scale (8 items, e.g., “When my best friend and I don’t agree on what to do, I might: a. Try to convince my friend, b. Listen to my friend and work it out, c. Get upset and go away to be by myself, d. Go along with my friend; α = .68 in the Portuguese adaptation study, as well as in the present study sample) of the Portuguese version of the scale (Pereira & Marques-Pinto, Reference Pereira and Marques-Pintoin press) part of the Relationship Questionnaire (Rel-Q) developed by Schultz, Selman, and LaRusso (Reference Schultz, Selman and LaRusso2003). This scale evaluated how self and others are coordinated while considering the consequences of different actions. Each item had 4 multiple-choice answers and the respondent had to choose and rate the most suitable response. The two results were combined into a mean composite result.
Well-being and school engagement measures
Pupils’ well-being was measured through a self-report questionnaire, the Mental Health Continuum Short Form (MHC-SF), developed by Keyes (Reference Keyes2002) and adapted to the Portuguese population by Matos et al. (Reference Matos, André, Cherpe, Rodrigues, Figueira and Pinto2010), which measured the emotional, psychological and social dimensions of well-being: the Emotional well-being scale (3 items, e.g., “Satisfied”; α = .85 in the Portuguese adaptation study, and α = .66 in the present study sample), the Psychological well-being scale (6 items, e.g., “That you had experiences that allowed you to grow and become a better person”; α = .83 in the Portuguese adaptation study, and α = .79 in the present study sample), and the Social well-being scale (5 items, e.g., “That the way that our society works makes sense to you”; α = .80 in the Portuguese adaptation study, as well as in the present study sample). Behavioral school engagement was measured by the teachers through the sub-scale Academic skills of the SSBS-2 (Merrell, Reference Merrell2002; 8 items, e.g., “Completes assigned activities on time”; α = .91 in the Portuguese adaptation study, and α = .88 in the present study sample) measuring competent performance and engagement in academic tasks.
Measures of participant responsiveness
These measures focused on three indicators of participant responsiveness: satisfaction, attendance, and homework completion. Satisfaction was evaluated through a self-report item filled in by the pupils at the end of the program (e.g., “Did you enjoy participating in the program ‘Experiencing Emotions’?”) using a five-point scale ranging from 1 (not at all) to 5 (very much). Attendance and homework completion were measured by the facilitator in each session through a checklist in which the number of sessions attended by the pupils and the number of home exercises completed by each pupil were registered. The name of each pupil was registered in the first column of the checklist; present pupils’ names were recorded in the second column; the third column served to register the names of the pupils who had completed the homework. These two indicators were also monitored in the control groups.
Procedure
The research was approved by the Scientific Council of the Faculty of Psychology of the University of Lisbon, the entity responsible, at the time, for the ethical and scientific evaluation of research projects. The Experiencing Emotions program was planned on the basis of a case study of a Portuguese middle-school (Pereira & Marques-Pinto, Reference Pereira, Marques-Pinto, Marques-Pinto and Raimundo2016), which granted social validity to the program, resulting in the preparation of a program manual. In school-based prevention programs, a manual usually defines the intended content (Schultes et al., Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014). The program was implemented on the basis of the manual in three Portuguese schools with 7 different groups of pupils. Control sessions were simultaneously established with 5 groups of pupils consisting of individual handicraft activities over 12 weekly hourly sessions, the same duration and frequency of the program. Handicraft activities in the control sessions consisted of learning how to create useful and decorative handmade objects (e.g., key ring). Pupils worked individually with the help of the facilitator, and no socio-emotional skills were involved in the control activities. The program and control sessions functioned as extracurricular activities in the schools under study, and were conducted after authorization had been received from the administrative school council of the three schools. The activities were announced at each school at the beginning of the school year, and the pupils who wished to participate voluntarily signed up according to their interest in getting involved in one or another type of activity. A request for written informed consent was subsequently sent to pupils’ parents to obtain the required permission. The groups were created according to the number of enrolments with a limit of 15 participants for the program groups and 20 participants for the control groups. A total of 105 pupils attended the sessions however, the sample of the present study only included the pupils who attended both evaluation points. Sessions of the program and of the control condition took place in an appropriate classroom at the schools under study, and were conducted by a psychologist. Both groups were asked to do homework which implied different tasks. Each pupil received a worksheet with the corresponding home exercise at the end of each session, after explanation of the exercise was given by the facilitator. In the program, the homework implied learning and training socio-emotional skills (e.g., playing an emotional-related mimic game with their family), and in the control condition the homework consisted of handicraft activities (e.g., taking clippings from magazines). The measures that evaluated program outcomes were filled in by the pupils of both groups in the classrooms at the schools under study, and by the teachers, at home, following standard written instructions and without knowledge of whether the pupils had participated in the program or in the control condition. The data from the outcome measures were collected by one of the researchers, at baseline and post-test, and demographic data was collected at pre-test. The facilitator was required to fill in a checklist which included registering the pupils’ presence in each session and the number of home exercises completed by each pupil, both in the intervention and in the control groups. At the end of the program, the pupils filled in a self-report item in order to assess their satisfaction with the program.
Data analysis
Program outcomes
The outcomes of the program on pupils’ socio-emotional skills and on their well-being and school engagement were analyzed in a prior study (Pereira & Marques-Pinto, Reference Pereira, Marques-Pinto, Marques-Pinto and Raimundo2016) through repeated measures analysis of covariance (ANCOVA) based on the General Linear Model (GLM). In the aforementioned study, the effects of the program were measured by comparing the results of the outcome measures according to the groups (intervention and control), while pre-test, age, gender, grade, and nationality/ethnicity differences were controlled. The dependent variables which measured pupils’ socio-emotional skills were the Perceiving and understanding emotion scale (self-awareness domain); Expressing and labeling emotion scale (social awareness domain); Managing and regulating emotion scale and Self-management sub-scale (self-management domain); Interpersonal skills sub-scale (relationship skills domain); and Interpersonal negotiation scale (responsible decision-making domain). The Emotional well-being, Psychological well-being, and Social well-being scales were the dependent variables referring to pupils’ well-being, and the Academic skills sub-scale measured pupils’ behavioral school engagement.
Participant responsiveness
The statistical analysis was initially conducted through a descriptive analysis of the indicators of participant responsiveness with reference to means, standard deviations and range. Attendance was coded according to the number of sessions attended by the pupils: 1 corresponded to the pupils who participated in less than ¾ of the total sessions, and 2 to the pupils who participated in at least ¾ of the total sessions. Homework completion was coded on the basis of the number of exercises completed at home by the pupils: 1 corresponded to the pupils who completed ¼ or less of the total home exercises; 2 corresponded to the pupils who completed more than ¼ and less than ¾ of the home exercises; and 3 corresponded to the pupils who completed at least ¾ of the home exercises. Pupils’ satisfaction with the program was also coded ranging from 1 (dissatisfied) to 5 (very satisfied). A frequency analysis regarding attendance, homework completion and satisfaction was performed. Bivariate correlation and hierarchical regression analyses between the outcome measure results and attendance and homework completion were then performed to ascertain whether higher levels of these dimensions were significantly related and predicted higher improvement levels of pupils’ socio-emotional skills, well-being and school engagement, by comparing intervention and control groups results. Satisfaction was not included in this analysis due to the fact that this dimension was only evaluated in the program, therefore, not enabling a comparison between intervention and control pupils. Statistically significant r values greater than 0.5 were considered strong correlations, between 0.5 and 0.3 moderate correlations, and those between 0.3 and 0.1 were interpreted as weak correlations (Cohen, Reference Cohen1988). In the regression analyses, effect size was measured through the Cohen d values. Values above 0.5 were considered large effects, between 0.3 and 0.5 were considered moderate effects, and between 0.1 and 0.3 were considered small effects (Cohen, Reference Cohen1988). In order to control the impact of the demographic variables (gender and age), these variables were introduced in the first step of the regression (model 1), and to control the results of the outcome measures obtained in the pre-test, these variables were introduced in the second step of the analyses (model 2). Attendance and homework completion were introduced separately in the third step of the regression, and each variable was introduced separately (models 3 and 4, respectively). Prior to performing the regression analyses, the relevant assumptions were tested. Firstly, the sample size was deemed adequate, given that five independent variables were included in the analyses (Austin & Steyerberg, Reference Austin and Steyerberg2015). The assumption of multicollinearity was also met, measured through the Tolerance and VIF values (Hair, Black, Babin, & Anderson, Reference Hair, Black, Babin and Anderson2010).
Results
Descriptive analyses of participant responsiveness
Results indicated high levels of participants’ satisfaction (M = 4.53, SD = .82) and attendance (M = 1.85, SD = .36), and a medium-high level of participants’ homework completion (M = 1.94, SD = .75).
Regarding the frequency of each coded indicator, 8 pupils revealed low attendance in the program (15%), and 45 pupils revealed high attendance (85%). Concerning homework completion, 16 pupils (30%) revealed a low rate of homework completion; 24 pupils (45%) revealed a medium rate of homework completion; and 13 pupils revealed a high rate of homework completion (25%). According to pupils’ satisfaction with the program, none of the pupils reported dissatisfaction; 2 pupils were somewhat dissatisfied (4%); 5 pupils were somewhat satisfied (9%); 9 pupils were satisfied (17%); and 37 pupils (70%) were very satisfied.
Participant responsiveness and its influence on program outcomes
The results presented in Table 1 revealed the correlations between the outcome results of the program on pupils’ socio-emotional skills, well-being and school engagement and attendance and homework completion, comparing the intervention and control group results. The most significant correlations found were moderate, and some of them weak, however they were all in the expected direction. Results revealed that in the intervention group, higher levels of attendance were significantly associated with an increase in the outcome results in the SEL domains of social awareness (Expressing and labeling emotion scale r = .283) and self-management (Managing and regulating emotion scale r = .418; Self-management scale r = .374). Higher levels of pupils’ attendance in the program were also significantly and positively associated with the emotional and psychological dimensions of well-being (Emotional well-being scale r = .377; Psychological well-being scale r = .281) and school engagement (Academic skills sub-scale r = .274). Higher levels of homework completion by the pupils who participated in the program were significantly associated with an increase in the post-test results of the SEL self-management domains (Managing and regulating emotion scale r = .319; Self-management scale r = .358) and relationship skills (Interpersonal skills scale r = .314), and also with an increase in school engagement (Academic skills sub-scale r = .426). No significant correlations were found in the control groups with any indicator of participant responsiveness.
Correlations between attendance and homework completion and the outcome measure post-test results in the intervention and control groups
* p < .05; **p < .01.
The results of the hierarchical regression analyses performed to determine if the two responsiveness indicators of attendance and homework completion would significantly predict the outcome results are subsequently presented.
Socio-emotional skills
Results presented in Table 2 and Table 3 describe the regression analyses referring to pupils’ socio-emotional skills, in the intervention groups and in the control groups, respectively. All effect sizes referring to significant results were large (> 0.5).
Results of the hierarchical regression analysis relative to socio-emotional skills outcome measures in the intervention group
* p < .05; **p < .01; ***p < .001.
Results of the hierarchical regression analysis relative to the socio-emotional skills outcome measures in the control group
* p < .05; **p < .01; ***p < .001.
In the intervention groups, model 1, regarding the predictive value of the demographic variables, did not contribute significantly to the explanation of any of the socio-emotional skills variables. Model 2 added the pre-test variables to the regression models and results indicated that higher values in the pre-test were predictors of higher results in the post-test in terms of all socio-emotional outcome measures. However, results of Model 3 revealed that introducing attendance enhanced the percentage of explained variance of the Expressing and labeling emotion scale outcomes to 43% (Model 2 significantly explained 39%), F(1, 48) = 10.666, β = .234, p = .04, d = .59; of the Managing and regulating emotion scale outcomes to 38%, F(1, 48) = 8.989, β = .254, p = .04, d = .57 (Model 2 significantly explained 34%); and also of the Self-management scale outcomes to 50%, F(1, 48) = 13.925, β = .320 p = .003, d = .87 (Model 2 significantly explained 41%). Taken together, these results indicated that the intervention group pupils’ higher attendance was predictive of better results in the aforementioned outcome measures of the self-management and social awareness SEL domains. Finally, Model 4, introducing homework completion, further enhanced the percentage of explained variance of the Interpersonal skills sub-scale to 26% (model 2 accounted for 20%) F(1, 48) = 5.448, β = .264, p = .04, d = .59, indicating that the intervention group’s higher rate of homework completion was predictive of better results in this outcome measure of the relationship skills SEL domain.
In the control groups, results revealed that Model 1 only contributed significantly to the explanation of the Interpersonal negotiation scale outcome, accounting for 15% of its variation, and that age was the demographic variable that contributed significantly to the regression model, F(2, 42) = 4.918, β = .328, p = .02, d = .71. However, and in line with the intervention groups, Model 2, introducing the pre-test variables, enhanced the percentage of explained variance of this scale to 31%, F(1, 41) = 7.585, β = .423, p = .002, d = .98, as well as of all the other socio-emotional outcome measures, indicating that higher results in the pre-test were predictors of higher results in the post-test. Model 3 and Model 4 results revealed that attendance and homework completion respectively were not significant predictors of the socio-emotional skills outcome measures in the control condition.
Well-being and school engagement
The regression analyses results related to pupils’ well-being and school engagement, for both the intervention groups and control groups respectively are presented in Tables 4 and 5. Large effect sizes were encountered (> 0.5) with regard to significant results.
Results of the hierarchical regression analysis relative to well-being and school engagement outcome measures in the intervention group
* p < .05; **p < .01; ***p < .001.
Results of the hierarchical regression analysis relative to well-being and school engagement outcome measures in the control group
* p < .05; **p < .01; ***p < .001.
In the intervention groups, Model 1 did not reveal any significant results for the well-being and school engagement outcomes. When the pre-test variables were introduced (Model 2), higher results in the pre-test were predictive of higher results in the Psychological well-being scale outcomes, F(1, 49) = 3.568, β = .364, p = .008, d = .76, accounting for 13% of the variance. Model 3, including attendance, enhanced the percentage of explained variance of this scale, accounting for 23% of the variance, F(1, 48) = 4.906, β = .349, p = .009, d = .76, indicating that higher attendance was predictive of higher results in the psychological dimension of well-being. Model 2, including the pre-test variable, also revealed significant results concerning the Academic skills sub-scale, measuring behavioral school engagement, F(1, 49) = 11.042, β = .621, p = .001, d = 1.55, and accounted for 37% of the variance. Model 3, introducing attendance, was, however, a better predictor, accounting for 41% of the variance in this scale, F(1, 48) = 9.839, β = .223, p = .04, d = .56. Model 4, introducing home practice completion, was also a better predictor than Model 2, significantly accounting for 45% of the variance, F(1, 48) = 11.745, β = .314, p = .005, d = 1.50. These results revealed that both higher attendance and home practice completion predicted higher behavioral school engagement. Finally, model 3, including attendance, significantly predicted the outcome results of the Emotional well-being scale, accounting for 15% of the variance, F(1, 48) = 3.257, β = .316, p = .02, d = .67. Taken together, the results revealed that higher attendance predicted higher results in both the psychological and the emotional dimensions of well-being.
In the control condition, results revealed that Model 1 only contributed significantly to the explanation of the Psychological well-being scale outcome, F(2, 42) = 3.986, β = –.328, p = .03, d = –.69, accounting for 12% of the variance, and that gender was the demographic variable that contributed significantly to this regression model. However, the introduction of the pre-test variable (Model 2) significantly enhanced the percentage of explained variance of this scale to 26%, F(1, 41) = 6.252, β = .423, p = .004, d = .92. Higher results in the pre-test (Model 2) were also predictive of better results in the Academic skills sub-scale, F(1, 41) = 9.295, β = .633, p = .005, d = 1.58, accounting for 36% of the variance. In this sub-scale, however, Model 4 was a better regression model, F(1, 40) = 10.404, β = .334, p = .006, d = .88, accounting for 46% of the variance, indicating that pupils’ higher rate of home practice completion was predictive of higher school engagement. Attendance (Model 3) was not a significant predictor of the well-being and school engagement outcome measures in the control condition.
Discussion
The main purpose of the present study was to analyze participant responsiveness in the Experiencing Emotions SEL program, and to explore its influence on program outcomes.
Regarding the overall participant responsiveness towards the Experiencing Emotions program, results revealed high levels of pupils’ satisfaction and attendance, and a medium-high level of homework practice completion. These results suggest that the program met pupils’ interest and satisfaction.
Several studies have highlighted the positive relationship between participant responsiveness and program outcomes (Baydar et al., Reference Baydar, Reid and Webster-Stratton2003; Low et al., Reference Low, Ryzin, Brown, Smith and Haggerty2014; Pettigrew et al., Reference Pettigrew, Graham, Miller-Day, Hecht, Krieger and Shin2015; Schultes et al., Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014; Watts et al., Reference Watts, Witt and King2008). Therefore, in the current study, it was hypothesized that higher levels of pupils’ attendance and homework completion in the program would significantly predict a greater increase in pupils’ socio-emotional skills, well-being and school engagement, in comparison with the control condition.
Results revealed that pupils’ higher attendance in the program predicted better results in the self-management and social awareness SEL domains, and that pupils’ higher rate of home practice completion in the program predicted better results in the relationship skills SEL area. In the control condition pupils’ attendance and home practice did not predict pupils’ socio-emotional outcome results. Even though the results were not found in all SEL domains, these findings partially support our hypothesis. Indeed, the results suggest that attendance may positively enhance the impact of SEL programs on skills involving the ability to regulate one’s emotions, thoughts and behaviors (self-management domain), and taking the perspective of others, understanding social and ethical norms, and acknowledging resources and support (social awareness domain). Additionally, the results also suggest that home practice completion may positively influence the impact of SEL interventions on the ability to establish and maintain healthy and rewarding relationships (relationship skills domain). Some of the home exercises of the program implied family participation, thus these types of exercises may explain why relationship skills were more enhanced, since family involvement in school seems to have a positive impact on social and behavioral skills and may help to build positive relationships (Weiss, Bouffard, Bridglall, & Gordon, Reference Weiss, Bouffard, Bridglall and Gordon2009).
The results also revealed that pupil’s higher attendance in the program predicted better results in emotional and psychological well-being, as well as higher results in school engagement, which gives support to the hypothesis of the current study. This result seems to indicate that a higher attendance rate in the program positively influences pupils’ satisfaction with life (emotional well-being) and feelings of personal accomplishment (psychological well-being), as well as pupils’ behavioral engagement in school. However, results concerning home practice completion showed that in both the intervention and control groups, pupils’ higher rates of home exercise completion predicted school engagement, which partially counters what was hypothesized. One possible interpretation for this result may be that since the evaluated school engagement variable refers to adherence, participation and engagement in school tasks and school-related activities (Christenson, Reschly, & Wylie, Reference Christenson, Reschly and Wylie2012), the accomplishment of exercises at home, in both conditions, that were related to the school setting may have served to enhance pupils’ engagement in the remaining school activities.
Overall, the findings regarding the relationship between participant responsiveness and program outcomes suggest that participant responsiveness positively influences program outcomes, supporting the results of previous research (Baydar et al., Reference Baydar, Reid and Webster-Stratton2003; Low et al., Reference Low, Ryzin, Brown, Smith and Haggerty2014; Schultes et al., Reference Schultes, Stefanek, van de Schoot, Strohmeier and Spiel2014; Watts et al., Reference Watts, Witt and King2008), particularly when the outcomes are direct effects of the program. In the present study, some of the direct effects of the program related to pupils’ socio-emotional skills were positively influenced by responsiveness indicators in the program condition, compared with the control condition. As for the distal effects of the program on school engagement, the influence of responsiveness was not only observed in the program but also in the control condition. These results suggest that the control after-school activities may also have had some type of effect on this distal outcome variable.
A positive association between pre-test and post-test results was found, suggesting that the pupils who had higher socio-emotional skills prior to the intervention were the ones who most developed their skills. Neuropsychology research has shown that children and youths with better executive function skills, which control thoughts, behavior and emotions, may be more teachable (Blair, Reference Blair2002), and that these skills are positively associated with socio-emotional competence (Pears, Fisher, Bruce, Kim, & Yoerger, Reference Pears, Fisher, Bruce, Kim and Yoerger2010). These findings may justify why children and youths with higher initial socio-emotional skills seem to be more able to enhance them.
Limitations regarding the present study should, however, be taken into consideration. Firstly, the fact that the research was conducted in a Portuguese context does not allow for the generalization of the results to all Portuguese middle-school pupils or to pupils from other countries. Another important limitation is related to the fact that the recruitment of the pupils depended on their voluntary participation. Hence, there may be some bias related to systematic differences about the group choosing the program activities or the control activities. Furthermore, satisfaction was not measured in the control condition therefore, not allowing for comparisons between both groups. Lastly, the current study did not measure other dimensions also considered by research as predictors of implementation quality, such as adherence, quality of delivery and adaptation.
Despite these limitations, the findings of the present study have important implications both for research and practice related to SEL interventions targeting children and youths. First and foremost, the results revealed high responsiveness rates on the part of the participants towards the Experiencing Emotions program, which is relevant not only since responsiveness influences program outcomes (e.g., Low et al., Reference Low, Ryzin, Brown, Smith and Haggerty2014), but also considering that it is a challenge to develop interventions that children and adolescents find appealing and motivating (Watts et al., Reference Watts, Witt and King2008), especially when they are optional such as most after-school interventions. Therefore, it seems that the use of Education through Art activities in SEL programs, as was the case in the present study, may be a promising strategy in order to promote child and youth responsiveness, particularly in after-school interventions. Increasing child and youth responsiveness to programs is crucial for the effective and efficient delivery of interventions in schools and in the community (Calear et al., Reference Calear, Christensen, Mackinnon and Griffiths2013).
Secondly, the current study adds important findings in terms of the relationship between specific indicators of participant responsiveness and specific outcomes of a SEL program. Attendance was positively related to self-management and social awareness SEL domains, to emotional and psychological well-being, and to school engagement. Home practice completion was positively related to the relationship skills SEL area, and also to school engagement.
Future studies regarding SEL programs should further explore how different aspects of implementation influence outcomes in terms of the five main SEL areas (self-awareness, social awareness, self-management, relationship skills, and responsible decision-making), and other dimensions usually associated with socio-emotional skills.
In short, the present study contributes to a growing body of research demonstrating the importance of implementation quality research as a component of program planning by highlighting the implications of participant responsiveness for program outcomes, thus contributing to further knowledge in the area of SEL interventions with children and adolescents. Furthermore, this study contributes to validating a new SEL approach which uses Education through Art activities in support of the idea defended by other studies that programs should feature children and youth voices by meeting their interests and satisfaction (Watts et al., Reference Watts, Witt and King2008), and also that artistic activities may help to accomplish this task (Wright et al., Reference Wright, John, Alaggia and Sheel2006).
