As young children enter the school setting, they are increasingly expected to navigate social interactions and modulate emotions in emotion-eliciting situations. For instance, the ability to decrease anger feelings in response to frustrating assignments or peer interactions, as well as the ability to promote interest in topics or activities they may not find naturally stimulating, help children succeed throughout the day. Emotion competence (emotion knowledge and emotion regulation; Buckley & Saarni, Reference Buckley, Saarni, Buckley and Saarni2006) is a construct that consistently predicts young children's abilities in a number of other developmental domains and is often required for successful adaptive functioning in the school context (Blair, Reference Blair2002; Denham, Reference Denham2006; Ladd, Herald, & Kochel, Reference Ladd, Herald and Kochel2006; Thompson & Raikes, Reference Thompson, Raikes, Perry, Kaufman and Knitzer2007). Greater emotion competence is associated with less psychopathology and better social competence, adaptive behavior, school readiness, and academic success, both predictively and concurrently (Bierman et al., Reference Bierman, Domitrovich, Nix, Gest, Welsh and Greenberg2008; Cook, Greenberg, & Kusche, Reference Cook, Greenberg and Kusche1994; Denham et al., Reference Denham, Bassett, Mincic, Kalb, Way and Wyatt2012; Izard et al., Reference Izard, Fine, Schultz, Mostow, Ackerman and Youngstrom2001, Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008; Southam-Gerow & Kendall, Reference Southam-Gerow and Kendall2002). Children with greater emotion knowledge are better able to identify emotions in themselves and others and understand the antecedents and consequences of emotions. This knowledge guides their interactions with peers and adults (Denham et al., Reference Denham, Blair, DeMulder, Levitas, Sawyer and Auerbach-Major2003; Denham & Burton, Reference Denham and Burton2003; Mostow, Izard, Fine, & Trentacosta, Reference Mostow, Izard, Fine and Trentacosta2002; Schultz, Izard, & Bear, Reference Schultz, Izard and Bear2004). Comprehension of emotions in themselves and others may lead children to anticipate and avoid conflict situations or implement other emotion regulation strategies (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008). Such emotion regulation skills are believed to occupy an essential role in children's adaptive and social functioning (Cole, Martin, & Dennis, Reference Cole, Martin and Dennis2004; Raver, Reference Raver2004).
In addition to normative age-related development, socialization and environmental factors also influence the development of emotion regulation skills. Young children from urban, low socioeconomic status populations grow up exposed to multiple long-term stressors, including higher levels of parental psychopathology, household instability, family conflict, community violence, and fewer available protective resources (Ackerman & Brown, Reference Ackerman, Brown and Kail2006, Reference Ackerman, Brown, Evans and Wachs2010; McLoyd, Reference McLloyd1998). The factors associated with living in poverty lead to a disproportionate risk for a variety of poor outcomes (Bassett, Denham, Wyatt, & Warren-Khot, Reference Bassett, Denham and Warren-Khot2012; Hackman & Farah, Reference Hackman and Farah2009; Mulligan, Hastedt, & McCarroll, Reference Mulligan, Hastedt and McCarroll2012). Children growing up in low-income households tend to exhibit less emotion knowledge and emotion regulation skills than do their more advantaged peers (Blandon, Calkins, Keane, & O'Brien, Reference Blandon, Calkins, Keane and O'Brien2008; Brown & Ackerman, Reference Brown and Ackerman2011; Li-Grining, Reference Li-Grining2007; Raver, Blair, & Willoughby, Reference Raver, Blair and Willoughby2012). Because these skills are important predictors of school readiness and overall adaptive functioning, early deficits are a cause for significant concern.
Prevention Programs
Fortunately, emotion competence skills can be improved with an increased focus on emotions and emotion regulation in childhood interventions (Domitrovich, Cortes, & Greenberg, Reference Domitrovich, Cortes and Greenberg2007; Fitzgerald & Edstrom, Reference Fitzgerald, Edstrom, Fitzgerald and Edstrom2006; Webster-Stratton & Reid, Reference Webster-Stratton and Reid2004). A number of universal classroom-based prevention programs have been successfully implemented with preschool children. Among them are Al's Pals Kids Making Healthy Choices (Lynch, Geller, & Schmidt, Reference Lynch, Geller and Schmidt2004), the Chicago School Readiness Project (Raver et al., Reference Raver, Jones, Li-Grining, Zhai, Metzger and Solomon2009), the Emotion-Based Prevention Program (EBP; Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008), I Can Problem Solve (Shure, Reference Shure2001), the Incredible Years Dina Dinosaur Classroom Curriculum (Webster-Stratton, Reid, & Stoolmiller, Reference Webster-Stratton, Reid and Stoolmiller2008), the Preschool Promoting Alternative Thinking Strategies Program (Domitrovich et al., Reference Domitrovich, Cortes and Greenberg2007), and the Second Step Preschool/Kindergarten curriculum (Frey, Hirschstein, & Guzzo, Reference Frey, Hirschstein and Guzzo2000). Although the outcome measures varied, a number of positive effects on children's development have been reported. Overall, these preschool prevention programs reported improvements to children's social and emotional competence, problem-solving abilities, and general classroom behavior.
Prevention programs that focus on improving emotion regulation also prove beneficial in promoting academic competence and school success. Improved regulation and behavior minimizes interference with cognitively stimulating activities and other teacher instruction (Raver et al., Reference Raver, Jones, Li-Grining, Zhai, Metzger and Solomon2009). Therefore, classrooms with a focus on improving emotion competence, self-regulation, and behavior are likely to result in better academic outcomes than are programs focused solely on academic content (Algozzine, Wang, & Violette, Reference Algozzine, Wang and Violette2011; Blair & Diamond, Reference Blair and Diamond2008).
At this point, supporting research for most published programs is composed of efficacy studies, relying on research staff for implementation, supervision, or support. However, not all efficacious interventions are effective or able to be successfully implemented in community settings. A legitimate cause for concern is how well these intervention programs are implemented and sustained beyond the initial program development and evaluation. As a next step, it is necessary to evaluate program effectiveness to determine the degree to which efficacious programs are able to improve child outcomes when delivered in real-world conditions. Limited research has begun examining the sustainability of evidence-based programs in preschools. In a recent sustainability study of an adapted version of the Second Step Preschool/Kindergarten curriculum (Wenz-Gross & Upshur, Reference Wenz-Gross and Upshur2012), six of seven classrooms completed all program lessons, with an average research-staff-rated fidelity score of 4.09 on a 5-point scale during a year with no programmatic support. In another sustainability study of the PATHS curriculum, research staff ratings indicated that 90% of the 20 teachers involved in an initial efficacy study continued use of the PATHS curriculum the following year (DeRousie & Bierman, Reference DeRousie and Bierman2012). These results are promising for the goal of program dissemination. However, additional comprehensive studies are needed to evaluate not only teacher fidelity but also child outcomes when the responsibility for program implementation is contained within the school system. Furthermore, sustainability tends to be considered the degree to which a program is successfully maintained for at least 2 years after the initial training and implementation year (Bumbarger & Perkins, Reference Bumbarger and Perkins2008; Elias, Zins, Graczyk, & Weissberg, Reference Elias, Zins, Graczyk and Weissburg2003). Therefore, it is crucial to continue evaluation after research staff have withdrawn for more than 1 year to determine what programs are truly feasible and worthwhile.
In addition, it is vital to evaluate and identify interventions that can be successful with children from disadvantaged populations, as well as be maintained in the community settings serving such populations. The current research began to address these limitations.
Present Study
The goal of this study was to evaluate the effectiveness of a theoretically coherent emotion-focused prevention program, the EBP, which was previously found to be efficacious in the same setting, an urban preschool Head Start system (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008). Two years prior to this study, EBP was introduced in this Head Start system as part of an efficacy trial that compared the effects of EBP to an established universal prevention program, I Can Problem Solve (ICPS; Shure, Reference Shure2001). ICPS is a social–cognitive program designed to teach children how to think in a problem-solving framework in order to improve interpersonal difficulties, aggression, frustration tolerance, and social withdrawal. Prior to the efficacy trial, ICPS had been in place in this Head Start system's curriculum for more than 12 years. After the completion of the efficacy trial, teachers continued administering EBP and ICPS in the same classrooms, with a brief 2-hr refresher training as the only support provided by research staff during the gap year.
The present study examined the level of teacher fidelity to the EBP intervention 2 years after the initial teacher training, with a 2-hr “refresher” 1 year later and again prior to this study. We expected that all teachers would be rated as demonstrating “acceptable” levels of program fidelity. This study also compared EBP against non-EBP classrooms to examine treatment effects on child outcomes 2 years after the initial intervention efficacy trial. While teachers in non-EBP classrooms were expected to administer the ICPS intervention, the implementation was not observed nor rated for fidelity. Therefore, the comparison condition is considered to be “as-usual Head Start” because the degree to which ICPS was implemented as prescribed is unknown. As with prior EBP results, we predicted that EBP would lead to greater increases in emotion knowledge and in regulation skills than in non-EBP classrooms. Next, also consistent with prior EBP results, we hypothesized that EBP would lead to greater decreases in maladaptive behaviors and increases in adaptive behaviors, as compared to as-usual Head Start classrooms. Finally, as in prior studies (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008), we also considered potential moderating variables of age, sex, verbal ability, and ethnicity in order to more fully understand the context under which the intervention programs are most effective. In addition, risk factors and classroom emotional support were also included as potential moderators. Because cumulative risk has the potential to impact children's outcomes (Ackerman & Brown, Reference Ackerman, Brown and Kail2006; Raver et al., Reference Raver, Blair and Willoughby2012) and the low socioeconomic demographic targeted in this study tends to experience a greater relative number of risk factors (Ackerman & Brown, Reference Ackerman, Brown and Kail2006, Reference Ackerman, Brown, Evans and Wachs2010), it is prudent to assess the degree of risk that participating children face. Similarly, emotional support in the classroom also affects children's behaviors and can also influence program effectiveness (Hirsh-Pasek, Golinkoff, Berk, & Singer, Reference Hirsh-Pasek, Golinkoff, Berk and Singer2009; Stoiber, Reference Stoiber2011).
Method
Design
Random assignment was established for the efficacy trial 2 years prior (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008) and retained for the current study. The previous trial utilized a cluster randomized design, with randomization occurring at the center level to avoid contamination of treatment condition within centers. Three Head Start centers were assigned to the treatment condition (EBP) and three centers to the comparison condition (ICPS). The number of classrooms per center ranged from 1 to 9, with a total of 13 classrooms in each condition.
The EBP Program
EBP was based on differential emotions theory (Izard, Reference Izard2001, Reference Izard2002, Reference Izard2009), which emphasizes the inherently adaptive and motivational functions of emotions and recognizes that poor emotion understanding and regulation may put children on a trajectory toward psychopathology. EBP consists of the teacher-conducted Emotions Course (EC) in the classroom, emotion tutoring and coaching teacher dialogues, and weekly parent messages aimed to reinforce the lessons taught in EC. These treatment components are designed to foster the acquisition and development of emotion knowledge, emotion regulation, and emotion utilization (i.e., emotion-related skills that help children harness the emotion motivation and arousal for constructive behavior). For a more detailed description of the development, components, and rationale of EBP, see Izard et al. (Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008) and Izard, Trentacosta, King, and Mostow (Reference Izard, Trentacosta, King and Mostow2004). Previous implementations have resulted in treatment-induced increases in emotion knowledge, emotion regulation, and adaptive behaviors, and decreases in maladaptive behaviors (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008).
The emotions course
EC lessons and activities are intended to help children learn about and practice emotion recognition, labeling, activation, regulation, and utilization. EC lessons focus mainly on joy/happiness, sadness, anger, and fear, but they also introduce the emotions of interest and contempt.
Each EC lesson is composed of three to five parts, which are delivered by the teacher during a large group instruction time throughout the week. Each weekly lesson begins with a puppet show that illustrates the main concept of the lesson. Throughout each lesson, the teachers elicit children's participation by asking them to label or demonstrate emotion expressions, share their ideas as to what causes them to feel the emotion being discussed, compare expressions of different emotions and different intensities, and draw or act out emotion expressions for their classmates. Teachers are trained to respond appropriately to the children's emotion experiences during EC games and activities. Each lesson ends with an interactive reading of an emotion storybook.
Emotion tutoring and coaching
Teacher–child dialogues describe methods for teachers to interact with and help children in the moment they are experiencing emotion dysregulation, both during and outside of the EC lesson time. The dialogues are intended to help teachers increase children's emotion competence by providing support and emotion-specific techniques to children who are having a difficult time regulating their emotions.
Parent messages
The EBP parent component consists of weekly handouts that teachers send home with each child. These parent messages summarize the EC lesson of the week and provide instructions for ways that parents can interact with their children to review and reinforce the material.
Participants
We recruited participants from an inner-city Head Start system in a mid-Atlantic state. Head Start administration incorporated the intervention into their curriculum, so all children in the treatment classrooms received EBP. Continuing longstanding practice, non-EBP teachers were scheduled to implement ICPS to all children in their classrooms, although the degree to which it was implemented correctly is unknown. Teachers distributed consent forms to all children. Of 465 children asked to participate, 295 consents (63%) were returned and parents of 277 elected to participate (94% of returned consents). Four children (1% of those consented to participate) were withdrawn from the Head Start system before data collection began, resulting in a sample size of 273 children at pretest. The final sample size was reduced to 248 children at posttest (91% of pretest sample) due to family relocations outside of the Head Start system (4%; 12 children), removal from the Head Start system due to chronic nonattendance (2%; 6 children), a change to a cross-condition classroom during the program implementation (2%; 5 children), and family request for removal from the study (1%; 2 children). From the final sample, children ranged in age from 3.17 to 5.17 years (M = 4.28, SD = 0.55) and were split evenly by sex (52% male). Parent-reported race and ethnicity indicated that 59% of the children were African American; 40% were Hispanic or Latino.
EBP was delivered by 13 teachers, 11 of whom completed training for and delivered EBP during the efficacy trial 2 years prior. The as-usual non-EBP condition also consisted of 13 teachers, 11 of whom were involved in the efficacy trial 2 years prior. Four teachers (2 EBP and 2 as-usual non-EBP) were new to the Head Start system. Because high rates of turnover are unfortunately common in early childcare settings, all classrooms and teachers were included to more realistically examine the effectiveness and sustainability of the intervention in a community setting.
Procedure
Following the original teacher training delivered as part of the efficacy trial 2 years previously and the 2-hr refresher session delivered 1 year later, Head Start administration arranged for two 1-hr “refresher” trainings for both EBP and as-usual groups prior to the current intervention implementation. Trainings took place as part of teacher in-service training.
Research staff collected preintervention data from mid-September through mid-November. Data collection began approximately 1 month after Head Start commenced to allow children to become acclimated and teachers to get to know their students. After preintervention data collection, teachers administered the 20-week programs and underwent fidelity checks. Classroom climate was observed and rated halfway through the intervention implementation. Postintervention data collection took place during April and May of the same academic year.
Interviewers conducted child assessments (“interviews”) during the school day at children's Head Start centers. The interviewers met with children individually for three sessions approximately 25 min long. Each child was administered tasks in a fixed order, with the verbal ability task first and the emotion knowledge measure second. After each task, interviewers rated children's attention to task demands. Children reported to be Spanish speaking by either parents or teachers received the verbal ability measure in both English and Spanish, and the language with the higher standard score was the language used to administer the remaining measures.
Trained independent observers conducted observations of children's social behaviors during playtime in their classrooms. Observers trained over a period of 5 weeks prior to conducting live observations. After receiving detailed oral and written explanations of observation categories and instructions for making ratings of behaviors (described in the Measures Section below), they first practiced using prerecorded videos and then conducted live practice observations of 3- to 5-year-old children in classrooms at a university-based preschool. Training was complete when each observer achieved an average of 80% agreement with one of the authors over three consecutive live 8-min observation sessions. Due to time and staff limitations, 40% of children from each classroom were randomly selected for observation. Observers blind to treatment condition conducted three observations on each child. Observations were conducted during free-play time in the classrooms, such that while some activities were semistructured by teachers, children were allowed to play in the center (e.g., dramatic play, block area, or teacher-led activity) of their choice.
Interviewers collected parent data at preintervention as part of a larger data collection module. Primary caregivers reported on family demographics and their own symptoms of depression. Interviews were conducted at Head Start centers at times convenient for parents. Interviewers met individually with parents for a 90-min session during which they read each measure aloud in either Spanish or English, whichever the parent identified as his or her primary language. Parents were compensated $75.
Teachers completed report measures on children's emotion regulation, behavior problems, social behavior, and emotion expressions. Teachers were compensated $10 per child at each assessment period.
Measures
Fidelity
During the program implementation, research staff observed each EBP teacher conduct a sample of five EC lessons, approximately one per month. Because observation days were scheduled through center managers, it is likely that teachers were aware of the days they could possibly be observed. Observers used a 6-point Likert scale (1 = poor, 6 = excellent) to complete ratings of fidelity to treatment components. Measured components included use of program techniques, coverage of lesson content, and faithfulness to the conceptual framework of the program. Observations lasted the length of morning circle time, approximately 10–20 min.
Multiple methods of data collection were utilized for child outcomes. Child assessments, independent observations, and teacher reports were collected at pre- and postintervention time points. Parent interview data were collected at preintervention, and classroom observations were conducted midintervention. All measures have been found to have acceptable to very good reliability.
Emotion knowledge
The emotion matching task (EMT; Izard, Haskins, Schultz, Trentacosta, & King, Reference Izard, Haskins, Schultz, Trentacosta and King2003) measured four facets of emotion knowledge: (a) recognition of emotion expressions, (b) expressive emotion labeling, (c) receptive emotion labeling, and (d) association of causes of emotion expressions of joy, sadness, anger, and fear. The original EMT has 48 items and was specially developed with ethnically diverse and well-standardized emotion expression photographs for use with multiethnic populations. Correlations of the EMT total score with widely used measures of children's emotion knowledge attest to its construct validity (Morgan, Izard, & King, Reference Morgan, Izard and King2010). Due to time constraints, the EMT was divided into split-half versions, EMT-A (α = 0.77 at pretest; 0.77 at posttest) and EMT-B (α = 0.80 at pretest; 0.76 at posttest). A one-way analysis of variance revealed no significant differences between versions, F (172) = 0.78, p = .38.
Emotion regulation
The Emotion Regulation Checklist (Shields & Cicchetti, Reference Shields and Cicchetti1997) measured children's emotion regulation abilities. Teachers rated children's abilities using a 4-point Likert scale. Two items on emotion knowledge were removed from the measure, resulting in a 22-item scale (α = 0.90 at pretest; 0.89 at posttest).
Behavior problems
Teachers used the Caregiver–Teacher Report Form (Achenbach & Rescorla, Reference Achenbach and Rescorla2000) to rate children's internalizing (α = 0.90 at pretest; 0.86 at posttest) and externalizing (α = 0.96 at pretest; 0.96 at posttest) behavior problems. The Achenbach scales are widely used measures with well-established reliability and validity. Teachers used a 3-point Likert scale to rate how characteristic each item was of the child.
Social behavior
Teachers rated children's social competence using the Adaptive Social Behavior Inventory (Hogan, Scott, & Bauer, Reference Hogan, Scott and Bauer1992). One item on emotion knowledge was removed from the measure. Teachers used a 3-point Likert scale to rate children on 29 items, resulting in prosocial (α = 0.93 at pretest; 0.92 at posttest) and disruptive (α = 0.82 at pretest; 0.80 at posttest) scale scores.
The Behavior and Emotion Expression Observation System (Izard, King, & Finlon, Reference Izard, King and Finlon2007), a structured behavioral observation coding system, was used to provide data on children's social behaviors during playtime in the classroom. Each observation period lasted 8 min, with observers making ratings at 15-s intervals, for a total of 32 ratings per observation period. At the end of every interval, the observers recorded the type of behavior the child was engaged in at that exact moment. Categories of behavior included positive social interactions (e.g., positive or neutral comments or gestures between the target child and an adult or peer), negative social interactions (e.g., reprimands, arguments, or statements or gestures indicating disapproval between the target child and an adult or peer), and solitary behaviors (e.g., individual engagement in an activity, watching peers while not otherwise interacting, or exhibiting daydreaming-like behaviors). Throughout data collection, approximately 13% of observations were simultaneously coded by two observers to assess reliability (Cohen κ = 0.86 at pretest, 0.95 at posttest). Children were observed during three free-play periods, across three separate days, and scores represent the percentage of total ratings that children received for each category of behavior.
Emotion expression
Teachers used the Emotion Expression Ratings Scale (Izard, Reference Izard2000) to report the frequency of children's emotion expressions in the classroom. Teachers rate how frequently children display the expressions of seven emotions using a 7-point Likert scale, resulting in a positive expression scale (interest and joy; α = 0.77 at pretest, 0.83 at posttest) and negative emotion expressions scale (sadness, anger, fear, shame, and guilt; α = 0.75 at pretest, 0.80 at posttest).
Potential moderators
Verbal ability
The Peabody Picture Vocabulary Test—Third Edition (PPVT-III; Dunn & Dunn, Reference Dunn and Dunn1997) and the Test de Vocabulario en Imagenes Peabody (TVIP; Dunn, Lugo, Padilla, & Dunn, Reference Dunn, Lugo, Padilla and Dunn1986) measured receptive vocabulary in English and Spanish, respectively. The TVIP was developed based on the PPVT to assess the vocabulary of Spanish-speaking and bilingual students. In both versions, children were shown four pictures and asked to point to the picture that represents a word spoken by the interviewer. All children received the PPVT-III. Interviewers administered the TVIP to children reported to be Spanish speaking by either parents or teachers. Standard scores were used to represent children's verbal ability. For children completing both the PPVT-III and the TVIP, the higher standard score was considered their verbal ability score. Verbal ability is included as a statistical control, because children with greater verbal abilities often perform better on measures of emotion knowledge (Cassidy, Werner, Rourke, Zubernis, & Balaraman, Reference Cassidy, Werner, Rourke, Zubernis and Balaraman2003).
Attention during testing
The Inattentiveness Scale of the General Assessment of Test Session Behavior (Glutting & Oakland, Reference Glutting and Oakland1991) included seven items rated on a 3-point Likert scale. Following the administration of the EMT (α = 0.79 at pretest, 0.80 at posttest), interviewers rated the child's attention during the testing session.
Contextual risk
Contextual risk represents variables that are likely to destabilize and negatively impact family functioning and child development (Brown & Ackerman, Reference Brown and Ackerman2011). The risk measure was composed of four dichotomous items obtained from a demographics inventory. Risk items included whether the parent had less than a high school education (28% of families), the family was primarily a single-parent household (44%), there was a change in the parent's intimate partner status in the previous 2 years (31%), and the family changed residences two or more times in the previous 2 years (15%). These risk variables covary with poverty and are contributing factors of stress within economically disadvantaged homes (Ackerman & Brown, Reference Ackerman, Brown and Kail2006). Because adversity impacts child outcomes in a cumulative nature (Ackerman & Brown, Reference Ackerman, Brown and Kail2006; Raver et al., Reference Raver, Blair and Willoughby2012), individual items were totaled to form the contextual risk score, ranging from 0 to 4.
Parental depression
The Quick Inventory of Depressive Symptoms (Rush et al., Reference Rush, Trivedi, Ibrahim, Carmody, Arnow and Klein2003; α = 0.77) is a self-report measure for depressive symptoms. Although the Quick Inventory of Depressive Symptoms was standardized as a 16-item measure, 1 item related to suicide was removed to decrease potential stress associated with completing the questionnaire and to remain compliant with institutional review board regulations. Parents reported on the severity of symptoms related to the domains that define a major depressive episode.
Classroom environment
Independent, trained observers followed the Classroom Assessment Scoring System (Pianta, LaParo, & Hamre, Reference Pianta, LaParo and Hamre2006) to observe and rate classroom levels of emotional support midway through the intervention period. Specifically, the emotion support composite (α = 0.94) rates positive climate, negative climate, teacher sensitivity, and regard for student perspectives. Emotionally supportive climates can support the development of children's emotion and social competence, while nonsupportive climates can exacerbate children's emotion and social difficulties (Hughes, Cavell, Meehan, Zhang, & Collie, Reference Hughes, Cavell, Meehan, Zhang and Collie2005; Mashburn et al., Reference Mashburn, Pianta, Hamre, Downer, Barbarin and Bryant2008). Approximately 11% of classrooms were simultaneously coded by two observers to assess reliability (87.50% interrater reliability within one point, consistent with Classroom Assessment Scoring System training guidelines). Authors initially denoted 5 as the cutoff score to differentiate low versus high emotional support. However, after conducting threshold analyses, Burchinal, Vandergrift, Pianta, and Mashburn (Reference Burchinal, Vandergrift, Pianta and Mashburn2010) reported that child outcomes continued to increase as the quality of emotional support increased, even above the score of 5. Therefore, the range of emotion support scores will be used rather than a dichotomous high versus low. In this study, 5 of the 21 classrooms earned a score below 5 (M = 5.64, SD = 0.65, range = 3.69–6.75), and there was no statistical difference in emotion support ratings between EBP and ICPS classrooms.
Data analytic strategy
When doing research within the structure of a school system, data are inherently clustered. Children are nested within classrooms and classrooms are nested within schools. Because nesting can lead to biased standard errors, often in the direction of reduced standard errors, inflated estimates of treatment effects are likely. To account for interdependence in the data due to clustering, we performed the primary data analyses using hierarchical linear modeling (HLM 6.08; Raudenbush, Bryk, & Congdon, Reference Raudenbush, Bryk and Congdon2004). HLM models were estimated for each of the main outcome variables. Because one Head Start center houses only one classroom and there were not enough centers (N = 7) to expect stable estimates of variance at the center level, we performed two-level analyses with children represented at Level 1 (within-class variance) and classrooms at Level 2 (between-class variance).
Along with preintervention score of the outcome variable, moderators that significantly related to the outcome variable were included in the final model and were entered as predictors at level 1. Models for the EMT also included research assistants' ratings of children on the inattention scale of the General Assessment of Test Session Behavior, which assessed children's attention while completing those tasks. A dummy-coded treatment condition variable (0 = as-usual, 1 = EBP) was entered at level 2. Treatment, sex of child (0 = female), and ethnicity (0 = Hispanic) were uncentered, and all other predictor variables were grand-mean centered. Following analyses of main effects, we considered the variables of age, sex, verbal ability, ethnicity, level of contextual risk, parental depression, and classroom emotional support as potential moderators of treatment effects.
Results
Initial levels of emotion competence and behavior
Table 1 reports means and standard deviations for participants' emotion knowledge, emotion regulation, and levels of behavior. No significant treatment differences on these variables were observed at preintervention. Table 2 reports descriptive statistics of moderator variables. Child ethnicity was significantly different between treatment conditions, with a greater percentage of Hispanic children in EBP classrooms and a greater percentage of African American children in the as-usual classrooms. Table 3 presents intercorrelations among outcome measures at pretest.
Means (standard deviations) of study variables separated by treatment condition
Note: Group means are presented with standard deviations in parentheses. EBP, Emotion-Based Prevention Program; Emotion knowledge, EMT total score; Emotion regulation, ERC total score; Problem solving, PIPS total score; Internalizing behavior, C-TRF internalizing scale score; Externalizing behavior, C-TRF externalizing scale score; Disruptive behavior, ASBI disruptive scale score; Negative expressions, EERS negative scale score; Obs. negative interactions, BEEOS negative interactions percentage; Obs. solitary behavior, BEEOS solitary behaviors percentage; Prosocial behavior, ASBI prosocial scale score; Positive expressions, EERS positive scale score; Obs. positive interactions, BEEOS positive interactions percentage.
Descriptive statistics of moderator variables
Note: EBP, Emotion-Based Prevention Program; Classroom emo. sup., CLASS emotional support scale score.
*p < .05. **p < .01.
Intercorrelations among outcome measures at pretest, verbal ability, and age
Note: Obs, Observed using the BEEOS observation system.
*p < .05. **p < .01.
Analyses of program sustainability and effectiveness
-
Hypothesis: Teachers would demonstrate acceptable EBP program fidelity.
Teacher fidelity to the EBP program ranged from acceptable to very good, with an overall average rating of 4.72 on the 6-point system (SD = 0.57). Average ratings from across the year for individual teachers ranged from 3.83 to 5.44, with ratings for 11 of the 13 teachers above a 4.0 (good). As shown in Figure 1, average fidelity ratings across teachers began and remained moderately strong, with no significant changes in implementation fidelity across the year.
Average Emotion-Based Prevention Program fidelity ratings across program implementation.
To examine the impact of teacher fidelity on EBP outcomes, HLM models were run with EBP classrooms only. Teacher fidelity significantly predicted children's skills in emotion knowledge, λ = 1.30; t (11) = 2.57, p = .026. Teachers who were more faithful to the EBP program had children who made greater gains in emotion knowledge than did children from classrooms where teachers were less faithful to the EBP program.
-
Hypothesis: EBP would lead to greater increases in emotion knowledge and regulation skills.
Table 4 presents final HLM models. Consistent with hypotheses, HLM analyses showed that compared to children in as-usual classrooms, children in EBP classrooms gained significantly more emotion knowledge over the course of the intervention period, t (24) = 3.36, p = .003.
Hierarchical linear modeling analyses for treatment effects on children's emotion knowledge and regulation skills
Note: Statistically significant EBP main effects and moderation effects are in boldface. EBP treatment, Sex, and Ethnicity were uncentered variables, with Head Start as usual, female, and Hispanic as the 0 reference groups, respectively. All remaining variables were grand-mean centered.
*p < .05. **p < .01.
No main intervention effect was present for teacher ratings of children's emotion regulation. Moderation analyses indicated that parental depressive symptoms significantly moderated a treatment effect, t (24) = 2.08, p = .048. For children of parents reporting greater depressive symptoms, EBP led to greater increases in teacher-rated emotion regulation abilities compared to those in the as-usual condition. There were no treatment group differences for children whose parents reported fewer depressive symptoms.
-
Hypothesis: EBP would lead to greater decreases in maladaptive behaviors and greater increases in adaptive behaviors.
Table 5 presents final HLM models. Supporting our hypotheses, teacher ratings showed that children in EBP classrooms made greater improvements in internalizing behaviors, t (24) = –2.82, p = .010, as compared to children in as-usual classrooms. In addition, treatment differences in teacher ratings of children's negative emotion expressions approached significance, t (24) = –1.99, p = .058, indicating that EBP led to children displaying fewer negative emotion expressions.
Hierarchical linear modeling analyses for treatment effects on children's maladaptive and adaptive behaviors
Note: Statistically significant EBP main effects and moderation effects are in boldface. EBP treatment, Sex, Ethnicity, and Emotional support were uncentered variables, with Head Start as usual, female, Hispanic, and low emotional support as the 0 reference groups, respectively.
†p < .10. *p < .05. **p < .01.
Moderation analyses provided additional support for the effectiveness of EBP. Children's levels of contextual risk significantly moderated a treatment effect on teacher-rated externalizing behaviors, t (180) = –2.34, p = .020. For children with higher levels of contextual risk, those in the EBP condition demonstrated a greater decrease in teacher-rated externalizing behaviors across the implementation period than those in the as-usual condition. Changes in teacher-rated externalizing behaviors for children with lower levels of contextual risk were comparable across interventions. Parental depressive symptoms significantly moderated a treatment effect on negative interactions observed in the classroom, t (133) = 2.12, p = .036. For children of parents with fewer depressive symptoms, EBP led to fewer observed negative interactions during playtime in the classroom than did ICPS, t (24) = –2.80, p = .010. There were no treatment group differences for children of parents with greater depressive symptoms.
Classroom emotional support significantly moderated the treatment effects on teacher-rated prosocial behaviors, t (22) = –2.61, p = .020, as well as children's observed solitary play behaviors, t (22) = 2.52, p = .020, and positive interactions in the classroom, t (22) = –2.44, p = .023. For classrooms with relatively low ratings of emotional support, EBP led to significantly fewer solitary behaviors and more positive interactions during playtime, along with greater gains in teacher-rated prosocial behaviors across the year, as compared with Head Start as usual. There were no treatment group differences for classrooms with relatively high ratings of emotional support in the classroom.
Discussion
Numerous studies point to the importance of emotional and social competence in young children's adaptive functioning, and a number of prevention programs have been successfully implemented in the preschool population. However, much of the published research at this point has relied on efficacy trials. Because efficacy should not be the end to intervention development, moving forward, research is beginning to examine the degree to which established prevention programs can be realistically sustained outside of the scope of program development and evaluation. Along with assessing teacher fidelity to program implementation, it is also necessary to evaluate the impact on child outcomes to determine program effectiveness when delivered in real-world contexts. In addition, the disproportionate number of poor outcomes in low-income, at-risk populations necessitates that research identify which programs lead to improved outcomes for these children.
The current study was an effectiveness trial of the EBP intervention in a low-income and culturally diverse population. In line with a focus on sustainability, data were collected 2 years following an initial efficacy trial. EBP was implemented completely by Head Start teachers in real-world conditions. As-usual Head Start classrooms comprised the comparison condition, in which teachers were instructed to continue delivering ICPS, although ICPS implementation was not observed nor rated for fidelity. Multiple methods of data collection, including teacher report, direct child assessment, and independent observation, were utilized to gather information regarding children's behaviors in multiple contexts and to help account for various biases. In addition, multilevel modeling was used to account for the dependency of data due to children being nested within classrooms.
During this second consecutive year of limited programmatic supervision and support, EBP fidelity ratings were consistent across the year. All teachers implemented the EBP program at “acceptable” levels, with 85% of teachers achieving fidelity at “good” or higher levels. While all EBP teachers demonstrated acceptable fidelity, greater fidelity to the EBP program predicted greater gains in emotion knowledge and behavioral regulation skills. Preschool teachers interviewed as part of the PATHS and Second Step sustainability studies indicated that they viewed addressing the social–emotional domain with their students as important to furthering children's development and improving classroom climate (DeRousie & Bierman, Reference DeRousie and Bierman2012; Wenz-Gross & Upshur, Reference Wenz-Gross and Upshur2012). It is likely that the belief that a program is leading to meaningful effects will increase teacher's motivation to continue following the protocol. Of note, fidelity to program implementation was not measured in ICPS classrooms. Therefore, results regarding ICPS treatment effects on outcomes need to be interpreted cautiously.
Consistent with the prior efficacy trial, EBP led to significantly greater gains in children's emotion knowledge in the current effectiveness study. The EBP intervention is in part composed of activities and techniques designed to increase emotion knowledge. In general, there is an increased teacher emphasis throughout the day on using language to help children connect words and feelings as well as engage in discourse regarding emotion feelings. Specific elements of the emotions course taught in the classroom include targeted time for children to practice discussing events and situations involving specific emotions. Games and illustrations that focus on labeling and describing emotion expressions and how they relate to feelings and actions are also incorporated. Improving children's emotion knowledge assists in the development of emotion regulation and increases the likelihood of greater success in social and behavioral domains.
Treatment effects on behavioral outcomes were also found, again consistent with the previous efficacy trial. Children in EBP classrooms showed a greater decrease in internalizing behaviors than did children in Head Start as-usual classrooms. Internalizing behavior problems in young children are often overlooked, because they are harder to identify and do not tend to lead to classroom disruption the way that externalizing behaviors do. However, internalizing behavior problems are similarly associated with negative outcomes, including poor social competence and lower literacy, language, and mathematics ability (Bulotsky-Shearer, Fantuzzo, & McDermott, Reference Bulotsky-Shearer, Fantuzzo and McDermott2008; Dobbs, Doctoroff, Fisher, & Arnold, Reference Dobbs, Doctoroff, Fisher and Arnold2006). It is likely that children who are more reticent and withdrawn participate in fewer social interactions and demonstrate less engagement in classroom activities that promote readiness skills (Bulotsky-Shearer, Dominguez, & Bell, Reference Bulotsky-Shearer, Dominguez and Bell2012; Hughes & Coplan, Reference Hughes and Coplan2010). One of the strengths of the EBP program is the attention toward identifying, addressing, and impacting internalizing behaviors.
EBP seemed to be especially advantageous for children more at risk for school problems. While EBP and Head Start as-usual led to similar changes in behaviors of children who were functioning with relatively few stressors, EBP led to greater improvements in outcome areas for children exposed to higher levels of stress. For children with a greater number of contextual risk factors, those who received the EBP intervention showed larger decreases in externalizing behaviors. For children whose parents reported more depressive symptoms, EBP led to larger increases in emotion regulation skills. EBP was specifically developed and adapted to address the needs of children living in disadvantaged populations. Together, emotion theory, teacher feedback, and observations of early implementations led to incorporating child-friendly and hands-on materials, culturally diverse illustrations, and a focus on culturally meaningful contextual situations.
Similarly, although there were no treatment group differences for classrooms rated higher in emotional support, EBP led to greater growth in observed and reported prosocial behaviors in classrooms rated lower in emotional support. In examining effects of a classroom-based intervention, it is important to consider classroom climate, because such contextual characteristics can influence program effectiveness (Stoiber, Reference Stoiber2011). In general, the quality of classroom climate affects both children's social and academic behaviors (Hirsh-Pasek et al., Reference Hirsh-Pasek, Golinkoff, Berk and Singer2009; Pianta, Reference Pianta1999). Teachers demonstrating higher levels of sensitive and emotionally supportive behaviors are believed to foster preschool children's social interactions, motivation to learn, and engagement in classroom activities (Chien et al., Reference Chien, Howes, Burchinal, Pianta, Ritchie and Bryant2010, Pianta, Reference Pianta1999). In this study, EBP seemed to provide a buffer for classrooms with lower levels of emotional support, promoting social behaviors that came more naturally in classrooms with higher levels of emotional support. It is possible that EBP provides teachers with the theory and a format in which they can increase their supportive behaviors. It would be worthwhile to examine classroom emotional support over the course of the treatment implementation to evaluate the possibility of treatment-induced effects on emotional support ratings.
In addition to the noted strengths of this study, there are also limitations and room for future improvements. By comparing the effects of multiple evidence-based programs rather than the effects of a targeted intervention against a control condition, we would be able to flush out programmatic strengths and weaknesses. While a second established universal prevention program, ICPS, was expected to be implemented, no method to examine ICPS program fidelity was included. Treatment effects are not comparable without considering the degree to which both programs are being implemented as intended. On another note, by observing EBP program lessons and not ICPS implementation, it is possible that a greater importance was conveyed to teachers about EBP, shaping teachers' behavior and beliefs regarding their instruction. However, raters did go into all classrooms on multiple occasions while observing classroom climate and individual children's social behaviors. In addition, contact between EBP and non-EBP teachers was not routine, and because randomization had occurred at the center level, teachers did not observe research staff's contact with the opposite condition. Comparable fidelity checks would, ideally, be conducted on both interventions across the year. Further, to account for teachers' previous trainings and experiences, it may be helpful to measure the extent to which the key elements of each intervention are incorporated by all teachers, regardless of treatment condition.
Because teachers have the unique position of observing children's daily behaviors as well as having the context of a peer reference group, they were asked to provide ratings of children's behaviors in their classrooms. However, it is also possible that biases factor into such ratings. Therefore, the above results regarding intervention effects on teacher-rated internalizing behaviors, externalizing behaviors, and emotion regulation should be interpreted with caution. In the future, it would be advantageous to include an additional method of data collection to support teacher ratings.
Along with examining program efficacy, effectiveness, and sustainability within the classroom setting, it is also important to evaluate whether intervention programs lead to long-term benefits for children. Prior trials have examined EBP's efficacy (Izard et al., Reference Izard, King, Trentacosta, Morgan, Laurenceau and Ewing2008), and this study demonstrates program effectiveness and sustainability. Collecting follow-up data from children who were administered EBP while in preschool and have now entered elementary school would allow researchers to assess the extent to which the skills taught were maintained and generalized to daily life.
Considering the ultimate goal to have an intervention that is able to be disseminated and run successfully in the targeted settings with the population for whom it was developed, there are a few additional steps that can and should be addressed with EBP. Developing a training package designed as part of the intervention materials would allow broader dissemination. Similarly, developing a system of monitoring and fidelity checks that can be implemented by in-house staff would promote longevity. Although this study demonstrated that EBP can be successfully sustained following a year of limited support from outside research staff, it is also necessary to determine the degree to which the program is effective and sustainable when beginning with more realistic in-house training and monitoring. It may also be beneficial to collect outcome data at additional time points during the program implementation. By including time points between pre- and posttest, we stand to gain better insight into the causal mechanisms and temporal relations of intervention gains. Such information could help strengthen and focus interventions.
Overall, this study demonstrates that EBP can be feasibly implemented in an urban Head Start system without large-scale support from a research team and still result in significant effects on child outcomes. These results add to previous findings that EBP leads to improved emotion knowledge and regulation skills, along with decreased problematic behaviors in low-income preschool children. Demonstrating program feasibility and effectiveness in real-world settings should aid in the dissemination of quality intervention programs.
