We need mental health strategies not only for children and parents, but for preserving the well-being of staff as well. (Zigler, Reference Zigler, Yoshikawa and Knitzer1997, p. 5)
Mental health supports have been considered a core component of effective early childhood programming, dating back to at least the advent of Project Head Start in 1965 (Zigler & Valentine, Reference Zigler and Valentine1979). However, the effective implementation of mental health supports infused into early childhood programs is often challenging and sometimes results in suboptimal uptake of services (Zigler, Piotrkowski, & Collins, Reference Zigler, Piotrkowski and Collins1994). Early childhood mental health consultation (ECMHC) has been promoted by both the US Department of Health and Human Services and Department of Education (2014) as one of the most promising systemic strategies for promoting mental health and reducing exclusionary discipline (expulsions and suspensions) in early childhood settings. ECMHC is an indirect service where trained mental health consultants work with adults in the child's proximal environment (e.g., classroom, home) to address behavioral challenges (Hepburn, Perry, Shivers, & Gilliam, Reference Hepburn, Perry, Shivers and Gilliam2013). Capitalizing on the growing momentum of ECMHC implementation across states (Caputo, Reference Caputo2016; Grossman-Kahn, Reyes, Maupin, & Gilliam, Reference Grossman-Kahn, Reyes, Maupin and Gilliam2018), the current study evaluates the effectiveness of Ohio's ECMHC system in a statewide randomized controlled trial (RCT). This is the second RCT of a statewide ECMHC program, the first being Connecticut's (Gilliam, Maupin, & Reyes, Reference Gilliam, Maupin and Reyes2016).
ECMHC as a promising practice
A foundational premise of mental health consultation is that a child's behavior is highly influenced through their interactions with the adults with whom they have authentic relationships (Caplan, Reference Caplan1970), including early childhood settings (Zigler & Gilman, Reference Zigler and Gilman1998). The focus of consultation should therefore be on strengthening practices that improve teaching and promote early relational health (Condon, Reference Condon2017). Below, we identify the various relational pathways by which ECMHC, theoretically, changes adult behaviors, reduces challenging behaviors, and ultimately eliminates expulsions.
Pathway 1: Strengthening adult–child relationships. Children's development is facilitated by their strong relationships with the caring and supportive adults who meet their needs for safety and emotional connection (Shonkoff & Phillips, Reference Shonkoff and Phillips2000), including their relationships with teachers. Close teacher–child relationships are associated with decreases in externalizing behaviors in young children, especially among those with the highest levels of challenging behaviors at the beginning of the year (Silver, Measelle, Armstrong, & Essex, Reference Silver, Measelle, Armstrong and Essex2005). By contrast, high levels of conflict in teacher–child relationships are associated with rapid growth in externalizing behaviors (Hamre, Pianta, Downer, & Mashburn, Reference Hamre, Pianta, Downer and Mashburn2008; Silver et al., Reference Silver, Measelle, Armstrong and Essex2005; Zhang & Sun, Reference Zhang and Sun2011). Effective ECMHC also focuses on supporting the relationship between children and their parents, through supporting better home–school collaboration (Zigler, Finn-Stevenson, & Stern, Reference Zigler, Finn-Stevenson and Stern1997).
Pathway 2: Improving teacher well-being by reducing job stress. Teachers’ affect and attitudes have a powerful influence on preschoolers (Chang & Davis, Reference Chang, Davis, Schutz and Zembylas2009; Cohen, Janicki-Deverts, Turner, & Doyle, Reference Cohen, Janicki-Deverts, Turner and Doyle2015). Teacher depression and job stress, in particular, are associated with lower quality care and education that lead to challenging behaviors (Jeon, Buettner, & Snyder, Reference Jeon, Buettner and Snyder2014; Zaslow, Tout, Halle, Whittaker, & Lavelle, Reference Zaslow, Tout, Halle, Whittaker, Lavelle, Neuman and Kamil2010). This is especially concerning because female preschool teachers are at higher risk for poor physical and mental health, particularly depression, than other women with similar sociodemographic characteristics (Whitaker, Becker, Herman, & Gooze, Reference Whitaker, Becker, Herman and Gooze2013). Teachers who reported greater depression and job stress tend also to report greater levels of problem behaviors in children (Friedman-Krauss, Raver, Neuspiel, & Kinsel, Reference Friedman-Krauss, Raver, Neuspiel and Kinsel2014; Hamre et al., Reference Hamre, Pianta, Downer and Mashburn2008). Preschool teachers with higher levels of self-reported depression and job stress are more likely to expel children (Gilliam & Shahar, Reference Gilliam and Shahar2006). Providing teachers with social emotional supports, such as ECMHC, lowers the likelihood of stress and teachers requesting expulsions (Zinsser, Zulaf, Das, & Silver, Reference Zinsser, Zulaf, Das and Silver2019).
Pathway 3: Increasing teacher self-efficacy. Given job demands, preschool teachers often report feeling overwhelmed and lacking a sense of self-efficacy in effectively addressing challenging behaviors (Zinsser, Christensen, & Torres, Reference Zinsser, Christensen and Torres2016). As preschool teachers are already overworked with little time for additional activities, they need relevant and actionable strategies that they can apply immediately in their classrooms (Helterbran & Fennimore, Reference Helterbran and Fennimore2004). Supporting teacher efficacy in behavior management by providing such strategies is a potential path to decreasing behavior challenges and lowering the likelihood of expulsion by reducing job stress and burnout (Schwarzer & Hallum, Reference Schwarzer and Hallum2008).
Pathway 4: Changing teacher beliefs and their appraisal of challenging behavior. Another potential path to reduce expulsion is locus of control or shifting responsibility for a child's behaviors from something external to the teacher (e.g., something in the child or the child's family) to something internal to the teacher (e.g., what teachers themselves can do in the classroom). Many of the factors that predict the likelihood of being expelled from preschool have little to do with child behaviors and more to do with how they appraise behavior or structure the environment (Gilliam et al., Reference Gilliam, Maupin and Reyes2016). One study found that preschool teachers’ fear of accountability, not the perceived disruptiveness of the child's behavior, predicted the likelihood of expelling the child (Gilliam & Reyes, Reference Gilliam and Reyes2018). Since teachers who are exhausted attribute their stress to specific children (Zinsser et al., Reference Zinsser, Christensen and Torres2016) or shift the blame to parents (Martin, Bosk, & Bailey, Reference Martin, Bosk and Bailey2018), consultation efforts that reframe the ways teachers appraise the source of challenging behaviors likely reduce expulsion (Vuyk, Sprague-Jones, & Reed, Reference Vuyk, Sprague-Jones and Reed2016).
Pathway 5: Increasing application of developmentally appropriate, child-centered pedagogy. A developmentally appropriate, child-centered classroom is one where the teacher considers children's development and knows what children need to learn to be ready for school (Kostelnik, Soderman, & Whiren, Reference Kostelnik, Soderman and Whiren2011). These include, for example, teachers’ ability to support children's vocabulary development, self-expression, and problem-solving, as well as teachers’ ability to attend and focus, facilitate smooth transitions, help children manage strong feelings appropriately, and foster excitement and curiosity in the classroom. This requires a teacher who understands the contributions children make to the curriculum with their ideas and questions, and who structures the curriculum so that individuality, creativity, and self-esteem flourish. Developmentally appropriate practice results in effective teaching and classroom management skills (Barbetta, Norona, & Bicard, Reference Barbetta, Norona and Bicard2005; Copple & Bredekamp, Reference Copple and Bredekamp2009; Walker, Reference Walker2009). When teachers apply developmentally appropriate and child-centered practice, in contrast to adult-centered teaching, children's immediate needs are met (Frey & Wilhite, Reference Frey and Wilhite2005), leading to fewer behavior challenges and increased learning (Burchinal et al., Reference Burchinal, Howes, Pianta, Bryant, Early, Clifford and Barbarin2008; Burts et al., Reference Burts, Hart, Charlesworth, Fleege, Mosley and Thomasson1992). It is unclear to what extent ECMHC promotes this pedagogical approach, but it certainly poses an opportunity and warrants exploration.
Pathway 6: Directly addressing challenging behaviors using evidence-informed strategies. Findings compiled from a meta-analysis of adult training programs (Kaminski, Valle, Filene, & Boyle, Reference Kaminski, Valle, Filene and Boyle2008) and a systematic review of experimental evaluations of early childhood programs (Carney, Stratford, Moore, Rojas, & Daneri, Reference Carney, Stratford, Moore, Rojas and Daneri2015) revealed the common ingredients of interventions shown to reduce both externalizing and internalizing behaviors in 0–8-year-old children. Strategies that have been shown to reduce externalizing behaviors include: (a) training adults on how to interact positively with children in everyday scenarios, including applying positive reinforcement and responding consistently to behavior; (b) real-time modeling by the interventionist; and (c) having trained facilitators or therapists deliver a standardized or manualized curriculum in training the adults. Conversely, strategies that have been shown to reduce internalizing behaviors include having teachers deliver the intervention, which specifically targets internalizing behaviors, directly to children. Ineffective strategies at reducing challenging behaviors included teaching adults to promote social skills or prosocial behaviors (e.g., sharing, remembering manners, making friends; Kaminski et al., Reference Kaminski, Valle, Filene and Boyle2008) and holding separate sessions for the child and the adult (Lundahl, Risser, & Lovejoy, Reference Lundahl, Risser and Lovejoy2006). ECMHC efforts should therefore address challenging behaviors deliberately, with interventionists providing modeling and coaching while the class is in session.
ECMHC as an emerging evidence-based practice
Emerging evidence suggests that both appraisals of children's challenging behaviors and expulsions may have more to do with the adults in a classroom than the children (Gilliam, Maupin, Reyes, Accavitti, & Shic, Reference Gilliam, Maupin, Reyes, Accavitti and Shic2016; Gilliam & Reyes, Reference Gilliam and Reyes2018). Systematic reviews suggest that ECMHC is associated with increased teacher self-efficacy in addressing challenging behaviors (Brennan, Bradley, Allen, & Perry, Reference Brennan, Bradley, Allen and Perry2008), reduced externalizing and internalizing behaviors and increased prosocial skills (Perry, Allen, Brennan, & Bradley, Reference Perry, Allen, Brennan and Bradley2010). Moreover, one study showed that about 80% of infants and toddlers at high risk for expulsion whose teachers received ECMHC were not expelled (Perry, Dunne, McFadden, & Campbell, Reference Perry, Dunne, McFadden and Campbell2008).
Despite growing evidence pointing to the effectiveness of ECMHC, only one statewide ECMHC system, Connecticut's Early Childhood Consultation Partnership (ECCP), has been evaluated in a rigorous RCT (Gilliam, Maupin, & Reyes, Reference Gilliam, Maupin and Reyes2016). Findings from the Connecticut RCT showed that although ECMHC was effective in decreasing externalizing behaviors in the preschoolers who prompted the ECMHC request, it had no discernible impact on random peers, children's risk of expulsion, or overall classroom mental health climate.
Ohio's system of ECMHC
Beginning in fiscal year 2016, the Ohio Governor's budget allocated funding for a statewide system of ECMHC. Known as Whole Child Matters, a name harkening to the whole child movement (Zigler & Bishop-Josef, Reference Zigler, Bishop-Josef, Singer, Golinkoff and Pasek2006), and spearheaded by the Ohio Department of Mental Health and Addiction Services, the overarching goal is to reduce expulsions in early care and education programs by nurturing young children's socialemotional development (Potoma, Schiavone, Phelps, & Kearney, Reference Potoma, Schiavone, Phelps and Kearneyn.d.). The Ohio ECMHC model requires a minimum of six site visits by mental health consultants to early childhood classrooms. This low dose is in stark contrast to the previously evaluated Connecticut ECMHC model, which is significantly more intensive and spans 14–16 weeks at approximately four hours per week (Bicio, Moy, & Starkey, Reference Bicio, Moy and Starkey2007; Caputo, Reference Caputo2016). As a means for creating and maintaining greater cohesion within the Ohio ECMHC system, all Ohio ECMH consultants utilized the Climate of Healthy Interactions for Learning and Development (CHILD®; Gilliam & Reyes, Reference Gilliam and Reyes2017) in all consultation relationships.
The current study
The current study examines the effectiveness of Ohio's system of ECMHC in a statewide RCT. The study is only the second methodologically rigorous RCT of a statewide ECMHC system. The current Ohio RCT extends on the prior Connecticut RCT by evaluating impacts (a) on teacher mental health, beliefs, and practices; (b) on random classmates; and (c) within the context of a CHILD-informed version of ECMHC in a large and diverse state.
Method
During late Summer and early Fall 2018, participating early learning programs were enrolled in the study through a combination of normal referral procedures and recruitment through state agencies. Eligibility criteria for early childhood programs were (a) center-based early learning programs serving 3–5 year-olds, (b) no prior receipt of ECMHC services, (c) agreement to random assignment to treatment or waitlist-control, and (d) agreement to participate in all data collection waves. Each program (i.e., director + teacher) was compensated $200 for participating in the study. Within each classroom, participants included one lead teacher, two “target children” (identified by the teacher as having most prompted the request for ECMHC), and two “random peers” (randomly selected by the researchers from the remaining class roster).
Participants
A total of 52 early learning and development classrooms from 24 Ohio counties met eligibility criteria and were enrolled and randomly assigned to either treatment or waitlist-control condition. One control program was lost to attrition, leaving 51 study programs in treatment (n = 30) and control (n = 21) conditions. Across the 51 classrooms, six teachers left and were replaced in the study, resulting in a total of 57 teachers across treatment (n = 32) and control (n = 25) conditions. Across the 204 children recruited into the study (2 target children + 2 random peers per program), 14 (6.9%) were lost to attrition, resulting in 190 children across treatment (n = 115) and control (n = 75) conditions. Baseline demographic analyses between conditions indicated no significant differences at community, program, and teacher levels. Treatment children, however, were more likely to be children of color (Table 1).
Sample demographics.
CDA = Child Development Associate; ECE= early childhood education; ECMHC = early childhood mental health consultation; GED = General Educational Development
† p < .10; *p < .05; **p < .01.
a percentage of children in the program receiving federal, state, or other government subsidy.
b 2013 rural–urban continuum code.
c Teacher Ns differ from center Ns because of replacements for teacher turnover.
Intervention
ECMHC was provided to the treatment group between October 2018 (pretest) and February 2019 (posttest). ECMH consultants averaged nearly five years of experience (M = 4.72; SD = 4.34). Programs received a median of six visits (M = 6.63; SD = 3.15). Median length of a consultation visit was 2.0 hr (M = 1.85; SD = 0.67). On a 1–4 scale with higher scores reflecting greater satisfaction, teachers were generally very satisfied and had high regard for the services they received (M = 3.51; SD = 0.49).
Measures
Classroom mental health climate
The CHILD Tool was used to assess classroom mental health climate via observers blinded to treatment/control condition and certified to reliability on the tool. The CHILD is a comprehensive observational assessment of the mental health climate of early care and education settings (Gilliam & Reyes, Reference Gilliam and Reyes2017). The CHILD partitions mental health climate into nine dimensions, further divided into 28 easily observable items. In addition to the nine dimensions, auxiliary dimensions of “warmth” and “attunement” were examined in this study. “Warmth” is the extent to which teachers convey investment in children's personal lives through facial expression, body language, and vocal tone, and by children seeking teachers as sources of comfort. “Attunement” is the extent to which teachers are deeply aware of the needs of individual children and make deliberate attempts to provide individualized attention and support, especially to those presenting greatest need. In the present study, Cronbach's α = .98 for the total score and .87 to .95 for individual dimensions.
Edward Zigler described “attunement” between children and parents by focusing on the level of adult–child synchronicity needed in a New York Times interview (Rierden, Reference Rierden1989). “There's a phenomenon called attunement. Here you have this baby that doesn't know how to read cues and this mother who doesn't understand this crying little blob. Before long, they learn to attune to each other. It's like a dance of the swans. They begin to synchronize with one another. That's a very important phase of growth and development.”
Teacher measures
Teachers rated the frequency of various symptoms of depression with the Center for Epidemiological Survey Depression Scale – 10 (CESD-10; Andersen, Malmgren, Carter, & Patrick, Reference Andersen, Malmgren, Carter and Patrick1994), a shortened 10-item version of the 20-item well-known CESD Scale. Teachers also rated the extent to which they felt able to bounce back easily from setbacks with the six-item Brief Resilience Scale (BRS; Cronbach's α for this study = .84; Smith et al., Reference Smith, Dalen, Wiggins, Tooley, Christopher and Bernard2008). Teachers rated their confidence in managing classroom behavior with the eight-item behavior management subscale of the Ohio State Teacher Efficacy Scale (OSTES; Tschannen-Moran & Hoy, Reference Tschannen-Moran and Hoy2001). The practices subscale of the PreK Beliefs & Practices Scale (PKBS; Marcon, Reference Marcon1999) was used to assess the extent to which teaching approaches are child-centered approach to teaching. Finally, teachers rated their perceived locus of control (at posttest only) when addressing challenging behaviors using the Framing Challenging Behaviors Questionnaire (FCBQ; developed by the first author for the purpose of this study). The FCBQ is a three-item measure scored on a 1–10 semantic differential scale for appraising beliefs about how challenging behaviors can be addressed: external locus of control on the lower end and internal locus of control on the higher end. Cronbach's αs for this study = .76, .95 and .79 for the PKBS, OSTES, and FCBQ, respectively.
Child measures
The Conners Early Childhood Behavior-Teacher/Childcare Provider Short Form (Conners, Reference Conners2009) is a well-known and validated 48-item measure that assesses teacher ratings of behavioral, social, and emotional problems in 2- to 6-year-olds. The Preschool Expulsion Risk Measure (PERM; Gilliam & Reyes, Reference Gilliam and Reyes2018) is a 12-item measure comprising four distinct but related factors associated with risk of expulsion. Cronbach's αs for this study range from .82 to .97. The Devereux Early Childhood Assessment for Preschoolers, Second Edition (DECA-P2; LeBuffe & Naglieri, Reference LeBuffe and Naglieri2012) is a well-known and validated social and emotional rating scale used to assess child protective factors and screen for behavioral concerns during early childhood. The 38-item assessment is formed by subscales for behavior concerns and protective factors, including initiative, self-regulation, and attachment/relationships. The Preschool Learning Behaviors Scale (PLBS; McDermott, Green, Francis, & Stott, Reference McDermott, Green, Francis and Stott2000) assesses learning-related behaviors appropriate for children aged 3–5.5 years old. The PLBS consists of 29 items formed by three subscales (competence motivation, attention/persistence, and attitude toward learning). The PLBS has high internal and test–retest reliability, with most mean coefficients respectively between .75 to .89 and .80 to.94.
Preliminary analyses
Missing data
Missing data (MD) across outcomes of interest ranged from 28% to 37% for children and 10% to 34% for teachers. At the child level, older children were more likely to have baseline MD in all outcome variables, and Black children were more likely to have MD in the PERM measure at posttest. At the teacher level, no pattern of MD was evidenced. MD were estimated using multiple imputation procedures in NORM (Schafer, Reference Schafer1999).
Baseline equivalence
Baseline equivalence was assessed following What Works Clearinghouse (WWC) guidelines (Institute of Education Sciences, n.d.). On three measures, baseline equivalence (Hedges g < 0.25 SD) was not achieved. Treatment teachers scored significantly higher on the PKBS (ES = 0.71, p = .010), and treatment children were rated higher on the Conners aggressive and mood & affect subscales (for both, ES > 0.40, p < .05). To address baseline nonequivalence and improve estimates, we explored covariates associated with primary outcomes of interest and sample characteristics associated with treatment status, and added these to impact models.
Analytic approach
For classroom-/teacher-level analyses, we ran ordinary least squares regressions, using pretest scores and program/neighborhood characteristics as covariates. For child-level analyses, to account for the nested structure of the data (i.e., children nested in classrooms), two-level multilevel models were estimated in HLM 8 (Raudenbush, Bryk, Cheong, & Congdon, Reference Raudenbush, Bryk, Cheong and Congdon2019) using multiply imputed datasets with restricted maximum likelihood estimation and robust standard errors.
Level 1 equation:
Where:
-
Yij = posttest score for child i in program j
-
β0j = conditional mean posttest scores for children in program j
-
β1j = mean pretest slope for children in program j
-
Pretestij = pretest score for child i in program j
-
HomeRiskij = cumulative home risk score for child i in program j
-
β2j = mean home risk slope for children in program j
-
Black = 1 for child is Black, 0 for child is not
-
β3j = mean slope for Black children in program j
-
rij = random effect representing the difference between child ij's posttest score and predicted posttest mean score for program j; rij ~ N(0,σ2)
Level 2 equation:
Where:
-
γ00 = conditional grand mean posttest score for control group
-
γ01 = treatment effect (i.e., conditional mean posttest difference between ECMHC and control group)
-
Treatmentj = 1 for ECMHC, 0 for control
-
Wtj = t additional program/neighborhood-level covariates
-
γ10 = grand mean pretest intercept
-
γm 0 = grand mean intercept for additional covariates
-
u 0j to u 1j = residuals; ~ N(0,τ)
Results
Classroom-/teacher-level impacts
At posttest two variables showed a trend (ps < .10; Table 2). ECMHC teachers had greater internal locus of control than did control teachers (FCBQ: ES = 0.55, p = .062). ECMHC teachers, however, reported less progressive practices than control teachers (PKBS: ES = −0.28, p = .085). Although not statistically significant but with ESs > |0.25|, small gains favoring ECMHC teachers over control teachers were detected in the OSTES (t = 1.66, ES = 0.38) and three dimensions of the CHILD: Individualized and Developmentally Appropriate Pedagogy (t = 1.36, ES = 0.30, ns), Social and Emotional Learning, and Staff×Child interactions (ts = 1.16–1.23, ESs = 0.27–0.35).
Adjusted pretest and posttest means for teacher outcomes based on regression analyses (N = 57).
† p < .10; *p < .05. Effect sizes ≥ |.25| appear in bold.
Note: CB = Child Behaviors, DR = Directions & Rules, ECMHC = early childhood mental health consultation, IDAP = Individualized & Developmentally Appropriate Pedagogy, Saff = Staff Affect, Saw = Staff Awareness, SCI = Staff×Child interaction, Sco = Staff Cooperation, SEL = Social & Emotional Learning, TR = Transitions.
a Hedges g
b Cohen's d accounting for pretest–posttest rs
Child-level impacts
Full sample
Adjusting for baseline scores and covariates (Table 3), significant effects were found for the DECA-P2 scales favoring children in the ECMHC condition: Initiative (ES = 0.58, p = .000), Self-Regulation (ES = 0.29, p = .031), Attachment Relationships (ES = 0.42, p = .008), and Behavioral Concerns (ES = −0.27, p = .040). As much as a 6-point gain was detected in the Initiative scale (t = 4.40, ES = 0.66, p = .000), and three points in the other DECA-P2 scales in the expected direction. Although not statistically significant, ECMHC teachers reported feeling less stressed by children's challenging behaviors. Although no significant effects were found on other measures associated with behavioral challenges, gain scores revealed promising trends, such as greater declines in the Conners Aggressive (t = −2.09, ES = −0.30, p = .057) and Atypical (t = −2.03, ES = −0.28, p = .050) scales.
All children: Adjusted pretest and posttest mean child outcomes based on HLM analyses (N = 190).
DECA-P2 = Devereux Early Childhood Assessment for Preschoolers, Second Edition; PERM = Preschool Expulsion Risk Measure; PLBS = Preschool Learning Behaviors Scale
† p < .10; *p < .05; **p < .01; ***p < .001. Effect sizes ≥ |.25| appear in bold.
a Hedges g
b Cohen's d accounting for pretest-posttest rs
Target children
For target children (Table 4), significant and moderate effects were found in the DECA-P2 Initiative and Attachment Relationships scales (ESs = 0.76 and 0.51, respectively, ps < .05). Although no other significant effects were found for target children, examination of gains over time showed promising trends such as declining challenging behaviors favoring the ECMHC condition.
Target children: Adjusted pretest and posttest mean child outcomes based on HLM impact analyses (N = 96).
DECA-P2 = Devereux Early Childhood Assessment for Preschoolers, Second Edition; ECMHC = early childhood mental health consultation; PERM = Preschool Expulsion Risk Measure; PLBS = Preschool Learning Behaviors Scale
† p < .10; *p < .05; **p < .01. Effect sizes ≥ |.25| appear in bold.
a Hedges g
b Cohen's d accounting for pretest–posttest rs
Random peers
For random peers (Table 5), those in the ECMHC condition received higher teacher ratings on the DECA-P2 Initiative scale (ES = 0.52, p = .033), and only expected-direction trends (ps < .10) were found for the other DECA-P2 scales (ESs ≥ |0.36|). A trend was also found for the PERM-Stress factor such that ECMHC teachers reported less stress with random peers’ challenging behaviors (ES = −0.64, p = .072).
Random peers: Adjusted pretest and posttest mean child outcomes based on HLM impact analyses (N = 94).
DECA-P2 = Devereux Early Childhood Assessment for Preschoolers, Second Edition; ECMHC = early childhood mental health consultation; PERM = Preschool Expulsion Risk Measure; PLBS = Preschool Learning Behaviors Scale
† p < .10; *p < .05. Effect sizes ≥ |.25| appear in bold.
a Hedges g
b Cohen's d accounting for pretest–posttest rs
Discussion
ECMHC is a burgeoning field, yet despite several studies supporting its effectiveness (Brennan et al., Reference Brennan, Bradley, Allen and Perry2008; Perry et al., Reference Perry, Allen, Brennan and Bradley2010) only one other ECMHC system has been evaluated in a rigorous RCT, limiting the development of an evidence-based set of models. As the second RCT of an ECMHC system, the present study builds significantly on the extant literature and examines the impact of Ohio's statewide ECMHC system, using the CHILD Tool as a consultation framing support. Overall, Ohio's CHILD-informed ECMHC system yielded positive and promising results, with statistically significant findings for both target children and random peers’ social and emotional outcomes. This is the first RCT of an ECMHC system to demonstrate positive effects for random peers.
Classroom climate and teacher outcomes
Consistent with findings from Connecticut's statewide RCT of ECMHC (Gilliam et al., Reference Gilliam, Maupin and Reyes2016), no impacts on teacher well-being (depression and resilience) were found. While mental health promotion is central to the ECMHC model, ECMHC is not direct psychotherapy for the teacher (Hunter, Davis, Perry, & Jones, Reference Hunter, Davis, Perry and Jones2016). Perhaps at a more intensive dosage, there might be an indirect effect whereby ECMHC reduces job stress and depression via an increase in teachers’ self-efficacy and/or decrease in children's challenging behaviors. Stress-reduction techniques and well-being strategies such as mindfulness and exercise have demonstrated promising results in reducing preschool teacher depression and job stress (Iancu, Rusu, Măroiu, Păcurar, & Maricuțoiu, Reference Iancu, Rusu, Măroiu, Păcurar and Maricuțoiu2018). Teachers in the ECMHC group reported trends toward increased internal locus of control and self-efficacy when managing challenging behaviors. Because expulsion is largely a teacher decision (Gilliam & Reyes, Reference Gilliam and Reyes2018), shifting the level of control teachers feel in addressing challenging behaviors is key to reducing expulsion (Heller et al., Reference Heller, Boothe, Keyes, Nagle, Sidell and Rice2011). Decreasing blame on external factors may improve teacher self-efficacy in managing challenging behaviors (Gilliam & Shahar, Reference Gilliam and Shahar2006).
Child outcomes
The most robust impacts of Ohio's ECMHC were on children's social and emotional skills, in particular protective factors such as child initiative, self-regulation, and attachment relationships. Teachers reported seeing both target children (children whose behavior prompted request for ECMHC) and randomly selected peers (children randomly selected for the present study) as more self-initiated. One-half to three-quarters of a standard deviation improvement on this scale for a relatively short intervention is impressive, albeit enhanced with the CHILD Tool. Target children in the ECMHC condition may have felt more empowered to make independent decisions as a result of changes in teachers’ beliefs and practices. For instance, perhaps teachers’ increased internal locus of control and self-efficacy helped them develop a strengths-based narrative for target children, in turn shifting the child's internal narrative from being “the problem child” to being “an independent child.”
Significant impacts also were found for self-regulation and attachment relationships when examining all children, both target children and random peers. When examining target and randomly selected children independently, growth in both protective factors was significant for randomly selected peers. This is the first RCT to show significant impacts on children's positive behavior or on random peers (Perry et al., Reference Perry, Allen, Brennan and Bradley2010). One nonexperimental ECMHC evaluation found impacts on positive behaviors, but only by the third year of implementation and with smaller effect sizes (Conners-Burrow, Whiteside-Mansell, McKelvey, Virmani, & Sockwell, Reference Conners-Burrow, Whiteside-Mansell, McKelvey, Virmani and Sockwell2012).
Limitations
Two programmatic limitations may have reduced impacts. First, although Ohio's ECMHC was fully operational at the time of the evaluation, its use of the CHILD Tool was in its pilot stage, perhaps leading to immature implementation (Fixsen, Naoom, Blasé, Friedman, & Wallace, Reference Fixsen, Naoom, Blasé, Friedman and Wallace2005). Many consultants were newly trained on the CHILD Tool. Only 84% of consultants reported applying the CHILD Consultant's Guide, and only 65% of the ECMHC teachers reported the CHILD being used during consultation. Second, teacher turnover rate (27%) during this study was more than twice national rate (13%; Whitebook, Phillips, & Howes, Reference Whitebook, Phillips and Howes2014). In one classroom, teacher turnover occurred twice from fall to winter. Such inconsistency adversely affects classroom environment, especially the development of attachment relationships (Hale-Jinks, Knopf, & Kemple, Reference Hale-Jinks, Knopf and Kemple2006), and makes effective ECMHC challenging, if not impossible. Moreover, teacher turnover may have significantly undermined dose receipt by teachers who left or joined programs mid-way through the study.
Six methodological limitations warrant discussion. First is the small sample size relative to the Connecticut RCT, which enrolled almost twice as many preschool programs as the current study. Regardless, the intervention was strong enough to detect variability in some child outcomes. Second, the timing of the intervention varied across programs with some programs receiving consultation later in the intervention period than others, perhaps leading to variability in dosage. Third, some variables violated strict adherence to What Works Clearinghouse standards for baseline equivalence (Institute of Education Sciences, n.d.), potentially undermining effects, particularly on challenging behaviors. Fourth, financial and geographical constraints may have limited the ability to conduct more thorough assessments. For instance, some blind raters had to travel at least three hours each way to conduct observations. Fifth, validly measuring expulsion intent may have been complicated in Ohio, due to signing of the Supporting Alternatives for Fair Education (SAFE) Act (H.B. 318) in August 2018, just two months prior to pretest data collection. The SAFE Act significantly limits expulsions and out-of-school suspensions for preschoolers and early elementary students to include only instances where the safety of other students or staff are threatened. Although these SAFE Act provisions did not take effect until the 2020–2021 school year, recent passage of this high-profile education law may have depressed scores on measures of expulsion intent. Sixth, racial/ethnic representation in the sample, although reflecting the state's demographics, includes a relatively small sample of Black children despite their disproportionate risk for preschool expulsion (Meek et al., Reference Meek, Smith, Allen, Catherine, Edyburn, Williams and Pontier2020). This finding is consistent with past reviews of ECMHC, highlighting inequities in ECMCH participation (Albritton, Mathews, & Anhalt, Reference Albritton, Mathews and Anhalt2019).
Implications
With different states implementing various models of ECMHC, it is important to identify the core components of implementation, and assess how they may differentially impact effectiveness (Fixsen et al., Reference Fixsen, Naoom, Blasé, Friedman and Wallace2005). Shedding light on implementation may help practitioners and policy-makers understand what makes ECMHC effective, for whom, and under what circumstances (Zigler, Reference Zigler, Yoshikawa and Knitzer1997). Current findings contribute to this understanding, particularly regarding ECMHC dosage and programming.
Dosage
Despite significant findings in child protective factors, an average of approximately 12 hr of classroom-based consultation provided through six visits over a three-month period did not have an impact on mental health climate, teacher well-being, or children's externalizing/internalizing behaviors and risk of expulsion. A more intensive model of ECMHC is likely needed to make an impact on these areas, particularly risk of expulsion. Such a model should be multilevel, focusing on the child, family, classroom, and program (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011). A more intensive model may also benefit from frontloading, with more initially frequent visits followed by gradual reduction over time (Gilliam et al., Reference Gilliam, Maupin and Reyes2016). Social and emotional learning programs that have shown an impact on classroom climate scores, for instance, are more intensive in that they often require teacher participation at workshops in addition to coaching sessions (Rivers, Brackett, Reyes, Elbertson, & Salovey, Reference Rivers, Brackett, Reyes, Elbertson and Salovey2013). More specifically, the average duration of social and emotional learning programs is 40 sessions (Durlak et al., Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011), and Connecticut's ECMHC dosage is 14–16 weeks for a total of up to 64 hr of consultation (Caputo, Reference Caputo2016; Gilliam et al., Reference Gilliam, Maupin and Reyes2016). By comparison, Ohio's standard dosage is unusually low.
Programming
Findings suggest that changing teacher mindsets around challenging behaviors should be a focal point of ECMHC. Some consultants reported being asked by teachers, “Why are you focusing on me and not the child?” Helping teachers understand that children's challenging behaviors involve an interaction between children and their environment (Sutherland & Oswald, Reference Sutherland and Oswald2005) may increase ECMHC efficacy by better aligning the expectations of teachers and consultants.
Other than being symptoms of unmet child needs (Frey & Wilhite, Reference Frey and Wilhite2005), challenging behaviors are also symptoms of systemic challenges. Identifying and addressing these systemic challenges may strengthen the impact of consultation. One systemic challenge is addressing the issue of teacher job stress, burnout, and turnover. Teacher job stress and burnout not only increase the likelihood of expulsion (Gilliam & Shahar, Reference Gilliam and Shahar2006), they also increase the likelihood of teacher turnover, which may seriously undermine the potential effectiveness of teacher-focused interventions such as ECMHC. Teacher turnover is predicted by the presence of several factors including having a negative relationship with supervisors and colleagues (Wells, Reference Wells2015). This type of systemic challenge requires intensive intervention that may be prerequisite to addressing challenging behaviors in challenging environments.
Finally, our findings underscore racial/ethnic inequities to participation in ECMHC, which is consistent with past reviews (Albritton et al., Reference Albritton, Mathews and Anhalt2019). To directly and deliberately address inequities in access to services, we recommend recruitment efforts that prioritize hard-to-reach areas. Exploring a blended model of consultation (e.g., in vivo and tele-mental health consultation) is another promising way to address issues of access, particularly in rural or otherwise geographically diffused communities. This approach addresses two key challenges: (a) the ability to provide services to more programs and children; and (b) the ability to follow-up more efficiently with teachers and programs. In a period of uncertainty and social distancing amid the COVID-19 pandemic, tele-health has become a necessary supplement to standard in-person practices (Cole, Pickard, & Stredler-Brown, Reference Cole, Pickard and Stredler-Brown2019). We suggest capitalizing on lessons learned from this experience to explore blended models of ECMHC.
Conclusions
Children with challenging behaviors often are placed in challenging environments, which requires a whole-child-whole-classroom intervention (Zigler & Hall, Reference Zigler, Hall, Carey and McDevitt1994). Ohio's ECMHC system, powered by the CHILD Tool in this investigation, shows positive impacts for all children, not just those at highest risk of being expelled. In addition, Ohio's ECMCH has shown promising trends in improving teachers sense of control and efficacy when managing challenging behaviors. Findings from this study extend the evidence for effective ECMHC programming. More rigorous evaluations of ECMHC are recommended, paying careful attention to implementation levers that make ECMHC both cost-effective and accessible for all who need it.
“Our subjects in all these studies are children. They are our partners in this learning enterprise, and we have a special responsibility to use this knowledge—not to fill up journals, but to make the lives of these children better.”
Edward Zigler (quoted in Perkins-Gough, Reference Perkins-Gough2007)
Acknowledgments
This paper is dedicated to Sterling Professor Edward F. Zigler (1930–2019), who charged us all to “stand firmly on firm data.” The authors are grateful for the contributions of the following individuals: Madeline Klotz for coordinating recruitment and data collection; Abigail Hemenway Deaver for editing drafts of this manuscript; Dr. Pilar Alamos for assisting preparing the data for analyses; Dr. Valerie Alloy, Holly Scheibe, Dr. Wendy Grove, and the various agencies providing ECMHC; and the staff, families, and children of the early learning programs that participated in this project. As Professor Zigler has been quoted, “I study children, and they're my subjects in my studies. They're my colleagues, really, all these little children. And I owe them.”
Funding Statement
This research was supported by the Caplan Foundation for Early Childhood, the Ohio Child Care Referral and Resource Association on behalf of the Ohio Department of Mental Health and Addiction Services, and the Heising-Simons Foundation. Funding sources had no involvement in the design, collection, analyses, and interpretation of the data.
Conflicts of Interest
The two authors of this paper have equity ownership in the CHILD Research and Education Consulting, LLC, which is developing products related to the research being reported. The terms of this arrangement have been reviewed and approved by Yale University in accordance with its policy on objectivity in research.
