Earlier this year, two articles appeared in top-tier journals within days of one another, both evaluating mechanisms children's adjustment outcomes following treatment with the Incredible Years (IY) series. One paper appeared in this journal, and reported modest but significant effects of IY parent training (PT) on children's externalizing behaviors in the Observational Randomized Control Trial of Childhood Differential Susceptibility (ORCHIDS) prevention program (Weeland et al., Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018). Of note, despite improvements in both parenting and child behavior, no mediating effects of parenting on externalizing outcomes were found. At first glance, this might seem surprising, but in a recent review, Forehand, Lafko, Parent, and Burt (Reference Forehand, Lafko, Parent and Burt2014) found that tests of parenting as a mediator of child behavior change were significant across less than half of prevention and intervention trials. Thus, the Weeland et al. findings are not anomalous. We write this comment out of concern that without elaboration, such findings could be misconstrued, leading to abandonment of well-established treatments for externalizing behavior (Eyberg, Nelson, & Boggs, Reference Eyberg, Nelson and Boggs2008; Menting, de Castro, & Matthys, Reference Menting, de Castro and Matthys2013).
The second paper, written by our group, reported posttreatment improvements of large effect size on multiple measures of children's externalizing behavior, and mediating effects of reduced negative parenting on children's posttreatment autonomic nervous system activity and reactivity (Bell, Shader, Webster-Stratton, Reid, & Beauchaine, Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018; see Webster-Stratton, Reid, & Beauchaine, Reference Webster-Stratton, Reid and Beauchaine2011). Results from this intervention are consistent with findings from several other randomized clinical trials (RCTs) of IY. Such trials, including reports from independent research groups, identify parenting change as a specific mechanism through which reductions in children's behavior problems and emotional lability are effected using IY and similar interventions (e.g., Beauchaine, Webster-Stratton, & Reid, Reference Beauchaine, Webster-Stratton and Reid2005; Forehand et al., Reference Forehand, Lafko, Parent and Burt2014).
Regardless of whether or not mediation is assessed, studies show that the IY series improves both parenting and child externalizing behaviors for racially and ethnically diverse families who experience a wide range of adversities, including maltreatment, socioeconomic disadvantage, welfare system involvement, and recent maternal incarceration (e.g., Baker-Henningham, Walker, Powell, & Meeks Gardner, Reference Baker-Henningham, Walker, Powell and Meeks Gardner2009; Hurlburt, Nguyen, Reid, Webster-Stratton, & Zhang, Reference Hurlburt, Nguyen, Reid, Webster-Stratton and Zhang2013; Kim, Cain, & Webster-Stratton, Reference Kim, Cain and Webster-Stratton2008; Leijten, Raaijmakers, Orobio de Castro, van den Ban, & Matthys, Reference Leijten, Raaijmakers, Orobio de Castro, van den Ban and Matthys2017; Menting, de Castro, Wjingaards-de Meij, & Matthys, Reference Menting, de Castro, Wijngaards-de Meij and Matthys2014; Reid, Webster-Stratton, & Beauchaine, Reference Reid, Webster-Stratton and Beauchaine2001; Webster-Stratton & Reid, Reference Webster-Stratton and Reid2010). Improvements in children's externalizing behaviors following IY persist for up to 10 years, and extend to clinically and educationally significant endpoints including criminal justice system involvement, antisocial personality, and reading literacy (Scott, Briskman, & O'Connor, Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Rinaldi, & Reid, Reference Webster-Stratton, Rinaldi and Reid2011). According to this literature, PT plays a crucial role in altering well-characterized, costly developmental trajectories to antisociality (Beauchaine, Zisner, & Sauder, Reference Beauchaine, Zisner and Sauder2017). Moreover, the efficacy of IY has been evaluated in several RCTs using data collected from multiple informants, including mothers, fathers, teachers, and blinded coders (e.g., Webster-Stratton, Rinaldi, et al., Reference Webster-Stratton, Rinaldi and Reid2011). In some of these studies (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Bell et al., Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018), effects of parenting change on child outcomes have been confirmed using stringent conceptual criteria for mediation (Kraemer, Wilson, Fairburn, & Agras, Reference Kraemer, Wilson, Fairburn and Agras2002) and/or modern statistical tests that assess direct and indirect (mediational) effects using bootstrap confidence intervals (Montoya & Hayes, Reference Montoya and Hayes2016).
Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) reported no significant mediating effects of parenting change on observed improvements in children's externalizing behavior following intervention with IY, despite increases in positive parenting, decreases in negative parenting, and increases in parental positive affect. The authors offer potential third variable explanations for child behavior change, including decreased family and parent distress, improved quality of life, and increases in parenting self-efficacy, all of which are plausible. In addition, the authors found that children's serotonin transporter linked polymorphisms (5-HTTLPR) moderated effects of IY on changes in negative parenting. Reductions in negative parenting were larger for families whose children carried two copies of the short allele of the 5-HTTLPR gene (s/s homozygotes) than for families of children who were either l/l homozygotic or heterozygotic.
This is an extremely important finding that addresses longstanding questions in the prevention and intervention literatures regarding which treatments work best for which children and families (Brestan & Eyberg, Reference Brestan and Eyberg1998; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). In our own work, we have commented on both (a) the need to better understand how children's individual differences affect their treatment response to IY (e.g., Beauchaine et al., Reference Beauchaine, Gatzke-Kopp, Neuhaus, Chipman, Reid and Webster-Stratton2013), and (b) the bidirectional and transactional nature of family relationship dynamics in high-risk families (e.g., Beauchaine & Zalewski, Reference Beauchaine, Zalewski, Dishion and Snyder2016). The finding that children's genetically mediated individual differences influence changes in negative parenting is a welcome addition to the literature, and Weeland et al. should be applauded for their painstaking research.
Our objective in writing this comment is not to challenge the important 5-HTTLPR finding. Instead, we are concerned about the seeming certainty of (based on null findings) their conclusion that parenting change is not a mechanism through which IY exerts at least some of its effects, and the ongoing need for well-established, efficacious treatments for externalizing conduct, regardless of specific causal mechanisms (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). Moreover, for reasons articulated below, mediation of child behavior change through parenting was likely not put to a strong test by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) given the relatively low dose of PT provided (even if commensurate with other prevention programs; see below), modest effect sizes in both parenting change and child behavior change, restricted severity of externalizing behavior among children at intake, and perhaps most important, misinterpretations of their cross-lag panel models.
In addition, the implication that parenting and children's genetic predispositions can or should be pitted against one another as explanations of treatment outcome runs counter to widespread recognition that gene–environment interplay affects externalizing behaviors (e.g., Samek et al., Reference Samek, Hicks, Keyes, Bailey, McGue and Iacono2015). This apparent pitting of causal influences, as exemplified in the title of their paper, is almost certainly unwitting given the sophistication with which the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) research team has previously addressed gene–environment interplay in this journal and elsewhere (e.g., Jaffee et al., Reference Jaffee, Caspi, Moffitt, Dodge, Rutter, Taylor and Tully2005). Results reported by Weeland et al. may reflect gene–environment interplay in action, but this possibility is not tested directly (only main effects of the 5-HTTLPR gene on parenting, and of parenting on child behavior are tested). This is understandable given very large sample sizes required to detect statistical interactions, especially when alleles are distributed unequally, as is the case for 5-HTTLPR (16.8% s/s, 50.8% s/l, and 32.5% l/l in this sample). However, without acknowledgment or discussion of the possibility of gene–environment interplay, the take-home message of the Weeland et al. paper, as indicated by the reactions of some of our colleagues, is that PT does not matter because children's genetic predispositions drive parenting change, which is unrelated to externalizing outcomes. As noted above, many treatment-outcome studies identify parenting as a treatment mediator, and many others demonstrate efficacy of PT, even when parenting fails statistical tests of mediation or is not evaluated as an intervening variable (for reviews see, e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). Thus, the Weeland et al. findings do not challenge the efficacy of PT (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014), a point that may not be clear to many readers. In sections to follow, we review likely explanations for divergent findings regarding parenting as a mechanism of change in PT programs (Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). We focus primarily on IY because it was evaluated specifically by Weeland et al. We then consider the utility of cross-lag panel models for use in testing mediation, particularly as applied by Weeland et al.
Heterogeneity in Effect Sizes for IY PT
By definition, well-established PT programs, including IY, yield consistent improvements in child externalizing behaviors (Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). Nevertheless, effect sizes vary considerably from study to study, ranging from small to large at both posttreatment and long-term follow-up depending on a host of predictors and moderators, including severity of child externalizing behavior at intake (greater severity at intake predicts more improvement), levels of critical, harsh, and ineffective parenting at intake (worse parenting at intake predicts less improvement), number of PT sessions attended (higher attendance yields more improvement), informant (parent reports yield larger effect sizes than teacher reports and behavior observations), and whether child training (CT) is included in prevention/treatment (PT+CT yields larger effect sizes than PT alone; see Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Kazdin, Esveldt-Dawson, French, & Unis, Reference Kazdin, Esveldt-Dawson, French and Unis1987; Menting et al., Reference Menting, de Castro and Matthys2013; Scott et al., Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Reid, & Hammond, Reference Webster-Stratton, Reid and Hammond2004).
Although we do not have space to review these predictors and moderators in detail, it is notable that Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) recruited a sample that (a) was high risk (≥75th percentile on externalizing behavior) and therefore less severe than participants in clinical intervention studies (often defined as ≥90th–95th percentile and/or clinical diagnosis); (b) was non-treatment-seeking, which can affect attendance and adherence; (c) attended a limited number of PT sessions (mean = 8.6, with 44 families attending zero); and (d) did not participate in CT. This is not a criticism of their use of or implementation of IY in the ORCHIDS study (Chhangur, Weeland, Overbeek, Matthys, & Orobio de Castro, Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). Prevention efforts among children with mild and subclinical symptoms are important for altering developmental trajectories to worsening externalizing behavior. However, numerous studies show that PT yields larger effects for children with more severe externalizing behaviors (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Conduct Problems Prevention Research Group, 2007; Menting et al., Reference Menting, de Castro and Matthys2013). Thus, smaller effect sizes should be expected in high-risk, non-treatment-seeking samples than in clinical samples, and generalizations about PT effects, including mechanisms, should be confined to the population from which a sample is drawn. Again, our concern is that readers may reach the incorrect conclusion that PT is unnecessary based on null effects of mediation.
Although treatment effect sizes are not reported by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018), they can readily be computed from Table 1 on p. 96. For parenting (negative affect, positive affect, negative parenting, and positive parenting), pre- to posteffect sizes (Cohen's d) ranged from d = 0.0 (negative affect) to d = 0.68 (self-reported negative parenting). For child behavioral outcomes, pre- to posteffect sizes were d = 0.19 for negative affectivity and d = 0.57 for externalizing behavior. These effect sizes are small to medium by Cohen's (Reference Cohen1988) standards, and were maintained at long-term (10-month) follow-up (see below). Such effect sizes for child behavior are within expected ranges reported by Menting et al. (Reference Menting, de Castro and Matthys2013) in their recent meta-analysis.
In contrast, pre- and posteffect sizes from Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) were slightly larger across parenting outcomes, ranging from d = 0.38–0.75 for positive parenting and d = 0.31–0.71 for negative parenting, as rated by blinded coders. These findings are reported in Table 2 (p. 139). Pre- and posteffects sizes for child behavioral outcomes were considerably larger than in the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) study, ranging from d = 1.02–1.32 (see Table 1, p. 138). These are large effects by Cohen's (Reference Cohen1988) standards, and were also maintained at long-term (1-year) follow-up. As reviewed above, larger effect sizes are expected in studies that enroll children with higher levels of externalizing behavior (Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Menting et al., Reference Menting, de Castro and Matthys2013). In the Bell et al. study, children were required to score at or above the 95th percentile on externalizing conduct, and therefore comprised a clinical rather than high-risk sample. Although direct effects of treatment on child behavioral outcomes are not required for mediation (Hayes, Reference Hayes2013), larger direct effects provide more opportunity for mediation to be detected (e.g., Kenny & Judd, Reference Kenny and Judd2014).
Dose Effects of PT
Several studies demonstrate that for IY parent training, numbers of sessions attended are associated with effect sizes in improvement of both parent and child behavior (Baydar, Reid, & Webster-Stratton, Reference Baydar, Reid and Webster-Stratton2003; Lavigne et al., Reference Lavigne, LeBailly, Gouze, Cicchetti, Jessup, Arend and Binns2008; Menting et al., Reference Menting, de Castro and Matthys2013). This is illustrated in the Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) article, where effect sizes for improvements in parenting were twice as large for parents who were assigned randomly to a 20-session PT condition versus a 10-session PT condition. As shown in Table 2 (p. 139), effect sizes for positive parenting were d = 0.75 versus 0.38, and effect sizes for negative parenting were d = 0.71 versus 0.31 in the 20-session versus 10-session groups, respectively. In the same RCT (Webster-Stratton, Reid, & Beauchaine, Reference Webster-Stratton, Reid and Beauchaine2013), changes in children's Child Behavior Checklist externalizing scores were over three times as large in the 20-session condition (d = 0.80) compared with the 10-session condition (d = 0.25).
To be fair, the Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) evaluation of IY also included CT, which exerts direct effects on child behavior. In turn, improvements in child behavior are likely to affect parenting. However, similar dose effects also emerge from PT-only applications of IY, which improve parenting in a linear, dose-response fashion (Baydar, Reid, & Webster-Stratton, Reference Baydar, Reid and Webster-Stratton2003). This raises questions regarding how much PT is necessary to confer maximum benefits for altering parenting behavior. Data from Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018), summarized in the paragraph immediately above, suggest that somewhat more than 10 sessions are needed, at least for clinical samples. Optimal numbers of PT sessions are less clear for high-risk prevention samples, for whom dose is often confounded with lower levels of parenting competence and higher levels of child externalizing behavior at intake (Pasalich, Witkiewitz, McMahon, Pinderhughes, & Conduct Problems Prevention Research Group, Reference Pasalich, Witkiewitz, McMahon and Pinderhughes2016).
Dose effects on children's externalizing behaviors are also observed when implementing IY parent training. For example, Webster-Stratton and Hammond (Reference Webster-Stratton and Hammond1997) reported that at least 7 sessions are required for significant change on a single outcome measure, and at least 9 sessions are required for significant gains on two outcome measures. In the Coping Power (prevention) Program, Lochman, Boxmeyer, Powell, Roth, and Windle (Reference Lochman, Boxmeyer, Powell, Roth and Windle2006) reported significant parent training effects at 8 sessions. It is important to note, however, that the minimum number of sessions required to reach statistical significance is a low threshold that represents meaningful benefit for at best a plurality of enrolled children (see, e.g., Atkins, Bedics, McGlinchey, & Beauchaine, Reference Atkins, Bedics, McGlinchey and Beauchaine2005). This is why a minimum of 14 sessions of IY parent training are recommended for prevention purposes, and why 22–23 sessions are recommended for intervention studies (Incredible Years, 2018a, 2018b).
Given findings regarding dosage effects outlined above, the average number of PT sessions attended by parents in the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) study (mean = 8.6, with 44 families attending zero) may not provide an especially strong test of parenting effects on child behavior. Underestimation of prevention effects is made even more acute when intent-to-treat analysis is used (i.e., when families who attended zero sessions are included; see, e.g., Lochman et al., Reference Lochman, Boxmeyer, Powell, Roth and Windle2006). Once again, these are not criticisms of either design or execution of the ORCHIDS study, which targets a high-risk prevention sample (see Chhangur et al., Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). Both the number of PT sessions attended and the effect sizes reported are fully consistent with those observed in other prevention trials using IY (e.g., Reid et al., Reference Reid, Webster-Stratton and Beauchaine2001). Nevertheless, hoping parents will attend 14 sessions is one thing, but achieving that objective is quite another. However, we again caution against generalizing findings from prevention studies, where attendance is often more variable than in intervention studies (see Lochman et al., Reference Lochman, Boxmeyer, Powell, Roth and Windle2006), to nonprevention populations. In this case, concluding that parenting is not a mechanism of child behavior change for IY based on a null outcome in a prevention trial may be an overgeneralization.
Mediation in Cross-Lag Panel Models
The ORCHIDS study is a true RCT, a major strength of its design (see Chhangur et al., Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). High-risk participants were assigned randomly to prevention and control conditions at pretest, and IY was then administrated to the prevention group, before posttreatment and 10-month follow-up assessments were conducted. Random assignment allows the authors to make causal inferences about intervention-induced changes in parenting and child behavior at posttest and 10-month follow-up, regardless of specific mechanisms. Group differences at posttreatment and 10-month follow-up indicate that something about the intervention (direct or indirect) yielded changes in both parenting and children's externalizing behavior (see Table 1, p. 96). As there was no prevention dose delivered directly to children, the only explanation for child behavior change is through some aspect of parental behavior change, even if not in the specific parenting domains assessed. Something that parents did differently had to be experienced by their children. Identifying specific mediational effects is more complex.
For a mediator to be specified in any intervening variable analysis, both conceptual and statistical criteria must be met (see, e.g., Agler, & De Boeck, Reference Agler and De Boeck2017; Montoya & Hayes, Reference Montoya and Hayes2016). Although reasonable people often disagree on conceptual criteria for and interpretation of mediation in correlational data, whether cross sectional or longitudinal (see, e.g., Cole & Maxwell, Reference Cole and Maxwell2003), conceptual criteria for mediation in RCTs are well established, and follow from foundational work by Kraemer et al. (Reference Kraemer, Wilson, Fairburn and Agras2002). According to these criteria, for parenting to qualify as a mediator, (a) parenting change must accrue during the course of treatment, (b) parenting change must correlate with treatment condition, and (c) parenting change must show either a main or an interactive effect on child behavior change. These criteria are more restrictive than broader definitions that are often applied to nonexperimental data, allowing for stronger statements about mediation because assignment of participants to groups in a RCT is random, and because changes in the mediator must be observed between pre- and postassessments. Thus, there is temporal resolution between the independent variable (assignment to treatment condition), the putative mediator, and the posttreatment outcome variable. In a RCT, mediation can only be inferred when these conceptual criteria are met and appropriate statistical tests are executed. Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) do not apply the Kraemer et al. criteria, nor do they test, at least as their analyses are described, the mediating effects that they state they test.
Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) run 24 cross-lag panel models to test mediating effects of parenting on children's externalizing behavior. The first set of 6 are intended to test mediating effects of parental negative affect, negative parenting (observed and self-reported), and positive parenting (observed and self-report). These are followed by 3 reanalyses of each model, one for each genotype (s/s, s/l, and l/l). The general form of these cross-lag models appears in Figure 1, in which path a represents the independent effect of the intervention on children's externalizing behavior at 10-month follow-up, path b represents the independent effect of the intervention on parenting at posttest, and path c represents the independent effect of parenting on children's externalizing behavior at 10-month follow-up.
Figure 1.General format of cross-lag panel models from Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018). The direct effect of the intervention on child externalizing behavior at 10-month follow-up, controlling for child behavior at posttest, is indicated by a. The indirect effect of the intervention on child externalizing behavior at 10-month follow-up through parenting at posttest, controlling for parenting at pretest, is indicated by b × c. As outlined in text, this model tests whether the intervention continued to exert changes (i.e., unique effects) on child externalizing behavior after it was over (at 10-month follow-up), and whether parenting mediates any such effects. The model controls for (i.e., statistically partials out), rather than testing direct effects of, the intervention on child externalizing behavior at posttest.
There are two statements that Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) make repeatedly that may follow from a common misunderstanding of path-analytic regression. The first is that prevention had no effect on children's externalizing behaviors at 10-month follow-up. This conclusion is inconsistent with data presented in Table 1 on p. 96, which show that posttest prevention effects were maintained at 10-month follow-up. In contrast, path a in all of their models represents the independent effect of prevention on children's behavior at 10-month follow-up, statistically partialling out children's externalizing behavior at posttest (see, e.g., Cole & Maxwell, Reference Cole and Maxwell2003). Put another way, these coefficients test whether prevention exerted additive (i.e., unique) changes in children's externalizing behavior, after IY was over. This is not equivalent to stating that IY had no effects on externalizing outcomes at long-term follow-up. Again, our intent is not to criticize; these are very common misinterpretations of cross-lag models and related statistical control techniques (see McDonough-Caplan, Klein, & Beauchaine, Reference McDonough-Caplan, Klein and Beauchaine2018). However, it is important that readers understand that the prevention program did exert enduring effects on externalizing outcomes at 10-month follow-up.
Second, as outlined in detail above, the authors conclude that changes in parenting did not mediate changes in children's externalizing behavior, based on findings observed across the 24 cross-lag models. However, these models do not test the mediating effect of parenting. Instead, the indirect path from intervention to child externalizing behavior at 10-month follow-up through parenting (b × c) tests whether parenting at posttest, statistically partialling out parenting at pretest, mediates intervention effects on children's externalizing behavior at 10-month follow-up, statistically partialling out children's externalizing behavior at posttest. Once again, children's externalizing behavior at posttest is partialled out of the regression equation. It may be the case that a preponderance of child behavior change was statistically partialled out of a test of mediating effects on child behavior change (for further discussion, see McDonough-Caplan et al., Reference McDonough-Caplan, Klein and Beauchaine2018).
For these reasons and others, some methodologists recommend that cross-lag panel models be used primarily for exploratory purposes (see Kearney, Reference Kearney and Allenin press), not for hypothesis testing with experimental data such as those collected in RCTs. More powerful methods to test mediation in RCTs are available (e.g., Montoya & Hayes, Reference Montoya and Hayes2016). When applied carefully, these do not remove overlapping sources of variance in outcome measures. In the present study, we cannot know whether prevention-induced changes in parenting mediated child behavior change without reanalysis of the ORCHIDS data.
Summary and Conclusions
We write this comment not to assail Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018), who present very important findings regarding moderating effects of children's genetic predispositions on parenting during an empirically supported prevention program for externalizing behavior. Specifying what works best for which families has clear implications for devising more effective, targeted prevention and intervention programs (Brestan & Eyberg, Reference Brestan and Eyberg1998). Thus, Weelend et al. should be applauded for their important work.
We consider and evaluate the conclusion that parenting is not a mechanism through which children's externalizing behaviors improve during empirically supported PT. Similar findings emerge from other prevention and intervention studies (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). Yet as we argue above, and as Forehand et al. also emphasize, even if parenting is not a causal mechanism of child behavior change in PT programs, such programs should be retained given demonstrated efficacy and effectiveness in forestalling development of costly externalizing outcomes (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Scott et al., Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Rinaldi, et al., Reference Webster-Stratton, Rinaldi and Reid2011).
It is possible and perhaps likely that at least some of the null effects summarized by Forehand et al. (Reference Forehand, Lafko, Parent and Burt2014) and reported by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) emerge from use of specific conceptual and statistical strategies used to test mediation. Such strategies vary widely across studies, and are not equivalent, equally powerful, or equally effective (see Agler & De Boeck, Reference Agler and De Boeck2017; Montoya & Hayes, Reference Montoya and Hayes2016). In the case of cross-lag panel models, what appear to be legitimate tests of longitudinal mediation may test something different than we think given statistical partialling of prior effects. Greater convergence of findings should emerge through more consistent use of established conceptual criteria for mediation in RCTs (Kraemer et al., Reference Kraemer, Wilson, Fairburn and Agras2002), and through concurrent modeling of known treatment moderators, including PT dose, severity of parenting problems at intake, and levels of children's externalizing behaviors at intake, among others.
You have
Access
Earlier this year, two articles appeared in top-tier journals within days of one another, both evaluating mechanisms children's adjustment outcomes following treatment with the Incredible Years (IY) series. One paper appeared in this journal, and reported modest but significant effects of IY parent training (PT) on children's externalizing behaviors in the Observational Randomized Control Trial of Childhood Differential Susceptibility (ORCHIDS) prevention program (Weeland et al., Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018). Of note, despite improvements in both parenting and child behavior, no mediating effects of parenting on externalizing outcomes were found. At first glance, this might seem surprising, but in a recent review, Forehand, Lafko, Parent, and Burt (Reference Forehand, Lafko, Parent and Burt2014) found that tests of parenting as a mediator of child behavior change were significant across less than half of prevention and intervention trials. Thus, the Weeland et al. findings are not anomalous. We write this comment out of concern that without elaboration, such findings could be misconstrued, leading to abandonment of well-established treatments for externalizing behavior (Eyberg, Nelson, & Boggs, Reference Eyberg, Nelson and Boggs2008; Menting, de Castro, & Matthys, Reference Menting, de Castro and Matthys2013).
The second paper, written by our group, reported posttreatment improvements of large effect size on multiple measures of children's externalizing behavior, and mediating effects of reduced negative parenting on children's posttreatment autonomic nervous system activity and reactivity (Bell, Shader, Webster-Stratton, Reid, & Beauchaine, Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018; see Webster-Stratton, Reid, & Beauchaine, Reference Webster-Stratton, Reid and Beauchaine2011). Results from this intervention are consistent with findings from several other randomized clinical trials (RCTs) of IY. Such trials, including reports from independent research groups, identify parenting change as a specific mechanism through which reductions in children's behavior problems and emotional lability are effected using IY and similar interventions (e.g., Beauchaine, Webster-Stratton, & Reid, Reference Beauchaine, Webster-Stratton and Reid2005; Forehand et al., Reference Forehand, Lafko, Parent and Burt2014).
Regardless of whether or not mediation is assessed, studies show that the IY series improves both parenting and child externalizing behaviors for racially and ethnically diverse families who experience a wide range of adversities, including maltreatment, socioeconomic disadvantage, welfare system involvement, and recent maternal incarceration (e.g., Baker-Henningham, Walker, Powell, & Meeks Gardner, Reference Baker-Henningham, Walker, Powell and Meeks Gardner2009; Hurlburt, Nguyen, Reid, Webster-Stratton, & Zhang, Reference Hurlburt, Nguyen, Reid, Webster-Stratton and Zhang2013; Kim, Cain, & Webster-Stratton, Reference Kim, Cain and Webster-Stratton2008; Leijten, Raaijmakers, Orobio de Castro, van den Ban, & Matthys, Reference Leijten, Raaijmakers, Orobio de Castro, van den Ban and Matthys2017; Menting, de Castro, Wjingaards-de Meij, & Matthys, Reference Menting, de Castro, Wijngaards-de Meij and Matthys2014; Reid, Webster-Stratton, & Beauchaine, Reference Reid, Webster-Stratton and Beauchaine2001; Webster-Stratton & Reid, Reference Webster-Stratton and Reid2010). Improvements in children's externalizing behaviors following IY persist for up to 10 years, and extend to clinically and educationally significant endpoints including criminal justice system involvement, antisocial personality, and reading literacy (Scott, Briskman, & O'Connor, Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Rinaldi, & Reid, Reference Webster-Stratton, Rinaldi and Reid2011). According to this literature, PT plays a crucial role in altering well-characterized, costly developmental trajectories to antisociality (Beauchaine, Zisner, & Sauder, Reference Beauchaine, Zisner and Sauder2017). Moreover, the efficacy of IY has been evaluated in several RCTs using data collected from multiple informants, including mothers, fathers, teachers, and blinded coders (e.g., Webster-Stratton, Rinaldi, et al., Reference Webster-Stratton, Rinaldi and Reid2011). In some of these studies (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Bell et al., Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018), effects of parenting change on child outcomes have been confirmed using stringent conceptual criteria for mediation (Kraemer, Wilson, Fairburn, & Agras, Reference Kraemer, Wilson, Fairburn and Agras2002) and/or modern statistical tests that assess direct and indirect (mediational) effects using bootstrap confidence intervals (Montoya & Hayes, Reference Montoya and Hayes2016).
Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) reported no significant mediating effects of parenting change on observed improvements in children's externalizing behavior following intervention with IY, despite increases in positive parenting, decreases in negative parenting, and increases in parental positive affect. The authors offer potential third variable explanations for child behavior change, including decreased family and parent distress, improved quality of life, and increases in parenting self-efficacy, all of which are plausible. In addition, the authors found that children's serotonin transporter linked polymorphisms (5-HTTLPR) moderated effects of IY on changes in negative parenting. Reductions in negative parenting were larger for families whose children carried two copies of the short allele of the 5-HTTLPR gene (s/s homozygotes) than for families of children who were either l/l homozygotic or heterozygotic.
This is an extremely important finding that addresses longstanding questions in the prevention and intervention literatures regarding which treatments work best for which children and families (Brestan & Eyberg, Reference Brestan and Eyberg1998; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). In our own work, we have commented on both (a) the need to better understand how children's individual differences affect their treatment response to IY (e.g., Beauchaine et al., Reference Beauchaine, Gatzke-Kopp, Neuhaus, Chipman, Reid and Webster-Stratton2013), and (b) the bidirectional and transactional nature of family relationship dynamics in high-risk families (e.g., Beauchaine & Zalewski, Reference Beauchaine, Zalewski, Dishion and Snyder2016). The finding that children's genetically mediated individual differences influence changes in negative parenting is a welcome addition to the literature, and Weeland et al. should be applauded for their painstaking research.
Our objective in writing this comment is not to challenge the important 5-HTTLPR finding. Instead, we are concerned about the seeming certainty of (based on null findings) their conclusion that parenting change is not a mechanism through which IY exerts at least some of its effects, and the ongoing need for well-established, efficacious treatments for externalizing conduct, regardless of specific causal mechanisms (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). Moreover, for reasons articulated below, mediation of child behavior change through parenting was likely not put to a strong test by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) given the relatively low dose of PT provided (even if commensurate with other prevention programs; see below), modest effect sizes in both parenting change and child behavior change, restricted severity of externalizing behavior among children at intake, and perhaps most important, misinterpretations of their cross-lag panel models.
In addition, the implication that parenting and children's genetic predispositions can or should be pitted against one another as explanations of treatment outcome runs counter to widespread recognition that gene–environment interplay affects externalizing behaviors (e.g., Samek et al., Reference Samek, Hicks, Keyes, Bailey, McGue and Iacono2015). This apparent pitting of causal influences, as exemplified in the title of their paper, is almost certainly unwitting given the sophistication with which the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) research team has previously addressed gene–environment interplay in this journal and elsewhere (e.g., Jaffee et al., Reference Jaffee, Caspi, Moffitt, Dodge, Rutter, Taylor and Tully2005). Results reported by Weeland et al. may reflect gene–environment interplay in action, but this possibility is not tested directly (only main effects of the 5-HTTLPR gene on parenting, and of parenting on child behavior are tested). This is understandable given very large sample sizes required to detect statistical interactions, especially when alleles are distributed unequally, as is the case for 5-HTTLPR (16.8% s/s, 50.8% s/l, and 32.5% l/l in this sample). However, without acknowledgment or discussion of the possibility of gene–environment interplay, the take-home message of the Weeland et al. paper, as indicated by the reactions of some of our colleagues, is that PT does not matter because children's genetic predispositions drive parenting change, which is unrelated to externalizing outcomes. As noted above, many treatment-outcome studies identify parenting as a treatment mediator, and many others demonstrate efficacy of PT, even when parenting fails statistical tests of mediation or is not evaluated as an intervening variable (for reviews see, e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). Thus, the Weeland et al. findings do not challenge the efficacy of PT (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014), a point that may not be clear to many readers. In sections to follow, we review likely explanations for divergent findings regarding parenting as a mechanism of change in PT programs (Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). We focus primarily on IY because it was evaluated specifically by Weeland et al. We then consider the utility of cross-lag panel models for use in testing mediation, particularly as applied by Weeland et al.
Heterogeneity in Effect Sizes for IY PT
By definition, well-established PT programs, including IY, yield consistent improvements in child externalizing behaviors (Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Menting et al., Reference Menting, de Castro and Matthys2013). Nevertheless, effect sizes vary considerably from study to study, ranging from small to large at both posttreatment and long-term follow-up depending on a host of predictors and moderators, including severity of child externalizing behavior at intake (greater severity at intake predicts more improvement), levels of critical, harsh, and ineffective parenting at intake (worse parenting at intake predicts less improvement), number of PT sessions attended (higher attendance yields more improvement), informant (parent reports yield larger effect sizes than teacher reports and behavior observations), and whether child training (CT) is included in prevention/treatment (PT+CT yields larger effect sizes than PT alone; see Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Kazdin, Esveldt-Dawson, French, & Unis, Reference Kazdin, Esveldt-Dawson, French and Unis1987; Menting et al., Reference Menting, de Castro and Matthys2013; Scott et al., Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Reid, & Hammond, Reference Webster-Stratton, Reid and Hammond2004).
Although we do not have space to review these predictors and moderators in detail, it is notable that Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) recruited a sample that (a) was high risk (≥75th percentile on externalizing behavior) and therefore less severe than participants in clinical intervention studies (often defined as ≥90th–95th percentile and/or clinical diagnosis); (b) was non-treatment-seeking, which can affect attendance and adherence; (c) attended a limited number of PT sessions (mean = 8.6, with 44 families attending zero); and (d) did not participate in CT. This is not a criticism of their use of or implementation of IY in the ORCHIDS study (Chhangur, Weeland, Overbeek, Matthys, & Orobio de Castro, Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). Prevention efforts among children with mild and subclinical symptoms are important for altering developmental trajectories to worsening externalizing behavior. However, numerous studies show that PT yields larger effects for children with more severe externalizing behaviors (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Conduct Problems Prevention Research Group, 2007; Menting et al., Reference Menting, de Castro and Matthys2013). Thus, smaller effect sizes should be expected in high-risk, non-treatment-seeking samples than in clinical samples, and generalizations about PT effects, including mechanisms, should be confined to the population from which a sample is drawn. Again, our concern is that readers may reach the incorrect conclusion that PT is unnecessary based on null effects of mediation.
Although treatment effect sizes are not reported by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018), they can readily be computed from Table 1 on p. 96. For parenting (negative affect, positive affect, negative parenting, and positive parenting), pre- to posteffect sizes (Cohen's d) ranged from d = 0.0 (negative affect) to d = 0.68 (self-reported negative parenting). For child behavioral outcomes, pre- to posteffect sizes were d = 0.19 for negative affectivity and d = 0.57 for externalizing behavior. These effect sizes are small to medium by Cohen's (Reference Cohen1988) standards, and were maintained at long-term (10-month) follow-up (see below). Such effect sizes for child behavior are within expected ranges reported by Menting et al. (Reference Menting, de Castro and Matthys2013) in their recent meta-analysis.
In contrast, pre- and posteffect sizes from Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) were slightly larger across parenting outcomes, ranging from d = 0.38–0.75 for positive parenting and d = 0.31–0.71 for negative parenting, as rated by blinded coders. These findings are reported in Table 2 (p. 139). Pre- and posteffects sizes for child behavioral outcomes were considerably larger than in the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) study, ranging from d = 1.02–1.32 (see Table 1, p. 138). These are large effects by Cohen's (Reference Cohen1988) standards, and were also maintained at long-term (1-year) follow-up. As reviewed above, larger effect sizes are expected in studies that enroll children with higher levels of externalizing behavior (Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Menting et al., Reference Menting, de Castro and Matthys2013). In the Bell et al. study, children were required to score at or above the 95th percentile on externalizing conduct, and therefore comprised a clinical rather than high-risk sample. Although direct effects of treatment on child behavioral outcomes are not required for mediation (Hayes, Reference Hayes2013), larger direct effects provide more opportunity for mediation to be detected (e.g., Kenny & Judd, Reference Kenny and Judd2014).
Dose Effects of PT
Several studies demonstrate that for IY parent training, numbers of sessions attended are associated with effect sizes in improvement of both parent and child behavior (Baydar, Reid, & Webster-Stratton, Reference Baydar, Reid and Webster-Stratton2003; Lavigne et al., Reference Lavigne, LeBailly, Gouze, Cicchetti, Jessup, Arend and Binns2008; Menting et al., Reference Menting, de Castro and Matthys2013). This is illustrated in the Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) article, where effect sizes for improvements in parenting were twice as large for parents who were assigned randomly to a 20-session PT condition versus a 10-session PT condition. As shown in Table 2 (p. 139), effect sizes for positive parenting were d = 0.75 versus 0.38, and effect sizes for negative parenting were d = 0.71 versus 0.31 in the 20-session versus 10-session groups, respectively. In the same RCT (Webster-Stratton, Reid, & Beauchaine, Reference Webster-Stratton, Reid and Beauchaine2013), changes in children's Child Behavior Checklist externalizing scores were over three times as large in the 20-session condition (d = 0.80) compared with the 10-session condition (d = 0.25).
To be fair, the Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018) evaluation of IY also included CT, which exerts direct effects on child behavior. In turn, improvements in child behavior are likely to affect parenting. However, similar dose effects also emerge from PT-only applications of IY, which improve parenting in a linear, dose-response fashion (Baydar, Reid, & Webster-Stratton, Reference Baydar, Reid and Webster-Stratton2003). This raises questions regarding how much PT is necessary to confer maximum benefits for altering parenting behavior. Data from Bell et al. (Reference Bell, Shader, Webster-Stratton, Reid and Beauchaine2018), summarized in the paragraph immediately above, suggest that somewhat more than 10 sessions are needed, at least for clinical samples. Optimal numbers of PT sessions are less clear for high-risk prevention samples, for whom dose is often confounded with lower levels of parenting competence and higher levels of child externalizing behavior at intake (Pasalich, Witkiewitz, McMahon, Pinderhughes, & Conduct Problems Prevention Research Group, Reference Pasalich, Witkiewitz, McMahon and Pinderhughes2016).
Dose effects on children's externalizing behaviors are also observed when implementing IY parent training. For example, Webster-Stratton and Hammond (Reference Webster-Stratton and Hammond1997) reported that at least 7 sessions are required for significant change on a single outcome measure, and at least 9 sessions are required for significant gains on two outcome measures. In the Coping Power (prevention) Program, Lochman, Boxmeyer, Powell, Roth, and Windle (Reference Lochman, Boxmeyer, Powell, Roth and Windle2006) reported significant parent training effects at 8 sessions. It is important to note, however, that the minimum number of sessions required to reach statistical significance is a low threshold that represents meaningful benefit for at best a plurality of enrolled children (see, e.g., Atkins, Bedics, McGlinchey, & Beauchaine, Reference Atkins, Bedics, McGlinchey and Beauchaine2005). This is why a minimum of 14 sessions of IY parent training are recommended for prevention purposes, and why 22–23 sessions are recommended for intervention studies (Incredible Years, 2018a, 2018b).
Given findings regarding dosage effects outlined above, the average number of PT sessions attended by parents in the Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) study (mean = 8.6, with 44 families attending zero) may not provide an especially strong test of parenting effects on child behavior. Underestimation of prevention effects is made even more acute when intent-to-treat analysis is used (i.e., when families who attended zero sessions are included; see, e.g., Lochman et al., Reference Lochman, Boxmeyer, Powell, Roth and Windle2006). Once again, these are not criticisms of either design or execution of the ORCHIDS study, which targets a high-risk prevention sample (see Chhangur et al., Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). Both the number of PT sessions attended and the effect sizes reported are fully consistent with those observed in other prevention trials using IY (e.g., Reid et al., Reference Reid, Webster-Stratton and Beauchaine2001). Nevertheless, hoping parents will attend 14 sessions is one thing, but achieving that objective is quite another. However, we again caution against generalizing findings from prevention studies, where attendance is often more variable than in intervention studies (see Lochman et al., Reference Lochman, Boxmeyer, Powell, Roth and Windle2006), to nonprevention populations. In this case, concluding that parenting is not a mechanism of child behavior change for IY based on a null outcome in a prevention trial may be an overgeneralization.
Mediation in Cross-Lag Panel Models
The ORCHIDS study is a true RCT, a major strength of its design (see Chhangur et al., Reference Chhangur, Weeland, Overbeek, Matthys and Orobio de Castro2012). High-risk participants were assigned randomly to prevention and control conditions at pretest, and IY was then administrated to the prevention group, before posttreatment and 10-month follow-up assessments were conducted. Random assignment allows the authors to make causal inferences about intervention-induced changes in parenting and child behavior at posttest and 10-month follow-up, regardless of specific mechanisms. Group differences at posttreatment and 10-month follow-up indicate that something about the intervention (direct or indirect) yielded changes in both parenting and children's externalizing behavior (see Table 1, p. 96). As there was no prevention dose delivered directly to children, the only explanation for child behavior change is through some aspect of parental behavior change, even if not in the specific parenting domains assessed. Something that parents did differently had to be experienced by their children. Identifying specific mediational effects is more complex.
For a mediator to be specified in any intervening variable analysis, both conceptual and statistical criteria must be met (see, e.g., Agler, & De Boeck, Reference Agler and De Boeck2017; Montoya & Hayes, Reference Montoya and Hayes2016). Although reasonable people often disagree on conceptual criteria for and interpretation of mediation in correlational data, whether cross sectional or longitudinal (see, e.g., Cole & Maxwell, Reference Cole and Maxwell2003), conceptual criteria for mediation in RCTs are well established, and follow from foundational work by Kraemer et al. (Reference Kraemer, Wilson, Fairburn and Agras2002). According to these criteria, for parenting to qualify as a mediator, (a) parenting change must accrue during the course of treatment, (b) parenting change must correlate with treatment condition, and (c) parenting change must show either a main or an interactive effect on child behavior change. These criteria are more restrictive than broader definitions that are often applied to nonexperimental data, allowing for stronger statements about mediation because assignment of participants to groups in a RCT is random, and because changes in the mediator must be observed between pre- and postassessments. Thus, there is temporal resolution between the independent variable (assignment to treatment condition), the putative mediator, and the posttreatment outcome variable. In a RCT, mediation can only be inferred when these conceptual criteria are met and appropriate statistical tests are executed. Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) do not apply the Kraemer et al. criteria, nor do they test, at least as their analyses are described, the mediating effects that they state they test.
Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) run 24 cross-lag panel models to test mediating effects of parenting on children's externalizing behavior. The first set of 6 are intended to test mediating effects of parental negative affect, negative parenting (observed and self-reported), and positive parenting (observed and self-report). These are followed by 3 reanalyses of each model, one for each genotype (s/s, s/l, and l/l). The general form of these cross-lag models appears in Figure 1, in which path a represents the independent effect of the intervention on children's externalizing behavior at 10-month follow-up, path b represents the independent effect of the intervention on parenting at posttest, and path c represents the independent effect of parenting on children's externalizing behavior at 10-month follow-up.
General format of cross-lag panel models from Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018). The direct effect of the intervention on child externalizing behavior at 10-month follow-up, controlling for child behavior at posttest, is indicated by a. The indirect effect of the intervention on child externalizing behavior at 10-month follow-up through parenting at posttest, controlling for parenting at pretest, is indicated by b × c. As outlined in text, this model tests whether the intervention continued to exert changes (i.e., unique effects) on child externalizing behavior after it was over (at 10-month follow-up), and whether parenting mediates any such effects. The model controls for (i.e., statistically partials out), rather than testing direct effects of, the intervention on child externalizing behavior at posttest.
There are two statements that Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) make repeatedly that may follow from a common misunderstanding of path-analytic regression. The first is that prevention had no effect on children's externalizing behaviors at 10-month follow-up. This conclusion is inconsistent with data presented in Table 1 on p. 96, which show that posttest prevention effects were maintained at 10-month follow-up. In contrast, path a in all of their models represents the independent effect of prevention on children's behavior at 10-month follow-up, statistically partialling out children's externalizing behavior at posttest (see, e.g., Cole & Maxwell, Reference Cole and Maxwell2003). Put another way, these coefficients test whether prevention exerted additive (i.e., unique) changes in children's externalizing behavior, after IY was over. This is not equivalent to stating that IY had no effects on externalizing outcomes at long-term follow-up. Again, our intent is not to criticize; these are very common misinterpretations of cross-lag models and related statistical control techniques (see McDonough-Caplan, Klein, & Beauchaine, Reference McDonough-Caplan, Klein and Beauchaine2018). However, it is important that readers understand that the prevention program did exert enduring effects on externalizing outcomes at 10-month follow-up.
Second, as outlined in detail above, the authors conclude that changes in parenting did not mediate changes in children's externalizing behavior, based on findings observed across the 24 cross-lag models. However, these models do not test the mediating effect of parenting. Instead, the indirect path from intervention to child externalizing behavior at 10-month follow-up through parenting (b × c) tests whether parenting at posttest, statistically partialling out parenting at pretest, mediates intervention effects on children's externalizing behavior at 10-month follow-up, statistically partialling out children's externalizing behavior at posttest. Once again, children's externalizing behavior at posttest is partialled out of the regression equation. It may be the case that a preponderance of child behavior change was statistically partialled out of a test of mediating effects on child behavior change (for further discussion, see McDonough-Caplan et al., Reference McDonough-Caplan, Klein and Beauchaine2018).
For these reasons and others, some methodologists recommend that cross-lag panel models be used primarily for exploratory purposes (see Kearney, Reference Kearney and Allenin press), not for hypothesis testing with experimental data such as those collected in RCTs. More powerful methods to test mediation in RCTs are available (e.g., Montoya & Hayes, Reference Montoya and Hayes2016). When applied carefully, these do not remove overlapping sources of variance in outcome measures. In the present study, we cannot know whether prevention-induced changes in parenting mediated child behavior change without reanalysis of the ORCHIDS data.
Summary and Conclusions
We write this comment not to assail Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018), who present very important findings regarding moderating effects of children's genetic predispositions on parenting during an empirically supported prevention program for externalizing behavior. Specifying what works best for which families has clear implications for devising more effective, targeted prevention and intervention programs (Brestan & Eyberg, Reference Brestan and Eyberg1998). Thus, Weelend et al. should be applauded for their important work.
We consider and evaluate the conclusion that parenting is not a mechanism through which children's externalizing behaviors improve during empirically supported PT. Similar findings emerge from other prevention and intervention studies (see Forehand et al., Reference Forehand, Lafko, Parent and Burt2014). Yet as we argue above, and as Forehand et al. also emphasize, even if parenting is not a causal mechanism of child behavior change in PT programs, such programs should be retained given demonstrated efficacy and effectiveness in forestalling development of costly externalizing outcomes (e.g., Beauchaine et al., Reference Beauchaine, Webster-Stratton and Reid2005; Eyberg et al., Reference Eyberg, Nelson and Boggs2008; Scott et al., Reference Scott, Briskman and O'Connor2014; Webster-Stratton, Rinaldi, et al., Reference Webster-Stratton, Rinaldi and Reid2011).
It is possible and perhaps likely that at least some of the null effects summarized by Forehand et al. (Reference Forehand, Lafko, Parent and Burt2014) and reported by Weeland et al. (Reference Weeland, Chhangur, Jaffee, Van Der Giessen, Matthys, Orobio De Castro and Overbeek2018) emerge from use of specific conceptual and statistical strategies used to test mediation. Such strategies vary widely across studies, and are not equivalent, equally powerful, or equally effective (see Agler & De Boeck, Reference Agler and De Boeck2017; Montoya & Hayes, Reference Montoya and Hayes2016). In the case of cross-lag panel models, what appear to be legitimate tests of longitudinal mediation may test something different than we think given statistical partialling of prior effects. Greater convergence of findings should emerge through more consistent use of established conceptual criteria for mediation in RCTs (Kraemer et al., Reference Kraemer, Wilson, Fairburn and Agras2002), and through concurrent modeling of known treatment moderators, including PT dose, severity of parenting problems at intake, and levels of children's externalizing behaviors at intake, among others.