Introduction
Exposure to adversity is common in children, both in the USA and internationally. Approximately half of all US children will experience at least one form of adversity – such as physical, sexual, or emotional abuse, neglect, exposure to interpersonal violence, or chronic poverty – by the time they reach adulthood (Benjet et al., Reference Benjet, Borges, Medina-Mora, Zambrano, Cruz and Méndez2009; Green et al., Reference Green, McLaughlin, Berglund, Gruber, Sampson, Zaslavsky and Kessler2010; Kessler et al., Reference Kessler, McLaughlin, Green, Gruber, Sampson, Zaslavsky and Williams2010; McLaughlin et al., Reference McLaughlin, Green, Gruber, Sampson, Zaslavsky and Kessler2012). Evidence from population-based and longitudinal studies demonstrates that experiences of adversity are associated with the onset of internalizing and externalizing psychopathology transdiagnostically, including anxiety, mood, disruptive behavior, posttraumatic stress, and substance use disorders (Cohen, Brown, & Smailes, Reference Cohen, Brown and Smailes2001; Green et al., Reference Green, McLaughlin, Berglund, Gruber, Sampson, Zaslavsky and Kessler2010; Kessler et al., Reference Kessler, McLaughlin, Green, Gruber, Sampson, Zaslavsky and Williams2010; MacMillan et al., Reference MacMillan, Fleming, Streiner, Lin, Boyle, Jamieson and Beardslee2001; McLaughlin, Conron, Koenen, & Gilman, Reference McLaughlin, Conron, Koenen and Gilman2010; McLaughlin et al., Reference McLaughlin, Green, Gruber, Sampson, Zaslavsky and Kessler2012; Weich, Patterson, Shaw, & Stewart-Brown, Reference Weich, Patterson, Shaw and Stewart-Brown2009). Exposure to childhood adversity explains about a third of mental disorder onsets in the USA (Green et al., Reference Green, McLaughlin, Berglund, Gruber, Sampson, Zaslavsky and Kessler2010; McLaughlin et al., Reference McLaughlin, Green, Gruber, Sampson, Zaslavsky and Kessler2012) and is associated with increased risk for a wide range of chronic diseases and early mortality (Brown et al., Reference Brown, Anda, Tiemeier, Felitti, Edwards, Croft and Giles2009; Chen, Turiano, Mroczek, & Miller, Reference Chen, Turiano, Mroczek and Miller2016; Felitti et al., Reference Felitti, Anda, Nordenberg, Williamson, Spitz, Edwards and Marks1998; Wegman & Stetler, Reference Wegman and Stetler2009). A substantial body of work has characterized the developmental mechanisms through which experiences of adversity influence risk for psychopathology and chronic diseases (Cicchetti & Toth, Reference Cicchetti and Toth1995; Danese & McEwen, Reference Danese and McEwen2012; McLaughlin, Reference McLaughlin2016; McLaughlin, Colich, Rodman, & Weissman, Reference McLaughlin, Colich, Rodman and Weissman2020; McLaughlin & Lambert, Reference McLaughlin and Lambert2017). Although knowledge of these mechanisms has grown dramatically, little progress has been made in translating those mechanisms into interventions to prevent the onset of psychopathology on subsequent consequences (McLaughlin, DeCross, Jovanovic, & Tottenham, Reference McLaughlin, DeCross, Jovanovic and Tottenham2019). Through our current work, we hope to carry on Edward Zigler's legacy of “be[ing] totally committed to the optimal development of each child” by developing such an intervention for children who have screened positive for adversity. In contrast to the scarcity of research available prior to the development of Head Start, considered “the prototype of effective early childhood intervention,” we are fortunate to now be able to draw on the substantial evidence base supporting early childhood intervention (Zigler & Styfco, Reference Zigler, Pfannenstiel and Seitz2001).
In theory, screening and early intervention could help adversity-exposed children by preventing future incidences of adversity along with its physical and mental health consequences. We agree with the belief that children who begin life healthy “grow to become contributing members of the society, [and] the small investment made in their early years will have compounded to reap a handsome dividend” (Zigler, Reference Zeman and Garber1996, p. 47). A 10% reduction in childhood adversity prevalence in the USA has been estimated to save $105 billion annually (Bellis et al., Reference Bellis, Hughes, Ford, Ramos Rodriguez, Sethi and Passmore2019). The broader the scale of screening and prevention efforts, the greater the potential impact becomes. Pediatric primary care is a promising setting for wide-reaching adversity screening and prevention efforts, as the majority of US children see a primary care provider annually (Chevarley, Reference Chevarley2003). Moreover, addressing child mental health concerns in a medical setting has myriad benefits, including reduced stigma, cost, and family discomfort, as well as improved outcomes (Asarnow, Rozenman, Wiblin, & Zeltzer, Reference Asarnow, Rozenman, Wiblin and Zeltzer2015; Godoy et al., Reference Godoy, Long, Marschall, Hodgkinson, Bokor, Rhodes and Beers2017; Kolko & Perrin, Reference Kolko and Perrin2014). Primary care settings are already increasingly integrating youth mental health care (Aupont et al., Reference Aupont, Doerfler, Connor, Stille, Tisminetzky and McLaughlin2013), with approximately half of all visits now involving some type of behavioral or mental health concern (Martini et al., Reference Martini, Hilt, Marx, Chenven, Naylor, Sarvet and Ptakowski2012). Simultaneously, there has been a movement encouraging pediatricians to screen for adversity within routine primary care visits (Garner et al., Reference Garner, Shonkoff, Siegel, Dobbins, Earls, McGuinn and Wegner2012; Pardee, Kuzma, Dahlem, Boucher, & Darling-Fisher, Reference Pardee, Kuzma, Dahlem, Boucher and Darling-Fisher2017). One encouraging example in primary care settings is the Safe Environment for Every Kid (SEEK) program (Dubowitz, Reference Dubowitz2014; Dubowitz, Feigelman, Lane, & Kim, Reference Dubowitz, Feigelman, Lane and Kim2009), which includes a validated short adversity screening tool and a brief intervention (5–15 min) of motivational interviewing and community resource referrals for families who endorse adversity experiences (Dubowitz et al., Reference Dubowitz, Feigelman, Lane, Prescott, Blackman, Grube and Tracy2007; Dubowitz, Prescott, Feigelman, Lane, & Kim, Reference Dubowitz, Prescott, Feigelman, Lane and Kim2008; Feigelman et al., Reference Feigelman, Dubowitz, Lane, Prescott, Meyer, Tracy and Kim2009; Lane et al., Reference Lane, Dubowitz, Feigelman, Kim, Prescott, Meyer and Tracy2007). Randomized controlled trials of SEEK have shown reduced rates of future child maltreatment (Dubowitz et al., Reference Dubowitz, Feigelman, Lane and Kim2009; Dubowitz, Lane, Semiatin, & Magder, Reference Dubowitz, Lane, Semiatin and Magder2012), and feasibility studies have shown high acceptability and satisfaction among providers and caregivers in primary care settings (e.g., Eismann, Theuerling, Maguire, Hente, & Shapiro, Reference Eismann, Theuerling, Maguire, Hente and Shapiro2019). However, despite all this progress, one concern about adversity screening is the dearth of interventions that can prevent the mental health outcomes associated with adversity exposure (Campbell, Reference Campbell2020). This scarcity is problematic, as it has been recommended that youth receive evidence-based prevention strategies that are matched to their screened risks for psychopathology (Hankin, Reference Hankin2020). There thus remains a major challenge for adversity screening efforts in primary care or elsewhere: we must respond to those youth identified as adversity-exposed and yet we lack effective strategies for preventing the associated psychopathology.
The absence of effective preventive interventions for children exposed to adversity does not reflect a lack of research attention to this issue. Dozens of psychosocial prevention and early intervention programs for maltreated children have been developed and tested, but meta-analyses examining them have not produced substantive evidence of intervention effectiveness (Churchill, Reference Churchill2016; van IJzendoorn, Bakermans-Kranenburg, Coughlan, & Reijman, Reference van IJzendoorn, Bakermans-Kranenburg, Coughlan and Reijman2020). There are some effective interventions for children experiencing psychopathology after exposure to trauma (e.g., abuse, exposure to violence), such as trauma-focused cognitive behavioral therapy (TF-CBT; Cohen, Deblinger, Mannarino, & Steer, Reference Cohen, Deblinger, Mannarino and Steer2004; Cohen, Mannarino, & Iyengar, Reference Cohen, Mannarino and Iyengar2011; Dorsey et al., Reference Dorsey, McLaughlin, Kerns, Harrison, Lambert, Briggs and Amaya-Jackson2017). However, these interventions are treatment level, and youth psychological treatments typically involve a large number of sessions (averaging 16.5 in a recent meta-analysis; Weisz, Kuppens, et al., Reference Weisz, Kuppens, Ng, Eckshtain, Ugueto, Vaughn-Coaxum and Fordwood2017), which is less practical for typical behavioral health session limits. More feasible for integration into primary care would be, for example, a brief (four-session) skills-based intervention that has demonstrated effective prevention of posttraumatic stress disorder (PTSD) symptoms in children who have experienced trauma (Berkowitz, Stover, & Marans, Reference Berkowitz, Stover and Marans2011). Nevertheless, adversity experiences are associated not only with PTSD but also with transdiagnostic psychopathology, including a wide range of internalizing and externalizing problems (Green et al., Reference Green, McLaughlin, Berglund, Gruber, Sampson, Zaslavsky and Kessler2010; Keyes et al., Reference Keyes, Eaton, Krueger, McLaughlin, Wall, Grant and Hasin2012; McLaughlin et al., Reference McLaughlin, Green, Gruber, Sampson, Zaslavsky and Kessler2012). Furthermore, such existing evidence-based interventions were designed as treatments for children who have already developed symptoms of psychopathology following traumatic events and thus primarily focus on symptom reduction rather than preventing the emergence of psychopathology. This distinction is important as the mechanisms that underlie symptom maintenance may differ from those that contribute to the emergence of psychopathology following experiences of adversity. As such, prevention practices that target the developmental pathways linking adversity to the onset of psychopathology – particularly mechanisms that are associated with transdiagnostic risk – may ultimately hold the most promise.
Many have argued that greater attention to the neurodevelopmental pathways linking childhood adversity with psychopathology is needed in order to improve our ability to effectively intervene with children who have encountered adversity (Bentovim et al., Reference Bentovim, Chorpita, Daleiden, Gray, Pizzey and Vizard2020; Cicchetti & Toth, Reference Cicchetti, Toth, Nolen-Hoeksema and Hilt2009; Manly, Kim, Rogosch, & Cicchetti, Reference Manly, Kim, Rogosch and Cicchetti2001; Masten & Cicchetti, Reference Masten and Cicchetti2010; McLaughlin et al., Reference McLaughlin, DeCross, Jovanovic and Tottenham2019; van IJzendoorn et al., Reference van IJzendoorn, Bakermans-Kranenburg, Coughlan and Reijman2020). Over the past decade, a substantial body of evidence has emerged on these mechanisms. As noted in recent reviews of this literature, developmental pathways that have garnered the most consistent evidence for being influenced by experiences of adversity and that, in turn, are associated with the later onset of psychopathology include threat-related social information processing biases (e.g., hostile attribution bias), heightened emotional reactivity, difficulties with emotion regulation, and blunted reward processing (McLaughlin, Reference McLaughlin2016; McLaughlin et al., Reference McLaughlin, DeCross, Jovanovic and Tottenham2019). Threat-related social information processing biases have been observed most consistently among children exposed to trauma and forms of adversity involving threat (e.g., violence exposure, abuse), whereas blunted reward processing has been observed most consistently in children who have experienced forms of adversity involving deprivation, including neglect, caregiver separation, and food insecurity. Heightened emotional reactivity and difficulties with emotion regulation have been observed in children exposed to many forms of adversity (for a review, see McLaughlin et al., Reference McLaughlin, DeCross, Jovanovic and Tottenham2019). Correspondingly, McLaughlin et al. (Reference McLaughlin, DeCross, Jovanovic and Tottenham2019) posit that these mechanisms could serve as important behavioral targets for interventions aimed at preventing psychopathology transdiagnostically. Zigler (Reference Zigler2000) also suggested that research on brain development had important implications for early childhood interventions – specifically, that such research has made us more certain than ever that adverse environments alter processes such as learning and emotion regulation. Regardless of such suggestions, we know of no existing prevention program designed to specifically target the developmental processes that are influenced by exposure to adversity and, in turn, predict the later emergence of psychopathology.
Current Research
In this article we describe a series of studies investigating the development and testing of an intervention program through pediatric primary care settings to prevent the mental health consequences of exposure to childhood adversity. Our planned intervention will incorporate the evidence-based SEEK screening and case management approach (Dubowitz, Reference Dubowitz2014; Dubowitz et al., Reference Dubowitz, Feigelman, Lane and Kim2009). We will then extend services by offering a brief intervention (four to six sessions) to families with children who screen positive for adversity exposure. As Zigler and colleagues have argued, parents and caregivers play a central role in children's behavioral development (Zigler, Pfannenstiel, & Seitz, Reference Zigler, Shonkoff and Meisels2008) and, correspondingly, their response to behavioral health interventions (Sun, Rith-Najarian, Williamson, & Chorpita, Reference Sun, Rith-Najarian, Williamson and Chorpita2019). Accordingly, our brief intervention will introduce coping strategies for the children and adolescents, and parenting strategies for caregivers.
Though we ought not simply repackage treatment strategies for preventive purposes, we unfortunately lack strongly supported prevention strategies for adversity-exposed youth. In contrast, the development of new intervention strategies for youth mental health treatments seems to have reached saturation (Okamura et al., Reference Okamura, Orimoto, Nakamura, Chang, Chorpita and Beidas2020). As such, the vast array of intervention components from youth treatment programs may still provide some candidates for our prevention purposes, though other prevention strategies likely remain to be discovered through empirical research. Given that experiences of adversity are associated with psychopathology transdiagnostically (e.g., Keyes et al., Reference Keyes, Eaton, Krueger, McLaughlin, Wall, Grant and Hasin2012), it is essential that intervention strategies are flexible enough to target a range of internalizing and externalizing problems and the variable needs of children and families. Accordingly, we selected two evidence-based interventions – each of which includes multiple intervention elements – from which to pull candidate intervention strategies: FIRST (Weisz & Bearman, Reference Weisz and Bearman2020) and PCIT-ED (Luby, Barch, Whalen, Tillman, & Freedland, Reference Luby, Barch, Whalen, Tillman and Freedland2018).
Selection of candidate intervention strategies
FIRST (Feeling calm, Increasing motivation, Repairing thoughts, Solving problems, Trying the opposite)
FIRST is a good fit to our goals for several reasons. First, it is transdiagnostic, encompassing intervention procedures for an array of problems and concerns, including posttraumatic stress, anxiety, depressed mood, and conduct problems. Second, it is designed for personalization – for tailoring to fit the specific combination of needs of each individual child and family. Third, it is streamlined, efficient and accessible to clinicians, organizing many specific evidence-based procedures within five broad principles of change. These five principles are Feeling calm, Increasing motivation, Repairing thoughts, Solving problems, and Trying the opposite (see Table 1 for details). Clinicians using FIRST are encouraged to identify which of these principles may be most relevant to the child they are working with, and then apply that principle in their work with the child, drawing from the specific procedures grouped beneath that principle, as shown in Table 1. Benchmarking trials of FIRST have shown rapid reductions in child problems and symptoms – assessed via standardized measures of internalizing and externalizing problems and via idiographic measures of the “top problems” most important to children and their caregivers – and with trajectories of improvement similar to those seen with markedly more complicated treatments that require substantially more training (Cho, Bearman, Woo, Weisz, & Hawley, Reference Cho, Bearman, Woo, Weisz and Hawley2020; Weisz, Bearman, Santucci, & Jensen-Doss, Reference Weisz, Bearman, Santucci and Jensen-Doss2017).
Specific intervention strategies within the FIRST principles
PCIT (parent–child interaction therapy)
PCIT is an intervention focused on improving parent–child interactions that has demonstrated large and sustained effects for treating youth with disruptive behavior problems (Brestan & Eyberg, Reference Brestan and Eyberg1998; McNeil & Hembree-Kigin, Reference McNeil and Hembree-Kigin2010; Thomas & Zimmer-Gembeck, Reference Thomas and Zimmer-Gembeck2007). Critical for our purposes, PCIT has also been shown to reduce the rate of future maltreatment when delivered to parents (or caregivers) who had abused their children (Chaffin et al., Reference Chaffin, Silovsky, Funderburk, Valle, Brestan, Balachova and Bonner2004; Timmer, Urquiza, Zebell, & McGrath, Reference Timmer, Urquiza, Zebell and McGrath2005). PCIT includes several intervention elements that are taught through behavioral observations of parent–child interactions and live coaching of these interactions by a therapist. PCIT includes two primary intervention components: child-directed interaction and parent-directed interaction. In child-directed interaction, parents learn a variety of techniques designed to enhance the parent–child relationship. These include techniques praise, reflecting, imitating, and describing the child's behavior, and providing contingent attention (i.e., attending to positive behaviors and ignoring negative behaviors). Parent-directed interaction provides parents with training in effective discipline strategies designed to increase child compliance and decrease disruptive behaviors. PCIT has been expanded to include additional elements designed to target internalizing problems (Carpenter, Puliafico, Kurtz, Pincus, & Comer, Reference Carpenter, Puliafico, Kurtz, Pincus and Comer2014). For our purposes, we focus on PCIT plus emotional development (PCIT-ED), which includes an additional module focused on parenting techniques to help children identify and regulate their emotional experiences (Lenze, Pautsch, & Luby, Reference Lenze, Pautsch and Luby2011). PCIT-ED has been shown to significantly reduce depression symptoms in both children and their caregivers (Luby et al., Reference Luby, Barch, Whalen, Tillman and Freedland2018; Luby, Lenze, & Tillman, Reference Luby, Lenze and Tillman2012). More importantly for prevention, another trial showed that, relative to a waitlist, PCIT-ED produced greater changes in reward processing, which were in turn associated with greater improvements in depression symptoms (Barch et al., Reference Barch, Whalen, Gilbert, Kelly, Kappenman, Hajcak and Luby2019). Thus, this research suggests that some interventions designed as treatments for youth psychopathology may also influence key developmental mechanisms involved in the link between adversity and psychopathology.
Study aims
The success of prevention and early intervention programs depends in large part on their design, and incorporated elements should be preventive as opposed to purely remedial (Zigler, Reference Zeman and Garber1996). On the one hand, many children and adolescents who screen positive for adversity exposure may already be symptomatic. Thus, these youth will likely be well served by intervention strategies from FIRST and PCIT-ED. On the other hand, many other youth who screen positive may not yet be experiencing clinically significant mental health symptoms. Therefore, we still need to identify which candidate intervention strategies are more likely to target not only symptoms but also the neurodevelopmental mechanisms that put a youth at risk of eventual symptom onset (i.e., threat-related social information processing biases, heightened emotional reactivity, difficulties with emotion regulation, and blunted reward processing; McLaughlin et al., Reference McLaughlin, DeCross, Jovanovic and Tottenham2019). Such an approach would be consistent with the American Psychological Association (APA) prevention guidelines to develop programs that reduce risks and promote human strengths, as opposed to strategies that focus only on symptom reduction (American Psychological Association, 2014).
To this end, we conducted a modified Delphi study to gather feedback on the intervention strategies that we selected as candidates for our prevention program for adversity-exposed youth. A lack of expert consensus about which youth mental health interventions are considered “evidence based” has been a longstanding issue related to the science-to-practice gap (Weisz, Sandler, Durlak, & Anton, Reference Weisz, Kuppens, Ng, Eckshtain, Ugueto, Vaughn-Coaxum and Fordwood2005), and that issue is even more salient when developing a novel intervention. Consensus methods can enhance decision making in health research by systematically assessing expert recommendations for topics that have a dearth of scientific agreement or evidence (Trevelyan & Robinson, Reference Trevelyan and Robinson2015). It has been recommended to combine the existing scientific evidence base with the current widespread professional norms (i.e., expert opinions) to maximize effective clinical care (Campbell, Roland, & Buetow, Reference Campbell, Roland and Buetow2000). It has also been noted that, when using Delphi methods in mental health research, the quality of input from experts depends on the quality of input provided to them (e.g., systematic reviews, personal experience) (Jorm, Reference Jorm2015). Systematic reviews are a gold standard for assessing the current state of effectiveness for youth treatments and prevention programs (Weisz et al., Reference Weisz, Kuppens, Ng, Eckshtain, Ugueto, Vaughn-Coaxum and Fordwood2005) and have been specifically recommended as a strong form of evidence for identifying effective components of child maltreatment interventions (Powell et al., Reference Powell, Bosk, Wilen, Danko, Van Scoyoc, Banman, Daro, Donnelly, Huang and Powell2015). We thus planned to synthesize feedback across survey rounds as informed by two sources of knowledge – experts and meta-analyses.
Our primary aim was to evaluate consensus on recommendations and strength of evidence ratings for the candidate intervention strategies. We recognize that reaching consensus on recommendations is not necessarily proof that those interventions are the “best” or “correct” recommendations (Trevelyan & Robinson, Reference Trevelyan and Robinson2015). Still, even if there is not always a single best solution, it is helpful to prioritize options from a list of candidates (Hall, Reference Hall and Lentz2009). We also believe that a more extensive approach to planning intervention design is consistent with the cautionary notes of Zigler regarding “hasty” prevention development and implementation (Zigler, Reference Zeman and Garber1996; Zigler & Styfco, Reference Zigler and Styfco2001). The findings of the current study could provide some foundational evidence to more strongly hypothesize which intervention strategies might be mechanism-targeting prevention approaches. The results of skills recommendations can be directly applied to our intervention development and, later, their actual effects on mechanisms and symptoms empirically tested in future clinical trials of the prevention program. An exploratory additional aim was to gain insight into how evidence-informed decision making in intervention development unfolds.
Method
All procedures were reviewed and approved by the Institutional Review Board of Harvard University.
Overview of the Delphi approach
Traditional Delphi studies aim to gather expert input iteratively over multiple rounds until consensus is achieved, with the first round generating qualitative data that then informs the subsequent items (Hsu & Sandford, Reference Hsu and Sandford2007). However, given that our aim was not necessarily to reach consensus but to assess for consensus, we conducted a modified Delphi study. First, we invited experts to recommend candidate intervention strategies based on the five broad principles from FIRST (Weisz & Bearman, Reference Weisz and Bearman2020) and three parenting practices from PCIT-ED (Luby et al., Reference Luby, Barch, Whalen, Tillman and Freedland2018). Second, we invited a subset of experts – members of our broader study team – to identify relevant systematic reviews and then consider the strength of evidence for intervention strategies based on meta-analytic support for practice elements. Finally, our team reviewed anonymous and objective summaries of the recommendations from experts and systematic reviews together. This approach was designed to produce more informed conclusions than we could reach as individuals, while addressing some limitations of group decision making (e.g., influence of relative authority figures, tendency towards group conformity or polarization) (Hsu & Sandford, Reference Hsu and Sandford2007; Surowiecki, Reference Surowiecki2004).
Survey 1
Participants
We identified experts whose research focuses on child and/or adolescent mental health in at least one of the following categories of relevance: developmental psychopathology research focused on mechanisms linking adversity with psychopathology; prevention science; intervention development or treatment-focused research; and dissemination and implementation science. Potential participants were identified through four sources: (a) the principal investigator (PI), coinvestigators (CIs), and established consultants of the intervention research project; (b) scholars nominated by these members of the research team as experts in the domains of interest; (c) members of relevant professional task forces and executive boards; and (d) PIs of relevant preventive intervention trials currently active on clinicaltrials.gov. Participants were eligible to participate if the majority of their work was in one of the categories of interest (as determined by their professional website or CV) and they were still active in their career (e.g., not retired, had published research in one of the categories of interest within the past 2 years). We focused on identifying experts across a range of populations served (e.g., racial and ethnic minorities, rural and urban populations, and so on). A list of potential participants was iteratively compiled and revised to limit redundancy and broaden representative coverage of expertise areas. Recruitment was conducted by the PI/CIs through email invitations that explained the purpose of the study, the rationale for expert selection, and anonymity of participation. The study PI/CIs and previously hired consultants did not receive financial compensation, while all remaining invited experts were offered a payment of $100.
In total, 46 experts, including the four PI/CIs, were invited to participate in the expert recommendations survey, of which 32 (69.6%) provided consent and completed the survey. Demographic data for the participating sample (n = 32) are provided in Table 2.
Demographics of participants in Survey 1
a Note: Respondents could select multiples, so percentages may exceed 100%.
Procedure and measures
The participants responded to an online survey that assessed demographic information and asked them to recommend intervention strategies that would be most effective in targeting our four developmental mechanisms of interest. The survey also asked participants to recommend intervention strategies for four types of psychopathology symptoms that commonly occur in children exposed to adversity, including symptoms of disruptive behavior problems, depression, anxiety, and PTSD. As such, intervention recommendations were provided for a total of eight targets – four about developmental mechanisms linking adversity and psychopathology, and four about mental health symptoms. In each subsection, the survey provided examples of adversity experiences that have been commonly associated with each developmental mechanism or symptom type, and included examples of how each developmental mechanism or symptom type might present clinically or subclinically (e.g., heightened emotional reactivity may present as rapid mood changes or being quickly consumed by intense emotions; depressive symptoms may present as sad mood or loss of pleasure in activities that used to be enjoyable). Examples of each developmental mechanism were taken from standardized measures or published instruments, including: the Behavioral Inhibition System/Behavioral Activation System scales for blunted reward processing (Carver & White, Reference Carver and White1994); the Emotion Reactivity Scale (Nock, Wedig, Holmberg, & Hooley, Reference Nock, Wedig, Holmberg and Hooley2008) for heightened emotional reactivity; the Children's Response Styles Questionnaire (Abela, Vanderbilt, & Rochon, Reference Abela, Vanderbilt and Rochon2004) and Zeman and Garber's (Reference Zapf, Castell, Morawietz and Karch2016) emotion regulation vignette coding system for emotion regulation difficulties; and the Social Information Processing Application (Kupersmidt, Stelter, & Dodge, Reference Kupersmidt, Stelter and Dodge2011) for social information processing biases. Examples for each of the mental health symptoms were adapted from the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychiatric Association, 2013) to remove “clinical-level” qualifying language. Anxiety symptom examples were informed by the symptom criteria in the generalized anxiety disorder, panic disorder, social anxiety disorder, specific phobia, and separation anxiety disorder sections of the DSM-5. Depression symptom examples were informed by the symptom criteria in the major depressive disorder and persistent depressive disorder sections. Posttraumatic stress symptom examples were informed by the symptom criteria in the PTSD section. Disruptive behavior examples were informed by the symptom criteria in the conduct disorder and oppositional defiant disorder sections.
The survey instructed participants to recommend specific intervention strategies that could be used for children and adolescents who have experienced adversity and were exhibiting either a specific developmental mechanism or symptom pattern. Participants were told that all youth would have received case management prior to intervention and that the goal was to develop a preventive intervention rather than treatment. Participants were told to select a single answer they felt best addressed the specific presentation and were reminded that there were no correct answers. For all recommendations, participants were asked to select their primary recommendations and secondary recommendations (optional) for the most relevant intervention principles from FIRST. Participants were provided with a written description and a visual of FIRST and its five core treatment principles – Feeling calm, Increasing motivation, Repairing thoughts, Solving problems, and Trying the opposite. Participants were asked to make separate recommendations for younger (i.e., children aged 8–12 years) and older (i.e., adolescents aged 13–15 years) presentations. Participants were also asked if they would make additional recommendations beyond the FIRST principles and if their recommendations would change for children and adolescents of certain demographics. Finally, participants were asked to separately rank the three parenting practices – Increasing positive reinforcement, Decreasing caregiver criticism and hostility, and Coaching of child emotional competence – in order of importance for a brief preventive intervention for adversity-exposed children and adolescents. Although training in specific discipline strategies is also a core element of PCIT, we did not include this in our list of parenting practices as these techniques are subsumed within the Increasing motivation principle of FIRST. Throughout the survey, participants were allowed to provide qualitative feedback or explanations for their responses.
Survey 2
Participants
Members of our research team (i.e., PI/CIs, staff, graduate students) were invited to participate via an email sent from the study lead (LRN). The recruitment email emphasized that participation was voluntary and no compensation was provided for completing the survey. Of the eight individuals who were invited to participate, six provided consent and completed the full survey. This survey intentionally took place after Survey 1 had closed to ensure that the study investigators provided their prior responses before systematically reviewing the relevant meta-analyses that were incorporated in Survey 2.
Literature search for relevant systematic reviews
Members of the study team identified meta-analyses through an informal literature search and article nomination process (i.e., no specified databases or mandatory key terms). The search aimed to identify recently published systematic reviews and meta-analyses on “common elements,” “practice elements,” or “dismantling studies” of clinical trials, with a particular focus on identifying articles that examined prevention programs, a range of internalizing/externalizing problems, a range of ages (i.e., early childhood through adolescence), and (when possible) samples of adversity-exposed youth, or interventions conducted through primary care gatekeeping. Included studies met the following criteria: (a) they examined the effects of prevention programs (at the universal, selective, or indicated level) and (b) they reported effect sizes by common elements or intervention strategies. Meta-analyses that examined both prevention and intervention programs were included if they reported effect sizes for prevention programs separately from treatment programs.
Round 1: Meta-analysis selection
The informal literature search yielded 15 potential meta-analyses. The study team members reviewed the identified articles and voted to select the final group of meta-analyses. They were instructed to vote for the meta-analyses that they felt were “most useful in informing” the design of a preventive intervention. They were told to consider relevance (i.e., prevention programs, special samples, age ranges), breadth of targets covered within and across meta-analyses, and the quality of analysis (e.g., meta-regression was considered highest quality). Meta-analyses were included if more than half of the reviewers (n = 6) voted for their inclusion, resulting in a final sample of seven meta-analyses (Caldwell et al., Reference Caldwell, Davies, Hetrick, Palmer, Caro, López-López and Welton2019; Filene, Kaminski, Valle, & Cachat, Reference Filene, Kaminski, Valle and Cachat2013; Leijten et al., Reference Leijten, Gardner, Melendez-Torres, van Aar, Hutchings, Schulz and Overbeek2019; Sanchez et al., Reference Sanchez, Cornacchio, Poznanski, Golik, Chou and Comer2018; Schleider & Weisz, Reference Schleider and Weisz2017; Singla et al., Reference Singla, Waqas, Hamdani, Suleman, Zafar, Zill-e-Huma and Rahman2020; van der Put, Assink, Gubbels, & Boekhout van Solinge, Reference van der Put, Assink, Gubbels and Boekhout van Solinge2018).
Round 2: Survey data collection
Participants were provided with links and were instructed to read the final set of meta-analyses. In addition, the survey instructions summarized key information from each meta-analysis, including a list of which age groups, settings, and clinical targets were reviewed, as well as key tables and figures. For each meta-analysis, participants were asked to rate the clinical practice elements/intervention components (e.g., psychoeducation, contingency management, relaxation) that had the strongest support indicated for treatment and prevention programs. Participants were also asked to indicate which of the five FIRST principles (Feeling calm, Increasing motivation, Repairing thoughts, Solving problems, and Trying the opposite) and three parenting practices (Increasing positive reinforcement, Decreasing caregiver criticism and hostility, and Coaching of child emotional competence) were supported by each meta-analysis. After responding to questions for each individual meta-analysis, participants were provided with a summary of their responses and a spreadsheet that aggregated all the effect sizes reported across the meta-analyses. Participants were asked to rank the top practice elements/intervention components that they felt had the “strongest evidence” (a) across all targets and treatment/prevention programs, (b) across all prevention programs only, and (c) and that were relevant for a “brief (two to six sessions) prevention-level service offered to adversity-exposed youth (age 8–15) and their families who were screened in via primary care.” Finally, participants were instructed to rate their perceptions of support for the FIRST principles and parenting practices on a 5-point Likert-type scale, ranging from “extremely effective” to “not effective at all.” For all questions there was an open-text field in which participants could write notes elaborating on their decision making. Those with a bachelor's degree were both currently enrolled in a clinical psychology doctoral program.
Survey 3
Participants
The same eight individuals invited for Study 2 were invited in the same way for Study 3, all of whom provided consent and completed the full survey. These individuals were selected to participate because of their knowledge of the ultimate goal of the project – to develop a brief intervention for adversity-exposed youth. All participants had completed one or both of the prior surveys. This sample was 75% female, 88% white with one respondent identifying as biracial, and the highest educational degree levels were doctoral (50%), master's (25%), and bachelor's (25%) degrees. Those with a bachelor's degree were both currently enrolled in a clinical psychology doctoral program.
Summarizing the findings from Survey 1 and Survey 2
The study lead (LRN) consulted with KM and JW, then compiled a summary of the results and qualitative feedback from Surveys 1 and 2. Based on findings from the prior two rounds, some recommendation results were collapsed across age groups rather than presented separately. The study lead distributed the results summary and instructed participants to focus their review on the quantitative results summaries from both surveys as well as overall qualitative feedback from Surveys 1 and 2. Participants were encouraged to read additional qualitative feedback on each target from Survey 1 as desired or needed.
Data collection
Survey 3 instructed participants to rate the FIRST principles and parenting practices in a similar manner to Survey 2. Based on the findings from Survey 1 that suggested recommendations did not differ significantly by age, participants provided recommendations irrespective of child age. Based on qualitative data from Survey 2, participants were also instructed to provide their recommendations in three different ways: first, by using responses from the aggregated evidence and recommendations from the results of the two prior surveys; second, by also considering their own knowledge and experiences; and third, by ranking the skills they would prioritize including in the planned intervention. Participants were also invited to provide open-ended responses explaining any changes in their recommendations for this final survey.
Data analyses
We used a similar statistical approach for analyzing the data from all three surveys. To evaluate recommendations, rankings, and ratings across respondents, it is important to differentiate between stability, agreement, and consensus (Gisev, Bell, & Chen, Reference Gisev, Bell and Chen2013; Trevelyan & Robinson, Reference Trevelyan and Robinson2015). For Delphi studies, the stability of responses can be assessed by measurements of inter-rater reliability, such as intraclass correlation coefficients (ICCs) (Trevelyan & Robinson, Reference Trevelyan and Robinson2015), whereas consensus of responses can be assessed with measurements of variation, such as interquartile range (Trevelyan & Robinson, Reference Trevelyan and Robinson2015), or measurements such as Kendall's coefficient of concordance (Gisev et al., Reference Gisev, Bell and Chen2013). As such, we first assessed the stability of responses by calculating Krippendorff's alphas (α) for nominal data and ICCs for ordinal data. We selected Krippendorff's α, as opposed to Fleiss’ K, because they yield similar results but α is more robust to missing data (Zapf, Castell, Morawietz, & Karch, Reference Witteman2016). We selected ICC(3,k) to test two-way mixed effects of average measures for multiple raters (Koo & Li, Reference Koo and Li2016) for assessing consistency (<.4 = poor, .4 = fair, .6 = good, .75 = excellent; Cicchetti, Reference Cicchetti1994). Next, for assessing consensus of responses, we calculated variation ratios for nominal data and Kendall's coefficient of concordance for ordinal data. The variation ratio represents spread of responses (VR = 1 − proportion of cases in the mode; Weisberg, Reference Weisberg and Weisberg1992)), with scores from 0 to 1 indicating less to more dispersion. Kendall's coefficient of concordance (W, or Wt when corrected for ties) can assess the relative strength of agreement (.3 = low, .5 = fair, .7 = high, .9 = very high) by 3+ raters for rank order data (Schmidt, Reference Schmidt1997). Given the number of participants and questions asked in Survey 1, multiple values of VR and α were calculated for assessing the recommendations overall, by subtype (e.g., primary vs. secondary), and by participant type (e.g., self-reported expertise). In addition, for Survey 1, we only ran chi-squared analyses to determine if the final presentation of recommendations should be collapsed or separated by age group. Finally, for all data types, we calculated frequencies to assess which responses were most commonly endorsed. All VRs were calculated in Microsoft Excel and all remaining analyses were conducted with RStudio version 3.6.0 (R Core Team, 2019) using the irr package (Gamer, Lemon, Fellows, & Singh, Reference Gamer, Lemon, Fellows and Singh2019).
Results
Survey 1
Frequency of FIRST skills recommendations
Examining recommendations for the eight targets individually, chi-squared analyses (or Fisher's exact tests in instances of low cell count) compared the frequency of skills recommendations for children (age 8–12 years) versus adolescents (age 13–15 years). These analyses showed that none of the recommendations significantly differed by age (p values from .28 to .99), so we collapsed the recommendations across both age groups. The frequencies of skills recommendations by target are presented graphically in Figure 1.
Frequencies of recommended FIRST principles by targets (Survey 1).
Stability of FIRST skills recommendations
The overall Krippendorff's α for primary skills recommendations was .27, suggesting little stability across the intervention skills recommended across all developmental mechanisms and symptom types. There was a similar α value for recommendations on developmental mechanisms overall (α = .27) and symptom types overall (α = .25). Examining α values for skill recommendations for individual targets (e.g., heightened emotion reactivity, anxiety), no stability was observed regardless of collapsing or separating recommendation by age groups (α values from −.03 to .00). The α values varied slightly between participant groups, with the highest stability of responses by study investigators (n = 4, α = .36), followed by external experts (n = 25, α = .26), and then study consultants (n = 3, α = .23). The α values also varied slightly by self-reported expertise, with the highest stability of responses by dissemination and implementation researchers (n = 14, α = .29), followed by treatment researchers (n = 17, α = .28), researchers focused on developmental mechanisms (n = 18, α = .22), childhood adversity researchers (n = 18, α = .22), and prevention researchers (n = 14, α = .18).
Consensus of FIRST skills recommendations
The overall variation ratio (VR) from primary skills recommendations was .39, indicating that 39% of recommendations fell outside each respective modal recommendation. For reference, the “at chance” VR for items in this survey would be .80, with lower values reflecting higher consensus. There was slightly better consensus for primary skills recommendations for developmental mechanisms (overall VR = .36) than for symptom types (overall VR = .42).
There was quite a range in VR values when examining primary skill recommendations by individual targets, with the highest consensus for disruptive behavior (collapsed age VR = .19) and the lowest consensus for PTSD symptoms (collapsed age VR = .58) (see Figure 1 for recommendations for each FIRST skill). Compared with the primary skills recommendations, secondary skill recommendations had worse consensus overall (VR = .72) and for individual targets (VR range from .67 to .75). As such, we discontinued the evaluation of secondary skills recommendations. The VR values varied slightly between participant groups, with the highest consensus of responses by study investigators (n = 4, VR = .27), followed by study consultants (n = 3, VR = .33), and then external experts (n = 25, VR = .26). The VR values also varied slightly by self-reported expertise, with the highest consensus of responses by dissemination and implementation researchers (n = 14, VR = .36), followed by treatment researchers (n = 17, VR = .39), childhood adversity researchers (n = 18, VR = .40), developmental mechanisms-focused researchers (n = 18, VR = .41), and prevention researchers (n = 14, VR = .43).
Ranking of parenting practices
Examining the ranking of recommended parenting practices, stability was excellent both for children (ICC(C,29) = .83) and adolescents (ICC(C,29) = .90). However, consensus was very low both for children (Wt = .17) and adolescents (Wt = .27). A chi-squared analysis revealed that the first ranked parenting practice did not differ by age group (p = .4); the rankings of parenting practices for each age group are presented in Figure 2.
Ranking of parenting practices for younger (ages 8–12) and older (ages 13–15) presentations (Survey 1).
Survey 2
Intervention skills broadly
From a compiled list of 33 intervention components/practice elements that were reviewed in the seven screened meta-analyses, respondents were asked which they would select as the top five with “strongest evidence” for different purposes. Stability across respondents (n = 6) was low for top five skills selected overall (α values from .29 to .37). However, consensus was very good for the selection of the top five skills overall (VR = .11), considering prevention programs only (VR = .11), and considering programs that were most similar to our planned early adversity intervention (VR = .09). For reference, the “at chance” VR for these cases would be .83. Moreover, about 20 options were never ranked by any respondent, showing some collective discrimination of elements that were clearly not in the top five. Figure 3 shows those skills ranked as the top five by respondents for our prevention program purposes.
Skills ranked as top five based on review of meta-analyses (Survey 2).
FIRST skills and parenting practices
Examining respondents’ strength of evidence ratings for the five FIRST skills, stability was excellent (ICC(C,6) = .88) and consensus was high (Wt = .85). The five intervention strategies ranked as having the strongest evidence were discipline/behavior management, parenting skills for positive reinforcement, improving parent–child relationship/interactions, problem-solving for parents, and relaxation skills (see Figure 4). Examining respondents’ strength of evidence ratings for the three parenting practices of interest, stability was excellent (ICC(C,6) = .96) and consensus was high (Wt = .81).
Strength of evidence ratings based on review of meta-analyses (Survey 2).
Survey 3
FIRST skills ratings and rankings
When respondents (n = 8) were instructed to rate each FIRST skill according to the evidence they reviewed from the results of Survey 1 (expert recommendations) and Survey 2 (meta-analysis review), stability was excellent (ICC(C,8) = .94) and consensus was high (Wt = .75) (see Figure 5 for average ratings). When respondents were instructed that they could also consider knowledge from their own experiences, stability was still excellent (ICC(C,8) = .85) but consensus was moderate (Wt = .60). Relative to the ratings presented in Figure 5, these second ratings based also on respondents' individual knowledge produced the following changes (as measure by standardized mean difference; SMD): Feeling calm had a small decrease (SMD = −0.43); Increasing motivation had a small increase (SMD = 0.23); Repairing thoughts, Solving problems, and Trying the opposite had a moderate increase (SMD = 0.57 to 0.66). Finally, when respondents rank-ordered the skills to prioritize for our planned intervention, stability was still excellent (ICC(C,8) = .97) and consensus was high (Wt = .82), with the rank order of FIRST skills exactly following the strength of evidence ratings order as per Figure 5.
Strength of evidence ratings based on respondents' review of prior rounds (Survey 3).
Parenting practice ratings and rankings
When respondents were instructed to rate each parenting practice based on evidence from Surveys 1 and 2, stability was excellent (ICC(C,8) = .98) and consensus was very high (Wt = .97) (see Figure 5 for average ratings). When respondents were instructed to also consider their own past knowledge, stability was again excellent (ICC(C,8) = .97), while consensus decreased but was still high (Wt = .88), with the following changes: Increasing positive reinforcement stayed at the maximum possible rating, Decreasing caregiver criticism and hostility had a small increase (SMD = 0.40), and Coaching of child emotional competence had a large increase (SMD = 0.86). Finally, when respondents rank-ordered the parenting practices, stability was still excellent (ICC(C,8) = .98) and consensus was still high (Wt = .89), with the rank order again exactly following the strength of evidence ratings order as per Figure 5.
Final stability and consensus of FIRST skills recommendations
The overall α value for skills recommendations was .60, suggesting moderate stability. There was also moderate stability for recommendations by developmental mechanisms overall (α = .58) and symptom types overall (α = .52). As for consensus, the overall VR was .22, an improvement over the consensus of Survey 1. The “at chance” VR for these items is .80. The consensus was identical for skills recommendations for developmental mechanisms (overall VR = .22) and symptom types (overall VR = .22).
Regarding skill recommendations for individual targets, we reached absolute stability (ICC(C,8) = 1.00) and consensus (VR = .00) for four targets – Feeling calm was recommended by all participants for targeting heightened emotional reactivity; Increasing motivation for targeting disruptive behavior; Repairing thoughts for targeting social information processing biases; and Trying the opposite for targeting depressive symptoms. For blunted reward processing, consensus was split (VR = .5) and thus stability was very low (α = -.12), with 50% (n = 4) of respondents recommending Increasing motivation and 50% recommending Trying the opposite. The α value was −.12 and the VR was .38 for the remaining three targets of emotion regulation difficulties, anxiety symptoms, and posttraumatic stress symptoms (see Figure 6 for details).
Survey 3: Frequencies of recommended FIRST principles by target.
Qualitative feedback across surveys
Although a comprehensive qualitative analysis is beyond the purview of this paper, selected responses may be helpful to contextualize quantitative findings. We have provided some exemplar quotes from each survey in Table 3 to demonstrate how respondents explained their decision making process or elaborated on rationales, qualifiers, or additions to their recommendations.
Selected qualitative data across surveys
Discussion
In the service of developing an intervention to prevent the mental health consequences of childhood adversity, we conducted a modified Delphi study to assess intervention strategy recommendations and ratings based on expert feedback and review of the existing intervention evidence base. Over three survey rounds, we found increasing consensus on intervention strategies regarding recommendations for individual targets, strength of evidence ratings, and ranking order. Certain intervention strategies for youth skills and parenting practices emerged as stronger candidates, some with consensus for targeting specific developmental mechanisms and symptoms. In addition, our findings produced useful information about the process of evidence-informed intervention decision making.
First, we were interested in investigating which candidate intervention strategies – specifically, youth skills drawn from FIRST (Weisz & Bearman, Reference Weisz and Bearman2020) and parenting practices drawn from PCIT-ED (Luby et al., Reference Luby, Barch, Whalen, Tillman and Freedland2018) – were recommended for the developmental mechanisms and symptoms we intend to target. To do so, we first asked a group of experts (Survey 1). Consensus in these recommendations can be interpreted from the lower VR values and higher W values within the survey rounds, and by relative decreases or increases between rounds (Gisev et al., Reference Gisev, Bell and Chen2013; Holey, Feeley, Dixon, & Whittaker, Reference Holey, Feeley, Dixon and Whittaker2007). For the youth skills recommendations in Survey 1, we found very low stability and low consensus overall, though one or two modal recommendations emerged for all eight targets. Overall, the most recommended youth skills in Survey 1 were Feeling calm, Increasing motivation, and Trying the opposite; Solving problems was rarely recommended. Regarding parenting practices recommended for our intervention overall, the experts in Survey 1 had excellent stability of rankings, but very low consensus. Low consensus reflects that respondents are divergent from each other, whereas high stability reflects that they are diverging in a consistent way. The parenting practice Coaching of child emotional competence was clearly ranked lowest of the three. Of note, consensus of the recommendations in Survey 1 did not appear to depend on youth age, respondent's affiliation with our study team, or self-reported expertise type. In Survey 3, we invited only study team members to complete these recommendations a second time after they had reviewed summaries of the recommendations from the broader group of experts. This time, stability and consensus substantially improved for recommendations of both youth skills and parenting practices. The final most frequently recommended skills were again Feeling calm, Increasing motivation, and Trying the opposite; Solving problems was never recommended. Youth skills recommendations reached 100% consensus in Survey 3 for four of the eight targets – Feeling calm for heightened emotional reactivity, Increasing motivation for disruptive behavior symptoms, Repairing thoughts for social information processing biases, and Trying the opposite for depressive symptoms – and was split 50/50 between Increasing motivation and Trying the opposite for blunted reward processing. For the final parenting practice recommendations in Survey 3, Increasing positive reinforcement emerged as the unanimous first parenting practice to prioritize and Coaching of child emotional competence remained the last.
We were also interested in investigating which intervention strategies were most strongly supported by existing empirical evidence. To do so, we identified and reviewed relevant meta-analyses and rated which intervention elements had the strongest evidence related to our intended prevention program. In Survey 2, although there were more than 30 practice elements and intervention strategies presented across the reviewed meta-analyses, the respondents showed good consensus for identifying the intervention elements that had the strongest effects. When asked which intervention strategies had the most evidence for our prevention program, at least two thirds of our study team selected the following within their top five: discipline skills/behavior management; parenting skills for positive reinforcement; improving parent–child relationships/interactions; problem-solving skills for parents; and relaxation strategies. Respondents were then asked to translate their selections into strength of evidence for our candidate FIRST youth skills and parenting practices, and the resultant ratings had excellent stability and high consensus. According to Survey 2, strength of evidence ratings were “extremely effective” for Increasing motivation (a skill that targets both youth and parent behaviors; see Table 1) and Increasing positive reinforcement (parenting practice), “moderately effective” for Feeling calm (youth skill), Solving problems (youth skill), and Decreasing caregiver criticism and hostility (parenting practice), and “slightly effective” for Trying the opposite (youth skill), Repairing thoughts (youth skill), and Coaching of child emotional competence (parenting practice). Strength of evidence ratings in Survey 3 again had excellent stability, while consensus for youth skills was still high but somewhat decreased, and consensus for parenting practices was very high, a relative increase. According to Survey 3, the final strength of evidence ratings stayed the same for Increasing motivation, Increasing positive reinforcement, Repairing thoughts, and Coaching of child emotional competence. The changes in strength of evidence ratings from Survey 2 to Survey 3 were an increase to “very effective” for Feeling calm and Decreasing caregiver criticism and hostility, an increase to “moderately effective” for Trying the opposite, and a decrease to “slightly effective” for Solving problems.
Implications for preventing mental health consequences of childhood adversity
Our primary aim in investigating these research questions was to evaluate intervention strategy consensus so that findings could inform the development of our prevention program for youth with varying presentations. As stability and consensus improved across surveys, there was some convergence with Feeling calm (from FIRST), Increasing motivation (from FIRST), Trying the opposite (from FIRST), and Increasing positive reinforcement (from PCIT) receiving the most frequent recommendations, higher rankings, and stronger strength of evidence ratings. The fact that these same intervention strategies were the most highly recommended/rated across all three surveys suggests that consensus is generalizable beyond our study team. These intervention strategies are also conceptually similar to the practice elements with strong effects in relevant meta-analyses. As examples, relaxation (which was included in two of the meta-analyses) is the central intervention strategy of the Feeling calm principle in FIRST, and positive attending, improving parent–child relationships, positive reinforcement strategies, and other behavior management skills (at least one of which was included in all seven meta-analyses) are core skills in the Increasing motivation principle of FIRST and the Increasing positive reinforcement element of PCIT. Although it is important to highlight that intervention strategies with high consensus do not guarantee that those strategies will necessarily be most effective (Trevelyan & Robinson, Reference Trevelyan and Robinson2015), the current findings certainly can help us prioritize our best candidates, given the currently existing evidence. In particular, given that our planned prevention program will be brief (up to six sessions), it may be useful to focus on Feeling calm, Increasing motivation, and Trying the opposite as the most relevant intervention elements from FIRST, while deprioritizing Repairing thoughts and Solving problems. Similarly, Increasing positive reinforcement is the parenting practice that could be prioritized above both Decreasing caregiver criticism and hostility and Coaching of child emotional competence.
One particularly encouraging aspect of our findings was reaching consensus on intervention strategies for some of our target developmental mechanisms. Many psychosocial interventions are designed for targeting symptoms, which necessarily involves more remedial intervention strategies rather than preventive strategies that reduce risks and promote strengths (American Psychological Association, 2014). Moreover, many clinical psychologists engage in intervention decision making in a symptom-driven way (Bakker, Reference Bakker2019) – part of the motivation for developing the APA prevention guidelines was that, despite the increased focus on prevention within clinical psychology, there is little understanding of the best preventive practices (American Psychological Association, 2014). Considering how little guidance and experience clinical psychologists have regarding prevention strategies, it was notable that we achieved 100% or 50/50 consensus for skill recommendations for three of our four developmental mechanisms (i.e., heightened emotional reactivity, social information processing biases, and blunted reward processing). Moreover, even before we reached consensus, these mechanisms already had modal skill recommendations in Survey 1, which indicates that the experts were not making recommendations randomly. It is not the case that each mechanism was simply treated as a proxy for some perceived parallel symptom (e.g., reward processing vs. depression); otherwise, the same four recommendations for symptom targets would have emerged for developmental mechanism targets. Qualitative data also supported this idea, as exemplified by one respondent describing how their intervention recommendation for depressive symptoms was informed by both the meta-analysis review (which covered depression targets) and recommendations “for blunted reward processing by experts – a mechanism closely related to depression.” Conceptually, the skills with strongest strength of evidence ratings largely involved risk reduction strategies (e.g., addressing positive parent–child relationship through Increasing positive reinforcement and Increasing motivation) and strengths promotion (e.g., Feeling calm as a coping strategy), as opposed to remedial skills that require some symptoms to be already present (e.g., exposure, cognitive restructuring). Thus, it appears possible for mental health professionals to apply their intervention decision making to planning a mechanisms-based preventive intervention.
Insights into evidence-informed clinical decision making
We had an additional exploratory aim, which was to gain insight into how evidence-informed decision making in intervention development unfolds. Our findings illuminate understanding of intervention decision making processes in four main ways.
First, the use of meta-analysis to inform intervention decisions is frequently recommended (Powell et al., Reference Powell, Bosk, Wilen, Danko, Van Scoyoc, Banman, Daro, Donnelly, Huang and Powell2015; Weisz et al., Reference Weisz, Kuppens, Ng, Eckshtain, Ugueto, Vaughn-Coaxum and Fordwood2005) yet rarely studied as a practice. Although meta-analyses and systematic reviews can, in theory, provide meaningful summaries of intervention evidence, there is little guidance on how to actually use them in practice when designing interventions. Our practice element ranking findings from Survey 2 demonstrate that even experts reviewing the same meta-analyses come to different conclusions about which intervention strategies emerged with the strongest evidence. Relatedly, respondents noted the difficulty of cognitively condensing and interpreting results across meta-analyses, especially for those with less familiarity with meta-analysis statistical methods. The difficulty in translating meta-analysis findings into intervention selections might explain some variation in the recommendations. For example, one respondent rated Solving problems as “moderately effective” based on some meta-analytic support of problem solving for caregivers, while another respondent acknowledged that same evidence but concluded low support for Solving problems because of other meta-analytic null results of problem solving for youth. Thus, although it is common practice to point to meta-analyses to decide best clinical practices, interpretation of meta-analytic findings is not actually a clear-cut task, even for a study team working closely together.
Second, our findings provide evidence that a Delphi study can help compensate for biases in decisions made by individuals alone or by groups without structured consensus building. The use of a Delphi study is intended to enhance group decision making by mitigating issues of bias and group dynamics (Hsu & Sandford, Reference Hsu and Sandford2007; Surowiecki, Reference Surowiecki2004). Moreover, much research has found that clinical decisions by individuals are affected by cognitive errors such as the availability heuristic, overconfidence, and hindsight bias (Lilienfeld & Lynn, Reference Lilienfeld, Lynn, Cautin and Lilienfeld2014; Witteman, Reference Whiteside, Sim, Morrow, Farah, Hilliker, Murad and Wang2020). To overcome these limitations, it has been recommended that we develop debiasing methods, or at least ways to systematically compensate for them (Lilienfeld, Reference Lilienfeld2019; Lilienfeld & Lynn, Reference Lilienfeld, Lynn, Cautin and Lilienfeld2014), which we believe could include the Delphi approach. The fact that the study investigators already had relatively high consensus on specific intervention recommendations in Survey 1, even before reviewing the meta-analyses in Survey 2, might indicate that some groupthink had already been occurring. The improved consensus reached in Survey 3 demonstrates how reviewing objective summaries of evidence and others’ ideas from prior survey rounds ultimately influenced the final decisions. One respondent confirmed this experience, saying “the aggregated results very much informed my thinking,” even allowing their review of prior results to override personal opinions when caught between two recommendations. Finally, when respondents provided their strength of evidence ratings in Survey 3, again based also on their own knowledge and experiences (i.e., beyond the information aggregated from experts and the meta-analyses), there was a decrease in consensus for both youth skills and parenting practices. This decrease was accompanied by a trend of respondents giving higher evidence ratings across intervention strategies, which makes sense given that our team selected these candidate skills in the first place. Together, our findings increase our confidence that a Delphi study can increase consensus in decisions about intervention selection while decreasing bias in group decisions.
Third, our findings are a reminder that aiming for consensus does not necessitate achieving consensus. Any Delphi study can be continuously iterative, so there must be a determination by researchers about when more rounds would no longer be useful (Hsu & Sandford, Reference Hsu and Sandford2007). Compared with more traditional Delphi methods, our modified Delphi study used a larger initial sample and then had only our smaller study team continue. Through this approach, our Survey 3 had the highest consensus levels and we reached maximum consensus for recommendations on half of our targets. However, our consensus on strength of evidence ratings slightly decreased from Survey 2 to Survey 3, along with the growing amount of information for respondents to review and consider. Given that we still achieved high consensus in Survey 3, it seemed that we may have reached a ceiling. This finding highlights how a consensus process may reach saturation, at which point perpetually synthesizing group feedback may provide diminishing clarity. Finally, consensus is not always the goal when there may not a single answer (Hall, Reference Hall and Lentz2009), with a good example being our 50/50 split on intervention recommendations for blunted reward processing.
Fourth, despite attempts at debiasing our clinical decisions, objective decision making is quite challenging. Qualitative feedback across surveys emphasizes this fact. Even though respondents were capable of providing intervention recommendations for the prevention mechanism targets, they still had some difficulty avoiding symptom-driven thinking. Some comments revealed that respondents were still considering clinical experiences or citing evidence related to treatment programs. For example, one respondent in Survey 2 admitted “biased interpretation from my own clinical understanding of working with [children]” after reviewing meta-analyses on prevention programs. Even by Survey 3, respondents were still referring to treatment recommendations (e.g., TF-CBT) and effective interventions for symptoms (e.g., “behavioral activation and exposure [for] mood and anxiety problems”) without making the distinction between treatment and prevention programs. Finally, the most notable example comes from some conflicting quantitative findings in Survey 3. On the one hand, we had 100% consensus for the recommendation of Repairing thoughts for the social information processing biases, despite also having high consensus in rating this skill with the lowest strength of evidence overall. On the other hand, none of the respondents recommended Solving problems for a single target, despite having rated Solving problems as “moderately effective” on average. In other words, within the same survey, we sometimes recommended intervention strategies for our prevention targets that contradicted our own rating of their strength of evidence for our overall prevention program. Though such recommendations may still hold some merit, these findings highlight inconsistence in how we apply clinical knowledge. Other research has found that that clinician intervention recommendations can be more reflective of personally held theories rather than current scientific evidence (de Kwaadsteniet & Hagmayer, Reference de Kwaadsteniet and Hagmayer2018). In summary, consensus does not solve all decision making problems and updating new knowledge into our pre-existing – and sometimes entrenched – frameworks remains a challenge for evidence-informed intervention decision making. These findings are somewhat reminiscent of sentiments from Zigler that it is difficult “to bring together politics, knowledge [..], and common sense” when developing standards for prevention content, and even “the best standards in the world are meaningless if they are not enforced” (Zigler, Reference Zeman and Garber1996, p. 38; Zigler & Styfco, Reference Zigler and Styfco2001, p. 10)
Limitations
Our study was subject to some limitations. First, interpretation of the findings was constrained by which intervention strategies we considered. As an example, we did not query respondents about computerized attention bias modification strategies, which are less commonly used in clinical settings, yet theoretically might be the most effective for threat-related social information processing biases (e.g., attention bias to threat; Bar-Haim, Reference Bar-Haim2010). The selected meta-analyses added to this constraint, as we likely missed some relevant reviews through our informal search procedures; furthermore, various intervention strategies are not equally represented in the published literature.
Second, the five FIRST principles, which served as our candidate intervention strategies for youth, vary in the breadth of skills contained within each module. Thus, while a respondent's recommendation of Feeling calm clearly involves relaxation strategies, a recommendation of Increasing motivation could be focused on one, some, or all of the encapsulated parent management skills (e.g., reinforcement strategies to promote positive behaviors, specific discipline strategies).
The third limitation was that the specific respondents varied across the three samples. However, we decided on this methodological approach to balance (a) representation – compared with most Delphi studies, we invited a larger group with greater diversity of research expertise and target populations (Cairns, Yap, Reavley, & Jorm, Reference Cairns, Yap, Reavley and Jorm2015), (b) feasibility – it seemed unrealistic to expect a group of experts to commit time to multiple surveys (Cairns et al., Reference Cairns, Yap, Reavley and Jorm2015; Hsu & Sandford, Reference Hsu and Sandford2007, and (c) quality of responses – our study team members had intrinsic motivation to thoughtfully review materials and could more readily provide recommendations with our specific intervention purposes in mind.
Fourth, not only does consensus on intervention recommendations from experts differ from effectiveness (as previously discussed), it also does not ensure acceptability. Client preferences predict their satisfaction with, completion of, and clinical response to intervention (Lindhiem, Bennett, Trentacosta, & McLear, Reference Lindhiem, Bennett, Trentacosta and McLear2014). Accordingly, it ultimately does not matter how evidence-based a preventive intervention is if primary care clinics and families who have experienced adversity do not want to participate. We are thus currently collecting stakeholder feedback from pediatric clinic providers and staff as well as families who have encountered adversity. Relatedly, to increase multicultural appropriateness as per APA prevention guidelines (American Psychological Association, 2014), we will (a) more thoroughly review expert qualitative feedback regarding multicultural recommendations and (b) specifically ask stakeholder families in our feasibility trial about cultural acceptability of materials and language adaptations.
Conclusion
Although many interventions have been developed to ameliorate the mental health consequences of childhood adversity, evidence of effectiveness for preventive interventions is scarce – particularly those that are brief and could be feasibly integrated into pediatric primary care. As such, we have yet to discover which prevention strategies are most effective for families with adversity-exposed youth. Simply borrowing and repackaging the most common treatment intervention strategies seems unlikely to advance effectiveness. Instead, our study team systematically considered evidence from experts and meta-analyses to determine which intervention strategies were the most relevant and most potent for our prevention purposes. Moreover, we invited recommendations on targeting four mechanisms related to the neurodevelopmental adaptations of childhood adversity exposure that may place youth at risk of later psychopathology. Given that not all adversity-exposed youth will be symptomatic at the time of screening, it is crucial that our prevention approach includes skills that are not dependent on symptom presentations. For example, behavioral exposures have shown stronger effectiveness than relaxation skills in the treatment of youth anxiety (Whiteside et al., Reference Weisz, Sandler, Durlak and Anton2020). However, exposures are essentially meaningless in the absence of impairing behavioral avoidance, whereas relaxation strategies, in contrast, can be used by anyone. Our findings thus give credence to that idea that strong intervention candidates for prevention programs will likely differ from those of treatment programs, in particular by including strategies that are conceptually more focused on risk reduction and strength promotion. Finally, our findings provided initial support for some candidate intervention strategies that could hypothetically target these developmental mechanisms. By developing our preventive program accordingly, we will now be able to empirically test the actual effects of the intervention strategies recommended through the current study. In the words of Zigler, once we “know what quality components are necessary to build successful interventions” we must then “put this wisdom to use” (Zigler, Reference Zeman and Garber1996, p. 42).
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0954579420002059.
Acknowledgments
We would like to thank our study team members – Shannon Dorsey, Arianna Riccio, Olivia Fitzpatrick, and Katherine Venturo-Conerly – for their contributions in the preparation of this study. We also thank Joan Luby for her ongoing consultation for this intervention development. Finally, we thank Beth Erickson and Deneen Vojta for their support of the ongoing implementation studies associated with this larger intervention research project.
Financial Statement
The presented research was supported by the Harvard University Dean's Competitive Fund for Promising Scholarship (KM). The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of the funding sponsor.
Conflicts of Interest
John Weisz is a coauthor of a treatment manual cited in this paper that will be used in the research described herein (Principle-guided psychotherapy for children and adolescents: The FIRST treatment program for behavioral and emotional problems, published in 2020 by Guilford Press (Weisz & Bearman, Reference Weisz and Bearman2020)). He receives royalties for its sales.
