Collaboration and Team Science

The benefit of collaboration and team science, broadly defined as experiences with other internal elements of ECHO, was the most common cross-cutting theme that emerged across the Intervention, Recipient, and Implementation Infrastructure domains. Many interviewees enjoyed being part of a national study that brings greater breadth and depth of expertise to the team and more exciting translational work and clinical research. One respondent explained:

Many felt that ECHO had enhanced the science of their specific cohort with its expanded focus aligned across multiple exposures and outcomes. However, some interviewees who felt they had willingly adapted their original study plans to meet overall ECHO goals were frustrated that some ECHO colleagues did not prioritize collaboration over their own research. Nonetheless, interviewees enjoyed opportunities to collaborate, both at large in-person and smaller group virtual meetings, which were seen as opportunities to learn from their ECHO colleagues. Many noted that while the first few years of the program felt chaotic, things had progressively stabilized and become more standardized and streamlined with greater focus and guidance.

Most expressed a desire for more opportunities for direct connection, learning, and sharing. Interviewees wanted more time to “build camaraderie within so that we could learn from each other” (#1009). Respondents felt there were opportunities for cohorts to support each other, acknowledging that many of the barriers were common, such as aligning the needs of similar groups of participants throughout the cohorts and leveraging the expertise of those working out in the field. There were requests for opportunities for problem solving and collaboration at the staff research level:

Another interviewee shared:

The interviewees spoke positively about the program’s collaboration: “ECHO has proven that you can have all these various cohorts and all these various components within a program work and be successful” (#1004), and, “I'm definitely learning… how to work with all these different cohorts that have different data structures and study structures and it’s been really interesting to kind of learn the sides of the science and contribute to that part of it” (#1018).

Communication and Decision Making

The theme of communication and decision making also emerged across the Intervention, Recipient, and Implementation Infrastructure domains. This theme encompassed responsiveness between ECHO components, internal meetings, requests, and infrastructure elements such as data harmonization. While those interviewed generally felt that ECHO is well organized, some reported varied experiences and impacts on their work. Communication among the ECHO components was characterized as initially complicated – one interviewee said ECHO, “needs better communication and a better decision making process (#1009)” – but progressively getting better. Multiple respondents perceived a long time to get answers to routine requests from the DAC or CC, which was said to delay staff entry into the field or create inefficiencies when data collection continues occurring in real time:

At the same time, some interviewees expressed difficulty with receiving multiple requests at the same time, all with seemingly urgent deadlines. Respondents sometimes felt:

However, interviewees recognized that the size of ECHO impeded timely, clear, and efficient communication:

Respondents identified the opportunity for greater transparency in decision making, like clarifying who in the organizational chart answers which questions when and whose needs are prioritized:

The size and complexity of the consortium added to the greater need for transparency considering the understanding that everyone has their own perspective, which can impede communication. One ECHO Program member commented, “It feels like sometimes you talk to people and they feel like their one small thing is the most important thing, but it’s like you’ve got to remember that everyone has their small thing” (#1018).

Descriptions of the development of the ECHO-wide data collection protocol and the current data processes (e.g., collection, cleaning, harmonization, etc.) reflected some of the challenges and opportunities in decision making. Respondents discussed how more work should have been done up front to streamline the protocol and expressed hope that it would be more aligned in the future:

Another interviewee added:

Interviewees mentioned the perceived burden on ECHO participants of the Program protocol as another opportunity for further alignment and greater flexibility. The sheer number of elements were often cited as a concern for families, “because the protocol is so much more extensive and they’re targeting multiple people in the same household, so the contact is more frequent, it has been hurting retention” (#1015). Respondents generally felt that streamlined decision making would help focus the protocol.

Diversity, Equity, and Inclusion (DEI)

Diversity, equity, and inclusion (DEI) emerged across multiple PRISM domains both explicitly (e.g., Intervention) and implicitly, in relation to implementation outcomes such as recruitment and retention. Although diversity was not specifically defined in the interview script, interviewees tended to interpret this term as being in reference to underrepresented or non-White groups. Interviewees recognized that DEI had been well prioritized. “DEI was obviously going on in our home base, but ECHO has really educated me a lot about it and given the opportunities for trying to advance that field” (#1010). ECHO’s focus on DEI some years after the start of the program left some cohorts feeling they couldn't respond as well because they could only recruit from the participants of their original study. Conversely, the ECHO cohorts that were still recruiting were happy to have the chance to meet the DEI-related goals. Other cohorts felt that access to more diverse families was lost both in the longer time it took to start implementing the protocol. However, most of the cohorts had recruited underserved populations initially, so interviewees felt that DEI was nevertheless supported and maintained.

In addition to ECHO participation, interviewees described different avenues to enhance and ensure DEI across ECHO. One interviewee described their cohort’s efforts to ensure biospecimens are representative of all children in their study sample:

Implementation Outcomes

Our data on implementation outcomes are both qualitative and quantitative; the interview themes are presented alongside the GOITs that have been mapped onto the RE-AIM measures (Table 1). The GOITs from our data collection period provide context for the interviewees’ experiences and perceptions. They also provide insight into the shared priorities and how the program itself assesses implementation. The GOITs were also a specific domain of interest in the interviews, so we present our qualitative findings about the GOITs to frame the results in the implementation domains.

Goals, Objectives, Indicators, and Targets (GOITs)

We present data from the GOITs for Year 6 (2021–2022) collected concurrently with the semi-structured interviews (February–May 2022). The GOITs were developed initially in the fifth year of ECHO by a task force of internal stakeholders, based on expert recommendations for evaluation and program-specific metadata, such as number of cohorts, proposals in the pipeline, etc.; the objectives are adjusted annually. For example, the task force initially identified the dissemination targets (60 publications and presentations per year) based on the number of cohorts in ECHO, and then used metadata from the publications pipeline to predict that target in future iterations. Similar to other ECHO infrastructure, interviewees felt the process of identifying and selecting GOITs improved over time:

In general, they felt the metrics are important:

While respondents felt the GOITs were a useful framework for articulating, monitoring, and focusing attention on program priorities—important given ECHO’s scientific and operational complexity—there were concerns about feasibility. Some interviewees expressed concern about achieving targets for enrollment, retention, and data collection that were impacted by things outside of their control (e.g., COVID, rurality). One interviewee explained that GOITs are “useful in knowing what the larger ECHO program is looking for, but I do not think it represents completely the work that’s being done at the local sites” (#1017). Others expressed additional concerns, “data collection goals are harder for rural sites relying on remote data collection,” (#1012) and “I think sometimes depending on the goal, it could take away from some of the science. If we say we want X number of publications--that’s great and that would be wonderful--but we do not want just to churn out publications for publication’s sake, because science can take a while” (#1020).

Reach

We defined the reach measure concept as the absolute number, proportion, and representativeness of individuals who are willing to participate in the ECHO Program, with a focus on recruitment strategies. The reach domain aligned with ECHO GOIT A.1: Race/Ethnicity of Pregnant Women and Child Participants Across the ECHO-Wide Cohort. At the time of the interviews, the ECHO-wide program was at or very close to the targets set for these metrics (Table 4). The related qualitative findings from the interviews addressed opportunities to enhance strategies for more diversity in recruitment and retention.

Table 4.

Goal A: enrollment and retention of participants

ECHO = environmental influences on child health outcomes; N = number.

Several interviewees expressed the need for more cultural competency in ECHO, especially in terms of the marked differences in how particular groups should be approached:

This case was also made in reference to enrolling and retaining indigenous families, which also requires acknowledgement of historical injustices, attention to trust building, and a greater investment in resources:

Resources and time were cited as critical both to reaching more underserved or minoritized communities and keeping them engaged in research.

Effectiveness

Effectiveness is the impact of the program on important outcomes, for which research publications using ECHO-wide data (versus single cohort studies) were a proxy measure. Table 5 shows data on (1) the number of ECHO-wide Cohort manuscripts published that were derived from proposals submitted during the previous grant year, and (2) the number of ECHO-wide Cohort presentations at scientific meetings. By the end of the data collection period, the program was not on target to reach its goals by the end of the year, likely due to delays in initial infrastructure (e.g., data harmonization) building. However, the ECHO Publications Committee received a 44% increase in manuscript submissions and an 867% increase in presentation submissions compared to those received during grant year 5. Our qualitative data focused on the broad impacts for the public and the scientific and research communities.

Table 5.

Goal D: publication and dissemination

ECHO = environmental influences of child health outcomes; N = number.

Most interviewees mentioned the broad hope that ECHO would result in healthier children and would benefit ECHO participants, families, and the public at large through better understanding the impact of exposures on child health. Interviewees cited the healthcare system, economic system, and entire country, while others described potential benefits focused on specific groups (e.g., children with a specific diagnosis). Several interviewees mentioned ECHO’s potential impact at the policy level:

Many interviewees described benefits to the scientific and research community, including best practices and lessons learned for team science, large-scale studies, and collaboration. Several interviewees mentioned how it would benefit individual researchers’ careers, especially junior investigators, in terms of experience and networking. “It will help people’s careers… it’s going to, you know, find whoever the next director of NIEHS is, or the NICHD” (#1008).

A few interviewees mentioned specific outcomes, exposures, or health conditions that ECHO research could impact significantly, including Attention-deficit/hyperactivity disorder (ADHD), autism-spectrum condition (ASC), and asthma:

ECHO research in genetics, genomics, and epigenomics was described as, “poised to make some true discoveries on health outcomes and maybe even therapeutics” (#1006), and “chemical exposure data from biospecimens we can assay that will give us rich exposure data from biospecimens that we can link to later child health outcomes” (#1025).

Several interviewees mentioned new research methods developed over the course of the program which would benefit the scientific community for years to come, including new measures and methods. As a result of ECHO, the Patient-Reported Outcomes Measurement Information System (PROMIS) team has developed new measures and validated others for younger ages, expanding our ability to understand impacts at earlier life stages [Reference Halfon, Forrest, Lerner and Faustman26]. The COVID-19 pandemic also prompted advances in remote data collection:

Some also saw benefits in the ECHO dataset itself as a long-term scientific resource; the sheer size and diversity of the sample, especially populations that have previously been excluded from research (e.g., Indigenous populations) [Reference Knapp, Kress and Parker27]. Interviewees described opportunities arising from the ECHO data infrastructure including the harmonization of the extant data, “harnessing past existing data to lead to new research questions” (#1013), the opportunities for intergenerational studies, and the ability to look at longitudinal data together with biospecimens. Going forward, deidentified data is available for investigators for scientific purposes by applying to the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Data and Specimen Hub (DASH) [Reference Hazra, Tenney and Shlionskaya28].

Adoption

We considered adoption in terms of both ECHO Program participants and ECHO internal stakeholders. With regards to the former, interviewees had considerable variation in their observations of participant experience; they were both encouraging and challenging. One cohort described having very committed families and gave the example of a dad continuing to participate when the mom passed. ECHO participants were described as engaged, with low attrition, but even when recruitment and retention was good, there remained the problem of getting participants to commit to the study visit schedule.

For internal stakeholder adoption, respondents mostly spoke positively about the investigative network and opportunity to do collaborative science, although some felt that there was not enough inclusion of team members working across multiple roles: “I have a doctorate and…I have been involved on one of the writing teams, but I think it’s very cliquey” (#1008). Interviewees shared great advantages to participating in a large research consortium, including many stakeholders willing to listen to ideas:

However, respondents noted inefficiencies such as investigators sometimes having to do things outside their expertise to meet Program goals. One interviewee offered their perspective on the reason for challenges with alignment:

Implementation

We considered implementation in terms of internal stakeholders’ fidelity to elements of ECHO, including data collection and analysis infrastructure. The GOITs that aligned with this concept concerned data, biospecimens, assays, and completeness of data collection to support program-wide research (Table 6). During our study, ECHO was meeting some goals while there was less progress towards others. The qualitative data reflected challenges with alignment and opportunities for enhanced collaboration.

Table 6.

Goal C: biospecimens and assays

DAC = data analysis center; ECHO = environmental influences on child health outcomes; GOIT = goals, outcomes, indicators, and targets; HHEAR = human health exposure analysis resource; N = number.

Multiple respondents cited challenges around flexibility with data submission and extant data harmonization. Some interviewees expressed difficulty managing diverse stakeholders with different research priorities:

However, most interviewees felt that ECHO-wide sharing and data management infrastructure were relatively well-organized and helpful, but that getting new people access and trained in the systems was often difficult given the turnover in such a long project, “I mean it takes six months to get a research assistant trained up. I mean, yes, you’re behind the curve when they start” (#1009).