Methods & materials

The impetus for the DDM emerged from our Ethics of Inclusion (EOI) Study, a qualitative investigation of conceptions of diversity and their operationalization in five PMR studies across three consortia funded by the NIH [Reference Lee, Fullerton and McMahon15–Reference Bentz, Saperstein, Fullerton, Shim and Lee20]. Our methods for the EOI Study included a content analysis of 76 documents, including study materials and funding announcements issued by the NIH; approximately 450 hours of observations of study activities, including site and consortium working group calls, and in-person or virtual meetings; and 125 in-depth interviews. Interviews included 102 initial and 23 follow-up interviews, each lasting 60–90 minutes, with funders of PMR research, PMR investigators, research team members, and research participants/participant advisory board members (see Table 1 for EOI interviewee demographic characteristics). Interviewees were purposively recruited based on their involvement in our five study sites or the PMR consortia to which the study sites belonged. All EOI data were analyzed using the principles of constructivist grounded theory [Reference Charmaz30]. (Additional information about sampling, recruitment, data collection, and analysis methods for the EOI Study can be found in [Reference Lee, Fullerton and McMahon15–Reference Bentz, Saperstein, Fullerton, Shim and Lee20].)

Table 1.

Demographic characteristics of EOI interviewees (N = 102)

To develop early drafts of the DDM, we inductively analyzed our EOI data to identify research activities and decisions that influenced and/or impacted goals of diversity, inclusion, and equity, and created a visual display of these decision-making nodes as part of a research life course. We then further analyzed our data to identify relationships between decisions about specific research activities and the prior “upstream” choices, events, or contexts that shaped those decisions, and their later “downstream” consequences. These analyses produced successive iterations of the DDM, as we continued to add newly identified decision-making nodes and arrows connecting them. The nodes in the DDM and their upstream and downstream connections to other research activities and decisions are not intended to be exhaustive but rather highlight common domains of research identified in our EOI data that were impacted by and influenced by goals of diversity.

Once we had a draft of the DDM, we were interested in receiving input from potential users and to gauge proof of principle: Did the DDM help to convey the importance of decision-making across the research life course? Did exemplar decisions and their upstream determinants and downstream consequences for diversity generate discussion and forward thinking about how to shape PMR studies for achieving goals of inclusion? To assess the potential utility of the DDM, we conducted three engagement sessions, each with a different stakeholder group. These included the regulatory and ethics resource of a clinical and translational science center at an academic medical center (referred to hereafter as “regulatory and ethics advisors”), precision medicine investigators who are part of a genetic research consortium (“researchers”), and a group of community partners with experience advising a clinical translational science institute (“community advisors”). We recruited these groups as follows: The regulatory and ethics advisors were part of an existing working group of a university’s NIH-funded Clinical and Translational Science Award (CTSA) program, and the researchers were part of an existing working group of a genetic research consortium. Both groups met regularly, and we obtained permission to join one of their meetings to conduct our engagements. We recruited the community advisors through a university’s CTSA community engagement core, which offers consultations with community advisors: upon our request, the CTSA staff issued an invitation to their pool of community advisors. Because of the relative dearth of EOI data from participants in precision medicine research studies, it was particularly important for us to seek input from community advisors. Our engagements involved 8 regulatory and ethics advisors, 11 precision medicine researchers, and 7 community advisors, for a total of 26 individuals across the three sessions. Only 4 individuals from the researcher group were also interviewees in our EOI Study; all other researchers, and all of the regulatory and ethics advisors and community advisors did not participate in EOI interviews.

During these stakeholder engagements, we began with a presentation of the DDM, explaining the concepts of a research life course, the path dependence of research decisions, and tradeoffs incurred when selecting different decision paths. We offered data-based examples of research decisions, and using the DDM, traced their tradeoffs, upstream determinants, and downstream consequences. We then elicited stakeholders’ reactions to the following discussion prompts: What questions and issues about study decisions did the DDM generate? Could they use it to follow operational and methodological choices, surface tradeoffs, and map their consequences through the research life course? Did the issues that surfaced resonate with their own experiences in PMR? Could they see the DDM helping to identify tradeoffs of research paths taken and not taken? Could they see the DDM being useful to them in their future work; if so, how, and what more work on the DDM would be needed? Team members observed and either audio-recorded or took extensive, near-verbatim notes on the stakeholder discussions. After the sessions were complete, the EOI Study team reviewed the recordings and notes and transcribed selected portions of the recorded sessions. We then itemized stakeholders’ comments and feedback, collated them by themes, and organized these into tables where we could examine commonalities and differences across stakeholder groups.

Results: The diversity decision map

In Figure 1, we show our current draft of the DDM. In constructing the DDM, it was important to us that it represent our understanding from our EOI data that PMR studies did not proceed linearly from beginning to end, neatly step by step, according to design or plan. Instead, many studies we investigated were built upon previous research efforts or already recruited samples or participants or were conducted in waves with multiple funding sources. Additionally, multiple phases of a PMR study often happened simultaneously as researchers made decisions, tried them out, encountered problems or changing conditions on the ground, and so pivoted and iterated. We also found that the cumulative effect of multiple decisions, that researchers perceived at the time to be relatively mundane, frequently had downstream impacts on the ability of a PMR study to reach diversity goals and implement inclusionary practices. That is, multiple small, ad hoc decisions made upstream shaped and constrained consequential outcomes, including participant diversity, the kinds of data collected, relations with participants or community advisors, and the potential to aggregate data with those of other studies. Research decisions also had a path-dependent and looping nature to them: choices made at one time about one study activity often looped back to shape other activities and options available in the future; similarly, decisions in a PMR study were shaped and constrained by the ecosystems and contexts in which the study was embedded.

Figure 1.

Diversity decisions across the research life course.

Note: arrows linking boxes and research stages are double-headed, to indicate mutual influence and dependence between study activities (i.e., earlier choices about study design and procedures shape subsequent options, and decisions at one stage may require amending previous methods and procedures). Not all possible arrows are depicted.

Thus, we constructed the DDM to show various research activities across the research life course in boxes, and through the arrows linking boxes to each other, indicate how these activities are connected to and influence one another.

At first glance, the DDM can appear to show a typical progression of research stages, for example, by depicting steps such as Proposal Preparation on the left side and Dissemination on the right side. However, our EOI data showed that studies did not simply start on the left and move to the right. Therefore, we included the many double-headed arrows linking boxes and research stages to indicate the complex mutual influence among them, both in the path-dependent sense where earlier choices about study design and procedures can shape subsequent options, but also in the iterative sense where decisions at one stage may require backtracking and amending previous methods and procedures. Not all possible arrows are depicted; the specifics of actual studies will inevitably shape whether and how boxes are connected, as well as the directionality of influence and impact.

The DDM is intentionally complex, to reflect the on-the-ground realities of research decision-making as it unfolds in relation to multiple competing demands and resource constraints. Also intentionally, the DDM is not meant as a prescriptive, normative guide for what specific research decisions can and should be made to optimize equity. Instead, we developed the DDM to serve as a useful heuristic and tool to enable stakeholders to identify tradeoffs and facilitate stakeholder dialogue about those tradeoffs, at multiple stages of a study and even when the research is well underway. To do so, we found that the DDM needed to allow stakeholders to locate where they are in the research process (much like a map reader who must pinpoint where they are before navigating their way). Stakeholders would then need to weigh competing interests, consider methodological and resource tradeoffs, anticipate the future consequences of different choices for inclusion and equity, and seek to make informed decisions. The DDM helps to guide the systematic consideration of the potential effects of study decisions about one set of research activities on others, by attending to how boxes are and might be connected to and influence others. Rather than focusing on recommending specific study decisions, best practices, or pathways in the abstract, our DDM seeks to support dialogue, transparency, and decision-making among research stakeholders who face complex on-the-ground constraints, conditions, and contingencies.

To further illustrate how the DDM might be used, we provide two examples based on significant findings from our Ethics of Inclusion Study (additional explication and demonstration of the DDM are also provided in an animated, narrated, and accessible slide show presentation [31]). In the first example (elaborated in our publications [Reference Jeske, Vasquez and Fullerton16–Reference Shim, Bentz and Vasquez17]) represented in Figure 2, the box labeled Population Sampling in the bottom middle of the map refers to decisions made about which populations the PMR study would focus on and how those groups are defined. Such choices can be shaped upstream in the research life course, for instance, by guidance from Funding Announcements (upper left on DDM) that may stipulate or even require that certain populations be included.

Figure 2.

Diversity Decision Map: population sampling example.

Additionally, population sampling can influence Team Formation (see Figure 3, on the left), that is, study team members whose characteristics reflect the populations targeted for participant recruitment might be hired and their involvement will then shape study activities such as Recruitment (top middle) strategies to enroll populations prioritized for inclusion [Reference Jeske, Vasquez and Fullerton16]. This, in turn, could also influence Site Selection (bottom left) where a study may decide to partner with specific clinics with reach and connections to the desired communities and groups targeted for enrollment [Reference Shim, Bentz and Vasquez17]. This example illustrates how the DDM can be used in studies already in progress: in this case, even though a decision about which populations would be enrolled has been made, the study team could start in the middle of the DDM to anticipate how that decision might shape downstream choices about team hiring and composition, and recruitment sites, in ways that impact diversity and inclusion.

Figure 3.

Diversity Decision Map: population sampling example (continued).

As another example (elaborated in our publication [Reference Shim, Foti and Vasquez18]) represented in Figure 4, we focus on the box labeled as Community Engagement found in the center left of the DDM. A series of decisions informs whether, when, and how community engagement is incorporated into the governance of the study throughout the research life course. That is, to what extent do community members participate in decision-making in study activities that are often understood to be shaped by community input, as seen by arrows connected to Recruitment and Retention (both in top middle), for example, and Population Sampling (bottom middle)? But we also link community engagement to other research stages such as Identification of Research Questions (bottom center), Measure Selection (near center), and Team Formation (middle left)—these may be less commonly recognized as connected to community engagement activities. Empirically, we found that decisions about whom to engage, when, and how shaped whether community advisors provided input on what sorts of research questions were pursued, which data were collected and how, and whether leadership and decision-making authority were given to community partners [Reference Shim, Foti and Vasquez18]. This reinforces that community engagement can happen at any stage of a study, and when and how it is conducted has implications for a multitude of research activities.

Figure 4.

Diversity Decision Map: community engagement example.

Finally, as seen in Figure 5, community engagement also had spillover effects on Data Analysis, Data Sharing (both middle right), and Research Translation (bottom right), where we found that some community-research partnerships shaped research questions and data collection to be highly local and specific in ways that limited the ability for study data to be aggregated with other data. And the degree of commitment to community engagement affected choices about Team Formation: strong commitments to an engaged approach led to community partners needing to be part of study leadership, whereas understandings of community input as advisory meant that community engagement could happen more separately [Reference Shim, Foti and Vasquez18].

Figure 5.

Diversity Decision Map: community engagement example (continued).

The stakeholder engagement sessions provided important feedback through which to assess the utility of the DDM (see Table 2). Across the different stakeholder groups, all agreed that a dynamic and user-responsive tool like the DDM was necessary, to facilitate intentional research decision-making that was collaborative and that would optimize the ability of studies to achieve their goals for diversity, inclusion, and equity. They saw the value of the DDM in tracing consequences of study decisions for diversity and inclusion, suggesting paths forward, and showing how important upstream factors and constraints shape research. They also felt the DDM provided a common orienting object that different stakeholders could use to raise important considerations (e.g., budget, team formation, community involvement, etc.) related to diversity and inclusion, and share ideas, questions, and dilemmas with each other. Regulatory and ethics advisors found the DDM useful for simplifying complex concepts as well as raising the challenges of determining meaningful community participation. Precision medicine researchers appreciated the potential for discussing the rationale for research pathways taken and not taken and the potential downstream impacts. Community advisors suggested the importance of dialogue about study-specific contexts and avoiding a researcher-centered understanding of tradeoffs. These sessions indicated that the DDM would catalyze the kinds of discussions and raise the kinds of questions about the consequences of potential research decisions on diversity and inclusion that we hoped it would. These conversations and debates served to support the proof of principle for the DDM.

Table 2.

Feedback from stakeholder engagement sessions

We also sought feedback from stakeholders on what about the DDM worked, what questions they had, and what could be improved. There were widely ranging opinions about whether the DDM was too prescriptive or too open-ended: some felt it represented particular pathways as being the “ideal” and others as “lesser options”; others thought it needed to indicate a recommended starting point (e.g., beginning with community engagement). Still others emphasized the need for the DDM to offer ‘middle road’ approaches, present benefits and challenges of different paths taken in balanced ways, and promote a stance of being curious, rather than presenting problems that have to be addressed; these latter comments align well with our intended purpose. Some stakeholders felt the DDM disproportionately centered researcher perspectives, and that its language and visuals were not appropriate for engaging community representatives. They offered a wide range of other comments for clarification and improvement, including indicating where “hard stops” or “veto power” might be integrated to prevent studies deemed to be problematic from moving forward, making the DDM more interactive, and providing accompanying user resources. Finally, stakeholders also pointed out contextual issues that would constrain the utility of the DDM, such as funder authority to reject certain pathways or community members not having sufficient decision-making authority.

Collectively, the stakeholder feedback led to revisions to the DDM, and were incorporated into our User Guide, along with an accompanying video [31], that we hope can facilitate cross-stakeholder discussions of research decisions and tradeoffs that may affect diversity- and equity-related outcomes. These materials provide the basis for key next steps to increase the DDM’s utility, including: incorporating additional research life course stages, making it more interactive, improving it visually, embedding more information as well as concrete examples and cases, and clarifying whether the DDM aims to enhance diversity and inclusion towards broader ends, whether that be statistical representation and/or equitable science.