Empirical Data

Methods

We conducted qualitative interviews with experts from four key stakeholder groups central to mHealth research:

• Patient and research participant advocates (“Advocate”)

• Researchers who are integrating mHealth technologies into their studies, including independent researchers and citizen scientists (“Researcher”)

• Regulatory and policy professionals (“Regulatory”)

• App and device developers (“Developer”)

Data collection and analysis are described in detail elsewhere in this issue.^{Reference Hammack-Aviran, Brelsford and Beskow5} In brief: Based on our knowledge of the issues and in consultation with the larger research team, we developed, pilot tested, and finalized a semi-structured interview guide centered around hypothetical scenarios involving two commercial mHealth apps collecting health, behavioral, and other data which may be shared for various purposes including research. Here we report findings in response to the following question:

When researchers are in a regulated environment, they are required to seek out external oversight. For example, if their research is funded by NIH, they are required to get approval from an Institutional Review Board, or IRB, which is tasked with making sure human subjects are protected.

• When people are conducting research that is not subject to these regulations, do you think they should still seek out some sort of external oversight? RW Why / why not?

A central concern for research falling into this gap is whether human subjects are being exposed to risk, or worse, being harmed. To assess this possibility, one must consider a series of questions: First, do the current regulations truly protect human subjects? Second, what is the scope and risk of unregulated research? Third, are there other mechanisms in place to protect human subjects?

We identified potential participants based on leadership positions in prominent organizations, institutions, and studies, authorship of influential papers, and nominated expert sampling.^{Reference Namey, Trotter, Guest and Namey6} We interviewed 41 experts representing a wide array of professional perspectives and demographic diversity (Table 1).

Table 1.

Participant characteristics (n = 41)

* Many of our interviewees have multiple areas of expertise and could have been recognized as belonging to two or more categories of stakeholder groups; this table reflects the primary perspective for which we identified them as experts.

^ Reflects >1 degree per interviewee, as applicable.

Interviews were conducted by telephone and professionally prepared transcriptions of the audio recordings were uploaded into NVivo 12 for coding and analysis using standard iterative processes.^{Reference MacQueen7} The Vanderbilt University IRB deemed this research exempt.

Commentary

The protection of human subjects participating in research is required and informed by federal regulations. But the construct of these regulations—with the Common Rule tethered to the use of federal funds and FDA regulations limited to products overseen by the FDA—creates a gap. Non-FDA regulated research conducted with no federal support is not legally required to meet regulatory standards for the protection of human subjects.

Thus, a central concern for research falling into this gap is whether human subjects are being exposed to risk, or worse, being harmed. To assess this possibility, one must consider a series of questions: First, do the current regulations truly protect human subjects? Second, what is the scope and risk of unregulated research? Third, are there other mechanisms in place to protect human subjects?

Regarding current regulations, the history of protection of human subjects in research can be traced back centuries, but it was virtually ignited in the mid-20th century with the Nuremberg trials that culminated in the Nuremberg Code in 1947,^{Reference Moreno, Schmidt and Joffe8} the bombshell article by Henry Beecher in 1966 outlining examples of unethical research published in leading medical journals,^{Reference Miller9} and the 1972 public revelation of the Tuskegee syphilis study.^{Reference Cobb10}

Prior to this, independent review and informed consent were promulgated by some individual entities, but agreed-upon standards and processes did not exist.¹¹ Some felt that investigators could and should be trusted to oversee their own research: “misconduct was a problem of rogue researchers…trust, not regulation would foster better research and clinical care.”^{Reference Jones, Grady and Lederer12} Even Beecher did not call for regulatory oversight, but rather focused on “the more reliable safeguard provided by the presence of an intelligent, informed, conscientious, compassionate, responsible investigator.”^{Reference Beecher13}

In the U.S., this changed in 1974 when the National Research Act formally codified IRBs for oversight of federally funded research with human subjects.¹⁴ Of note, this Act recognized the problem of unregulated research by calling for “… an investigation and study to determine the need for a mechanism to assure that human subjects in biomedical and behavioral research not subject to regulation by the Secretary are protected” (Section 202(3)).

Similar international initiatives emerged at the same time: In 1975, the World Medical Association’s Declaration of Helsinki¹⁵ added Principle 2: “(T)he design and performance of each experimental procedure involving human subjects should be clearly formulated in an experimental protocol which should be transmitted to a specially appointed independent committee for consideration, comment and guidance.” In later iterations, approval by an independent ethics committee (current Principle 5) was required and this has become the international norm.

This evolution in research regulations and norms is understandable given the context of egregious examples of self-regulated unethical research.^{Reference Moon and Khin-Maung-Gyi16} Did those conducting such research ever consider that their studies were unethical? Did their enthusiasm for, as well as their intellectual, career, and/or fiscal investment in, their research blind them to potential concerns? Can self-regulation adequately address the complexity of research bias, e.g., small-study effect bias, citation bias, industry bias, financial conflicts of interest, pressure to publish bias, early career bias? Most would likely find it difficult, if not impossible, to truly recognize and acknowledge the degree to which biases influence their own actions.^{Reference Fanelli, Costas and Ioannidis17} To quote one of the interviewees in our study, a review is better done by “somebody more objective than the person doing the research.”

Consistency of review is important and supported by proscribed regulatory criteria that must be met for approval. For example, risks must be minimized and reasonable in relation to anticipated benefits (if any) or knowledge gained; informed consent must be sought and documented (if not waived); there must be adequate provisions for monitoring, protection of confidentiality, and safeguards for vulnerable populations.¹⁸ In assessing these criteria, study hypothesis and design are essential considerations; after all, if it is bad science, it is unethical.¹⁹

While there has been international endorsement of independent and consistent oversight and approval, there are numerous criticisms of the current process, including those identified by our interviewees. Some of these focus on burden: that the process is too bureaucratic, too time consuming. Some question details of the process, perceiving a one-size-fits-all approach without flexibility or titration of oversight to different levels of risk. Some raise the issue of IRB members’ expertise and training to adequately handle new areas of research. Some question the value of the current system and ask for hard evidence. Some simply ask for a different process.^{Reference Abbott and Grady20}

Response to these criticisms is challenging. Although federal regulations and guidance documents inform the oversight process, there is tremendous variability in interpretation and implementation.^{Reference Silverman, Hull, Sugarman, McWilliams, Green and Kastner21} This continues despite attempts at accreditation to encourage standardization.^{Reference Summers, Feige, Gallin, Ognibene and Johnson22} Measuring/proving the value of IRB review has been elusive.^{Reference Grady23}

Although performance metrics such as turnaround times can be measured, the critical ask is for evidence that the current process improves the protection of human subjects — and response to this is generally anecdotal, rather than concretely evidentiary. Examples of protocol changes required by the IRB that, had they not been made, would likely have resulted in harm to subjects can readily be found. While compelling, these do not provide a systematic counterargument.

There have been long-standing concerns about the lack of flexibility in the regulations — a fundamental and unresolved one being that biomedical and social-behavioral research are inherently different and should not be covered by the same system of protection.^{Reference De Vries, DeBruin and Goodgame24} The explosion of mHealth apps, often associated with behavioral research, has fueled this controversy. However, the Common Rule does in fact incorporate flexibility, elucidating categories of research that are exempt from the regulations, identifying criteria for waiver of consent requirements, and assigning level of review based on whether the risks involved are greater than minimal.^{Reference Menikoff, Kaneshiro and Pritchard25} Even so, not all IRBs take full advantage of the flexibility that exists.

Proposals for change fall into several buckets,^{Reference Emanuel26} many of which were captured by our interviewees’ comments on approaches to independent oversight. One calls for training and certification to empower researchers to oversee their own research. This could be supported by a process to get feedback from experts, and perhaps include posting protocols online to elicit suggestions. In such a system “responsibility for ethical conduct during the study would be shared by both the researchers and the peers who agreed that the plan would adequately protect participants.”^{Reference Bloss27} This system is predicated on an available group of peers who are trained and possibly certified in bioethics and research. Other suggestions involve the creation of new formal or informal entities that would either augment the IRB (e.g., data boards, data security committees^{Reference Ienca28}) or take the place of the IRB.

With any of these proposals, the devil is in the myriad details and whether key attributes of review (such as those identified by our interviewees) could be met. For example, would these new mechanisms completely replace the current system or would they only be applied to a particular segment of research (perhaps based on risk)? Would these systems require independence of review? Would they be voluntary? What authority would these new processes have? What standards would be used for the assessment of a particular protocol or for certification of a peer reviewer? Would approval be required? How would new processes be sustained over time? In the example of online peer review, what incentives would be needed to sustain the number and quality of reviewers needed?

If alternate approaches were promulgated, how would that affect the consistency of review that is expected by researchers, institutions, funders, and publishers? If ethically acceptable alternatives were developed, who would determine which system would be available to specific research and/or researchers?

Turning to the issue of the scope and risk of unregulated research, further questions arise. Is it materially distinct from “traditional” research? Are the risks smaller or fewer? Are unregulated and regulated researchers different in ways that matter regarding ethical obligations to protect participants?

The scope of research not subject to regulation is limited by specific attributes that definitively bring certain activities into the regulated sphere, e.g., most research procedures conducted in a hospital setting, collaboration with colleagues who work for institutions that receive federal funding, research that falls under FDA-regulation. (Of note, the lack of clear understanding regarding the FDA’s enforcement discretion in mHealth adds some confusion.^{Reference Shuren, Patel, Gottlieb, Cortez, Cohen and Kesselheim29}) Further, research that can be conducted without meeting regulatory standards is limited by journal editors who routinely require documentation of compliance with federal regulations as a condition for publication.

But the scope of unregulated research and associated risk is broad, with interventional studies with drugs and devices at one extreme. But more common is research with big data, artificial intelligence, and, relevant to this study, mHealth apps, for which a good portion of development is done in the commercial sector by company employees as well as by independent entrepreneurs who are not covered by the regulations. The risk-benefit ratio of this research merits consideration. The main risk to subjects is the potential loss of privacy and confidentiality resulting from the necessary access to large amounts of data (observational and self-reported data as well as information in publicly accessible and/or highly controlled databases). Ensuring adequate protections is difficult as the definition of identifiability changes, technology facilitates re-identification, and hacking and cyber-attacks increase.^{Reference Vayena30} The benefit of the research must justify exposing any subject to these risks. And, despite logistical challenges, some feel strongly that individuals whose data are accessed should provide informed consent, or at least be notified of the research and its risk.^{Reference Moore31} When mHealth research is regulated, IRBs address these issues. When mHealth is unregulated, it is uncertain how risks and benefits are even considered. As some interviewees noted, “unregulated research may be riskier than the same study in a regulated environment.”

With regard to mechanisms other than IRBs for protecting human subjects, researchers conducting unregulated research will not be nested in conventional research infrastructures available at regulated academic/research centers. In addition to regulatory oversight, these systems support researcher development and good behavior with vetted local policies, structured institutional norms, and consequences for noncompliance.^{Reference Salloch32} For unregulated researchers who work with small independent research staffs or on a research team embedded in a larger organization focused on product development, it may be difficult to find similar support. This is particularly concerning for unregulated researchers who are new to research and have little to no experience with or understanding of human subjects protections. Their facility with basic ethical standards may be limited; their recognition of human subjects issues may be flawed and potential risks not considered. In the absence of independent review, bias may be an issue especially when the investigator has a stake in the product under development.

Unregulated research does pose risk to human subjects and those human subjects have the right to protection. We agree with the large majority of experts who participated in our qualitative interviews: over 90% said independent oversight is important, and many of these said the current system of oversight is the best option — while also acknowledging ample opportunities for improvement. If the risk of much unregulated research is trivial, assignment to an exempt category may be appropriate in some cases and in others the review could be expedited. Alternate processes could be entertained if they are standardized, sustainable, and can be evaluated. There is a duty to do no harm, and history has well demonstrated the limitations of self-regulation in realizing this duty.

In the absence of legislative action, unregulated research will continue to be a reality. The people and entities conducting this research must become intimately familiar with ethical norms and standards for the protection of human subjects. The future of these activities relies on the trust of everyone involved in the research³³ and part of that trust is demonstrating that the rights and welfare of human subjects are being protected. Could the current system of oversight be improved? Absolutely. But today it remains the only process that consistently protects humans in research.