A good way to start—or perhaps to stop—a conversation at any political science convention is to mention that you are on an Institutional Review Board (IRB). The likely reaction will be a recitation of horror stories of IRB imperiousness, nitpicking, delays, and the complete failure to recognize the burdens imposed on researchers. Most people believe that the process does little to protect the welfare of research subjects and is unable to prevent research controversies, such as the recently retracted Science article on attitudes toward same-sex marriage (Konnikova Reference Konnikova2015; LaCour and Green Reference LaCour and Green2014, retracted June 2015).
The complaints about IRBs are well known: mission creep, as boards move from protecting human subjects to micromanaging research projects and insisting on changes unrelated to the actual federal rules; bureaucratic delays, as projects are held up by procedural snags and red tape; and insistence that researchers comply with administrative requirements that are pure dead weight, constituting a failure (or refusal) to consider the effects of the IRB process on researcher time or even the ability to conduct research (Burris Reference Burris2008; Citro, Ilgen, and Marrett Reference Citro, Ilgen and Marrett2003; Gunsalus et al. Reference Gunsalus, Bruner, Burbules, Dash and Matthew Finkin2007; Klitzman Reference Klitzman2015; Schrag Reference Schrag2010). The lofty goal of protecting subjects has created a system that “has grown into a self-referential, unresponsive, and legalistic bureaucracy” (Burris Reference Burris2008, 65). None of this would be a surprise to those critical of the efforts to draft regulations in the 1970s, who warned that the rules designed for biomedical research were ill-suited to the social sciences. A previous symposium in PS (Hauck Reference Hauck2008) identified many issues, and little appears to have changed since then.
My goal is not to recapitulate these complaints but rather to analyze how it looks from inside an IRB. My views are informed not only by my experience as a researcher who has gone through the process multiple times but also from the vantage point of five years on the Educational, Social, and Behavioral Sciences (ED/SBS) IRB at the University of Wisconsin (UW), Madison, including two years as chair. During those five years, I spent considerable time speaking to faculty and listening to their complaints, meeting with principal investigators (PIs) to work through issues with their protocols, calming graduate students who were certain that a notice of noncompliance for a minor administrative violation was a career-ender, and responding to frustrated colleagues infuriated at what they perceived as unnecessary and even arbitrary demands and restrictions. I got into the habit of asking political scientists at other universities about their experiences with their own IRBs, and I frequently discussed in the ED/SBS IRB how we would have addressed those projects. I also am part of a loosely organized group of faculty working on institutional reforms at UW.
This experience has led me to three conclusions. First, the IRB process is not as bad as it is frequently portrayed, at least as it operates at the UW (although I know that some of my colleagues will dispute this). It is hardly ideal, to be sure, and we sometimes struggled to apply to social-science research a framework designed for medical research when the projects we examined often presented minor risks to subjects. However, we made an effort to be judicious, were careful to avoid mission creep, and focused our attention on those studies and activities that we believed were most likely to pose actual risks to subjects. We did not micromanage research projects or insist on methodological changes for their own sake. I do not recall any cases in which we rejected a protocol or in which a researcher abandoned a project rather than make the changes that we required.
Moreover, we were willing to question the rules that we were expected to implement and push back against what we deemed unreasonable demands. For example, the UW requires all PIs doing research involving non-English-speaking subjects to submit consent forms in the relevant language. This is an appropriate accountability check because even if there is no IRB member who speaks the language, we have a written record and the ability to check the documents if any concerns arise. During one external review, we faced pressure to require all foreign-language consent forms to be back-translated into English by independent professional translators. Such a rule, we were told, was necessary to ensure that consent forms contained what PIs claimed. The IRB Chair objected in the strongest terms, arguing that it would do nothing to protect subjects, would only serve to impose a monetary cost as a condition of submitting a protocol (it was understood that PIs would have to pay for it themselves), and that the UW trusted researchers to act ethically. The UW policy remains unchanged. Another review faulted us for returning too many protocols for minor changes that could be reviewed by a subcommittee of two IRB members rather than deferrals that had to go back to the full IRB, as well as for having too many unanimous votes. Still another review objected to the fact that some consent forms were signed in pencil rather than pen and wanted a requirement that all consent forms be signed in ink. Such a rule would do nothing but cause PIs to roll their eyes in exasperation.
What I told my frustrated colleagues is that they cannot use a “reasonable person” standard to judge the IRB process. They had to apply an “unreasonable bureaucrat” standard: that someone authorized to punish the university might question the IRB’s work years later, perhaps after a serious adverse event that questions our entire process; who demands justification for each decision that the IRB has made; and who is not likely to give us the benefit of any doubts. When something goes wrong, the individual researcher is not the one who alone bears the cost. It is the institution and other researchers that face the most serious harm if the federal government imposes the nuclear option of shutting down all research.
What I told my frustrated colleagues is that they cannot use a “reasonable person” standard to judge the IRB process. They had to apply an “unreasonable bureaucrat” standard: that someone authorized to punish the university might question the IRB’s work years later, perhaps after a serious adverse event that questions our entire process; who demands justification for each decision that the IRB has made; and who is not likely to give us the benefit of any doubts.
Second, the process can be improved primarily by recognizing that many procedural requirements are designed to address risks that are so unlikely—and so slight if they do occur—that it would be reasonable to dispense with them. Must minor alterations to interview questions be reported and justified? Is it a problem if a survey has a higher than expected response rate and results in 650 respondents instead of the originally intended 600? At the institutional level, policy makers should expand the categories of research that are exempt from review. Footnote 1 For example, this could involve a policy that the chance of a minor “dignitary harm” to a subject is so transitory or otherwise innocuous that it will not be considered a substantive risk. The difficulty, of course, is what happens at the margins, where benign ethnographic or survey research spills over into more sensitive areas, or where a seemingly trivial harm turns into something more serious. During my five years on the IRB, I was aware of only a few problems that could have resulted in harm to subjects, all involving actual or potential breaches of confidentiality. Most involved lost or stolen laptops or disks and file cabinets left unlocked. The most serious breach involved the unauthorized sharing of data with another researcher, including information about personality disorders, in a study of a vulnerable population. Even in these cases, the latter of which we deemed serious enough to report to the federal Office of Human Research Protections (OHRP), I witnessed no evidence of actual harm to a subject. Complaints that came into the office tended to be minor, such as people irritated at receiving an unsolicited mail invitation to participate in a survey that they regarded as intrusive.
Third, IRBs frequently are blamed for problems unrelated to the IRB process, either because research became controversial for reasons other than risks to subjects or because it never was subjected to IRB review. Footnote 2 This type of elision blurs important distinctions among different types of research and causes spillover effects when problems in one area are used to justify restrictions imposed on all research.
AN OVERVIEW OF THE EDUCATIONAL AND SOCIAL AND BEHAVIORAL SCIENCES IRB AT UW
The costs and benefits of the IRB process heavily depend on the institutional context, the amount of research that a university conducts, and the consequences for a serious lapse in the process. The UW, for example, received $879 million in external research funding in 2013–2014, including $660 million from the federal government (University of Wisconsin, Madison 2015, 62). In 2012–2013, the UW was ranked 14th nationally among all universities in federal funding for science and engineering research and development (University of Wisconsin, Madison 2015, 66–7).
The UW has three IRBs: a Health Sciences IRB, an Educational, Social, and Behavioral Sciences IRB, and a Greater than Minimal Risk IRB, which considers protocols involving more significant risks. The UW is at the more stringent end of the spectrum of review because, in practice, we subject all human-subjects research to the IRB process irrespective of funding source. The UW has two Federal Wide Agreements (FWAs): one with the US Department of Health and Human Services and another for research funded by the US Department of Defense. The UW also is accredited by the Association for the Accreditation of Human Research Protection Programs.
The Educational, Social, and Behavioral Sciences IRB reviews approximately 800 new protocols annually and another estimated 900 as continuing review of existing protocols. Of the new protocols, only about one in eight is reviewed by the full committee, with the remainder reviewed administratively or by a subcommittee of two IRB members. Approximately half of the new protocols are from students (i.e., mostly graduate students doing dissertation research).
IRB committee meetings and proposal reviews collectively add perhaps another 750 hours annually for the 14 non-staff IRB members; factor in another 10,000 hours annually for five full-time IRB staff. Allowing for additional consultation time, it is plausible that the UW spends annually, conservatively, at least 25,000 to 30,000 person-hours on the basics, with most of that burden falling on researchers. Footnote 3
A “back-of-the-envelope” calculation provides a rough idea of the time costs involved in the process. It is a reasonable estimate that preparing and submitting a medium-complexity protocol for IRB review takes about 10 hours; more detailed projects take longer. Factor in another 2 to 5 hours responding to IRB questions even for an expedited protocol, and another approximate 10 hours for protocols that are deferred or require modifications. A conservative estimate is an average of 15 to 20 person-hours for each new protocol; however, complex protocols with multiple PIs or sites easily could be four or five times higher, which suggests about 12,000 to 16,000 hours annually of researcher time for initial submissions. Submitting materials for the protocols undergoing continuing review or involving changes each will take at least 1 to 2 hours—an additional 1,800 hours. IRB committee meetings and proposal reviews collectively add perhaps another 750 hours annually for the 14 non-staff IRB members; factor in another 10,000 hours annually for five full-time IRB staff. Allowing for additional consultation time, it is plausible that the UW spends annually, conservatively, at least 25,000 to 30,000 person-hours on the basics, with most of that burden falling on researchers. Footnote 3 Unfortunately, we do not know the actual number because there is no mechanism for tracking how much time is spent on the process.
HARD CASES MAKE BAD LAW, AS WELL AS DIFFICULT IRB POLICY
It is easy to criticize IRBs for vastly overstating risks and demanding unnecessary changes to protocols. What from the outside appears to be overreach or even absurdity can look very different from the inside, where concerns that are in tension and definitions that can be ambiguous must be balanced. Consider the following case that our IRB addressed: a PI proposed a study of near-death elderly people in Tibet about what happens when they die. The local religious and cultural custom is that death occurs in stages over several hours, even after what in the West would be considered irreversible physical death. The research involved interventions and rituals performed on individuals who under our definition of death were dead but under their definition were still alive.
Is this example considered human-subjects research? If the subjects are dead, then they clearly are no longer human subjects and the IRB had no reason to be involved. We struggled with the question of whether we even had the authority to consider this. On the one hand, the subjects were dead at the time the research would take place, at least by our definition. On the other hand, the Common Rule, UW policy, and basic ethics required us to give special consideration to local cultural norms. If the subjects were still alive according to the local religious and social practices, then we had an obligation to respect those practices. This also was a vulnerable population (i.e., elderly, terminally ill, and non-English speaking), which required special consideration. We concluded that this project was human-subjects research, and we relied on the consent process to respect the autonomy of subjects and address any risks.
This may seem to be an obvious or even ridiculous case. An IRB critic might think it absurd to spend any time pondering what it means to be dead. Our interpretation of the rules and application to this protocol were reasonable in my view, but a different IRB may have concluded that the project did not constitute human-subjects research.
All of the difficult cases we addressed involved this type of ambiguity, including the following examples from protocols we reviewed:
-
(1) UW faculty and staff are mandated reporters of child abuse and neglect as well as sexual assault, whether observed or revealed. Under this policy, an otherwise benign home visit or interview could trigger a mandated report if a PI or another investigator observes or learns of evidence of abuse, neglect, or sexual assault. It is obvious that this is a risk to a research subject. How should the risk be addressed in a consent form? How much detail is necessary? Does this requirement and risk mean that all social-science research that involves children or people who may have children must be subject to full review?
-
(2) Studies dealing with risky behaviors or sensitive topics may apply for a Certificate of Confidentiality (CoC) from either the National Institutes of Health or the US Department of Health and Human Services. These are legal instruments designed to protect researchers from being forced to divulge identifying information about subjects “in any civil, criminal, administrative, legislative, or other proceeding, whether at the federal, state, or local level.” Footnote 4 A CoC enhances our confidence that subject data will remain confidential; however, does it offer meaningful protection to a research subject if a prosecutor or federal investigator went to court to demand it? The answer, we were advised, is that no one is certain because there have been few legal tests.
-
(3) A PI proposed a study involving interviews with people who have been charged with a crime or are the target of a criminal investigation to discuss the activities that led to the charges or investigations. Should the IRB be concerned about the risk that participants could be subject to prosecution or hurt their ability to defend themselves if a district attorney demands the researcher’s notes? Is informed consent sufficient to address these risks?
-
(4) How high does compensation have to be before it becomes coercive? Compensation of, for example, $1,000 for a 10-minute survey would be rejected as absurdly high, but what about $100 for a 5-hour study with a home visit? If $100 is acceptable, is $200 or $225? Should researchers be required to offer compensation for such a study as a matter of respect for individuals? If so, how much time can a study take before a PI is required to offer it? Does it make a difference whether the subject population is wealthy suburban homeowners or people recruited from a downtown homeless shelter?
-
(5) Should survey research be exempt from review, even when the surveys are conducted by private contractors who may want to keep the data they collect, along with identifiers, and use it for their own research?
-
(6) For research that we think poses some risk, should we require researchers to modify their studies to reduce those risks, or is it sufficient to merely disclose the risks in the consent form? If we rely on the consent form to address the problem, how much detail must be included? At what point does a consent form morph into a legalistic document that, similar to an end-user license agreement, no one reads?
-
(7) How should we evaluate risks in a project that studies attitudes toward government in a country with an authoritarian regime? Can we rely on the proposition that people in those regimes understand the risks of criticizing their government and that those risks are therefore (in the Common Rule definition) “not greater in and of themselves than those ordinarily encountered in daily life” (45 CFR 102(j))?
-
(8) A researcher who wants to study an African tribal region states that it is necessary to obtain permission from tribal leaders to work in a particular village. Does that permission have any impact on the residents’ ability to decline to participate in the research (i.e., because they know that the elders have approved it)? Can we ignore the risk of coercion because local customs conceptualize permission and consent differently than we do?
These examples have three common features. The first is that they all required judgment, because it often was unclear which policy or ethical standard applied or because multiple regulatory requirements and ethical imperatives conflicted with one another. The second is that they all have more than one reasonable answer. Despite the complexity of the regulations, many critical terms are subject to different interpretations—from the basic properties of “risk” to even the definitions of “research” and “human subject.” My personal view was that we did not have to be perfect—only that we make the best decision with the available information after careful consideration of risks and benefits. However, another IRB, quite reasonably, may have come to different conclusions.
A meaningful reform proposal would take far more space than available in this article, but if I were to offer one pragmatic suggestion for improving the IRB process, it would be to plead for better data.
The third commonality is that these cases are not easily addressed by simply using rules and policies. Even in an institution that is prepared to accept the risk of exempting a wider range of studies or practices—or to declare a policy of exempting all survey research—it is not difficult to construct cases that are at the margins of what qualifies as exempt or as risky. Most of those risks are minor, others are so unlikely that we can ignore them, and still others fall into categories that perhaps ought not to trigger close review. Most could be exempted, but not all—and even then there must be a process for resolving the close cases and for addressing adverse events that are unlikely but not impossible. That is the hard part.
CONCLUSION
The fact that the IRB process confronts ambiguities and competing standards is little comfort to researchers who believe their work is being delayed unnecessarily. If anything, this is a reason to clarify and address these ambiguities and tensions rather than to simply shrug our shoulders and say “it’s complicated.” A meaningful reform proposal would take far more space than available in this article, but if I were to offer one pragmatic suggestion for improving the IRB process, it would be to plead for better data. Even decades after the regulations have been in place, in social-science research we lack the most basic information about what we are doing and whether it actually does any good. The 2007 Illinois White Paper stated the issue as follows:
[E]xamination reveals that virtually no scientific evidence is brought to bear on any aspect of the debate about how IRBs function. Unrealistic and untested assertions abound. At the microlevel, this includes, for example, how IRBs decide what is adequate and respectful informed consent, what subjects perceive as risk, and what kinds of benefits to subjects and their communities make the relationship fair. At the macrolevel, there is virtually no research on the functioning of IRBs (numbers of protocols reviewed, numbers of serious abuses by discipline, common turnaround time, etc.) or of the effectiveness of the IRB system in protecting human subjects (Gunsulas et al. Reference Gunsalus, Bruner, Burbules, Dash and Matthew Finkin2007, 639).
Are we really protecting subjects from harm? Do all of the compliance and reporting requirements make a material contribution to the welfare of research subjects? Does the IRB process impede research that would improve society? My answers are most likely yes, no, and possibly, respectively. However, the real answer is “nobody really knows”—and there is little prospect of significant improvement until we do.
