Protection of Human Subjects Long Overdue

The horrors of medical experiments performed on captive populations in World War II were so egregious and became so well known that it is surprising that regulation was so long in coming. The Nuremberg Code, emerging from the post-war trials, set standards for judging physicians and scientists involved in those experiments on concentration camp internees. Subsequent to Nuremberg the 1964 Declaration of Helsinki was issued providing further ethical guidelines for research.Footnote ¹ But it was the Tuskegee case that finally motivated action in the U.S. That study began with the reasonable intention of determining if the treatments for syphilis then in use, which were often very dangerous, were better off not being used. Nearly 400 largely illiterate African Americans, however, were not given informed consent, were not told that they had the disease, and, most importantly, were not treated for the disease even after penicillin was shown to be safe and effective in 1947. The Tuskegee study ended when the press exposed it in 1972. In the United States, the National Research Act was signed into law in 1974, nearly 30 years after the end of the war.Footnote ² The act created the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. It was not until 1979 that The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research, was issued, which begins by making explicit reference to the biomedical experiments carried out during the war and to the Nuremberg War Crime Trials and includes reference to the Tuskegee project. Finally, in 1981 the Department of Health and Human Services (DHHS) of the Food and Drug Administration (FDA) approved a set of regulations that are known as the Code of Federal Regulations (CFR), Title 45 (public welfare), Part 46 (protection of human subjects), and 56 (institutional review boards). It was not until 1991 that this regulation was widely adopted by other agencies of the federal government, 17 in all, and became known as the Common Rule.

One wonders why it took so long for the U.S. Congress to establish the protection of human subjects involved in research studies. No doubt there is an interesting story here, linked to the power of medical schools and the pharmacology industry, but that subject is beyond the focus of this essay. Fortunately, despite the long delay, a nationwide system has come into place that protects human subjects in research. Moreover, the U.S. rules have sparked similar rules in many other countries, including some developing countries.Footnote ³ Violations will always occur, but one can be confident that those violations are the exception, and are likely to be quickly detected, reported, and corrected. It is inconceivable that anything like the Tuskegee study could be repeated today.

Today human subjects research at U.S. colleges and universities (and all other institutions, for that matter) in the United States is governed by CFR Title 45 Part 46. It is commonplace, however, to hear social science and humanities researchers working on non-federally funded human subjects research, or research that has no funding of any kind (such as small-scale studies), expressing great surprise (and not a little irritation) that their work is subject to the Common Rule. Although some universities of late have taken the position that such nonfederal research is indeed not covered by the regulations, most, on advice of their risk-management attorneys, insist on universal compliance irrespective of the source of funding.Footnote ⁴

An argument can be made that non-federally funded research is not covered by the rules. Any reading of these regulations makes it clear that they do not apply to research that is not funded by the U.S. government. The very first section of the CFR, “To what does this policy apply,” states:

In other words, the human subject regulations apply only to research on human subjects that is conducted or supported by the federal government. It does not apply to research that is not conducted by or supported by the federal government. The great majority of all colleges and universities in the U.S. have extended the regulations to apply to all human subjects research, supported by the federal government or not. The reasoning often used to make this blanket expansion of the application of federal human subjects research standards is that federal grants to a given college or university make research on human subjects at that institution as a whole subject to such regulation, just as colleges and universities apply nondiscrimination rules to all aspects of hiring and student admissions even when the particular employees or students are not receiving federal funds.

Even though the case can be made that nonfederal research should be exempt, this exemption is groundless if one is serious about the basic principle of protection of human subjects. If one accepts the fundamental principle that human subjects should be protected, then it is easy to argue that the issue of the source of support, or indeed the presence or absence of support, should be irrelevant. Consider a small-scale medical study involving only a handful of patients carried out by a private hospital and not supported by any grant, federal or otherwise. Human subjects in such a study most certainly deserve to be protected, and the source of the funding, if any, should have no bearing on their rights as patients.

In sum, nearly all research on human subjects is now regulated, as it should be, and patients (and subjects) are now, quite literally, “safe in their beds.” Yet this does not mean that all is well with the protection of human subjects when it comes to the social sciences, and in particular to the area of survey research, the methodology I use in my own work. In the sections of this essay that follow, I contend that while researchers all need to be aware of the principles and regulations for protecting human subjects and apply them rigorously in their own research, it is also important take a close look at the risk/reward equation in the way in which the regulations are being applied and enforced.

The Belmont Report: Problems of Assessing the Risk/Reward Ratio

The regulations that govern the operationalization of human subjects protection provide for the establishment of campus-based institutional review boards (IRBs) rather than a centralized, federal-level board. In effect, the government outsourced to campuses and other research institutions the regulation of human subjects protection, a policy that at the time it was implemented in 1981 seemed like a very wise decision. The argument in favor of such decentralization was that the regulations would become, in effect, self policing, with each campus being able to respond to the particular and local conditions of the research. The idea was to avoid a big brother police officer for the protection of human subjects and to simultaneously recognize the enormous diversity of research designs, methods, objectives, and settings.

Decentralization of the regulation of human subjects protections, while a good idea in theory, has run into serious problems in practice. One problem is the lack of uniformity in the way the IRBs at each campus interpret and enforce the regulations, all of which derive from the three “basic ethical principles” established in the Belmont Report (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979): (1) respect for persons, (2) beneficence, and (3) justice. Each of these principles confronts problems in their application. For example, the respect for persons provision has given rise to excessive and erroneous interpretations. The Belmont Report notes that some individuals have “diminished autonomy,” such as children and prisoners, and when those groups are objects of studies they receive intensified scrutiny from IRBs, as they should. But this principle has also been applied to pregnant women (Subpart B of Title 45, Part 46), such that some IRBs have insisted that in surveys the first question that must be asked of females is: “are you pregnant?”Footnote ⁵ Given the multitude of difficulties that survey researchers face in persuading subjects to respond to surveys, it is easy to see why a question such as this one would have a chilling effect on many female respondents, encouraging them to terminate the interview.

Similarly the Belmont Report justice principle, designed to protect the poor and various minorities from having to bear more than their fair share of the costs of experimentation, has been poorly interpreted by some IRBs. Investigators are asked to justify why some individuals are not included in a survey of public opinion, and provide detailed explanations for why they were excluded. For survey research based on sampling this is an absurd standard. How does one attempt to explain to randomly non-sampled individuals why they were not chosen to answer a survey? However, the mere requirement imposed by these IRBs to explain non-inclusion in a study could open the door to class action lawsuits formed by groups of those excluded.

But no component of the Belmont Report has been more troublesome than the beneficence requirement. This principle finds its applied dimension in the report in the section on “The Nature and Scope of Risks and Benefits.” Here we face “The requirement that research be justified on the basis of a favorable risk/benefit assessment … Accordingly, so-called risk/benefit assessments are concerned with the probabilities and magnitudes of possible harm and anticipated benefits.” Belmont goes on to correctly recognize, however, that, “Only on rare occasions will quantitative techniques be available for the scrutiny of research protocols.” Nonetheless, even though risk can only rarely be measured, the Belmont Report requires that “the idea of systematic, nonarbitrary analysis of risks and benefits should be emulated insofar as possible.” In other words, even though it cannot really be done, it must nevertheless be done and the calculation of a project's benefits outweighing its risk is one of the three elements that must be used to determine if an IRB will approve or reject a protocol.

Much of the critique of this component of the Belmont Report and its impact on IRB decision making has been on assessing the risk side of the equation, but it is appropriate to look first at the benefit side, for if there is no benefit, it would seem that no risk, however infinitesimal, could be justified. How might one go about determining benefit? Few IRB protocols are brash enough to suggest that the study promises to revolutionize the field of study. Indeed, many protocols are based on research grant proposals in which overstatement of promised findings is almost surely to be met by disbelief from the funding committee. This would sharply reduce the chances for funding (which would, of course, obviate the need to submit an IRB protocol in the first place), and therefore it is quite likely that the principal investigator (PI) will make only minimalist claims for the potential benefit of the research. But does the PI really know, or even have a good hunch? No doubt oncologists hope that their research will produce the breakthrough that will finally unlock the dark secrets of a cure for cancer, just as democracy experts hope that their research will yield the holy grail of explaining why some nations democratize and others do not. Yet the reality is that not only is the probability slight that a single research project will transform the researcher's field, it is almost certain that nobody would know before the research is done, and sometimes for many years after, just how important a given discovery might be. After all, Nobel Prizes are usually awarded years, if not decades, after the research is carried out.

Risk, which is the measurement of the cost side of the equation, is likely easier to measure in the medical sciences than are the potential benefits. Enough may be known from animal studies, or prior human studies, to be able to estimate the probability of serious harm. But in the social sciences, how can we assess risk? In social science experimental studies there is the chance of risk, as the infamous Stanley Milgram obedience-to-authority experiments at Yale demonstrated, in which subjects thought that they were giving powerful electric shocks to those who would not obey (Miller Reference Miller1986). However, the problem of assessing risk is especially vexing for all of those who rely on large-N studies, typically in the field of survey research. Ironically, when only a handful of subjects are used in a campus laboratory-based experiment, the IRB is likely to approve the project with no objection. But survey research, which invariably relies on large-N studies, is viewed with suspicion by many IRBs simply because the risk, however small, is seen as being replicated 1,000 or more times, since most samples strive for confidence intervals of ±3% or better. Protocol analysts, who are used to seeing laboratory experiments and focus groups with samples of fewer than 100, are often taken aback when they confront the large sample sizes inherent in most survey research. And when they do, they question why such a large sample is needed. As a result, it is not at all uncommon to have IRB protocol analysts ask survey researchers to cut down their sample sizes.

The risk/benefit ratio has been addressed from the perspective of academic freedom in a detailed study published recently by the American Association of University Professors (AAUP) that states: “What is deeply troublesome is the fact that research on human subjects must obtain IRB approval whether or not it imposes a serious risk of harm on its subjects” (American Association of University Professors 2006). The concern has to do with the extensive information that those seeking exempt status often request that goes far beyond assurance to the respondents that they will not reveal their identities and that they have the right to refuse to answer or terminate the study at any time. The AAUP goes on to make the following recommendation:Footnote ⁶ “we recommend that research whose methodology consists entirely of collecting data by surveys, conducting interviews, or observing behavior in public places be exempt from the requirement of IRB review.”

When Research is Non-research

Perhaps one of the most perverse consequences of the regulations governing human subjects research is that while they closely regulate social science studies that have an extremely low risk of producing harm, they leave unregulated studies that have a far higher (but still immeasurable) risk of producing harm to human subjects. Note that I refer here to studies rather than research because it is that very term, research, that is the source of this perversity of consequences.

The federal government understandably had to delimit what is covered and what is not covered under the regulations. In those regulations, the government made the unfortunate decision, however, to regulate research rather than studies. It then found itself in the position of defining research and in CFR 46.102 did so in the following manner: “Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge.”

This definition of research is unfortunate because of two key words in the definition quoted above: systematic and generalizable. When a study is conducted that is not systematic, and is not designed to produce generalizable knowledge, the study is not considered research from the point of view of the government.Footnote ⁷ Much of what political scientists do is indeed both systematic and aimed at producing generalizable knowledge, even if it does not end up doing so, and so when human subjects are involved, the work comes under the aegis of human subjects protection regulations, as it should. But what of other styles of research that are less systematic, or not systematic at all? A classic case would be Fenno's “soak and poke” approach to his studies. Richard Fenno (Reference Fenno1996 18) describes this method, which has led to so many important findings, this way:

Does this mean that for the purposes of IRBs that Fenno's work is not research? Those who elected him to the presidency of the American Political Science Association and to the National Academy of Sciences apparently have no problem calling his studies research.

Some studies in political science, especially case studies, do not aim to produce generalizable knowledge. Some monographs written in the area studies tradition (e.g., “The Politics of Paraguay”) may not have been aiming for the production of generalizable knowledge, but others may have. Which authors of such monographs would be required to undergo an IRB review and which would not? And what of an author who began the research with no intention of producing generalizable knowledge, but after the fieldwork was complete suddenly saw that the work was indeed generalizable? IRBs strongly discourage, and in most cases refuse to grant, retroactive approval. Would this author, after having the light bulb go on and seeing generalizable knowledge emerging, be forced by the IRB to jettison the entire project?

The great irony in all of this is that in the humanities, or indeed in the humanistic areas of the social sciences, many scholars have little or no knowledge of human subjects regulations, and when told about them argue strongly that they are not subject to them. Indeed should they hear of them and should they approach their own IRBs, they are often quickly told that since their studies are not systematic and/or are not aiming for the production of generalizable knowledge, they are not doing research as defined by CFR 46 and therefore need not apply, not even for exempt status. From the point of view of the federal regulations, because humanistic studies are not classified as research, they do not represent any risks to human subjects. This perspective has been reinforced by the American Historical Association, which has formally declared that “oral history interviewing activities, in general, are not designed to contribute to generalizable knowledge and therefore do not involve research as defined by the Department of Health and Human Services regulations at 45 CFR 46.102(d) an do not need to be reviewed by an institutional review board” (Townsend Reference Townsend2004). This position was officially accepted by the Department of Health and Human Services in a letter in 2003.Footnote ⁸

These rulings and positions create two different sets of problems. First, and most important from the point of view of human subjects research, the risks of such studies in my view is no less than the large-N research that political scientists do. Long before human subjects regulations and the invention of IRBs, survey researchers in all fields instinctually knew that by guaranteeing anonymity they would encourage frankness on the part of respondents. After all, voters know that they have a “secret ballot,” so why should they give up that right to an interviewer? As a result, political scientists who carry out surveys have been aware for decades of the importance of guaranteeing anonymity to their subjects. Once human subjects regulations were established, little changed in the administration of surveys other than the addition of informed consent statements at the outset of each interview, which more or less told respondents what they already knew, namely that they could refuse the interview or refuse to answer questions. The positive impact of the IRB regulations on survey research is to ensure that everyone in the community is aware of the ethics and regulations.

But what of humanistic research where the tradition of anonymity is not the norm? In the field of oral history, as well as in the newer testimonio literature popularized in Latin American literature, many authors use the real names of those interviewed, which can present serious risks for those respondents. Historians are not only exempt from IRB control, they have no requirement or even need to take human subjects protection training and pass tests on their knowledge of the principles and rules. Literature faculties often have no knowledge at all of human subjects protection.Footnote ⁹ As a result, unlike social science departments in which knowledge of human subjects protections is widespread, some humanists may be naive about the risks involved in disclosing names of subjects. One can imagine many kinds of risk to respondents. One such risk is dismissal of employment from an employer who either might not like the views expressed in the oral history or testimonio or deems them harmful to the company's welfare. Potential employers might look at the oral history information and deny a position based on the statements contained therein. Another risk could be ostracism at work or in one's neighborhood for expressing politically unpopular views. One can even imagine law enforcement officials using oral histories to prosecute individuals for revelations that suggest criminal behavior.

A second risk of such studies is to the humanists themselves. Tenure decisions and pay raises are based at many institutions, to varying degrees, on research. What of the faculty member who is in a field that the federal government has declared, and the professional association has agreed, does not conduct research? Deans and provosts alike could take the position, “since you don't do research you can't get tenure here.” Of course the individual and department could turn around and say, “but of course I do carry out research,” at which point the institution's IRB could begin an investigation into noncompliance with the federal regulations. That is if an individual had in fact carried out research and had not gotten IRB clearance, this could spell real trouble for that person. IRB decisions cannot be appealed, so once a process of this nature begins on a given campus, it would be the responsibility of the local IRB to protect the institution from the dreaded cutoff of federal funds to the entire institution, as is provided under 45 CFR 46.123, and take action against the humanists who have circumvented their policies by first having declared that they do not do research and then declaring that they do. Far fetched? Federal whistleblower regulations, as augmented by the Sarbanes-Oxley Act in 2002, as well as state rules, create precisely the environment in which such a nightmare could occur.

When Public Behavior Becomes Private

When one reads the “observation of public behavior” forms on many university web sites, it is hard not to imagine that they were heavily influenced by George Orwell. In common parlance, public behavior is just that, something we do with the knowledge that others out there can and do observe us. If we did not want it to be observed by others, we would not do it in public. Town councils, regulatory agencies, and courts frequently hold open hearings, in many cases televising them on community TV stations. C-SPAN broadcasts several channels of TV and 24-hour radio programs that provide very public views of these activities. When a member of the general public attends council meetings or flicks on C-SPAN, no approval is required. When a political scientist does so, then the long arm of the IRB comes into play. Researchers who decide to utilize any of these sources for their research (presuming that the purpose is to attempt to develop generalizable knowledge) need first to apply to their IRBs for permission. Of course, we have other public behaviors, such as when we take a walk, go the mall, pray in church, and dine in a restaurant. Individuals are not intending to provide their personal information under those circumstances, but since their names are not public, only a genuine effort to invade their privacy would produce that information, which is clearly a violation of IRB rules as well as probably many federal, state, and local statutes.

Under most circumstances, requesting permission to observe public behavior represents only a small hardship for the investigator, but what happens when such behavior emerges serendipitously? Consider the researcher carrying out a study in a foreign country when suddenly a protest march occurs, or a coup d'état is attempted. What is one to do? Observing the event(s) without first obtaining IRB approval would potentially jeopardize one's research career. On the other hand, IRBs are not set up to provide instantaneous approvals of protocols, even if the researchers had the wherewithal to download the proper forms and submit them without being distracted from observing the history being made in front of their eyes.Footnote ¹⁰

Can IRBs take their responsibility to regulate to the extreme? A quick perusal of the web will reveal countless horror stories of ones that have. Anecdotes abound, but I cannot resist one of my own. A very senior IRB official at one university, in order to impress upon a political science faculty member his omnipotence, asked, “Do you ever use the library to read books about President Bush?” When the response was affirmative, he said, “Unless you file for IRB approval before opening those books, you will be held in violation, since Bush is a human, is living, and the books almost certainly contain personal information.”

Added Difficulties in Comparative Politics

Some IRBs make life especially difficult for those who do studies abroad. One IRB has the following language on its web site:

Consider the difficulties this requirement places in the path of researchers. First, one needs written authorization to carry out the research. Who would grant such authorization? In countries where IRBs exist, presumably they could do so, which means having the study authorized by two IRBs. Then there would be the potential for conflicting and irresolvable differences in the way human subjects must be protected since there is no prior guarantee that the two sets of regulations would be compatible. In countries where such IRBs do not exist, who can authorize the research? Consider the U.S. prior to the establishment of human subjects regulations. What authority could have granted permission to a foreign political scientist wishing to carry out a survey of public attitudes in this country? Who would de Tocqueville have asked for approval of his research? In foreign environments, just as in the U.S., there are almost always multiple and conflicting jurisdictions that could conceivably approve or deny research studies. Does one ask the central government, the local government, or both? Finally, the quotation above requires, in addition to permission, a statement that will attest that “responses to questionnaires or interviews about political, economic, cultural, or religious topic will not affect the participant's reputation, employability, or financial standing.” Who could provide such assurances? The mayor of the town, the governor of a state, the president of the country? And what if it is a sample to be carried out in several dozen locations? Does each mayor, town council, and governor need to grant permission?

Exempt Status: Guilty until Proven Innocent

The Kafkaesque nature of the human subjects protection regulation as they are being implemented nationwide today is that the authors were wise enough to exempt many of the kinds of research political scientists undertake. Regulation 46.101(b) states quite clearly that exempt from IRB regulation is

Survey researchers, as already noted, have a long tradition of unlinking (or not even recording) the identities of the respondents, and I can think of no book or article in which that tradition has been violated. One anticipates, therefore, that surveys should be exempt from IRB control.Footnote ¹² Indeed, that is how IRBs ruled for many years, but, according to Katz (Reference Katz2007 799), “Procedural immunity from IRB oversight was progressively effaced and then decisively abolished around 2000, as campuses came to reflect a view communicated by federal human subjects administrators that ‘self-exemption’ was contaminated by conflicts of interest.”

In effect, even though political scientists conducting educational tests and surveys are exempt from federal regulation, they are not, after all, exempt because the federal government believes we cannot be trusted. What is so strange here is that in countless other important ways, we are trusted by that same federal government. When we grade tests taken by our students, we are not allowed to discriminate on the basis of race, creed, national origin, sexual preference, etc. Yet we are not asked to sign a statement saying that we will not discriminate before (or indeed after) we grade each exam or before we determine final grades. We hold office hours, but are not asked to submit an application prior to each office hour, not even prior to the start of each term, to the affirmative action offices on our campuses that we will not sexually harass students. We submit articles to conferences but are not asked to submit signed statements saying that we did not plagiarize the material. Beyond our lives as professors, we have many obligations to obey the laws, and breaking them can have severe and immediate consequences. Thus we take driving license tests showing that we know the rules of the road and are competent to drive, but each time we turn on the ignition, we are not asked to certify that we will obey the traffic laws. Failure to do so, of course, annually results in most of the over 40,000 annual deaths from automobile accidents. These accidents produce a fatality rate of about 14 per 100,000, more than twice the current national murder rate, a level of real risk that wildly exceeds any imaginable harm that surveys could possibly be causing, regulated by IRBs or otherwise. Bus drivers and train operators take great responsibility in their hands when they transport passengers, but do not need to recertify themselves each and every time they begin a new journey. Yet that is what political scientists are being asked to do. We cannot be trusted to have studied the rules and to have passed examinations proving that we know the rules and to have regularly been retested to make sure that our knowledge had not faded over time. Rather, we must take the additional step of submitting a protocol for each and every study we carry out.

Equally troubling is the ongoing control that IRBs have over the evolution of a study. Consider the problem of the survey researcher. Almost all surveys undergo many refinements and revisions as pretests and trials produce responses that cause us to make changes in questionnaire wording and/or coding. Many IRBs, however, demand that “amended protocols” be submitted when even a single comma is changed in a questionnaire. Consider the implications for this hyper vigilance on the real world of survey research, especially in a foreign context. A surveyor could be in the field and discover that in a certain region a different word is used for a local institution. In order to change that single word, the PI would need to call a halt to the survey, submit an amended protocol explaining the reason for the change and attach a revised questionnaire, and then await the decision of the IRB. If no snags emerge, the survey might be able to resume with only a week's delay. Of course, during that week, salaries and per diem are paid, and given the lean budgets that most of us work under, this would of necessitate cutting down the fieldwork by a week, thus reducing the sample size. This would now require another protocol modification, but it will also increase Type II errors, since the reduced sample size would increase the chance that a true finding will emerge as statistically insignificant.

The risk/reward ratio is clearly entirely out of whack. Once IRB's nationwide began making the assumption that social scientists cannot be trusted to act ethically and protect human subjects from harm, then the full weight of the regulatory burden has fallen on their shoulders. What are the benefits for applying for exempt status for each and every survey? What are the benefits of requesting permission to make changes in questionnaires? What are the risks that are being minimized by this scrutiny?