Atul Gawande is a surgeon who began creating vivid word paintings of the performance of the American health care system in The New Yorker's pages while he was still a resident. Those of us who are immersed in, and charged with measuring, that system—as I have been for the last decade—find his “Annals of Medicine” column's evocations of heroic system successes and frightening system failures true to the processes we more prosaically try to analyze and evaluate. In December 2007 Gawande published a New Yorker essay called “The Checklist,” followed just days later by an op-ed piece in the New York Times (Gawande Reference Gawande2007a; Reference Gawande2007b).
“The Checklist” gave a very human face to an increasingly challenging dilemma: the unintended collision of two beneficent principles, the protection of human subjects that is entrusted to institutional review boards, and the improvement of the quality, safety, and effectiveness of health care that is demanded of our health care system. The story is as follows: Peter Provonost, a critical care physician at The Johns Hopkins University, decided to apply the kind of checklist long established as the gold standard for safety in aviation to intensive care units (ICUs). Provonost's checklist was meant to tackle a common and very dangerous problem in ICUs: “line infections,” or the introduction of bacteria into the bloodstream via the intravenous lines through which patients receive fluids, nutrition, and medications. Lines save lives and are an indispensable part of critical care, but line infections take lives as well. Provonost's checklist looks breathtakingly simple, with just five steps: doctors should wash their hands, cleanse the patient's skin, put sterile drapes over the entire patient, wear a sterile mask, and put a sterile dressing over the line once it is in. A year after Provonost had introduced his checklist at Johns Hopkins (including empowering nurses to stop any doctor who missed any step), the results were astonishing: the team believed that the checklist alone had prevented more than 40 infections, saved eight lives, and saved two million dollars in that single year by not having to pay the costs of treating line infections that didn't occur. Provonost and his team developed and used more checklists and conducted proper before-and-after evaluations of their effects (Gawande Reference Gawande2007a; Reference Gawande2007b).
In 2003 the Michigan Health and Hospital Association invited Provonost to introduce three of his checklists into the state's ICUs. The results—in lives and dollars saved—were astonishing, and Provonost and his colleagues published their findings. The Johns Hopkins IRB had ruled the checklists were exempt from federal regulations and requirements for consent. The Department of Health and Human Services' Office for Human Research Protections (OHRP), however, shut the program down. OHRP ruled that the introduction of the checklists, followed by the measurement of their effects and publication of the findings, constituted human subjects research, for which the Johns Hopkins and Michigan actors had been improperly exempted from IRB monitoring and consent requirements. OHRP required Johns Hopkins and the Michigan Health and Hospital Association to obtain informed consent from every patient a checklist would be used for before the program could resume, an impossible burden (Gawande Reference Gawande2007b; Miller and Emanuel Reference Miller and Emanuel2008).
Research versus Improvement
The definition of research is the collision point. Because 45 CFR 46, the part of the Code of Federal Regulations governing human subjects research known as the Common Rule, defines research as “a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge” (HHS 2005b, §46.102; italics added), OHRP ruled that the use of checklists without IRB approval and informed consent violated human research ethics. This problem—of determining when quality-improvement innovation is research, how quality-improvement strategies can be tested if they are not systematically measured, and how the knowledge of such improvements can be shared if it is not generalizable (usually interpreted as publishable)—is increasingly urgent. Surely both goals—of research ethics and quality improvement—fundamentally further the good of individuals and society and the collision is an unfortunate one. Political science research can contribute to avoiding the collision and harmonizing the goals.
For the purposes of this essay, I will conflate several concepts that have particular meanings to health services researchers—quality of care, health outcomes measurement, patient safety, effectiveness, and performance improvement—in the general construct of quality improvement.Footnote 1 Scholars of public policy, political economy, and public administration will be familiar with the notions of continuous quality improvement that moved from W. Edwards Deming's applications of his theory in postwar Japan to the business world in the 1980s. Quality improvement similarly began making its way into health care in the late 1980s and early 1990s because of a growing awareness that our hugely expensive and complex health care system was not delivering the quality of care we would expect for the investment. Systems approaches to improving the safety and quality of health care became the paradigm by the mid 1990s (Lohr and Schroeder Reference Lohr and Schroeder1990; Berwick Reference Berwick1989; Chassin, Galvin, and the National Rountable on Health Care Quality Reference Chassin, Galvin and Quality1998; McNeil Reference McNeil2001).
A quality-improvement perspective on the processes of health care systems was legitimated as early as 1982, when the Department of Health and Human Services (HHS) permitted a waiver of human research protection regulations for “demonstration projects” of Medicaid costs and service delivery (HHS 2005a).Footnote 2 Today using quality-improvement strategies is becoming one of the leading imperatives facing academic medical centers and other large health care systems, and one of the largest foci of health services research. This has been spurred on by the Center for Medicare and Medicaid Services's “pay for performance” and “value-based purchasing” programs, the recent Medicare “coverage with evidence development” standard for reimbursement (Pearson, Miller, and Emanuel Reference Pearson, Miller and Emanuel2006), the Joint Commission for the Accreditation of Healthcare Organizations regulatory standards, and even the recertification and re-licensure of physicians and surgeons. But how can we manage and improve what we do not measure? And how, having measured improvement, can we fail to report on the results? Finally, acknowledging that such measurement is research, is it reasonable or appropriate to hold this research to the protection standards required for dangerous, unproven, or experimental projects?
IRBs were not brought into being primarily to govern political science research. Instead, IRBs were the long-delayed response to a growing public demand to prevent tragedies of abuse like the Tuskegee experiments, the Nuremburg Trials, the Willowbrook School scandal, and other distortions of “medical advancement” from happening again, (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979; Davis Reference Davis2008). Political science research became subject to the Common Rule definitions of “behavioral” research (HHS 2005b), but our work has never been the primary focus of human protection concerns.
This essay is not about the influence of IRBs on political science research. It is, instead, about the tremendous influence political science research could have on IRBs—or, rather, it is about our discipline's ability to contribute to improving IRBs for the public good. The collision of human research protections and quality-improvement research in health care calls for the kind of triangulated analysis that political science, and political scientists, are supremely fitted to undertake.
Our need for constructive, translatable scholarship on solutions—especially political solutions—to the collision is great. Gawande's story of the Provonost checklist is dramatic, but not atypical (for two other reflections on the collision see Doezema and Hauswald Reference Doezema and Hauswald2002; Davidoff and Batalden Reference Davidoff and Batalden2005). Public and private sectors alike are leaning on health care systems to demonstrate improvement in the quality, safety, and effectiveness of health care. But how does one demonstrate improvement without doing what the Common Rule (HHS 2005b) defines as research that requires human subjects protection—that is, without producing generalizable knowledge? And how are overwhelmed IRBs to cope with this explosion of new research?
IRBs are understandably, and necessarily, driven by their limiting case: the possibility of invasive or dangerous procedures performed with vulnerable individuals or populations, without adequate regard for the hallmarks of protection, respect for persons, beneficence and nonmaleficence, and justice elucidated in what is widely known as the Belmont Report, and adopted from the time of the Belmont Report's release as our fundamental ethical principles (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research 1979). It is not surprising, given IRBs' role as the protector of the vulnerable, that the general IRB perspective on “minimal risk” and risk-benefit comparisons is a conservative one.
IRBs are also terrified. All IRB professionals I know work in real fear that their IRBs could be the next ones to cause an entire university's research to be shut down. Shutdowns in recent years at Harvard, Duke, and Johns Hopkins give IRBs every reason to be fearful of making what OHRP considers to be an error. The reasonable suspicion that researchers regard IRBs as obstacles to, rather than facilitators of, research, must further IRB professionals' sense of being embattled.
Reviewers on IRB committees are our very hardworking colleagues who are usually not given adequate release time to meet their committee responsibilities, and who are not able to benefit from truly extensive and nationally standardized training, nor do they have anything like a consensus metric for evaluating the spectrum of risk in different research contexts and for different populations.
All these sources of strain might determine the conservative approach to human subject protections. When social science research (including quality-improvement research) occurs in a biomedical context, or when health care and health policy require evaluation, the conservative stance can become dysfunctional. IRB assessments of my own students' work provide a clear example of one of the ironic and unintended consequences of the absence of agreed upon and broadly understood metrics for assigning risk in different research contexts. IRBs have a comparative lack of familiarity with how social science methods—such as those used in quality-improvement research—may differ from some other methods of clinical research in the risks they pose to subjects.
One of my jobs as the co-associate director of the University of North Carolina's joint MD-MPH (master of public health) program is to advise and direct the original research of the medical students and physicians who wish to conduct policy analyses for their masters' papers. As with so much policy analysis, we generally construct triangulated research designs integrating content analysis of public documents and media accounts with textual analysis of in-depth interviews with elite stakeholders. These are classic cases of “interviewing up,” where the respondent, far from being a powerless or vulnerable subject, is a good deal more powerful than the interviewer. In these cases one might hope that the IRB would be prepared, if necessary, to protect the researcher from any recrimination by a powerful respondent rather than the more usual stance of protecting the research subject from unwarranted invasion (Sieber Reference Sieber1989).
My medical student advisees and young postgraduate physician fellows commonly interview stakeholders who might be leaders of academic medical centers or of industries, representatives of advocacy groups or peak associations, highly placed civil servants or congressional staff. The interviewers ask no personal questions of any kind, much less any questions about the respondents' health. The respondents are chosen on the basis of the position they occupy, rather than any personal characteristics. The interview questions my students ask are entirely focused on respondents' perceptions of public, political, and policy phenomena. By any definition such elite interviews cannot reasonably be construed as putting the respondents at any more than minimal risk.
Because we conduct this research from the institutional bases of the Schools of Medicine and Public Health, our IRB submissions start at a central point in the biomedical IRB where they might be reviewed by peers who are familiar with and trained in clinical trial research, or behavioral research of a sensitive or invasive nature. Although I supervise each student's IRB application, every application presents the same general methodological presentation, and the semi-structured interview protocols are substantially similar, my students' IRB determinations range from declarations that the project is “not human subjects research” at all to “research that requires full consent,” with every intermediate determination. I frequently have students working simultaneously on very similar projects, one of whom must go through a tedious consenting process taking as much as four to five minutes of the beginning of telephone interviews (with the busy elites at the other end chafing at these long and unnecessary prefaces to the first question), while another student researcher is not required to secure any kind of consent at all. The single source of variation across these cases is not the research question or the method, but the IRB reviewers and their familiarity (or lack thereof) with in-depth interviewing and the unique protections already available, via one's position, to the “powerful research subject.” The source of variation in judging quality-improvement research, as the checklist case makes clear, is the same: widely differing interpretations of the meaning of and restrictions on research.
A Political Science Research Agenda
My students and I always manage to cope with whatever result we obtain from the IRB. The point of this vignette is not to criticize IRBs. Its message, rather, once again invokes the collision of the normative perspectives of advancing the public good by enacting stringent human research protections, and of advancing the public good by facilitating quality improvement and evaluative research that can lead to real and positive health system change. But IRBs alone cannot avert the collision of these two public goods. Political scientists can make a valuable contribution to the solution by creating and executing a new research agenda whose goal is to supply analyses that can redirect the courses of these normative imperatives from collision to integration.
Our expertise in so many relevant areas—from philosophy and ethics to policy analysis, from the most sophisticated quantitative techniques to the richest qualitative methods, from the study of underrepresented groups to the analysis of power—prepares us to examine improved human subjects research policy on every level and from every perspective. Our own discipline is a rich polyglot, drawing insights from the use of varied strategies all connected by our shared appreciation of the enduring questions of power, the allocation of resources, the regulation of conflict, and interest in human society. With such familiarity and with such a diversity of methods and theoretical approaches, political science is well prepared to analyze and make normative (but evidence-based) recommendations about the politics of human subjects research. We can help define what it is, and the circumstances under which it is generalizable knowledge, even though it may not necessitate a conservative approach to protections. We can construct frameworks to achieve a more precise understanding of how to balance risks and benefits. Those frameworks might even lead us to formulate what would amount to a multidimensional risk scale. Finally, political science can contribute to the construction of theoretical and methodological underpinnings for the content of truly national standards for IRB training curricula. These would improve IRB reviewers' understanding of different research methods to go beyond mere compliance with federal regulations and become real resources and decision aids for hard-pressed reviewers who may have to evaluate research they aren't familiar with.
First, as physician-political theorist Ezekiel Emanuel and his colleagues in the Department of Clinical Bioethics at the National Institutes of Health have already begun to do,Footnote 3 we can craft a careful analysis of the limits on the Common Rule's definition of research as generalizable knowledge. Here, of course, ethicists have long been weighing in. Nonetheless, scholars of political theory, public law, public policy, and civil rights and underrepresented groups can contribute a great deal to the debate. Surely vulnerable populations must be given a full range of rights and protections to guard them against a medical research establishment that has shown itself capable of egregious violations of patients' rights. When, however, does protection have the unintended effect of making everyone an inadvertent experimental subject? Protection is so burdensome that it prevents evidence about improvements in the health care we receive from being measured, communicated, and critically appraised.
Second, and following from the first, an agreed-upon national consensus about the meaning of risk in varying research contexts would support consistent approaches to the evaluation of research from different disciplines, involving differently positioned research subjects. This consistency could contribute to real efficiency, making better use of IRB resources—especially their human capital—and better use of investigators' efforts. The praiseworthy desire to protect truly vulnerable subjects must be retained, but some subjects are not vulnerable, such as the powerful. Other subjects, though vulnerable, may not be at risk from research, such as when that research is constructed to measure improvement in ongoing health care processes or to build an evidence base for analysis of safety and effectiveness. A standardized, multidimensional risk scale to measure different research proposals against would permit more sophisticated grouping of applications by methodological approaches and the nature of the research questions they entail. This entails dimensions of risk. They include the level of invasiveness of different methods, the relative vulnerability of the populations, the burden of administration of the research for respondents and investigators, and other factors. What is left to be done is to synthesize those dimensions into a global and yet finely grained whole. Political science's long experience with measuring and accounting for multidimensionality, and our intellectual tradition of scholarship on rights, balance of interests, and public-private tensions, prepare the discipline to engage in such epistemological research. Such a multidimensional risk scale could ease the emergence of consensus about how many protections any given research project—whether for quality improvement or otherwise—requires. Political scientists from several subfields have much to contribute to the construction of this multidimensional metric of risk and benefit.
We should not merely conclude with the development of a multidimensional risk scale. That scale should become part of the core of newly standardized curricula and protocol for IRB reviewers. Further, standardized curricula and protocols need to be supported by resources adequate to assure their widespread implementation, and IRB reviewers themselves need more support, in the form of release time or additional compensation, or both. Both of these needs—for standard curricula and for more resources—may call for changes in IRB governance, and political scientists have perhaps the keenest of vantages on questions of governance.
We can evaluate evolving IRB regulatory policy. The final authority over human research protection, of course, rests in the HHS's Office for Human Research Protections, itself governed by the Common Rule 45 CFR 46 (HHS 2005b; 2008). As is so often the case in the American public-private hybrid system, however, much of the governance of IRBs emerges from self-regulation. Institutional IRB certification mechanisms are administered by the quasi-public Association for the Accreditation of Human Research Protection Programs (AAHRPP), which was created by a close partnership between the Association of American Medical Colleges and the Consortium of Social Science Associations, among other organizations. AAHRPP's (2008) accreditation standards are thoughtful and wide ranging, but they understandably emphasize compliance with 45 CFR 46 (HHS 2005a; 2005b) requirements for IRB creation and maintenance over, for example, the development and promulgation of standardized reviewer training for weighing risk.
Maintenance of AAHRPP accreditation, moreover, requires the institution to demonstrate that it has committed adequate resources to monitor human subjects research. But those resources must be found by the individual institution, as an increasingly challenging unfunded mandate. How will the pressures on institutions to produce more research of all types—including quality-improvement measurement in health care—be accommodated within relatively fixed IRB budgets? What new sources of revenue might be possible? Political scientists can probe the politics of IRB funding. Similarly, Public Responsibility in Medicine and Research (PRIM&R) has become an increasingly valuable learned association for IRB professionals and offers them certification programs among other educational opportunities, but once again those programs appear to be focused on compliance questions rather than on creating a common epistemology of the philosophical, ethical, and political foundations of human subjects research. Political scientists are ideally prepared to contribute to the latter.
Conclusion: The Checklist Survives, but …
Gawande brought Provonost's and his Johns Hopkins and Michigan colleagues' checklist—and the apparently unreasonable halting of its use on human research protection grounds—to the attention of the public in December 2007 by presenting the story in two publications widely followed by policymakers, The New Yorker (where he could present the story in rich detail), and the New York Times, where he could make a blunt argument in favor of having OHRP's position reversed. By February 2008 Emanuel, the director of the Department of Bioethics at the National Institutes of Health, and his colleague Franklin Miller, had developed an exquisitely and carefully reasoned argument refuting the grounds on which OHRP had suspended checklist use. They also specified the grounds on which it could be used ethically and legally to continue to promote the public good of improved patient safety and quality of care. Miller and Emanuel published their arguments as a “Perspective” in the New England Journal of Medicine, one of the most significant ways of reaching elite health policymakers (Miller and Emanuel Reference Miller and Emanuel2008). Only two months after Gawande published his two accounts, and just six days before the Miller and Emanuel piece appeared in print in the New England Journal, OHRP, which perhaps had heard from their colleagues at the National Institute of Health's Bioethics Department, issued a press release. The release said in part that
Michigan hospitals can continue implementing a checklist to reduce the rate of catheter-related infections in intensive care unit settings (ICUs) without falling under regulations governing human subjects research … the Johns Hopkins project has evolved to the point where the intervention, including the checklist, is now being used at certain Michigan hospitals solely for clinical purposes, not medical research or experimentation. Consequently, the regulations that govern human subjects research no longer apply and neither Johns Hopkins nor the Michigan hospitals need the approval of an institutional review board (IRB) to conduct the current phase of the project.
(OHRP 2008)The press release quoted Dr. Ivor Pritchard, acting OHRP director: “We do not want to stand in the way of quality-improvement activities that pose minimal risks to subjects … HHS regulations provide great flexibility and should not have inhibited this activity. The regulations are designed to protect human subjects.” Finally, the press release concluded, “HHS is reviewing the application of these rules to evidence-based quality-improvement activities, like the Johns Hopkins project, and whether any changes are needed to encourage such work while safeguarding the rights and welfare of human subjects in research” (OHRP 2008).
Our present policy structure governing human research forces the collision of two normatively significant goals, human research protections and improvement of health care quality and effectiveness. Key stakeholders are becoming more aware of the consequences of this collision course. Neither goal can be abandoned. Each goal, however, requires more nuanced understandings of how both might be realized with the greatest benefit to individuals and the public. The whole human research policy structure can benefit from a rigorous evaluation. All these are projects political scientists should undertake.
