Evidence-Based Medicine: Characteristics

Evidence-Based Medicine begins with the laudable desire to improve the quality of data used to evaluate diagnostic and therapeutic maneuvers. Most significantly, EBM recommends a shift from traditional, observational studies (with few or no controls) to randomized studies (with maximal controls). EBM is a self-conscious drive to make medical practice more scientific.

Stirrings in the medical community began in the mid-1980s, when some researchers argued that questions about medical care could not be analyzed effectively using observational data alone.¹

As the EBM movement gained momentum, enthusiasts soon argued that in the absence of EBM clinicians were practicing in the dark. Undergraduate and postgraduate training programs incorporated EBM in medical school and residency program curricula, and centers for evidence-based practice were established in most medical specialties. Evidence-based journals were launched. The EBM idea became so institutionalized that virtually all physicians—even those professionalized decades earlier— became aware of its underpinnings, premises, and conclusions.

EBM institutionalizes a new hierarchy of evidence. At the apex of a so-called quality pyramid is the randomized controlled trial. Relegated to a lesser status are observational studies. At the very bottom are case reports (usually consisting of a single or several detailed, patient cases). As one might guess, the data derived from “lesser-quality” evidence are given short shrift. So certain are some EBM proponents of the merits of this approach that their attitude borders on hubris. One group of prominent EBM enthusiasts asserts, “If you find that the study was not randomized, we suggest that you stop reading it and go on to the next article” (Sackett, et. al. 1997, 94).

Precursors to, and the Emergence of, EBM

The rise of the EBM movement has been the signal development of the past two decades of medical research. It is a development well worth viewing in historical context.

Before 1900, health workers had scant comprehension of human physiology and biochemistry. Clinical practice was largely devoid of scientific grounding, allowing the clinician to treat symptoms only; armed with a meager and ineffective pharmacology, he had to wait for the natural history of the disorder to play itself out. To the physician who operated in relative isolation from his colleagues, the macro-patterns of epidemiologic and physiologic processes remained invisible and scarcely informed his everyday practice.²

Over the next 100 years, medical care in advanced industrial countries improved exponentially, in great part because of the improved understanding of basic medical sciences that subsequently was translated into useful, bedside practice. Clinicians could make better practical use of their understandings of how the human body works, what happens to it when it becomes ill, and how therapy could change for the better the natural history of disease.

Why did the dramatic shift to EBM occur? First, physicians gradually had begun to communicate differently. At the beginning of the last century, tremendous variation in medical practice existed. Broad-scale discussion of the causes, natural histories, and treatments for disease was all but absent, given the difficulties of such information exchange. The falling costs of professional communication dovetailed with a recognition that physicians were part of a larger professional community working on similar problems. Over time, as researchers began to trade experiences with wider circles of colleagues around the world, an unprecedented array of data became available—much of which entailed contradictions that begged explanation.

This onslaught of information invited medical researchers to change strategy. At minimum, it raised awareness of a need for more systematic approaches; for many, the solution was to reduce the “noise” of multiple and contradictory data-flows by constructing highly controlled experiments. The need to systematize evidence had become apparent.

The pooling of resources, knowledge, and data began to occur globally. Today, it is no longer surprising if a clinician from Bulgaria and a clinician in California practice the same quality of medicine, using the same techniques and the same drugs.³

Change in funding patterns played a second role in inducing the shift to EBM. The National Institutes of Health and pharmaceutical companies increasingly drove research agendas. Enormous expenditures began to occur on an unprecedented scale. Of course, there was little new in the government's role in funding research. What was novel was the role of the private sector. The pharmaceutical industry found that by engaging laboratory-based techniques it could develop new products that, when approved by government agencies, rapidly became profitable. That the average consumer knows phrases like “compared to placebo” is testimony of the shift to EBM procedures and the power of marketing by drug companies. It is a matter of some debate to what extent these companies are tapping already-existing consumer demand, and to what extent they, in fact, create such demand through marketing, but the point remains: the drug industry has benefited from EBM and has pushed its techniques.

Is the Proof in the Pudding? EBM's Mixed Results

One of the peculiarities of social science research is how difficult it is to trace its impact outside academe. Shapiro (forthcoming, 1) argues that “tangible advances in knowledge” are hard to identify in the social and human sciences, let alone any impact of that knowledge. This stands in some contrast to research in neighboring disciplines. If improved public health and quality of life can be attributed to medical advances, then medicine must be doing something right. If such research proves useful, then its assumptions and methods have served society.

Indeed, some of the consequences of EBM have been profoundly positive. Examples abound. Coronary artery bypass grafting developed from randomized controlled trials that assessed the relative efficacy of medical versus surgical treatments of heart disease. The introduction of newer and more effective antibiotics is now almost always preceded by clear and direct evidence from randomized controlled trials that such antibiotics are superior to those previously used. Advances such as these emerge from strictly controlled experiments.

EBM has also stimulated the need to provide high quality evidence for alternative (especially herbal and non-Western) therapies. In the U.S., most practitioners had long ignored alternative therapies. EBM forced the issue by subjecting alternative treatments to laboratory testing. Medical schools established Departments of Alternative and Complementary Medicine, and the National Institutes of Health created a separate Institute of Alternative Medicine. Thus, practitioners have (if grudgingly) come to accept alternative medicine as a parallel field of medical practice, albeit one that requires close examination to evaluate its various claims.

Not all of EBM's consequences, however, have been positive. While improving data quality in medical research is the stated goal, the result has been to narrow the range of data sources considered acceptable for research purposes. As a practical matter, data from the lower tiers of the “quality pyramid” are often ignored, even when observational studies provide valuable information that randomized, controlled trials cannot generate.

This neglect of “lesser-quality” data sources occurs because of a broad perception that only randomization can eliminate researcher bias. In fact, two review articles in the prestigious New England Journal of Medicine report quite the opposite: observational studies gave results similar to those of randomized, controlled trials. The frequently heard claim that observational studies bias results was not borne out (Concato, Shah, and Horwitz 2000, Benson and Hartz 2000). Medical research stands to benefit from EBM, but only if other forms of evidence are not excluded.

The obstacles, however, are not merely intellectual; they are also practical. EBM has become the sole method of data collection for the pharmaceutical industry, which drives much of the research for effective therapies. The only way for a drug company to secure FDA approval to market a new product is to demonstrate that it has an increased marginal efficacy over existing remedies. This is difficult under any circumstances, but it is made easier if the sample size is 10,000–15,000 patients. Such studies, tremendously expensive to conduct, are nonetheless pursued because they can successfully show efficacy. The result has been an explosion of “me too” drugs—i.e., agents that differ only in a minute molecular sense from existing remedies—but that offer some statistically demonstrable effect over existing agents. The problem is that while the confidence about the effect may be high, the substantive effect may be infinitesimally small.

This means that resources and time are spent on a number of research agendas that might be better directed elsewhere. Consider, for example, drugs that lower levels of cholesterol. It is not that each successive drug markedly improves public health. Rather, each successive drug, differing from previous similar agents only slightly, offers the promise of profits, enabled through randomized, controlled trials. By contrast, problems that lend themselves neither to profit, nor to “top-quality” data are underexamined. Far fewer research dollars are spent on finding cures for such serious conditions as leprosy and infantile diarrhea, which are the scourge of the developing world. This occurs for two reasons. First, it is unethical to create a control group (i.e., not to treat a disease when remedies exist); therefore, a randomized trial does not occur. Second, there is little monetary profit in doing so.

EBM has also lowered the quality of patient care. Physicians now rarely possess the flexibility and creativity they once needed to be effective. Clinical judgment is discarded in favor of algorithms and general guidelines. Too frequently, medical schools produce doctors who are familiar with recent laboratory research, but who simultaneously fail to recognize the complexity of clinical situations. A valuable tool to the experienced physician, clinical judgment now runs the risk of becoming a relic of the past.

Why does this occur? Focusing on the statistical probabilities that substance X will have therapeutic effect Y on population cohort Z, EBM brings us much needed information about population-based relationships. Physicians, however, take care of individuals, one at a time. Statistical relationships only get them so far; at some point each physician must be guided by the unique and particular circumstances that each patient brings to the examining room.

Doctors must weigh the use of any given drug against its known sideeffects, against the propensity of his/her particular patient to adhere to the drug program, against the possibility that the patient's other illnesses may obviate the effectiveness of a drug, and against other social and psychological factors that impinge on patient care decisions. Ideally, physicians should possess all relevant medical information to make clinical decisions, information that EBM helps to generate. However, physicians deal with human beings. In great part, their decisions can not be determined by EBM-derived data alone, but must depend upon a myriad of other important factors. The physician is not a technician.

If clinical judgment is not obsolete, neither is it obsolescent; it will continue to define successful medical practice in the future. As world populations age, almost all patients will have comorbidities —other illnesses that impinge on decisions driven by EBM data. A 76-year-old with prostate cancer, chronic lung disease, severe osteoarthritis, and atrial fibrillation must have his/her therapy tailored to the combined effect of the illnesses. EBM-based therapies are silent on critical issues: which condition will limit his longevity most, how well he will follow instructions, or what interference might be expected among drug therapies. Such issues are the norm.

Could EBM address these comorbidities and unravel complex clinical situations? Never completely. At some point, the use of EBM alone fails the individual patient, whose complicated co-mingling of disease processes and psychological and social factors constitute a unique case. Enormous space remains for clinical judgment.

Parallel Developments in Political Research

What does this example tell us, as political scientists? We argue that it clearly speaks to parallel developments in our own field: specifically, to the rise of quantitative and rational choice approaches within the discipline.⁴

Names may be telling. The drive in medicine for “evidence-based” research was, among other things, a coup of nomenclature. By making the call to root research in “evidence” —which, as a practical matter, became synonymous with randomized, controlled trials—proponents of EBM implied that other approaches were based in something that did not deserve such a label.

Names also tell us something about political analysis. Within the international relations sub-field, for example, Realism orchestrated a similar coup. Non-realist approaches, by implication, focused on non-reality, just as practitioners before EBM used non-evidence. The name “rational choice” given to a cluster of related methodologies implies a similar claim: rationality is not just a heuristic for how people behave. It is also a commentary about the sensibility of one approach vis-´-vis other, presumably less rational, alternatives.

Even if one downplays the significance of names, the intellectual perspectives to which they apply do matter. They weigh, sometimes heavily, on how our discipline develops. Some developments may represent genuine advances worth pursuing further; others may entail negative consequences worth curtailing.

The creation of rigid hierarchies of data collection is a consequence worth curtailing—both in medical and in political research. Like the shift to EBM, the quantification of political analysis has been rooted in a desire to improve data quality. Indeed, many political scientists have benefited from greater computing capacity, information storage and recall on an unprecedented scale. These technological advances have made reasonable the expectation of systematic research, as data gathering and analysis have been facilitated. At least in theory, a wider array of evidence should be available for analysis because each datum can be recorded, stored, recalled, considered, and reconsidered at the touch of a few buttons.

However, if evidence that is quantifiable is preferred for its easier storage, recall, and analytic manipulation, data that are easily quantified lie atop a dataquality pyramid.⁵

The in-roads made by rational choice theorists follow a similar trajectory. Just as EBM sought to make research more scientific, rational choice scholars have intended to inject logical rigor into the study of politics. And the intention is laudable: by making the rationality assumption, logical deductions are laid bare for peer evaluation and debate. Here too, however, the shift has a flip-side. The sophistication of the method in some research circles has become the central point of study. At the extreme, much rational choice scholarship—in its drive to theorize parsimoniously—is not rooted in empirical evidence. Real-world complexity becomes anathema. When evidence is adduced, it may be so superficial as to be susceptible to accusations of “curve-fitting.” That is, the researcher may search for data that supports the model, thus skewing the evaluation of empirical results (Green and Shapiro 1996). Moreover, parsimony means that something is being left out, and the rational choice model privileges study of that limited range of questions for which “rationality” is a reasonable assumption. Are important questions about political life given short shrift, as a result? As with the EBM example, the good intention to improve the conduct of research can have noteworthy, negative side-effects.

When the range of what is deemed acceptable evidence is narrowed, much of our analytic purchase on political life hangs in the balance. The principle victims are those theoretical topics not easily studied through ascendant methods of data-collection. We know that culture, belief systems, and other “subjective” factors have political impact, and yet the bulk of political science research downplays their significance. Many analysts contend that this neglect is for lack of “data”; what they rarely underscore is that while data (narrowly construed as that which can easily be gathered, generated, or coded as a quantity) may be scant, evidence (more broadly defined as information of relevance to a research question) may be available in abundance.

There are other parallel consequences. Just as the physician who is armed with the latest knowledge of statistical relationships may not be able to treat the individual patient in clinical situations, theoretical sophistication does not necessarily equip us to contend with unique cases. Witness, for example, the paucity of cumulative knowledge among academic political scientists that might have shed light on the complex conflict in Afghanistan.

Some will dismiss individual cases as idiosyncratic. Even if we do so— setting aside for the moment how any valid theoretical generalization can be built without sound knowledge of individual cases—the neglect of context creates other problems. If research is to be relevant to policymakers, or if it is to be relevant in the classroom, theoretical knowledge is insufficient. Addressing the individual cases that policymakers inevitably face requires context-sensitivity, and theoretical sophistication cannot substitute for deep knowledge of individual cases.⁶

Comparability and Conclusions

We have argued that the rapid development of new technologies and approaches to data collection has led to methodological excesses both in the medical sciences and in political research. The medical example sounds a note of caution for parallel developments in our own discipline. In this conclusion, we ask whether medical research and political research are as comparable as we have assumed.

Our discussion has proceeded as if the conduct of medical research and that of political research were essentially comparable. We took as axiomatic that in both medicine and political research data gathering is largely unproblematic, interpretation of data is likewise a straightforward process, research findings cumulate over time, and science shifts and progresses accordingly. Let us relax this set of rosy assumptions and consider the implications that follow.

First, the causes of research shifts are quite different in political research. In medicine, increasingly rapid and dense communication helped spur the EBM movement: as researchers encountered contradictory data flows they sought to resolve tensions through more systematic analysis. Similar communicative changes may not have similar effects for political research. To be sure, rapid and dense communication among researchers is changing our enterprise. Notwithstanding barriers of language, education, and training that inhibit the free flow of information about politics, new technologies speed data across the country and around the globe. But, does this increased communication invite researchers to consider—and consider ways to reconcile—data that they otherwise would not have encountered?

We are not sanguine on this point. Almond's description of “separate tables” of political research seems to grow in relevance with each passing year (Almond 1988). With greater specialization and sub-specialization, fewer of us read across sub-fields except to “resolve” an apparent “paradox”—i.e., something that narrowly defined fields of inquiry encounter in great abundance. It seems that a data explosion can produce greater communication and more systematized research programs; it can also generate increasing specialization that raises the entrance costs for newcomers.

Second, political analysts actively contest the suitability and utility of the variables we use. Medical researchers, by contrast, generally agree on what constitutes a medical fact. Either a patient has cancer or does not. Either a given treatment is therapeutic for the cancer or it is not. To be sure, there are disagreements in medicine. Medical ethicists, for example, tackle the thorny problems of whether, and at what moment, a terminal patient should be spared life-sustaining treatment and whether the high cost of an extraordinary remedy is justified (and who should bear the costs). But these are less common in medicine than in political science.

For us, ambiguity is the stuff of daily research. While the nature of certain outcomes can be beyond dispute (elections, for example, are either won or lost—notwithstanding the Bush-Gore entanglement in 2000), the bulk of political phenomena are “essentially contested.” That is, what constitutes a war (the “war on drugs” and the “war on terrorism” come to mind for their ambiguity) is open to more questions than what constitutes a cancer. The same goes for state collapse, ethnic violence, and political participation. Our variables are fuzzier; theirs are crisper. We deal in a highly contested social world; they deal in a less-contested biological one.

Our lines of inquiry are inseparable from the analyst performing the inquiry. What Nietzsche (quoted in Ball 1975: 160) sardonically called “immaculate perception” may be more possible in disciplines other than our own. This might not matter if analysts had no professional or personal stake in protecting their lines of inquiry. As it stands, however, standard practice is to hide ambiguity and contestability behind the rhetoric of great conceptual precision and empirical certainty.

Not only are our variables fuzzier, but our entire endeavor lacks a sense of common purpose. Medicine is ultimately evaluated on the basis of how well it contributes to human health; other considerations are secondary. For our part, we evaluate our profession on the basis of a number of diverse criteria: how well it serves policymakers, how well it theorizes, how well it trains future political scientists, how much it contributes to civic education, and how well it critiques existing power relations (for examples). Lacking a single evaluative criterion, our discipline plays host to a plurality of ultimate ends to which research is directed.

Given these differences, the EBM example should make us shudder. If medical and political research were essentially comparable, the EBM analogy would suggest that we apply a mild corrective to what we do. As it stands, we should mount a dramatic reconsideration of the roots, direction, and consequences of much of our analytic enterprise. Even in a lesscontested field like medicine, which enjoys some comfort about variables and purpose and greater certainty about research findings, a dramatic research shift generated impediments to further advances. The terrain of political science research is much more contested; it may be particularly more likely to fall prey to fads and excesses that have little to do with intellectual debates. Ignoring the lesser certainty about our variables, our findings, and our unity of purpose that is inherent to the study of the political world can only exacerbate the problem.

There is, however, reason to end on an optimistic note. If we play our cards differently, the contested nature of our discipline is its potential strength. Intellectual pluralism can be a bastion against excesses of any kind (methodological, ontological, epistemological). We can use our diversity of perspectives, backgrounds, and expertise to nip in the bud the emergence of any kind of hegemony that might otherwise retard the intellectually honest pursuit of knowledge about the political world.

astric part of note-Check?

The authors would like to thank Scott McClurg, Michael Schatzberg, Kate Weaver, Joseph Wong, and four anonymous reviewers from PS for their insightful comments on earlier drafts. The authors alone bear responsibility for the final product.