In this book, Kevin A. Clarke and David M. Primo characterize and criticize the corporate culture of the contemporary US political science profession. Employing qualitative methods, they write as participant observers in the field. In this review essay, I will draw attention to one of the central arguments in their book, viz., that the current conception of good work in the profession is too constraining, and that their proposal for a more liberating conception of good work should be adopted. While calling for more methodological openness, this is not the usual appeal for toleration of mixed methods in political science. As we will see, Clarke and Primo take the debate in a new direction by including descriptions of the deleterious effects of the dominant methodological paradigm on the profession.
Clarke and Primo observe that within the profession’s corporate culture, “it is well understood that political science is a science” (p. 20). In this understanding, “science comes from ruthlessly subjecting speculative theories to data,” and the core of science is believed to be “the three-step method: propose, derive, and test” (pp. 21, 23). “This procedure is known as the hypothetico-deductive [H-D] method, and it is central to much current research in political science” (pp. 11, 25).
There is a widespread belief among political scientists, observe the authors, that this three step method “is how ‘real’ scientists (chemists, physicists, biologists) go about their work” (p. 21). They note that “the three step method is practiced in all subfields,” and by “top political scientists” (pp. 26, 22, 51). In their survey of articles published in leading political science journals, and their review of oft-cited substantive books and widely-used “undergraduate textbooks written by practicing political scientists,” they found that tribute is frequently paid to this method “as the gold standard of political science research” (p. 26). Consequently, the currently dominant view of good work in political science is that which adheres to the H-D vision of science.
But the ideas for this understanding of science were based on “nineteenth-century classical physics.” Now, in the first decades of the twenty-first century, the nineteenth-century model of physics “holds little relevance for modern physics and even less for the modern social sciences” (p. 10). Nevertheless, political science remains “enamored of the three-step method” (p. 21; cf. 44). The authors are concerned that this narrow bias is being systematically entrenched in the profession by such elite graduate programs as the Empirical Implications of Theoretical Models workshop, “which through its two summer institutes is responsible for training a generation of graduate students [in H-D methodology]” (p. 22; cf. 48–51). The H-D misinformed vision of science has resulted in a corrupted notion of good work in the profession, and Clarke and Primo seek to expose the misguidance.
They write that
our book is not an attempt to limit what the field views as good work … Whereas the definition of good work in political science has narrowed to comprising a formal [or theoretical] model [tested by] an empirical model … [throughout the book] we provide a number of examples … of theoretical models that are useful without being tested and empirical models that are useful without having been derived from a formal [or theoretical] model. One of our goals is to get the field to acknowledge and appreciate a wider range of scholarship as being good and useful (p. 19).
The Clarke and Primo critique of the H-D three-step model of science has several elements. Chief among these are that the deductive logic that the steps rely on is inherently faulty, and that deduced implications, hypotheses, and predictions are not actually tested against data, but models of data, which undermines the certainty of the results. Clarke and Primo also contend that the H-D method fails to produce interesting insights, and that the three steps are often not actually followed in practice. Because H-D ideology is enforced by the profession’s power structure, the intellectual integrity of career-oriented researchers could be compromised. Let us take a closer look at these and other points.
In compliance with their idealized image of the H-D method, empirically oriented political scientists attempt to deduce an implication (or hypothesis or prediction) from a theory, and then test that hypothesis against political reality (as shown by statistics), supposedly in an effort to refute the hypothesis. But this practice misconstrues the logic involved.
Clarke and Primo offer an extensive criticism of the logic behind the act of deducing an implication from a theory and then testing it against known data. There is room for uncertainty first in “the deductive connection between the theory and the hypothesis,” and second in “the deductive connection between the hypothesis and the data” (p. 106). Because deductions from theories tend to be truth preserving of the theory, rather than refuting or confirming anything, users of the method often commit the fallacy of Affirming the Consequence.
For example, a test that confirms an implication does not necessarily confirm the theory from which the implication (or hypothesis or prediction) was deduced. Suppose theory A has implications B, C, and D. Just because C is true does not necessarily mean that A is true. Likewise, just because B is false does not necessarily mean that A is false. To make either claim would be to commit the fallacy of Affirming the Consequence. “This logic holds regardless of the statistical approach—falsificationist, verificationist, or Bayesian—taken” (pp. 16–17; cf. 117). The possibility that this logical fallacy has been committed, perhaps unwittingly, in a claim of confirmation or falsification, undermines the reliability of all such claims.
While the H-D model envisions the testing of “theory,” or implications deduced therefrom, Clarke and Primo argue that theories are also models. Under the H-D notion of good work, implications deduced from formal or theoretical models are “matched with an empirical model that serves as a test” (p. 18). But the major weakness in the H-D claims that these steps will produce reliable political knowledge is that “we never test theoretical models [or hypothesis deduced therefrom] with data; we test theoretical models with models of data” (pp. 3, 134).
Data models, like all models, are representations of reality, partial constructs based on research undertaken for some purpose, and not the reality itself. Testing, then, is actually the comparison of one model to another. Given these limitations, “assigning the data model to be the final arbiter of the theoretical model is untenable” (p. 3). Comparing one partial model with another can overlook the missing features of both, thus creating the illusion of reliable knowledge.
Clarke and Primo then ask, “what can one learn about the truth of a model from testing an implication of that model?” Their answer is “nothing that we did not know before” (p. 35). Thus, another problem with the H-D approach to science is that “the logic of the test prevents us from drawing interesting conclusions about the model” (p. 3).
Here is why: logically, a valid deduction from a premise must already be contained in the premise; so, the valid deduction does not add any new knowledge. Consider the following valid argument: “All men are mortal. Socrates is a man. Therefore, Socrates is mortal.” But everyone already knows that conclusion. So, in “all deductive arguments … if the premises are true, then the conclusion must be true.” This is “the truth-preserving nature of deductions” (p. 35). For this reason, “the logic of the test prevents us from drawing interesting conclusions about the model” (p. 3).
Clarke and Primo have observed that under pressure from the elite H-D advocates in “political science today . . . all models … are increasingly being judged on the basis of ‘testability’ rather than usefulness” (p. 102). Those elites have considerable power within the profession. They sit on hiring and tenure committees. As editors they control access to many of the top journals in the discipline. They advise funding sources. Book publishers know that the success of books often depends on the approval of the elite H-D adherents. Clarke and Primo raise the question as to whether this power structure affects the intellectual integrity of political scientists who want to have their research funded and published; indeed, who must “publish or perish.”
They note, for example, that for show “political scientists … often talk in terms of falsificationism” (p. 117). In practice, however, few “actually test their models in a logically consistent manner” (p. 116). Despite their ritualistic tributes to the H-D ideology, “strict falsification is not really what political scientists do. … Rather, they attempt to verify or confirm their … models” (p. 120). Even where seemingly falsifying evidence is found, it is sometimes ignored, and “in reality competing theories often remain to fight another day” (p. 102). The pressures to appear to conform to the H-D model of “testing” often result in an obsequious glossing over of methodological deviations.
Journal editors who are adherents to the H-D ideology of science often exercise “desk rejects,” and select like-minded peers for reviewing submissions. Clarke and Primo have found that these journals tend to favor studies with statistical findings that at least meet a “0.05 significance level” (p. 98). Consequently, to appear to conform to this constraining H-D bias, “researchers may massage their data analysis or present selected specifications that give results just meeting that threshold” (p. 98). “A statement that the ‘data’ support one or more of the theoretical models invariably follows” (p. 116).
In this way, the power structure of the profession puts pressure on the integrity of researchers, thus re-defining “good work” as that which passes the censors. Once the dominant myth has been dethroned, “researchers who clearly have other purposes in mind will no longer have to defend their models on H-D terms” (p. 103).
Clarke and Primo also see the H-D standard of good work as encouraging a simplistic form of either/or thinking. That is, political science work either conforms to the three step rule of “propose, derive, and test,” and so is good, or it does not, and so is somehow less than professional. But such a standard is overly constraining, indeed, inappropriate, for the main tool of political science methods—models.
“Talking about models in terms of truth and falsity is a category mistake” (p. 170). Like maps, models are representations of a selected aspect of political reality. They are necessarily partial, have limited accuracy, and are made to satisfy the maker’s purposes. These methodological realities should be recognized and accepted. That is why the authors “argue for a different type of model assessment and evaluation that moves the discipline away from our current true-and-false dichotomy and toward an assessment based on usefulness” (p. 51).
How is usefulness to be determined? Clarke and Primo note that “there exists a single overarching goal that serves to unite the disparate purposes of social scientific models: explanation” (p. 140). Indeed, they find widespread consensus in the political science profession, among practitioners of both quantitative and qualitative approaches, that “explanation is the crucial goal of political science” (p. 141). Thus, good work is that which makes useful contributions to the profession’s aim of explaining behavior and events in a political system.
The key question for assessing the goodness of political science work is not whether it passes H-D testing, but whether it contributes to our store of political science knowledge and understanding of the political system, both of which are necessary precursors to explanation. While empirical testing certainly has its place in the work of the profession, supported or verified claims are only useful insofar as they contribute to the general aim of advancing the ability of the field to understand and explain the political phenomena occurring within a political system. That might seem to some a rather mundane and obvious claim, but it is precisely the advocacy of this standard that makes the authors liberators within the corporate culture of political science.
Clarke and Primo suggest several ways to assess how well a model has satisfied the general aim of political science, to produce useful knowledge. One is by its “fecundity. That is, does the model produce a series of important insights or just one?” New understandings can “move [research] in completely new directions, or … develop several new insights” (p. 100).
“Second, how important are the insights (p. 100)?” A model that produces several trivial insights, or things already known, is obviously far poorer than a model that produces just one richly fruitful insight. Thus, no matter how perfectly executed the testing method may be, its quality as good work would be minimal if it fails to offer any significant insights into our understanding of the political system (p. 62). The authors suggest that a model that does not ape the H-D method, but still produces important insights, “may be more valuable than [an H-D] model that produces several vapid results” (p. 100). That opinion may have particular poignancy among those political scientists who worry that the profession has a public reputation for being more a source of political trivia than of political sagacity.
Another measure of an explanation’s or a model’s goodness is how well it is accepted in the profession. If it generates discussion, gains praise, is often cited, used in classes, etc. then it has substantial goodness. Currently, however, editors, publishers, and pace-setting graduate programs are excessively biased in favor of the H-D standard of “‘testability’ rather than usefulness” (p. 102). The H-D biased selection of articles for publication artificially skews the use of citations as a measure of goodness. But once the elites allow the profession to have a broader conception of good work, peer review will become less myopic. Then citation frequency will become a more objective measure of goodness.
There are some unintended consequences of the profession’s un-self-critical commitment to the H-D model of science which were not directly addressed by the authors. One is the unquestioned belief that the prominent role of mathematics in natural science ought to be emulated by social and political science. Another presupposition compelled by that commitment is the mechanistic theory of human nature underlying some “behavioral” models of human action. The colliding atoms model of nineteenth-century physics is compatible with mathematical analysis, and with the so-called “cover law” theory of explanation, but even the hint that it can enrich our understanding of human nature and of political interaction defies common sense. For clarity and depth of understanding, human agency, resourcefulness, and creative meaning-making require an explanatory model that is more isomorphic with its subject matter.
Rather than prescribe a single alternative, Clarke and Primo hope to initiate a discussion about the deleterious effects in political science of the H-D model. Thomas Kuhn wrote of “paradigm revolutions” as the source of fundamental change in the history of natural science. However, the Clarke and Primo invitation to discussion appears to express their belief that “talk therapy” is the most promising way for our profession to recover from its Physics Envy.
