Formalization

Many scholars, most notably the proponents of “analytic-narratives,” recommend that case study analysts formalize the theories they test (Bates et al. Reference Bates, Greif, Levi, Rosenthal and Weingast1998). Formal analysis does often generate interesting claims about strategies, sequencing, information sets, and outcomes. The implied claim of the analytic narratives approach is stronger, however. It is that rigor in theory testing in case studies stems primarily from the power of formalization, which can generate “within case” predictions so precise that the risk of false positives is negligible. This is why analytic-narratives scholars (like most formal analysts) do not test their models against fully specified alternative theories, but instead recommend a lexicographical approach: they start with a preferred theory, formalize it, maintain it if it tests out, and revise it if it does not.Footnote ² This procedure implies that the analyst (and the reader) has extraordinary confidence in the precision, reliability, and transparency of evidence presented for or against the predictions of a single model. Were this not the case, any analytical narrative would risk imparting a blatant bias in favor of the preferred (initial) theory. Analytical narratives require, therefore, correspondingly rigorous standards of evidence and citation in order to conduct replicable tests of its extremely precise claims.

Case Selection

Perhaps the most salient methodological issue in the mind of most scholars embarking on qualitative analysis today is the need to select cases properly. Students are taught to maximize cross-case causal inference, generally by maximizing variation on the independent variable. This quasi-statistical approach is the overriding focus of two among the most influential recent primers on case study method, by Theda Skocpol and collaborators, and by Gary King, Robert Keohane, and Sidney Verba (Skocpol Reference Skocpol1984; King, Keohane, and Verba Reference King, Keohane and Verba1994). The assumption is that strategically chosen small-n comparisons support causal inferences and thus reduce the need for intensive primary-source analysis of individual cases. Yet shrewd case selection alone is among one of the least significant sources of causal leverage. While it can surely offer insight, even King, Keohane, and Verba concede that case selection offers no substitute in most cases for within-case analysis, which can, if the data and theory are good, permit us to multiply the number of direct observations exponentially: by tens, hundreds, or thousands, as compared to the number of cases (King, Keohane, and Verba Reference King, Keohane and Verba1994, 208–28). If so, then high-quality, transparently sourced evidence again becomes the vital link in any qualitative study that claims to test such fine-grained within-case claims in a replicable manner.

Process Tracing

Both analytic narratives and small-n studies, we have just seen, ultimately rest on rigorous methods of testing predictions against fine-grained within-case observations. Scholars in the process-tracing tradition have offered books of admirable advice on how to conceptualize and structure such case studies to maximize valid causal inference (George and Bennett Reference George and Bennett2005; Trachtenberg Reference Trachtenberg2006; Van Evera Reference Van Evera1997). Yet in practice such work is often extremely difficult to do without access to detailed primary evidence, which is required to reach conclusive results about detailed points of motivation, strategy, information, beliefs, and sequencing (e.g., Milner Reference Milner1989; Khong Reference Khong1992; Christensen Reference Christensen1996; Moravcsik Reference Moravcsik1998, Reference Moravcsik2000a, Reference Moravcsik2000b; Goldgeier Reference Goldgeier1999; Owen Reference Owen2000; Bass Reference Bass2008). Unless the qualitative evidence presented in such analyses is rigorous, precise, and transparent, outside observers have no way to assess whether the basic methodological techniques have been properly employed, much less whether the empirical conclusion is convincing.

Yet this is not generally the case today. Current practice in some areas of political science permits citations to be imprecise, vague, and secondary, rather than precise, annotated, and primary. It is still acceptable, even common, for citations to lack page numbers or any other reference to any specific elements. As scientific citations spread and word limits tighten, references increasingly lack any precise annotation describing how the evidence supports the textual claim. Such scientific footnotes thus overtly discriminate against good qualitative work. They encourage “abuse of sources” and discourages “replication by those who might otherwise be inclined to check claims … against what those sources actually say” (Lustick Reference Lustick1996a, 6). Even where proper citations are provided, it is usually prohibitively expensive for most readers to locate and check the source. Much qualitative work cites the conjectures and interpretations—“conclusionary evidence” from other secondary sources such as scholars, commentators, journalists, or even participants—as data to test theories, without reference to primary-source validity. An additional problem is that often this evidence is cited selectively (Lustick Reference Lustick1996b)—a particularly dubious practice when sources are critical to reconstruct the preferences, strategies, calculations, and intentions of specific political actors. There is no way to ascertain this if one does not know how the inference was drawn.

The lack of any evidentiary standards undermines the application of other qualitative techniques in political science more generally. An objective standard of transparent citation is a precondition for further methodological advances in case study analysis—just as proper treatment and presentation of statistical data is a precondition for the proper use of advanced econometric techniques. Without any collective access to evidence and data, how can any efforts to analyze qualitative data well be monitored, evaluated, or rewarded? What incentive do young scholars have to invest in presenting and analyzing data rigorously? While close to 50% of coursework in some top political science graduate programs is now given over to methodological training, only the exceptional elective focuses on qualitative methods—usually concerned largely with small-n research design, not case study analysis. Case studies are too often an afterthought in dissertations where much of the methodological effort goes into quantitative and formal elaboration. More subtly, the lack of clear and transparent standards cast an elitist pall over historical analysis. Some younger scholars find the qualitative research community intimidating and hierarchical: established scholars, who have “read more books” and “know the sources,” seem always to retain the upper hand in applying vague standards. The quantitative and formal research community, with sharper and more explicit standards and open sources, seems more inviting.

Overall, we are left with a curious circumstance. Despite admirable theoretical advances in recent qualitative methodology, in practice political scientists acknowledge many rules for selecting and theorizing a case study, but almost none for actually doing one. The central problem lies with the absence of rigorous standards for the collection, citation, and presentation of sources. Because a chain is only as strong as its weakest link, the result is non-replicable and non-cumulative work.

1. Rigor: Any critical and contested substantive empirical point in a scholarly case study should be backed by a precise and annotated citation to one or more presumptively primary sources

Under active-citation rules, scholars would be obliged to provide concrete evidence for controversial empirical claims, annotated to explain precisely how the source supports the textual claim. The need to footnote precisely and annotate implies the use of a classic, not “scientific,” form of footnotes, preferably on the page. Presumptively these sources would be primary in nature, though in certain cases secondary evidence might be acceptable. Typical primary sources would include a government or private “document” from an archive, an oral history, the transcript of an interview, notes from participant observation, or a published primary source. Yet primary sources might also include those previously published in secondary sources: recollections of specific events published in a memoir or newspaper or a primary source cited in reliable work of scholarship or journalism. Under some circumstances social scientists draw conclusions from bodies of secondary literature, in which case they would be obliged to cite secondary sources in the same way—a point to which I return below.Footnote ³ As with primary sources, secondary-source citations would be annotated to explain why and in what way the particular evidence cited supports the textual claim.

2. Transparency: Citations must contain a hypertext link to a reproduction or transcript of some part of the source

Today nearly every journal article has a parallel electronic version. In that version, and in parallel electronic versions of the notes of scholarly books, all citations to controversial source material would be linked via hypertext to a reproduction or transcription of some part of the source. This does not mean that a link must be provided to the original source online, which would clearly often be impossible. Instead the text of some part of the source would be copied onto a journal, publisher, or third-party Web site. The ideal, to be sure, would be to post a visual copy of the entire original source, plus translation into the language of the article, if needed. For the foreseeable future, however, many conditions (e.g., copyright, archival permission, confidentiality and human-subject concerns, intellectual property and professional development concerns, and sheer length) render this unrealistic as a general standard (Parry and Mauthner Reference Parry and Mauthner2004). Instead, a permissible minimum would be set at a textual transcription (with translation) of sufficient text to support the point being cited, with additional material to establish the context—under normal circumstances, a realistic rule of thumb might be a paragraph. As per prevailing historiographical norms, moreover, authors are presumptively responsible for not citing from documents out of context.Footnote ⁴

1. Higher-Quality Scholarship

The immediate requirement of rigorous, transparent evidence would encourage, enforce, and reward higher standards in historical and qualitative analysis. As in the very best empirical work from the disciplines of history and law, authors would be obliged to investigate and report the primary-source basis of any empirical claim they make (in the case of secondary-source citation, the contours of a scholarly debate they gloss). Authors will be better able to show, and readers will be better able to assess in real time, whether the evidence cited in an article provides a prima facie case for the interpretive or theoretical claims being advanced. Greater transparency and replicability would help unleash the full potential of analytic narratives, fine-grained process-tracing methods, and strategic case selection. When proper adherence to such methodological standards becomes a transparent act that others can observe and evaluate, expert use of the method and superior qualitative data collection can be properly recognized and rewarded. “Researchers who anticipate transparency will be motivated to carry out data collection and analysis in a systematic, replicable way,” as Elman, Kapiszewski, and Vinuela (Reference Elman, Kapiszewski and Vinuela2010) rightly note. Those who do not or cannot meet more rigorous standards will face greater incentives to improve their qualitative methodological skills.

2. Wider Criticism

The immediate availability of evidence to each reader will facilitate scholarly debate. This can take the form of criticism, replication (“research auditing”), and review essays (King Reference King1995). Currently assessing, criticizing, or debating the empirics of a qualitative article is prohibitively expensive for anyone except fully expert readers who already know the sources. In political science that means no one at all can debate the issue. This places qualitative IR, comparative, and American politics at a disadvantage vis-à-vis quantitative studies, where data sets are routinely available. It creates an unnecessary aura of aloofness. By revealing critical evidence at a single click, active citation would democratize the field, permitting potential critics to make an immediate assessment of the evidence for empirical claims and its relationship to the research design, theory, and method. Flaws like selective citation, poor use of sources, or contextually inappropriate interpretation would become far easier to document. Livelier and more engaged debate would likely ensue.

3. Richer Secondary and Meta-Analysis

Active citation would encourage “secondary” analysis of evidence, that is, the use of data for alternative purposes. Today nearly every qualitative IR scholar starts essentially from scratch, since the raw evidence used in existing studies is, in practical terms, unavailable. “Most data generated by American qualitative and multi-method social science,” Elman, Kapiszewski, and Vinuela (Reference Elman, Kapiszewski and Vinuela2010) note, “are used only once.” This stands in striking contrast to quantitative studies, where each scholar can build on previous data-collection efforts: the number of available data sets increases over time. Active citation would erase this imbalance. The evidence of their predecessors, in electronic form, would be available to prospective scholars.

Active citation would create incentives for scholars to uncover and reveal new evidence. To conduct a study, a scholar would need only engage in a secondary reanalysis of already cited sources and then provide a marginal increment of new evidence—as generally occurs in historiographical debates. If, for example, one scholar supports an interpretation with a paragraph of text, and another scholar suspects that the entire document, box of documents, or governmental debate from which it came, read in context, does not support the claim, that scholar would have an incentive to publicly reveal the entire document or other documents. The mark of a healthy scientific research tradition is precisely that over time ever increasing amounts of data are revealed in this way (e.g., Levy, Christensen, and Trachtenberg Reference Levy, Christensen and Trachtenberg1991).

The expanding network of available data would also facilitate meta-analysis, in which comparative analysis of similar situations in various settings (countries, issues, time periods) could be conducted using different evidence/data, perhaps with addition of new sources provided by the investigator. Declining start-up costs for each new scholar working on a topic would encourage new scholars to join the “club” by contributing new data, just as combining pre-existing statistical data sets with new data reduces the costs of doing quantitative work on existing topics.

4. More Intensive Interdisciplinary Engagement

Placing qualitative political science on a more transparent foundation would open debates to a wider range of voices, interpretations, and perspectives, including an expansion of opportunities for interdisciplinary interaction. Certain branches of law, history, and sociology, we have seen, employ higher qualitative research standards with regard to citing, documenting, and presenting evidence than those currently prevailing in political science. This has influenced the discourse in these fields, which is more attentive to hermeneutical and interpretive issues, and reflects much greater detailed knowledge about specific issues. Creating incentives for political scientists to engage legal and historical scholars in a similar way—but also by further improving the citation techniques in fields like history, which have not yet made full use of the potential of electronic media—may encourage the formation of an interdisciplinary critical mass of scholars employing similar methods, standards, and evidence. At the same time, scholars in other fields might benefit from engagement with political science theory, as well as methodological standards on formalization and case selection.

1. Is active citation infeasible for journals?

Active citation is realistic, since it simply requires political scientists to adopt and extend best practices already employed by journals in other disciplines. Even if this were not already the case, technological developments in electronic dissemination make marginal evolution easier than ever. In legal academia, law reviews employ active citation to case law and law-review materials. Legal scholars customarily reproduce critical passages of legal text verbatim in citations, annotate to explain the significance of the passage, and include hypertext links—just as proposed here. Increasingly, social and natural science journals increasingly use hypertext links to journal articles. Journals in biology, physics, psychology, and other empirical sciences not only regularly require that raw data be made available, but now routinely contain provisions for additional information—experimental, visual, even video—to be included as background information.Footnote ⁶ Just as virtual journals now dominate most disciplines, the days of hard-copy academic books are surely numbered, which promises to extend the practice further. In any case, all scholars submit manuscripts electronically—so expanding the footnotes in a Web site run by the journal is a marginal change. The only immediate expense would be the welcome evolution away from abbreviated “natural science” citation forms; but even this simply marks a return to more the classic discursive footnotes that prevailed a few years ago.Footnote ⁷

2. Does the active-citation requirement impose an excessive logistical burden on scholars?

The requirement to display an electronic copy of sources places a significant but tolerable burden on scholars. Like natural science academics, quantitative scholars in the social sciences already meet standards to post or provide data. Many legal academics and historians, as we have seen, already adhere to a qualitative standard approaching active citation. The requirement to post a transcript of part of the source would extend it modestly, yet not excessively. Some international historians have already voluntarily begun providing expanded citations and documentary materials of this kind to accompany books (Trachtenberg Reference Trachtenbergn.d.; Tønnesson Reference Tønnesson2009). Electronic scanners, laptops, and electronic recording are now widely in archives, interviews, and in handling published text, meaning that such material is—or should be—relatively easy to access. (Where such technology is banned, an exception might be made to substitute notes for verbatim text.) Even those whose workload would increase in the transition would gain offsetting benefits, namely access to an ever-expanding virtual archive of qualitative evidence provided by other scholars, and the collective disciplinary prestige stemming from the increased rigor of qualitative scholarship.

3. Does the presumption of primary-source documentation impose excessive rigor on scholars who cite secondary sources?

The active-citation standard does not ban citation of secondary sources. It treats citation of primary sources found in secondary scholarly and journalistic sources as equivalent to archival primary sources. It acknowledges that some social scientists treat the conclusions of secondary authors as a database, sampling from them as if they comprised a primary-source database. Those who do this may continue to do so. The active-citation standard would require only, to the extent secondary conjectures and interpretations are employed as a database for comparative study, proper citation, annotation, and reproduction of decisive passages from each secondary source.Footnote ⁸ This is a far more demanding citation standard than is currently practiced. It is also important to remember that often the only reliable way to weight competing claims in secondary historiographical debates is by considering the relative persuasiveness of the primary evidence that supports each one—as historians customarily do in such cases. To be utterly replicable, a scholar defending such a weighted assessment might want then to cite primary, rather than simply secondary sources.

4. Does active citation undermine the “intellectual property rights” of scholars?

A prominent issue in any debate over replicability is the tradeoff between the right of a scholar to fully exploit new evidence and the value in permitting other scholars to see it (Box-Steffensmeier and Tate Reference Box-Steffensmeier and Tate1995). In practice, different disciplines have struck this balance in different ways. In law, important legal materials are already public, and thus available equally to all academic analysts. In most natural and quantitative social sciences, a condition of publication is generally that full data must be made available to all legitimate users who request it. In history and the humanities, published and unpublished sources must be precisely cited: the former are generally available from libraries, while the latter may be more difficult to obtain. The active-citation standard strikes a compromise. By requiring that a paragraph of text, rather than the entire document, be transcribed, the author retains control over inessential evidence. This is roughly the same “partial revelation” standard as has prevails in history: scholars must reveal the evidence they use to draw inferences, but not necessarily the rest—as long as the entire document does not undermine the claim.Footnote ⁹

5. Does active citation do violence to the historical, ideological, cultural, personal, research, and gendered context in which textual evidence should properly be interpreted?

Some might argue that the interpretation of sources is a fundamentally reflexive or hermeneutical process, in which much “local knowledge,” contextual understanding, or deep expertise is required to grasp the full meaning. Providing one-paragraph snippets in citations, even with annotation, only encourages a superficial understanding of qualitative evidence. There are those—not least among traditional historians and ethnographers—who would prefer that scholarly debates remain less open and democratic. This is an important consideration for many, and it should be taken seriously.

On balance, it is plausible to expect that active citation would constitute an improvement over current practices. Bold interpretations already propagate among qualitative scholars, often the basis of minimal evidence. Active citation would seek to expand the range of evidence being considered (via citation), the depth of engagement with material (via hypertexted sources), and the contextual richness of claims one can make on behalf of it (via annotation), as well as expanding the number of voices involved in debates.Footnote ¹⁰ Currently the ability to engage in any meaningful empirical debate is limited to the very few who have invested heavily in particular topics and already know the sources. In political science often no debate at all takes place, surrendering topics to other methodologies. Active citation can be understood as a way of empowering qualitative scholars by transforming traditional hierarchies of document control and publication into an open virtual network, in which plural streams of evidence and interpretation, narratives, and debates, can emerge—while still imposing discursive rules that require substantial commitment from serious participants in the scholarly debate. Active citation promises to validate traditional scholarly virtues of hermeneutical precision and subtlety, and expand the community of those who practice them. Traditional historians and ethnographers should thus welcome it.

6. Would active citation encourage free riding by those who do not produce new evidence or engage in serious analysis?

As the stock of available primary sources expands, some scholars may simply cherry-pick sources from previous studies, generating poorly crafted case studies that would have the patina of primary-source content without actually reflecting new research, broad sampling, or contextual understanding. Might bad research drive out the good?

This is a valid concern, but should not be an overriding one. Some resulting qualitative work may be low quality. Yet overall active citation promises to enhance, not degrade, the ability of the scholarly community to distinguish high-quality scholarship. Good work should float to the top of a larger pool. As has occurred in quantitative political science, we should expect the general quality of research and data to rise over time. The goal of active citation is not to lower standards but to raise them. We must have faith in the ability of scholars to meet the challenge.

Still, this final concern reminds us that reform of citation practice is not an end in itself. It is an instrument to help enhance the internal and external validity of case studies—to render them richer, more varied, and more rigorous instruments to help us tell new stories, advance varied interpretations, and refine new theories. The real benefit of active citation can only be realized in combination with other scholarly practices to bolster case study analysis, ranging from analytic narratives to anthropological interpretation. Qualitative scholars must collectively enforce more rigorous process-tracing methodology: fair treatment of a full range of alternative interpretations (theories), rigorous and creative derivation of numerous implications (observations), unbiased sampling and attention to the relative quality of various sorts of evidence (data), and an appreciation for interpretive subtlety in grasping context. Yet transparent and rigorous citation is a necessary first step. Without it, other methodological steps to improve qualitative analysis will not be genuinely replicable or beneficial—and will go unrewarded by the profession. Only more transparent and replicable work can level a disciplinary playing field in which qualitative methods, though widely practiced, are treated as second class. To this end, it is time for political scientists to embrace new technologies that can help refine and perfect the qualitative best practices that have long prevailed in neighboring academic disciplines.