ADAPTABILITY AND ACCESSIBILITY THROUGH ATI: ANNOTATIONS AS ANALYTIC EXOSKELETON

In an oft-cited metaphor, John Lewis Gaddis contrasted the styles of political scientists with that of his fellow historians. “We recoil from the notion that our writing should replicate, say, the design of the Pompidou Center in Paris, which proudly places its escalators, plumbing, wiring and ductwork on the outside of the building, so that they are there for all to see,” Gaddis wrote. “We do not question the need for such structures, only the impulse to exhibit them” (quoted in Bennett, Elman, and Owen Reference Bennett, Elman and Owen2014, 662).

ATI offered an opportunity to moderate this impulse. My annotations supplied the more technical aspects of my analysis such that they functioned as a kind of electronic exoskeleton reflecting the underlying logic of my inferences. These annotations contained the Bayesian process-tracing method I used to weigh my evidence against alternative explanations for President Johnson’s choice to spurn major escalation in Vietnam in March 1968. This method extends Bayesian logic to qualitative analysis: researchers assign prior probabilities to well-specified rival hypotheses that reflect their initial beliefs about the viability of each explanation. Researchers then incorporate evidence in ways that progressively update their confidence that one of these hypotheses accounts for the outcome observed in the case under study (Bennett Reference Bennett, Andrew and Checkel2015; Fairfield and Charman Reference Fairfield and Charman2017).

The judgments that researchers make in assigning probability estimates when updating can be somewhat subjective. Therefore, Bayesian process tracing requires scholars to provide considerable justification—sometimes several paragraphs long—for their judgments in assigning estimates to the likelihood of encountering evidence under each rival explanation. First, researchers must “inhabit the world of each hypothesis” to determine with which explanation a piece of evidence is most consistent (Fairfield and Charman Reference Fairfield and Charman2017, 369). Researchers then must weigh how well this evidence “speaks” in favor of this hypothesis relative to each of its rivals.

For example, General William Westmoreland recalled in his memoir that President Johnson had said he “does not have the horses”—meaning votes among his policy advisers—to supply an additional 205,000 troops in Vietnam (Westmoreland Reference Westmoreland1976, 358). I interpreted this evidence as most consistent with my hypothesis that internal bureaucratic bargaining determined Johnson’s decision to deny the troop request. When assigning my likelihood estimate, I considered the reliability of Westmoreland’s retrospective account and his instrumental incentives to shift blame for Vietnam’s outcome onto Johnson’s civilian advisers. I then weighed how well this evidence spoke in favor of this hypothesis relative to each of three competing explanatiens for Johnson’s decision: (1) a structural explanation focused on the benefits and costs of deploying more troops to Vietnam; (2) a domestic explanation focused on his incentives to avoid political culpability for a war he initiated; and (3) an individual-level explanation focused on his preoccupation with his domestic legislative agenda. I then assigned likelihood estimates for encountering this evidence under each rival hypothesis, and I justified each assignment with several paragraphs of accompanying text.

These justifications often are of little interest to general readers, and interspersing them in a qualitative case study can weigh down prose in ways that would cause historians of Gaddis’s ilk to recoil. However, supplying them is crucial for making Bayesian analysis transparent to specialized audiences that are seeking to evaluate inferences more rigorously. ATI freed me to use my annotations to report the formal assignment of likelihoods and progressively updated posterior estimates while reserving the main text for a more cohesive narrative. Moreover, providing this information in annotations alongside the main text—rather than relegating it to a separate appendix—made it easier for readers to track the more technical expositions of my inferences and map them onto the analysis.Footnote ²

ATI thus helped me to demonstrate the rigor of my work and to achieve the inferential transparency that more discriminating reviewers might seek without alienating more general readers. To extend Gaddis’s metaphor, the placement of piping and other structural elements on the Centre Pompidou’s exterior makes more room for artwork featured inside.Footnote ³ ATI allowed me to achieve similar benefits. Its versatility allows authors who employ Bayesian process tracing or other analytic methods to display their work’s methodological trappings without sacrificing readability.Footnote ⁴

AUTHOR ACCOUNTABILITY THROUGH ATI

ATI had another unanticipated benefit: keeping me honest. I suspect I would not be the only scholar to confess a desire to see the evidence I gather conform to my preferred explanation. Employing ATI helped me to overcome this motivated reasoning by holding me accountable to the evidence I analyzed. Knowing that ATI would make my underlying evidence readily accessible to readers encouraged me to weigh evidence more carefully, in much the same way that the prospect of sharing datasets incentivizes quantitative researchers to take greater care in avoiding coding errors.Footnote ⁵

For example, according to notes from a March 1968 meeting, President Johnson told his generals that he did not “give a damn about the election” and that he “will be happy just to keep doing what is right [in Vietnam] and lose the election” (Johnson Reference Johnson and Kent1968[2002]). I initially inferred that Johnson might have been less sensitive to public opinion than prevailing interpretations suggested. However, while preparing an annotation that supplied a more extensive excerpt and hyperlink to the underlying data source, I noticed that Johnson contradicted himself later in the same meeting, fretting that the military’s troop request would prove politically perilous in an election year.

ATI prompted me to temper the inference I drew from what I initially interpreted as near-smoking-gun evidence; instead, it amounted to more of a smokescreen. I may have arrived at these revisions without using ATI. However, annotating as part of the editing—rather than the drafting—process helped me be better accountable to the evidence I gathered. I elaborate on the value of using ATI while editing in the following section.

TRANSPARENCY AS TRIBULATION? CHALLENGES AND OPPORTUNITIES WITH ATI

Several scholars have raised important concerns about efforts to achieve greater transparency in qualitative research. This section explores two common critiques: (1) variation in the shareability of data, and (2) the time investment required to achieve greater transparency. Neither critique is unique to ATI and, although not a panacea, ATI nevertheless can alleviate the demands associated with transparency in the following ways.

First, limitations on data sharing can make it difficult for researchers to make evidence accessible. In some cases, it may be infeasible to supply sufficient evidence to substantiate certain arguments. For example, as Saunders (Reference Saunders2014, 693) noted, “absence of evidence” claims can be difficult to make fully transparent without providing readers access to the whole corpus of evidence examined. Moreover, legal and ethical considerations also can constrain the amount and types of data that researchers can share (Kapiszewski and Kirilova Reference Kapiszewski and Kirilova2014). Sometimes scholars need to limit access to their data when engaged with human subjects whose identities must be protected (Monroe Reference Monroe2018). A related concern is that increasing transparency requirements might discourage scholars from researching topics for which there is little or no shareable data in order to avoid difficulties in publishing their work (Elman, Kapiszewski, and Lupia Reference Elman, Kapiszewski and Lupia2018, 43).

ATI allows—but, importantly, does not require—researchers to provide access to underlying data sources. When data are not shareable, ATI offers researchers the opportunity to achieve transparency in other ways. As Myrick (Reference Myrick2021) notes in this symposium, annotations can be used to shift the emphasis from data sharing to production transparency. By describing processes through which data were generated, annotations can help authors explain why data remain unshareable. Another analytic note might describe the full scope of documents consulted in an archive to buttress an “absence-of-evidence” claim. Moreover, ATI enhances accessibility of certain types of data (e.g., images, audiovisual recordings, and other multimedia) in ways that other strictly text-based techniques could not.

Second, some scholars have concerns that the substantial time investment required to make qualitative work transparent exceeds the benefits associated with doing so. The additional burden of producing annotations to meet ever-increasing transparency norms, as Subotic (Reference Subotic2016, 3) noted, could hinder the progress of early-career faculty who are expected to publish quickly. Moreover, this time investment could disadvantage members of the profession who are disproportionately burdened by resource constraints and non-research requirements, including contingent faculty, faculty of color, and women faculty (Guarino and Borden Reference Guarino and Victor2017; Mitchell and Hesli Reference Mitchell and Hesli2013; Orr and Czastkiewicz Reference Orr and Czastkiewicz2019; van Assendelft et al. Reference van Assendelft, Laura, Gay and Sanbonmatsu2019).

As an early-career researcher, I share these concerns about the time investment imposed by increased transparency expectations. In my experience, I have found that ATI does not demand much more of me than I already was doing in preparing my manuscripts. I would have written my Bayesian analysis justifications with or without ATI; the annotations simply offered an alternative mechanism for displaying this text. Moreover, although workflow processes are highly individualized, most qualitative researchers already organize their primary and secondary material (i.e., their evidence) in ways that are accessible to them when they are writing or revising. The additional work required to implement ATI does not extend too far beyond those tasks. Knowing that my annotations would be scrutinized more closely by readers required that I finetune some of my research notes more than I would have otherwise. However, integrating ATI into my workflow from the outset made this effort much less daunting. From my perspective, ATI does not require more work; rather, it results in more polished work.

Moreover, choosing when in the writing process to annotate can have an important effect on the amount of time invested in using ATI. I initially began annotating while writing my first drafts. However, I found myself becoming overwhelmed with decisions about whether material belonged in the main text, a footnote, or an annotation. Therefore, as Mayka (Reference Mayka2021) recommends in this symposium, I shifted my annotating to the revision phase, simply flagging areas during the drafting phase where I anticipated an annotation might be useful. Shifting the bulk of my annotating to later in the writing process helped me to correct shortcomings in my analysis and avoid sinking time into annotations that later might prove unnecessary. I then also could use annotations to supply information that I had cut to conform to article word limits, which again required little additional time investment because this text already had been written.

If there was any anxiety surrounding my use of ATI, it emerged when trying to assess whether I was annotating too little or, in some cases, too much. If I annotated too much, I was concerned that I would overwhelm readers and prevent them from engaging with the more important annotations containing my Bayesian analysis. One way to address this latter concern is for authors to categorize their ATI annotations according to their function. For instance, scholars who use both qualitative and quantitative methods could differentiate for their readers those annotations that, for example, offer a translated excerpt in its original language from those that justify a particular coding decision or point estimate. This way, readers who are interested in annotations of a certain type could identify them more readily. To further extend Gaddis’s metaphor, Paris’s Centre Pompidou does not only display its internal infrastructure on its exterior; it also uses a color-coding scheme that categorizes piping and wiring according to the elements that they circulate (e.g., blue for air and yellow for electricity).

More anxiety inducing was my concern about whether I had annotated enough. If I annotated too little, I was concerned that I would not meet what I imagined to be a sufficient threshold for transparency expected by the discipline. Of course, no such standard currently exists. However, there is little question that—as with many types of research—greater transparency is being expected in qualitative work. To avoid imposing inappropriate or infeasible standards, it is important to ensure that no single standard mandates what qualifies as sufficiently transparent research. Among the virtues of qualitative work is the rich diversity of its research traditions, and each community of scholars should be able to decide which thresholds constitute openness in its work.

CONCLUDING THOUGHTS

As I and others in this symposium suggest, ATI’s versatility provides researchers with a highly adaptable framework that can accommodate different methods, approaches, and types of data.Footnote ⁶ Through annotations that contextualize evidence and provide access to underlying data sources, ATI helps researchers to effectively “check their work” to ensure that inferences comport with the evidence on which they rely. Finally, ATI can help scholars increase transparency in ways that embrace methodological pluralism and avoid overly burdensome time investments. Anticipating when and how to use ATI can help keep manageable the time investment required to achieve greater levels of transparency.

Moreover, just as ATI can help scholars meet emerging norms of transparency in qualitative research, it also has the potential—as Myrick (Reference Myrick2021) illustrates—to facilitate transparency in quantitative research. For instance, data-replication files from a large-N study could include an annotated version of a spreadsheet that justifies various coding decisions. Given that ATI is still in the initial stages of development, early adopters have an opportunity to shape the conventions governing its use and to think creatively about the contexts in which it can be employed.