HOW NEW?

It is easy to become beguiled by the nifty technology associated with ATI. However, we must not lose sight of the fact that this technology is a tool for something that qualitative scholars have striven to achieve: establishing the validity of their data and allowing others to follow their steps—the core ideas behind transparency and replication.

Traditionally, this task is handled in footnotes, appendices, annotated bibliographies, and sections of a manuscript (e.g., a chapter, a prologue, or the acknowledgments) that comment on sources and how they support (or conflict with) the author’s claims. From this perspective, ATI is simply an extension of current practices.

If ATI is novel, it is by degrees. It offers more space for commentary (which, for all intents and purposes, is limitless) and the opportunity of linking directly to sources, which may be of many varieties (e.g., textual, oral, pictorial, or video)—a point explored by Siewert (Reference Siewert2021). These links flow directly from the text, only a click away, so there is no need to search online or depend on libraries and archives. The links can be accessed directly and instantly. (Ultimately, we can hope that a link will take us to the precise location—of what might be a long manuscript or transcript—that is relevant to the writer’s argument.)

This is no mean feat. It has been said that qualitative research suffers a reputation problem within the social science community. “You can prove anything with a case study” is the view of many (Moravcsik Reference Moravcsik2010, 29). The problem is that readers are at pains to evaluate the evidence provided by the author. Indeed, without direct access to the evidence, readers are at the mercy of the author, whose interpretation is unchallenged and unchallengeable (at least, without significant footwork). Insofar as ATI facilitates access to evidence, it makes the ideal of transparency manifest and the ideal of replication possible. In this respect, ATI seems well suited to achieve what every qualitative scholar aims to achieve. If scholars are confident of their interpretation of sources, they should welcome any approach that facilitates easy access to those sources for readers so that their interpretation can be vetted—and, it is hoped, consensus can be reached within the scholarly community.

That said, ATI should not be oversold. It is no panacea and does not solve—or even attempt to solve—many of the challenges currently facing qualitative research. This includes ascertaining the validity of sources, knowing how much source evidence should be provided, knowing how to deal with divergent sources, and figuring out how to secure the confidentiality of sources. It is important to understand these limitations lest the advantages of ATI be overhyped—the dismal fate of many new techniques. These challenges are orthogonal to the goals and functioning of ATI.

WHAT DOES DATA VALIDITY MEAN?

In quantitative research, data validity is fairly easy to achieve (at least compared to qualitative research). We cite the original source (if an extant dataset) or state the protocol by which data were collected (if an original dataset). There is no attempt to deal with the validity of the data at the level of individual observations. For example, the Varieties of Democracy (V-Dem) project describes in great detail how expert coders are recruited; how they perform their coding tasks; the questionnaire they fill out; and the method of aggregating datapoints across coders, years, and countries (Coppedge et al. Reference Coppedge, Gerring, Glynn, Knutsen, Lindberg, Pemstein, Seim, Skaaning and Teorell2020). However, there is no attempt to ascertain whether an individual datapoint (i.e., the coder’s rating or all coders’ ratings for a particular variable/country/year) is correct. Claims to validity rest primarily on following a defensible procedure, not on the results obtained by that procedure.

Not so for qualitative researchers, who must defend each datapoint. In many respects, this is a higher standard than that which quantitative research faces. At any rate, it is a more ambiguous standard.

The saving grace is that, typically, there are not as many cases to keep track of in qualitative research. Whereas V-Dem enlists thousands of coders, millions of codings, hundreds of variables, and tens of thousands of variable/country/year estimates, a typical qualitative research project has a more narrow focus—for example, on a single or several units (e.g., countries, organizations, or groups), a single time period, or even a single event. Of course, there (typically) are numerous datapoints emanating from every case, and these datapoints are of different types. They are all (by presumption) relevant to the central inference, but they are non-comparable, which means they cannot be reduced to the two-dimensional space of a flat dataset (Gerring and Thomas Reference Gerring, Thomas, Badie, Berg-Schlosser and Morlino2011).

Establishing the validity of a single case is quite different from—and, arguably, more challenging than—establishing the validity of a dataset. For the latter, we typically explain the method of data collection, examine possible sources of bias, report reliability tests, and perhaps run convergent validity tests. For the former, there are no well-established rules.

Complicating matters further, in qualitative research, the line between data collection and data analysis usually is more difficult to define. One is entangled in the other; this means that qualitative scholars must wage a simultaneous, two-front war.

ATI will not solve any of these time-honored problems of qualitative research. However, it may nudge authors toward greater self-consciousness and integrity, a point acknowledged by authors in this symposium (Myrick Reference Myrick2021; Siewert Reference Siewert2021). “Knowing that ATI would make my underlying evidence readily accessible to readers encouraged me to consider my evidence more judiciously, and make my inferences more explicit, than I might have done in ATI’s absence” (Milonopoulos Reference Milonopoulos2021).

HOW MUCH IS ENOUGH?

Qualitative researchers seeking to explain and justify their data face the problem of deciding how much information about sources should they provide to readers. By opening the gates to endless pieces of evidence, as well as commentary on that evidence, ATI seems to invite evidence overload. (This point is raised by several symposium contributors.)

However, we cannot blame ATI for a problem that is inherent in all qualitative research. ATI is the messenger, not the cause, of this conundrum. It forces us to recognize a problem that is nascent in any venture aiming at the truth. How do we know we got it right? What do/should readers know to fairly evaluate our conclusions?

In the pre-ATI world, there were severe space limitations. After all, there is only so much information that we can fit into a footnote or an appendix—and appendices easily become unwieldy. (How do we identify the piece of a 200-page appendix that is relevant for a particular point?) Traditional referencing also is limited insofar as it does not integrate alternate media or sources as they appear in primary documentation. It is this type of limitation that makes a journal article an unfriendly location for qualitative work, which often requires extensive discussion of sources (Mayka Reference Mayka2021).

With ATI, these limitations are overcome, which means that decisions about what and how much to present are now unavoidable. Authors can no longer ignore these core questions because of space limitations. Instead, they are forced to justify their decisions. This means that each field must develop standards of evidence that are relevant to their chosen topics.

In this respect, ATI may be viewed as liberating, lifting technological restrictions that heretofore constrained scholarly communication. ATI cannot answer these questions; it only can offer an approach with which they might be answered—but this is a major step in the right direction.

The invention of the internet raised the question of what to do with essentially limitless communication options. It did not tell us whom to communicate with or about what. The same might be said (in a less revolutionary fashion) for ATI.

HOW TO DEAL WITH EVIDENCE DIVERSITY?

Another issue that ATI does not address is contrary evidence. Commonly, writers cite evidence for their proposition, leaving aside—or noting only briefly—evidence that does not support or that perhaps even contradicts their argument. This is acceptable if it is understood that evidence is being used illustratively; that is, “There is some evidence to back me up; someone out there agrees with me.” However, the claim often is stronger: “The evidence, on balance (taken as a whole), agrees with me.”

To my knowledge, qualitative scholars have not) developed a protocol for citing an array of evidence—supportive opposing, or perhaps neutral in import. It is not clear how we could or should represent this messy reality.

Naturally, if all sources were equal in status, we simply could classify them in bins: (1) supporting, (2) opposing, or (3) ambivalent. However, this is rarely the case. Typically, some sources are more trustworthy or more relevant to the question than others. Moreover, some datapoints probably should be regarded as redundant because they rely on the same primary source or the same informant. This type of judgment requires interpretation and, hence, commentary, which is when ATI is helpful (Mayka Reference Mayka2021).

The technology alone will not solve the problem of divergent evidence; it merely offers space for solving the problem. Researchers will need to develop a protocol for how to address the ubiquitous situation of conflicting pieces of evidence. However, at least now there is a technology that makes it possible to have this discussion in the depth and detail that it deserves, which means there is no excuse (except for time burdens on researchers and readers) for not doing so. It also means that qualitative researchers can present a more nuanced and faithful view of the evidence. They do not need to overclaim because they can explore the subtleties behind their conclusions. This approach should prove more persuasive than the claim that is implicit in much qualitative research: “Trust me, I have pored over all the relevant evidence and explored all the angles.”

HOW SHOULD CONFIDENTIALITY BE ASSURED?

All scholarship, and especially qualitative scholarship, faces a problem with respect to the confidentiality of sources. If informants must speak openly (i.e., “on the record”), they may speak untruthfully or may not speak at all. Yet, if informants are granted confidentiality, it is not possible to replicate the findings (at least not in an obvious sense). Every scholar who has conducted research with human participants has faced this dilemma, an issue discussed by Myrick (Reference Myrick2021).

It goes without saying that ATI will not solve this problem; it does not make it better or worse. However, it does make it more real. In the pre-ATI and pre–Qualitative Data Repository (QDR) world, it was easy to avoid the issue because publication venues did not have established protocols for making qualitative research materials public. There was no place to put our interview transcripts, field notes, and archival documents. As a result, informants often enjoyed confidentiality.

Now, as a result of ATI and QDR, it is possible to post qualitative source material online so that it can be accessed by readers and to connect specific points in a published article to specific source materials. From this perspective, we might say that ATI and QDR have created a problem where none existed.

However, it is a good problem to have. Frequently, there are no problems of confidentiality that would prevent posting qualitative data so that other researchers can access them. Moreover, confidentiality sometimes can be secured by anonymizing materials. In these instances, ATI offers a useful option; where it does not, the author must demur.

The tradeoff between confidentiality and transparency is an enduring one. It is deep, it is philosophical, and there are no easy solutions. ATI forces us to recognize this, and it forces the argument out of the realm of technology and into the realm of science and ethics, where it belongs.