RESEARCH TRANSPARENCY AND ATI

Research transparency—that is, describing in detail how the data underlying a piece of scholarship were generated and analyzed and sharing those data ethically and legally—is an important goal across scientific disciplines. Broad acceptance of the value of transparency springs from four key contributions that it makes to the reliable production of new knowledge. First, open data and materials allow scholars to illustrate the power and rigor of their work. Second, transparency increases the clarity of scholarship, augmenting the comprehension of diverse readers, from scholars to policy makers to activists and other key actors. Third and relatedly, transparency facilitates evaluation, allowing scholars to assess the limitations and recognize the strengths of one another’s research, potentially enhancing their confidence in its claims and conclusions. Fourth, and most generally, transparency democratizes access to knowledge and promotes its accumulation, encouraging and empowering scholars to build on one another’s findings and to reuse shared data in their research and teaching.

Recent investments in sociotechnical infrastructure—that is, capabilities, arrangements, and institutions arising from the interdependent relationship between social and technological systems (Sawyer and Jarrahi Reference Sawyer, Jarrahi, Topi and Tucker2014)—have empowered unprecedented levels of transparency and addressed challenges in achieving it (Elman, Kapiszewski, and Lupia Reference Elman, Kapiszewski and Lupia2018; Miguel et al. Reference Miguel, Camerer, Katherine Casey, Esterling, Gerber and Glennerster2014). In the social sciences, most of this new capacity has been directed to and mainly facilitates transparency in quantitative inquiry. Due to the varied forms that qualitative data take (i.e., text, images, audio, and video) and the way they are deployed in published work, scholars whose research entails generating and analyzing such data require different mechanisms and strategies to achieve transparency (Karcher, Kirilova, and Weber Reference Karcher, Kirilova and Weber2016; Mannheimer et al. Reference Mannheimer, Pienta, Kirilova, Elman and Wutich2019).Footnote ²

Qualitative data typically are analyzed, and used to support claims in scholarship, individually or in small groups: an archival document, interview recording, newspaper article, video clip, or small groups of such sources serve as distinct inputs to an analysis. Yet, increasingly stringent journal-article word-count limits and publisher concerns about the length of book manuscripts make it difficult for authors to use large amounts of text as supporting material. This complicates the deployment of the very forms of evidence that are the lifeblood of much qualitative work. Furthermore, rather than presenting results in a single matrix as authors of quantitative work do, those who deploy qualitative data often interweave data, analysis, and conclusions across the span of their written work. Figure 1 compares how data are analyzed and deployed in quantitative versus qualitative research.

Figure 1

Data and Analysis in Quantitative versus Qualitative Research

Note: Quantitative analysis from Bolsen, Leeper, and Shapiro (Reference Bolsen, Leeper and Shapiro2014); qualitative analysis from Snyder (Reference Snyder2015).

ATI accommodates these features of qualitative inquiry. Developed by the Qualitative Data Repository (QDR) and the software nonprofit Hypothesis, ATI is a flexible, versatile approach to achieving transparency in qualitative and multi-method research.Footnote ³ ATI uses open-web annotation, which allows for the generation, sharing, and discovery of digital annotations across the web. Authors who use ATI digitally annotate particular claims or conclusions in their manuscript. Annotations may include full citations to the data sources underlying empirical assertions; “analytic notes” that clarify how authors generated or analyzed their data and/or how they support their inferences or interpretations; excerpts from data sources; and, potentially, links to the data sources themselves (figure 2). For scholars who use lengthy footnotes to clarify and illustrate points, ATI annotations are a natural extension of their writing practices (see also Gerring in the symposium conclusion).

Figure 2

ATI at a Glance: How a Passage in the Text of an Article Annotated with ATI Appears to Readers

Source: O’Mahoney (Reference O’Mahoney2019).

Annotations are displayed on the same publisher web page (or PDF) as the text of the digitally published article or book that they accompany. They can be used by any publisher on any publishing platform. Annotations (and underlying data sources) are curated and preserved by a repository where readers can further investigate and potentially access them for reuse (e.g., for evaluating the relevant publication, answering new questions, or teaching). Immediately proximate to the text and instantly available to readers, annotations enhance scholarship in many ways, all without increasing word count.

As the contributions to this symposium show, ATI also can be used to increase the transparency of multiple types of qualitative and multi-method research. ATI annotations can help authors to show the data that underlie their claims or to provide details about how they collected or analyzed the data (as noted previously); to discuss reflexivity and positionality; or to provide additional context for a case study. Thus, ATI is a versatile approach to making qualitative research transparent (developing a methodological appendix is another approach). Of course, as some symposium contributors note, scholarly use of ATI can have positive consequences that extend beyond transparency. Moreover, the technology used by ATI—open-web annotations—has broader applications: it can be used to comment on or carry out peer review of digital manuscripts, and it can facilitate teaching and learning. Nonetheless, the core goal of using ATI is to enhance the transparency of qualitative and multi-method work.

ATI’S POTENTIAL

As highlighted in the contributions to this symposium, ATI has several distinctive advantages as an approach to transparency, and it produces various broader benefits for research. Myrick suggests that anticipating the use of ATI encouraged her and her coauthor to approach the collection and interpretation of evidence more methodically than they otherwise might have done. Also, as Mayka and Myrick both note, ATI helps authors to “invite readers in” to the social world of study by enabling authors to provide rich description and use participants’ own words as evidence. Simultaneously, ATI enables scholars to ethically pursue openness by helping them to protect their research participants (because the data underlying ATI annotations are stored in a trusted repository).

Symposium contributors also found that ATI increases research integrity: by helping scholars to identify and encouraging them to resolve contradictions between the argument they are developing and their evidence—that is, by holding them accountable to their data—ATI helps them to overcome “motivated reasoning,” thereby increasing the accuracy of their assertions and inferences (Mayka, Milonopoulos, and Siewert). Moreover, contributors suggest that ATI increases the accessibility of their scholarship: by acting as a digital exoskeleton, annotations can provide technical details of the method used in a manuscript, increasing the work’s evaluability and its utility to readers interested in learning how the relevant method works. Furthermore, because of the proximity of annotations to the text to which they relate, they are easier to access than are alternative transparency mechanisms such as footnotes and methodological appendices, making viewing annotations less disruptive to the reading experience. Indeed, readers who are mainly curious about a work’s core findings and insights can hide the annotations completely (Milonopoulos, Myrick, and Siewert). In aggregate, the symposium contributions demonstrate that ATI is an adaptable, flexible approach to transparency that can be useful in many types of qualitative inquiry.

Of course, given ATI’s novelty, challenges remain. Various contributors (e.g., Mayka and Siewert) highlight the time that they dedicated to creating annotations and thus did not dedicate to other career-enhancing activities. There also is consensus on the importance of establishing a solid workflow to facilitate ATI’s integration into research processes (Mayka, Myrick, and Siewert).Footnote ⁴ Relatedly, some contributors (e.g., Mayka) struggled to identify the best point in the writing process at which to annotate, noting how early annotation can be helpful but also distracting and potentially inefficient because some initial annotations may be cut during the editing and review process. Likewise, contributors grappled with what to annotate—and how to determine if they are annotating “too little” or “too much,” given the absence of consensual norms regarding the use of ATI (Milonopoulos). Siewert offers a promising set of criteria that authors can use to evaluate and calibrate their use of ATI. Contributors also note how uncertainty about whether and how journals will integrate annotations into the peer-review and publication process complicates the composition of annotations (Myrick). We agree that many key questions about ATI and its use remain unanswered. As the approach becomes more widely adopted, norms for its effective use in multiple types of qualitative and multi-method inquiry will emerge. We hope this symposium represents a step toward that goal.