BACKGROUND AND MOTIVATIONS FOR COMMITTING TO TRANSPARENCY

Our paper explored HRD strategies through a 2015 human rights campaign called #Freethe20 organized by the US State Department to free 20 female political prisoners. To evaluate the effectiveness of #Freethe20, we compared release outcomes of women featured in the human rights campaign to two comparable groups: (1) a longer list of women initially considered by the State Department for inclusion in the campaign; and (2) a database that we constructed of female political prisoners that were imprisoned in the countries targeted by #Freethe20. After collecting data on release outcomes, we showed that women featured in #Freethe20 were statistically more likely to be released from prison relative to women in the two comparable groups. The paper then explored why the campaign was successful by tracing evidence of public diplomacy using quantitative data collected on media coverage of the #Freethe20 women and private diplomacy using qualitative data collected during interviews with US officials. We found little evidence that public diplomacy was solely responsible for the campaign’s success. Instead, public pressure facilitated coordination within the foreign policy bureaucracy, driving private diplomacy and the use of specific “carrots and sticks” to secure releases.

We decided at the outset of the project to increase the transparency of our research in a few ways. Moravcsik (Reference Moravscik2014) distinguished among three dimensions of research transparency: data, production, and analytic. These refer, respectively, to the ability of researchers to share their underlying data, describe how they were produced, and explain how they were analyzed and interpreted. To increase data transparency, we planned to share evidence that we collected in an online repository. Along with the interviews, our main source of qualitative data was a set of resources (e.g., government press releases, newspaper articles, and non-governmental organization reports) that we collected to track the release outcomes of women featured in #Freethe20 and other comparable cases. We used these resources to construct a dataset of female political prisoners in 13 countries imprisoned between 2000 and 2015, which was the basis of our quantitative analyses. To increase production transparency, we planned to provide detailed notes about the procedures that we followed to identify political prisoners and to code their cases.

However, the primary reason that we used ATI was to increase analytic transparency. We added digital annotations to the manuscript to explain and support our conclusions about when HRD strategies were more or less effective. Many of these annotations contained longer excerpts from interview transcripts, which provided context for the quotes appearing in the manuscript. Annotating also allowed us to engage in greater depth with conflicting evidence from interviews, a process we describe in the following section. In addition to increasing analytic transparency, a secondary function of the annotations was to supplement technical material to make the paper’s quantitative analyses more accessible to readers.

BENEFITS OF TRANSPARENCY IN MIXED-METHODS RESEARCH

We identified three main benefits to committing to transparency early in the research process. First, doing so forces researchers to approach the collection and interpretation of evidence in a more methodical way (see Milonopoulos Reference Milonopoulos2021). Given that we anticipated needing to justify claims and provide detailed quotes, my coauthor and I were more meticulous about taking notes and saving and organizing documents to code outcomes for political prisoners. For example, because we did not audio-record any interviews with government officials, detailed notetaking was important; therefore, two researchers conducted each interview, one to facilitate and one to take notes.

In the writing process, using digital annotations forced us to think carefully about our underlying assumptions and gave us more space in the manuscript to be transparent about conflicting pieces of evidence (see Mayka Reference Mayka2021; Milonopoulos Reference Milonopoulos2021). For instance, policy makers that we interviewed disagreed about the long-term impacts of the release of political prisoners on human rights practices. Some viewed releases as “token” actions taken by repressive regimes, whereas others believed they had symbolic or structural impacts that extended beyond outcomes of individual cases. This debate—although not central to the argument of our paper—was important for thinking through potential implications of government-led human rights campaigns such as #Freethe20. Using ATI allowed us to illustrate how policy makers wrestled with the tradeoffs of prioritizing advocacy around political prisoners relative to other human rights issues.

Second, a commitment to transparency can increase the credibility of a researcher’s conclusions. In early discussions about what an evaluation of the #Freethe20 campaign would look like, we were hesitant to start data collection. We anticipated that a “medium-N” study of 20 female political prisoners would be difficult to execute in an article-length project. If findings about the efficacy of the campaign were mixed, it could be important to provide detailed narratives of each case—which would require a lengthier analysis than allowed by a journal article.

We thought that making our research more transparent could resolve this issue in multiple ways. First, we could use annotations to add more nuance and context to the article. As Mayka (Reference Mayka2021) notes, a major advantage of ATI is that it allows researchers to write in greater depth than permitted by a journal’s word limit. Second, we thought that we could bolster the credibility of our claims by sharing the qualitative data that we collected, enabling other researchers to independently assess our conclusions. After completing our data collection and analysis, we were surprised that the results of the #Freethe20 campaign were more clear-cut than anticipated. The data that we collected showed that women featured in the campaign had significantly better release outcomes than their peers.

As a result, our paper—and our use of ATI—became more about understanding which HRD strategies made the campaign effective through interviews with government officials. We supplemented brief quotes in the manuscript with longer annotations containing interview excerpts that probed policy makers’ evaluations of the campaign. For example, an initial goal of #Freethe20 was to use a strategy of “naming and shaming” (Hafner-Burton Reference Hafner-Burton2008; Murdie and Davis Reference Murdie and Davis2012) to mobilize foreign publics and civil-society organizations within target states. We asked policy makers what evidence they had that the #Freethe20 campaign actually accomplished this objective. Our discussions revealed that it was unlikely that the public attention generated by #Freethe20 was sufficient to mobilize foreign publics. Interviewees instead expressed regret about not focusing efforts on in-language translations of messages from the campaign. Our quantitative analyses about media coverage and online searches of #Freethe20 women following its September 2015 launch also demonstrated that the campaign had little sustained international attention. This evidence suggested that “naming and shaming” alone did not independently drive prisoner releases. Instead, we found that public diplomacy around #Freethe20 launched private diplomatic efforts by the US government to secure release of the women featured in the campaign. Using ATI allowed us to increase transparency around our research process by explaining how our findings evolved throughout the course of data collection and interviews.

A third benefit to using ATI is that it can make technical academic work—whether quantitative or qualitative—accessible to a broader audience (a point also made by Milonopoulos and other contributors to this symposium). Much of what we know about contemporary US diplomacy, particularly with respect to private diplomacy—also called “quiet diplomacy” (Forsythe Reference Forsythe1995; Reference Forsythe2000)—comes from government officials who provide narratives about their experiences. Because our project was rooted in the foreign policy bureaucracy, the findings may be interesting to practitioners. However, our analysis of both the outcomes of the #Freethe20 campaign and the media coverage around it involved advanced quantitative methods, including less intuitive regression models such as count models and survival models to characterize media coverage and release rates, respectively. The ability to add digital annotations that explain these models in greater depth improved the accessibility of our work outside of academia.

This process made it apparent how annotation is a valuable tool for mixed-methods researchers interested in speaking to multiple audiences. One strategy could be the use of ATI to create different categories of digital annotations (see Milonopoulos Reference Milonopoulos2021). For instance, with regard to research methods, one set of annotations could provide technical details relevant for “experts” and another set could use layman’s terms to break down complex methods for “novices.” In addition to increasing data, production, and analytic transparency, another potential benefit of annotations is increasing access to scientific work.

CHALLENGES TO WIDER ADOPTION OF ATI

A movement toward greater transparency in qualitative research also raises interesting challenges for researchers. A core question we struggled with was: How should we balance calls for greater transparency with concerns about privacy? This issue was salient for our project because we relied heavily on interviews with US government officials to draw conclusions. We initially wanted to anonymize and share the transcripts to increase the transparency of our research. However, given the expertise of the policy makers whom we interviewed, it was apparent that releasing transcripts—even anonymized—would not protect their identity.

Other researchers using interview methods are likely to encounter similar obstacles to transparency in their research (see Gerring Reference Gerring2021; Mayka Reference Mayka2021). First, for those interviewing political elites about policy making, Hall (Reference Hall2016, 33) noted that “much will be left unsaid” without guarantees of anonymity. One way to resolve tension between transparency and privacy may be to emphasize production transparency more than sharing data. Rather than publishing transcripts, researchers could instead create and share an interview-methods appendix that explains in detail how the data were collected (Bleich and Pekkanen Reference Bleich, Pekkanen and Mosley2013; Elman, Kapiszewski, and Lupia Reference Elman, Kapiszewski and Lupia2018).

Second, should we develop any shared standards for data transparency across quantitative and qualitative research? As first-time users of ATI, it was difficult to know how in-depth our annotations should be and how much qualitative data to share (Milonopoulos and others in this symposium raise similar concerns). On the one hand, the flexibility of ATI is perhaps its greatest benefit: it allows researchers to use annotations for whatever they believe is relevant to their manuscript. On the other hand, researchers intrigued by the idea of using annotations in their work may find the absence of shared standards somewhat disorienting. As new adopters of this approach, we wrestled with which passages to annotate and found it difficult to strike the correct balance without a framework on which to rely.

Standards for sharing quantitative data are imperfect; however, during the past decade, political science journals have adopted policies that encourage or require authors to provide replication files consisting of the underlying data and replication code (Key Reference Key2016). It is less common to share all exploratory code or log files, which could overwhelm readers interested in the analyses. It is unclear, however, whether an analogous solution could be developed for qualitative data. For example, if we found conflicting information about whether a female political prisoner was released, should we share the information that we think is more credible or share both pieces of information? What is the balance between being transparent in the research process and “oversharing” qualitative data such that it becomes burdensome for another researcher to sift through? As Siewert (Reference Siewert2021) notes, it is not evident that sharing more information necessarily enhances transparency.

Third, how do we overcome challenges specifically related to mixed-methods research so that scholars who conduct this work are not dissuaded from adopting these practices? Three such challenges concern integrating elements of quantitative and qualitative transparency, introducing ATI into a researcher’s workflow, and navigating the peer-review process. With regard to the first concern, mixed-methods researchers may be confused about how to integrate annotations and qualitative data with quantitative replication files. As previously noted, many political science journals have requirements for data storage and replication that are specific to quantitative work. Scholars working on mixed-methods projects may find it cumbersome or time-intensive to store quantitative and qualitative data in separate repositories.

The second difficulty comes from adapting annotations into a mixed-method researcher’s workflow. Setting aside the fact that annotating a manuscript can be labor-intensive (a point echoed by Mayka, Siewert, and others in this symposium), researchers who write in LaTex because of its versatility in presenting quantitative work may find it challenging to adopt ATI. In Microsoft Word, adding “Comments” is an intuitive way to create annotations that later are digitized. These comments are not distracting when editing the text because Microsoft Word displays them in the margins. Adding annotations in LaTex is more visually challenging because “commented” text does not appear in a compiled LaTex document. One solution is to use an online LaTex editor such as Overleaf, which provides “Comment” functions similar to those in Microsoft Word. However, some researchers may prefer not to write in an online environment. These issues seem minor relative to broader debates about the importance of transparency in qualitative research, but practical concerns about workflow—which also are highlighted by Siewert (Reference Siewert2021)—may discourage researchers from adopting ATI.

Third, with respect to the peer-review process, researchers have expressed concerns that new advances in qualitative transparency will be unfamiliar to editors and reviewers (Parkinson and Wood Reference Parkinson and Wood2015). It is easy to see how annotations—much like lengthy technical appendices—could be perceived as an additional burden on reviewers. For mixed-methods researchers, adding qualitative annotations as well as an appendix with robustness checks from quantitative analyses could easily triple the length of a manuscript. Researchers curious about ATI may be hesitant to adopt the approach if they anticipate that annotations could be met with skepticism or confusion in the review process. Proactive messaging from journal editors about the value of ATI or similar approaches is one possible way to assuage these concerns.

CONCLUDING THOUGHTS

A movement toward qualitative transparency provides exciting opportunities for mixed-methods researchers, but there also are various challenges in moving forward. In our experience using ATI for the first time, many of these difficulties were related to practical questions, such as how to most effectively share data, balance privacy and transparency, and navigate new workflows. Nevertheless, the growing community of researchers using ATI is a testament to the enthusiasm for the approach. Continued experimentation with ATI in the research and writing processes is perhaps the most productive way to advance these important conversations about qualitative transparency.