%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
The early 2000s witnessed the beginning of a renaissance in qualitative research methods in the discipline of political science (e.g., Collier and Brady Reference Collier and Brady2004; George and Bennett Reference George and Bennett2005). This renaissance has included the development of more systematic and analytically explicit approaches to using qualitative evidence for descriptive and causal inference. Unfortunately, however, the teaching of qualitative research methods has not kept pace with their development. In particular, a recent study of the methods curriculum in 25 top political science doctoral programs between 2010 and 2015 found that qualitative methods instruction tends to take a passive rather than an active form. In marked contrast to the teaching of quantitative research methods, students rarely learn qualitative analytic methods by practicing their use on data generated and shared by other researchers (Emmons and Moravcsik Reference Emmons and Moravcsik2019, 4). Although not necessarily representative of the broader discipline, these findings suggest an important gap in the instruction of qualitative methods at the graduate level.
This article discusses a novel approach to teaching qualitative methods that uses more “active learning” through students engaging with scholarship that has been annotated using Annotation for Transparent Inquiry (ATI). ATI is a new approach to achieving transparency in qualitative and multi-method research that allows authors to annotate specific passages in a digital publication in order to clarify methodological challenges and choices, add detail about evidence or analysis, and/or link to data sources that underlie claims. Learning methods through engagement with annotated articles allows students to interact with original data and to better understand, evaluate, and critique how authors collected, analyzed, and used those data to draw inferences and develop arguments. Compared with traditional pedagogical techniques, this approach thus leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research.
Compared with traditional pedagogical techniques, this approach thus leads students to learn research methods in a way that more closely approximates how they will use those methods in their own research.
Our discussion in this article is somewhat speculative. To date, few of the teaching strategies that we propose have been used in practice, for reasons we discuss. Nonetheless, the case studies we present in this article demonstrate that initial efforts to use these techniques have yielded promising pedagogical outcomes. These results, in tandem with the broad consensus in the education literature on the value of active-learning techniques, recommend continuing to develop ATI-oriented teaching strategies. We hope that this article catalyzes further exploration and experimentation.
We begin by offering a few observations on qualitative methods instruction in political science. We elaborate in particular on how infrequently methods for analyzing qualitative data are taught via active learning using data generated and shared by other researchers. We then briefly describe ATI and its potential to facilitate the active learning of qualitative analytic methods, illustrating that potential by describing the experiences of three methods instructors. We conclude with a short discussion of limits to this pedagogical approach.
THE STATUS QUO AND THE CHALLENGE
Active learning—defined as the use of “instructional activities involving students in doing things and thinking about what they are doing” (Bonwell and Eison Reference Bonwell and Eison1991, 2)—is widely considered to be an effective pedagogical technique that facilitates skills acquisition and generates student interest (Rehak and Hmelo-Silver Reference Rehak, Hmelo-Silver and Peppler2017; National Research Council 2000). Whereas active-learning techniques often are used to teach strategies for collecting qualitative data (e.g., archival research, interviewing, and ethnography), recent research suggests that it is unusual for students to learn how to analyze qualitative data by practicing the use of particular methods on data that other scholars generated and shared (Emmons and Moravcsik Reference Emmons and Moravcsik2019, 4).Footnote 1 In the instruction of quantitative methods, by contrast, students often analyze shared datasets to complete exercises and problem sets, or seek to reproduce the findings of research articles with open data and materials using computational methods (Janz Reference Janz2016; King Reference King2006).
Two factors likely contribute to this gap in qualitative methods instruction. First, in contrast with the numerous textbooks treating quantitative analytic methods, the books used most often in the instruction of qualitative analytic methods tend to focus on the epistemological underpinnings and inferential logics of those methods rather than the practical steps involved in using them. Moreover, few of these books include problem sets or other practical exercises.Footnote 2 Second, practicing using analytic methods with data gathered and shared by other scholars is limited by the paucity of shared qualitative data. Relatedly, qualitative datasets specifically created for teaching are practically nonexistent.
To date, then, few graduate students have had the opportunity to learn qualitative analytic methods by practicing their use on shared data. Conditions for using this promising teaching practice are beginning to improve, however, as scholars develop new approaches for sharing their qualitative data in ethical, legal, and meaningful ways (Kapiszewski and Karcher Reference Kapiszewski and Karcher2020). The remainder of this article suggests pedagogical strategies that capitalize on these developments and that should enhance the instruction of qualitative analytic methods.
A SOLUTION: ANNOTATED SCHOLARSHIP AS A TEACHING TOOL
To make the teaching of qualitative analytic methods more active, we propose that instructors invite their students to engage with scholarship that has been annotated using ATI.Footnote 3 Developed by the Qualitative Data Repository (QDR) (see qdr.syr.edu) and the software nonprofit Hypothesis, ATI is an innovative approach to achieving transparency in qualitative and multi-method research. Authors who use ATI digitally annotate empirical claims or conclusions in their manuscripts. ATI uses open-web annotation, which allows for the generation, sharing, and discovery of digital annotations across the web. Annotations may include full citations to underlying data sources; “analytic notes” clarifying how authors generated or analyzed their data and/or how the data support inferences or interpretations; excerpts from data sources; and links to the data sources when they can be shared ethically and legally. Annotations, in other words, surface the “analytic scaffolding” and evidentiary underpinnings that rarely are included in published qualitative work due to space constraints. Annotations are displayed on the same web page as the digitally published article or book that they accompany; the annotations and underlying data sources are curated and preserved by a data repository (figure 1).Footnote 4
ATI at a Glance: How a Passage in the Text of an Article Annotated with ATI Appears to Readers
We highlight three strategies that instructors of qualitative analytic methods might use to integrate ATI-annotated scholarship into their teaching.Footnote 5 One strategy entails students examining the analytic scaffolding that ATI reveals; a second involves them engaging in reanalysis and analytic extensions. Both of these approaches rely on scholarship that has been annotated using ATI and work best when the annotations include links to a rich set of underlying qualitative data. A third strategy calls on students—typically advanced graduate students—to annotate their own work using ATI.
Instructors might use the first strategy—employing ATI to show students the mechanics of qualitative inference in action—to introduce a particular qualitative method. In this approach, instructors draw on annotations that elaborate on the analysis of qualitative data—in tandem with materialsFootnote 6 and data shared by the author—to highlight and discuss with their students the logic of inquiry and the type of analysis in the exemplars. Students thus learn qualitative research methods by retracing and examining an author’s analytical steps (and, potentially, missteps). For instance, students can see how authors identified and deployed causal process observations (CPOs) to advance narrative process tracing; learn about the mechanics of Bayesian process tracing from annotations that discuss how priors and likelihoods were assigned and posterior estimates were progressively updated; and gain insight into the conceptualization and coding that underlie the process of calibration in Qualitative Comparative Analysis (QCA).
In the second pedagogical approach, instructors devise a set of structured exercises that call on students to actively engage with an author’s analysis through the annotations provided in a publication. These exercises resemble the reproducibility exercises commonly used in quantitative methods instruction. For instance, students can reanalyze the supporting data provided in ATI annotations, using the same method that the author used, to evaluate the strength of particular claims made in the text of a publication or to regenerate and validate tables and figures.
The third teaching strategy involves translating active learning into active research. In this approach, students who are developing their own empirical work use annotation to be more explicit and to offer more detail about their analytical choices—that is, how they generated, analyzed, and deployed evidence to support their claims.
EXAMPLES OF TEACHING WITH ATI
One author (Jacobs) piloted an ATI-supported exercise using the first two strategies in a graduate qualitative methods class in fall 2019. Focusing on annotated portions of one of two pieces of scholarship (Saunders Reference Saunders2011, chap. 4; O’Mahoney Reference O’Mahoney2017), students selected individual CPOs for which an annotation provided either excerpts or reproductions of source materials. In the assignment writeup, students first retraced the author’s steps, summarizing how the author sought to link the observation to an empirical implication of the article’s or book’s theoretical argument. Second, students critically evaluated the inferences that the author drew from the evidence presented in or linked to in the annotation. The annotations allowed students to assess the original inferential reasoning more thoroughly or to develop more plausible alternative explanations than they could have done from the article text alone. By engaging with the more complete evidentiary record, students gained insight into how authors had selected observations from their source materials and how the meaning of a statement or observation might depend on the context in which it appeared. Moreover, reanalyzing the data presented in the article and found in the source material demonstrated the different inferences that might be drawn from a given observation, depending in part on assumptions that the analyst made and readings of the context in which the observation arose.
By engaging with the more complete evidentiary record, students gained insight into how authors had selected observations from their source materials and how the meaning of a statement or observation might depend on the context in which it appeared.
In a similar exercise used during multiple semesters in an upper-level undergraduate course focused on primary-source–based inquiry, Robert Adcock (American University) asked students to assess the annotated evidence in the Saunders (Reference Saunders2011) chapter. Students evaluated the accuracy and precision of the summary or use of the shared source materials in the chapter’s text and the sufficiency of the evidence as support for claims.Footnote 7 Students also were asked to think about whether—in aggregate, given their number, type, and authoritativeness—the sources could be plausibly considered representative of the broader set of potential sources, particularly given the growing variety and number of primary sources available online (discussed earlier in the course). The goal of the exercise was for students to acquire—via actively engaging with Saunders’ meticulous use of primary sources—a working model of what they should aspire to in using and presenting primary-source evidence as they drafted their own research papers in subsequent weeks of the course.
Juan Masullo (Oxford University and Leiden University) integrated ATI into his instruction of process tracing using the first strategy.Footnote 8 In one exercise,Footnote 9 Masullo led students to discover the architecture of process tracing by asking them first to read passages of an article annotated using ATI without the annotations, identifying evidentiary and analytic weaknesses, gaps, or silences. He and his students then reread those passages with the annotations, assessing how well the annotations addressed the pre-identified issues and contextualized and characterized the value of evidence.
We can easily imagine useful extensions of these exercises: students might be asked to reanalyze data shared in ATI annotations using an analytic method different from the one an author used. For instance, students could apply formal Bayesian process tracing to an article that analyzed evidence informally to see if they arrive at the same results.
Finally, in an exercise using the third pedagogical strategy, Masullo encouraged graduate students working on their dissertation to annotate it with ATI to reveal the analytic architecture of their research. In correspondence with us about this exercise, Masullo emphasized the importance of guiding students to annotate with purpose: to justify citing evidence or discuss its probative value; to describe analytically relevant aspects of interview respondents’ profiles; and to make more explicit the logic of process-tracing tests or Bayesian analysis.Footnote 10
In summary, the growing availability of ATI-based publications enables more active forms of learning qualitative analytic methods. Active learning of these methods can generate a better understanding of how scholars draw descriptive and causal inferences from qualitative evidence, making students more sophisticated consumers of qualitative research. Moreover, active engagement with shared research data helps students to learn to use qualitative methods by applying them.
Active learning of these methods can generate a better understanding of how scholars draw descriptive and causal inferences from qualitative evidence, making students more sophisticated consumers of qualitative research.
CONCLUSION
This article describes general strategies for introducing active-learning elements into qualitative methods instruction using ATI. It discusses the experiences of three instructors who used ATI in the classroom, recounting their impressions of how using this teaching technique improved learning outcomes. All three instructors continue to use and refine their approach. As more articles annotated with ATI are published, these and other instructors will be able to use that scholarship to teach a broader range of analytic methods.Footnote 11
As noted, there are similarities between the logic of teaching qualitative methods with ATI and the logic of teaching quantitative methods via reproducibility exercises. The usefulness of reanalysis exercises using ATI, however, does not hinge on that parallel. When authors use ATI, students can evaluate authors’ findings by assessing whether those findings can be reproduced from the data made available. Yet, because most qualitative analysis involves a degree of subjective interpretation, different qualitative analysts might reasonably draw different inferences, even when using the same methods to analyze the same data. In addition, some qualitative methodologies—particularly those operating within an interpretivist paradigm—eschew empiricist notions of researcher-independent objectivity. In these research traditions, reproducibility is incoherent as an evaluative standard. Where reproducibility is not a relevant benchmark, however, students are still likely to arrive at a better understanding of methods if they have the opportunity to apply them to the evidence underlying a publication and to compare their reasoning and findings to those presented by the original authors.
Even within a replicationist paradigm, learning qualitative methods with ATI has limits. First, not all work annotated with ATI is equally suitable for teaching. The initial experiences of teaching with ATI described in this article suggest that annotated publications that provide at least some associated primary material will be most useful pedagogically. Likewise, annotations that effectively “lift the curtain” on the research process—for example, by discussing the evidentiary value of data cited in the text, commenting on how the author adjudicated between contradictory pieces of evidence, and highlighting choices made during data collection—can be particularly enlightening for students. At the same time, we emphasize that even the most robustly annotated work cannot provide visibility into every choice that authors made when selecting sources or data from all of the information they encountered or collected. Moreover, the sharing of evidence is always partial: just as quantitative replication datasets typically include only those data used to generate the reported results, annotations typically include only the observations and materials referenced in the text, leaving out the potentially large set of sources and evidence consulted but not referenced.
Second, instructors of qualitative methods must assess the time required to effectively teach with ATI. Adcock, Jacobs, and Masullo all found that engagement with ATI could be readily integrated into lessons on core topics such as comparative historical analysis and process tracing, and they reported that students needed little time to become familiar with ATI. In the context of a course dedicated to qualitative methods—especially one intended to empower students to use these methods themselves—we believe ATI can be integrated into instruction without sacrificing other material. In a general research methods course, with only a few sessions focused on qualitative methods, the time required to set up an effective lesson using ATI might be prohibitive.
Third, and more broadly, although the initial experiences recounted in this article are encouraging, to date we have limited evidence of the efficacy of using ATI to teach qualitative methods. We hope that this article encourages more instructors of qualitative research methods to bring ATI into their classroom so we all can continue to develop, assess, and capitalize on its pedagogical promise.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://educate.apsanet.org/resource/11-30-2021/assignments-for-using-annotation-for-transparent-inquiry-ati-to-teach-qualitative-research-methods.
ACKNOWLEDGMENTS
We are grateful to the symposium organizers for inviting us to participate and to the scholar who reviewed our article and provided useful feedback. Kapiszewski’s and Karcher’s contributions to this article are based on work supported by the National Science Foundation under Grant Nos. 1823950 and 1946272.
