LEARNING BY DOING?

At the root of the argument in favor of synthesizing undergraduate teaching and research is the claim that students learn best by doing. While we are in favor of active learning, we do not think that simply carrying out a research task, isolated from the research design and epistemological justification which motivated it, can teach students much about how to do social science.

Collaborative social science involves a team effort. Of course, it is not necessary for every team member to do the same task, and not every task requires the same level of skill. Table 1 suggests four ideal types, depending on combinations of two dichotomous conditions: whether the research task is simple or sophisticated, and whether the student has been trained in methods.^{Footnote 6}

Table 1

Why Methods Training Should Precede Engaging in Research Tasks

With regard to the distinction between simple and sophisticated research tasks, even without a methodological grounding, with straightforward instructions undergraduate students can carry out simple research tasks—identifying a set of relevant newspaper articles or coding a set of documents, for instance. But without methodological preparation, in contrast to the socialization value we mentioned earlier, the educational value of conducting the task rarely extends beyond the immediate content of the activity: the necessary contextualizing knowledge remains buried beneath the project’s tasks and practices, even as it drives them. By contrast, if students first have learned about the “how” and the “why” of the type of research being conducted, even the simplest of tasks is imbued with this additional meaning. To put it another way, methods training converts even simple discrete research tasks into acts of social inquiry. The distinction becomes starker when contemplating more sophisticated research tasks, which are likely to be completely beyond the reach of students with no preparation. In short, those who have been trained in methods can aspire to be full partners with their faculty mentors, and hence learn a great deal more from the experience.

Of course, we do not dismiss the possibility that faculty might try to teach students the underlying epistemology, requirements, logics, and rules of the methods they are using as students assist them with their research. However, it is unrealistic to expect faculty to continually halt forward motion on their projects to take the necessary time to do this. Moreover, even if they did, grasping those methods would be complicated for students who are struggling to understand complex new contexts, topics, concepts, and theories.

Another practical reason we are dubious about teaching undergraduates research skills by engaging them in faculty research concerns scale. Any individual faculty member can only involve a few undergraduates in research in a way that would be helpful for the faculty member and meaningful to students. In particular, the bigger the team, the less clear the intellectual payoff for the students. As such, even if undergraduates could learn research methods through participating in faculty research, only a few will have the opportunity to do so.^{Footnote 7}

In short, it is very difficult for undergraduates to learn most research methods from undertaking rote tasks in a research project that uses them without engaging with the project’s motivating epistemology. This engagement requires the prior attainment of the requisite methodological knowledge base through dedicated classroom instruction. This is an especially serious problem for qualitative methods because courses fully focused on qualitative research methods are rare at the graduate level, and even less frequently taught to undergraduate students.^{Footnote 8}

THE BENEFITS OF TRAINING UNDERGRADUATES IN RESEARCH METHODOLOGY

Training undergraduates in social science methods entails more than simply teaching them to carry out research tasks, use data-collection techniques, or use analytic tools. For each social science tradition, the potential value of research is defined by foundational theories of knowledge, which in turn imply a set of practices. More formally, ontological claims are bracketed by the epistemology (and associated prescriptive methods) that produced them. Scholarship is rule-bound for a reason: badly done social science produces misinformation, not knowledge. The purpose of research skills is to enable scholars to follow the rules of their tradition and to produce knowledge with the value added of the type promised by the underlying epistemology.

Accordingly, teaching methods implies conveying the rules by which a research tradition operates and why they must be followed. It involves explaining how social science knowledge is derived, how intellectual consensus forms, and how claims are contested and conventional wisdom overturned. It also entails discussing how correlation is established and causal inferences are made, and from what angles and with what advantages different types of observational and experimental methods can contribute to those tasks.

While this knowledge is a prerequisite for effectively engaging in social inquiry (and for learning from that experience), such training benefits all students. Methods courses help students to comprehend the empirical arguments invoked by instructors in their substantive courses and to be more active and sophisticated consumers of political (and other social) science scholarship. They do so by providing students with the building blocks of social science and by enhancing their “information literacy” more broadly (Marfleet and Dille Reference Marfleet and Dille2005). Students trained in methods can identify concepts and make sense of how they were operationalized, comprehend and distinguish between different kinds of claims, understand the use of complex data, and interpret findings in various contexts. Most broadly, students who have been trained in methods are better placed to grasp the ephemeral, changeable, contestable nature of much social science knowledge (Bernstein and Allen Reference Bernstein and Allen2013, 11; Clark Reference Clark2011, 138).

TEACHING QUALITATIVE METHODS TO UNDERGRADUATES: A PROPOSAL FOR AN ALTERNATIVE FORM OF ACTIVE LEARNING

While the need to train undergraduates seems uncontroversial with regard to quantitative methods, it is a new approach for qualitative research in political science. A first step, then, is to envision what such undergraduate qualitative methods classes might look like. While graduate-level courses offer some clues, the traditional model would require substantial revision to be effective at the undergraduate level. Instead, the quantitative tradition might provide a better paradigm for teaching qualitative research methods to undergraduates.

Graduate qualitative methods classes typically combine prescription and examples. The prescriptive literature lays out the epistemic justifications for a given method and details the steps necessary to achieve its desired results. Methods texts and instructors usually point to applications of the method as examples and seek to describe how authors used it to arrive at their conclusions. This turns out to be a less-than-ideal way to teach research methods. Although most qualitative scholars state their research design and discuss their data and analysis, the mechanics that underlie the analytic method frequently remain “implicit” (Mahoney 2007, 135): even authors of exemplary scholarship rarely fully describe how they arrived at their conclusions. Accordingly, using such exemplars to teach methods requires instructors and students alike to reverse engineer the procedures. While sub-optimal at the graduate level, this model is even less likely to succeed at the undergraduate level.

A more promising model for teaching qualitative research methods at the undergraduate level can be found in an established practice of the quantitative tradition. Teachers of statistical techniques have long held that students internalize methods more quickly and completely when they use them. Rather than begin with “real” research, however, statistics texts routinely include exercises linked to “pedagogical data”—common, stylized data specifically prepared for teaching purposes—that maximize student engagement with, and focus their attention on, the methods being taught.^{Footnote 9} This is still a form of “learning by doing,” but the “doing” is structured by the texts and their exercises and data. That is, method is privileged over substance initially, with the balance shifting over time towards more authentic research tasks and data.^{Footnote 10} Teaching in a comparably progressive fashion is a well-regarded technique to overcome students’ anxiety about methods (Bos and Schneider Reference Bos and Schneider2009, 381–82).

Using data to teach qualitative methods should produce similar benefits. We propose designing individual courses and multicourse sequences that involve two step-wise processes: (1) a progression from engaging in simple text-based exercises to engaging in real research tasks and (2) a more gradual transition from being taught and practicing research skills with stylized data to doing the same with authentic data.

To give an abstract overview of the first transition, text-based exercises to “real” research tasks, after introducing students to the logic and rules of a particular qualitative analytic method, instructors illustrate the mechanics of the method through relatively simple research tasks, which students practice by doing exercises using pedagogical data. For instance, students may be asked to code a set of interview transcripts on particular dimensions, or to identify information in those transcripts that could be used to measure some concept. Over time, students can practice more complicated research tasks and skills or engage in more than one task at a time. For example, they could use process tracing to identify the steps and mechanisms in a causal chain using a series of archival documents. As they learn different methods (and how they fit in the broader research process), students can apply them to real social science problems, asking and answering substantive research questions (i.e., producing scholarship) relevant to their subfield interests.

The second transition, moving from pedagogical data to authentic data to learn and practice research skills, should occur more gradually. A “pedagogical data project,” as we envision it, is a coherent aggregate of qualitative data, usually derived from research data but revised for teaching purposes, streamlined, simplified, and organized for instruction. Because pedagogical data are designed to help students to develop research skills, cutting the connection between the data and the social world is justified: doing so allows the data to be adapted to maximize their utility for teaching a particular research skill. Such data will be accompanied by standard documentation detailing the processes through which they were generated and simplified, and materials oriented toward instruction.^{Footnote 11}

As this sequence suggests, we are very sympathetic to the advantages of students eventually working with actual research data. Research data, with all their intricacies and idiosyncrasies, are authentic: they reflect empirical reality and the original research experience more faithfully than do pedagogical data. To the extent pedagogical data are inauthentic, their use by students cannot produce new knowledge. Authentic data are also more likely to be drawn from the instructor’s own materials than are pedagogical data, allowing the instruction to be personalized in a way that is more difficult with data that are obtained elsewhere. Nonetheless, we believe it will usually be advisable to use authentic data only later in undergraduate methods instruction. Toward the beginning, students should work with data that were prepared for pedagogical purposes, allowing their attention to be tightly focused on the acquisition of research skills.

TEACHING QUALITATIVE METHODS THROUGH DATA: SOME CHALLENGES

Our proposal that qualitative methods be taught to undergraduates in dedicated courses using pedagogical data faces at least three major challenges. First, few texts are suitable for teaching qualitative political science research methods to undergraduates, and even fewer that invoke pedagogical data. Second, some pedagogical data will be derived from real research data, yet political scientists who generate qualitative data rarely share them. Third, even if qualitative social scientists did—or will—share their data, neither clear processes for converting research data into pedagogical data nor parameters for evaluating the pedagogical value of the resulting data have been developed. We discuss each challenge, and how it can be addressed, in turn.

Few texts teach qualitative research methods in the way we have proposed. Even the now near-dormant Sage blue books series may not have been sufficiently directed to an undergraduate audience. However, a new series recently introduced by Cambridge University Press may help to address this gap.^{Footnote 12} As planned, the series will comprise short, straightforward, and accessible “user guides” offering concrete instructions for using particular techniques for collecting and analyzing qualitative data. Each book will feature rigorous exercises to be conducted using associated pedagogical data (and their accompanying documentation). Each will also offer multiple references to exemplary articles and books that use the methods under analysis and, where possible, access to the data that underlie those studies.

The second major challenge concerns data availability. Pedagogical data can only be derived from authentic data if scholars share them. Unfortunately, norms and standards for doing this have been slow to emerge among qualitative political scientists. As a result, the discipline lacks a stock of data that could be modified for classroom use at anything approaching reasonable transaction costs. In part, this is because the ethical and legal challenges (related to human subjects and copyright in particular, see Bishop Reference Bishop2012) that attend sharing all data impact the sharing of “disaggregated” qualitative data more directly.^{Footnote 13} However, several initiatives are under way to develop strategies for addressing these significant concerns (see, for example, Lupia and Elman Reference Lupia and Elman2014).

A final challenge concerns our incomplete understanding of the desiderata for pedagogical qualitative data, and how to achieve them.^{Footnote 14} Teaching qualitative methods in the way we propose requires developing a clear understanding of how genuine research data and pedagogical data differ, and strategies and standards for deriving one from the other. This is a long-term project that we hope members of the qualitative methods community will embrace.^{Footnote 15}

These are not the only challenges to the approach we propose, of course. We can anticipate several additional misgivings, for instance, that undergraduates have just eight semesters (or 12 quarters) into which a great deal has to be fit; that political science undergraduate programs already struggle to teach students the basics of the world’s increasingly diverse political systems; and that proposing and gaining approval for new (untried) courses can be difficult. Notwithstanding these challenges, students will only be able to effectively engage in research (and learn from doing so) if they have previously taken dedicated methods courses, including on qualitative methods.