WHAT WE KNOW, AND WHAT WE ONLY THINK WE KNOW

Past research on undergraduate methods training teaches us a great deal about successful approaches to teaching the course, and constantly reminds instructors to keep their chins up—the course “is one of the toughest teaching assignments in academia, but it can also be one of the most productive” (Adams Reference Adams2001, 3) is a typical sentiment. Indeed, there are many good normative articles that offer great advice for teaching undergraduate methods courses, usually based on years of first-hand experience (see, for example, Rodgers and Manrique Reference Rodgers and Manrique1992; Hubbell Reference Hubbell1994; McBride Reference McBride1994; Reference McBride1996; Adams Reference Adams2001; Clawson, Hoffman, and McCann Reference Clawson, Hoffman and McCann2001; Hojnacki Reference Hojnacki2001; Janda Reference Janda2001; Lewis-Beck Reference Lewis-Beck2001; Poggione Reference Poggione2001; Stone Reference Stone2001; Reinke Reference Reinke2005; Turner Reference Turner2005). These articles have provided instructors with valuable insights about best practices in scope and methods courses and, we suspect, have helped provide a better classroom experience for both students and faculty. The shortcoming of this literature, when taken as a whole, is that it provides little systematic knowledge of how scope and methods is taught. The descriptions and recommendations are typically idiosyncratic, based on the experiences of one or two instructors at one or two universities.

In addition to these useful didactic pieces, there have also been a small number of descriptive works attempting to assess the state of the discipline in scope and methods training. Some offer broad overviews of the state of scope and methods training, usually with a prescription for change. Mayer (Reference Mayer2002), for example, presents an epistemological history of political science over the past few generations. He argues that a misplaced emphasis on statistics and the outward trappings of science has caused political science education to overlook more fundamental issues: “if we are to pursue a scientifically based discipline, we must first understand and reach consensus on what that is and then clearly communicate that understanding to all of our students” (Reference Mayer2002, 122). Though Mayer's advice to the discipline is valuable, and his historical descriptions likely accurate, his assessment of the present state of methodology education does not appear to be based on any objective empiricism. Other works on methods education, though empirical in nature, focus narrowly on specific content elements, such as the incorporation of information literacy (Marfleet and Dille Reference Marfleet and Dille2005). Still others, like Schwartz-Shea (Reference Schwartz-Shea2003) and Bennett, Barth, and Rutherford (Reference Bennett, Barth and Rutherford2003), offer thorough empirical accounts of methods offerings in graduate programs.

Finally, Thies and Hogan (Reference Thies and Hogan2005) present recent empirical evidence on undergraduate scope and methods courses. Surveying a broad cross section of American political science departments, Thies and Hogan find that departments with Ph.D. programs are less likely to require a methods course (46.8%) than those without (64.5%). Prior to their survey, many political scientists may have assumed the opposite was true. We speculate that this finding may mean the former programs are less interested in teaching undergraduates to be social scientists, or that the latter programs are more likely to see such training as a key component of a liberal arts education. Whatever the departmental motivations might be, the data suggest that important divisions exist in how political scientists view the role of undergraduate methods. This discovery indicates the need for a more detailed investigation.

Indeed, though we have a sense of how some individual courses are taught, and we now know how often they are required, political scientists have relatively little empirical knowledge of the content of undergraduate scope and methods courses. This lack of information has led some to make assumptions that may or may not be accurate. Michael Lewis-Beck (Reference Lewis-Beck2001, 7), for example, notes that the “scope and method” format for undergraduate instruction, with a “review of leading theoretical paradigms and its going over of different study approaches” is “pretty much a thing of the past.” He argues (Reference Lewis-Beck2001, 8) that this has largely been replaced by an emphasis on “understanding tools for the scientific explanation of politics—computers, mathematics, statistics.” Though Lewis-Beck's statements are made in passing and are not the main focus of his article, the claim raises an interesting question: just what is the substance of undergraduate political science scope and methods courses in American universities today? What do we know, and what do we only think we know? Upon teaching undergraduate methods for the first time, one professor observed: “I wasn't sure about what the objectives of a research methods course should be or what material to include and what to exclude” (Poggione Reference Poggione2001, 10). We suspect many an instructor has confessed a similar ignorance when faced with this question.

The importance of the question of content is further highlighted by battles over the “best” or “correct” methodology for political science. While these discussions have largely focused on the content of the leading journals and the training of graduate students, it may also be of some interest to know how such academic debates filter down to the undergraduate classroom. Are scope and methods classrooms being overrun with rational choice theorists? Are qualitative approaches being taught to undergraduates? On one hand, it may be the case that the positivist vs. postpositivist debate has been largely confined to academic journals and that when it comes to educating undergraduates in the fundamentals of political science research, we are largely in agreement. On the other hand, this debate may rage just as fiercely in the classroom. In order to answer this question, we asked instructors what they teach.

SURVEY DESIGN AND RESULTS

Though the findings of previous studies are enlightening, there are more pieces to this puzzle that need to be put in place if we are to get a clear picture of the scope and methods curriculum. To that end, we developed a survey for undergraduate scope and methods instructors (see Appendix).Footnote ² The survey asks about topics covered, assignments, and demographic data regarding the course, instructor, and institution. An introductory letter was mailed electronically to department chairs who were asked to identify faculty in their department who taught a scope and methods type course and provide us with e-mail addresses for them. Chairs provided approximately 150 addresses, which we used to distribute the survey instrument. We received 106 unique responses, for a response rate of 70%.Footnote ³

Respondents and their institutions were demographically varied: 60% of respondents taught at public institutions while 40% taught at private institutions. About 60% were tenured and 34% were on the tenure track.Footnote ⁴ The mean number of departmental majors was 263 (with a range from 10 to 1,100) and the average class size for the scope and methods class was 29, with 76% of respondents indicating that the class was recommended for students during their sophomore or junior year.Footnote ⁵

Table 1 indicates the most frequently mentioned titles for the scope and methods course. Though many titles were similar and have been grouped together in the table, of the 106 responses, there were 62 unique course titles. This result suggests that, while there is great variety in exact titles, political scientists have coalesced around certain important words to include in the title of this type of course. Of the 62 unique responses, 54 (87%) included some form of one or more of the following words: research, scope, methods, analysis, inquiry, and empirical. Though titles alone can be misleading, the data seem to suggest that research methods play a bigger role than does an examination of the scope of the discipline.

Table 1.

Most Common Scope and Methods Course Titles*

* Course titles are grouped by most salient words. For example, Research Methods includes courses with that exact title, but also courses with titles like Political Science Research Methods. The category “other” consists of titles mentioned only once.

Beyond course titles, we also asked which books instructors assigned for the class. The most frequently listed required texts are presented in Table 2. The most striking features of this list are the dominance of the first two titles and the diversity of the rest. Two titles were mentioned in 27 responses each. Political Science Research Methods (Janet Buttolph Johnson, H. T. Reynolds, and Jason D. Mycoff) and The Essentials of Political Analysis (Phillip H. Pollock III) each accounted for over 16% of the total. Beyond those two leaders, 18 titles received between two and 11 mentions and 51 of the 168 responses—over 30% of the total—were unique. Though the survey reached a relatively small sample of courses, the results suggest that scope and methods instructors see a wide range of texts as suitable for their course.Footnote ⁶ Indeed, though they could list more than one, 106 respondents produced a list of 71 distinct titles that they require for the scope and methods course.Footnote ⁷ This textual variety suggests that instructors have found a multiplicity of suitable tools for conveying course content.

Table 2.

Most Frequently Assigned Books in Scope and Methods Courses*

* Versions of the same text are grouped in the same line (e.g., second and third edition). The most recent edition/title is listed here. In the cases of Philip Pollock, Earl Babbie, and Michael Corbett, multiple distinct titles are grouped because respondents did not always make clear which text they were using by that author. The total sums to greater than 106 because respondents were allowed to list up to four required texts. The category “other” consists of titles mentioned only once.

Of course, class titles and books assigned are not always indicative of the exact content of a course, so the survey also asked respondents to indicate topics covered and projects assigned by selecting from a list of possibilities. The results are displayed in Table 3 and Table 4. Some topics appear to be covered nearly universally. Measurement, elements of a research design, the logic of scientific reasoning, causality, sampling, and survey research are included in over 90% of the courses sampled. An additional nine topics are covered by at least half of the respondents. Taken together, it seems reasonable to conclude that these 15 topics form a core of what political scientists teach undergraduate majors about the scope of political science, the research methods of the discipline, and its various modes of inquiry and analysis.Footnote ⁸ With that premise, a few observations are in order.

Table 3.

Percent of Respondents Covering Selected Topics in Course

N = 106

Table 4.

Percent of Respondents Requiring Selected Assignments

N = 106

First, very few instructors view this course as an appropriate venue for incorporating political ideology or the history of political thought. Perhaps because these topics often raise normative concerns, they do not tend to be covered in a course that nearly all respondents saw as an appropriate venue for discussing scientific reasoning.

Second, some methodologies clearly enjoy a wider acceptance than others in this course. At least 70% of respondents include survey research, experiments, quasi-experiments, and quantitative analysis. Over half cover interview techniques and case studies. On the other hand, less than half of the respondents mentioned the comparative method, archival research, qualitative analysis, ethnographic research, formal modeling, or program evaluation. It is not clear whether these results indicate the instructors' preferred methodological approaches, or merely what they deem relevant to the undergraduate experience. In any event, it is clear that some methods are taught more than others. Instructors teach quantitative approaches, surveys, and experiments to undergraduates far more often than qualitative or formal approaches.

Finally, the fact that 95% of courses address the elements of a research design and 83% cover the components of a research paper suggests that most instructors expect students to conceive of political science research as part of a written academic project. Table 4 further indicates the form that this written project takes. While just under two-thirds of respondents require a research design/proposal paper and just under half require a complete research project, cross-tabulation of responses for those two questions indicates that 79% require either one or the other of those two assignments. Thus, it seems safe to conclude that there is a fair amount of uniformity regarding the need for students to write their own scientific research, either with or without inclusion of their own data analysis. Moreover, 23% of respondents indicated that the scope and methods course fulfills a department or university writing requirement.

In addition to writing, most courses in the sample require students to work with data. Over three-quarters require the use of statistical software and about one-quarter require individual or collaborative student data collection. Of those two types of data collection, cross-tabulation indicates that 34% of courses require just one, while only 7% require both. In other words, instructors are much more likely to focus on data analysis than data collection, though a significant minority do both.Footnote ⁹