Although training in political science research methods at the undergraduate level has become increasingly common, it still may be insufficient. A survey by Parker (Reference Parker2010) indicated that only 28% of political science programs in the United States required training in research methods. The percentages were even lower in Canada (20%) and Australia (10%), which support a more liberal arts approach to undergraduate studies. In some European countries (e.g., the Netherlands, Finland, and Norway), fewer than half (44%) of political science departments required undergraduate courses in research methods.
Different factors may prevent programs from offering standalone courses in research methods. In programs that offer such courses, research methods training may be superficial or inadequate (Clark Reference Clark2011; Ryan et al. Reference Ryan, Saunders, Rainsford and Thompson2014). For example, in European institutions, research methods training may take the form of short seminars and workshops (Garner, Wagner, and Kawulich Reference Garner, Wagner and Kawulich2009). Even when dedicated research methods courses exist, students may not have adequate hands-on experience with the fundamentals of research design to gain the confidence needed to design independent research projects.
Even when dedicated research methods courses exist, students may not have adequate hands-on experience with the fundamentals of research design to gain the confidence needed to design independent research projects.
It is possible to integrate the teaching of research skills into content-focused courses to address these limitations and provide students with more opportunities for hands-on experience (Adriaensen, Coremans, and Kerremans Reference Adriaensen, Coremans and Kerremans2014; Dickovick Reference Dickovick2009; Morehouse et al. Reference Morehouse, Lovecký, Read and Woodman2017; Welch and Panelli Reference Welch and Panelli2003). This article describes how we integrated the core elements of research design into a third-year elective course on political participation without sacrificing the delivery of content. We provide evidence of teaching effectiveness using data from student surveys that we specifically developed to tap into their self-assessed knowledge of and confidence with the building blocks of research methods. In addition, the evaluation of different components of the research proposals that they submitted as their final assignment demonstrates that many students were able to apply core research design skills. Our findings demonstrate that it is possible for students to make significant improvements in their research methods knowledge and skills without taking a standalone methods course (Arikan and Milosav Reference Arikan and Milosav2023).
THE CONTEXT
Political participation has been offered in our program as a third-year elective course since the 2017–2018 academic year. During the 2021–2022 Spring term, we retained the course content and reading materials but also included more discussions and in-class activities on devising research questions, conceptualizing and measuring key variables, constructing testable hypotheses, and selecting the appropriate methods and data for testing hypotheses. We also provided optional learning materials on reading and writing research articles, conducting literature reviews, and differentiating between qualitative and quantitative research designs. These learning aims emphasized substantive goals relative to both the course content and the research skills (see online appendices I and II).
Of the 44 students enrolled in the course, 26 were visiting students (59%), mostly from other European universities. There were also some visiting students from North American and East Asian institutions. Both visiting and non-visiting students from different study pathways and degree programs (e.g., law and political science, business and economics) were eligible to take the course. The heterogeneity in their degree programs and backgrounds provided an excellent opportunity to evaluate the effectiveness of integrating research methods training.
APPLICATION AND EXAMPLES
As shown in table 1, both the lecturer and the teaching assistant (TA) introduced new research methods concepts to students in every session along with the substantive topic. For example, in weeks 4 and 5, the lectures featured a discussion of explanatory research questions and social science theories as well as their importance. In reviewing the required readings, the lecturer drew attention to the formulation of the research question and the qualities of explanatory questions in empirical research. The lecturer also emphasized how the authors used social science theories to inform their hypotheses. The students worked in pairs to formulate hypotheses based on existing political participation theories. This activity allowed them to develop their research methods skills while discussing the week’s substantive material.
Topics and Research Design Concepts Covered in Lectures and Tutorials
Note: *There are two hours of lectures per week and one-hour fortnightly tutorials led by the TA.
The TA developed discussion topics and learning activities that supported the learning goals of the course. For example, the first tutorial covered reading and writing social science articles. The TA described the main components of research design in the context of a research article and then asked students to work in pairs to identify the key variables, theoretical framework, and results from the assigned articles. During the third tutorial, students discussed correlation versus causation and theory building versus hypothesis testing in a research design exercise. Working in small groups, they first chose a dependent variable of interest for the following question: “What are the effects of online political participation on Y in the context of the ongoing war in Ukraine?” They next chose a theoretical framework and formulated testable hypotheses based on the selected dependent variable of interest. This exercise enabled the students to discuss the substantive topic of the week (i.e., the internet and political participation) while practicing essential research design skills.
The final assignment was a research proposal, which accounted for 60% of the overall grade.Footnote 1 The lecturer provided detailed information about the assignment during the lectures and posted the guidelines and the grading criteria to the course’s virtual learning platform (see online appendix III). Starting with week 6, the lecturer also encouraged students to ask questions related to the final assignment and gave them the opportunity to discuss their ideas and research plans with their peers during class sessions. The students also had the opportunity to submit their research question or proposal outline to the lecturer for feedback until the end of the teaching term (i.e., week 12). In preparation for the final assignment, the TA offered additional feedback and answered their questions in the last tutorial.
EVIDENCE OF TEACHING EFFECTIVENESS FROM STUDENT SURVEYS
In the second week of the course, we invited students to complete a survey about their background in research methods and their self-assessed knowledge of and confidence with research-related skills (Survey 1). We also posted an announcement and the survey link on the virtual learning platform for absent students to complete later. All students responded to the first survey (100% response rate). Overall, about 42.0% of the students in the course indicated not having prior training in research methods, 53.5% indicated that they had taken a research methods course in the past, and 4.5% were unsure.Footnote 2 Furthermore, 35% indicated that they had taken a qualitative methods course and 53.5% had taken a quantitative methods course (including statistics and data analysis). In addition, 18.5% of students were registered for a research methods course during the same term that they were taking our course.
We asked the students to take a second survey to answer the same questions toward the end of the term (Survey 2). Of the 44 enrolled students, 28 responded to this survey (64% response rate). Similar to previous studies, the response rate was reflective of declining attendance toward the end of the course but still could be considered acceptable (Clark Reference Clark2011). Participation in Survey 2 was lower among students with prior methods training: 42% who took Survey 2 indicated that they had previously taken a research methods course as opposed to 54% in Survey 1. Moreover, only 27% of the students who took Survey 2 stated that they had taken a quantitative methods course as opposed to 54% in Survey 1.
We calculated the difference in the students’ self-assessed knowledge of and confidence with research-related skills at the beginning and the end of the term to provide evidence of the effectiveness of our approach (table 2).Footnote 3 The findings concerning all students (see the columns under “All Students” in table 2) indicated an increase in their knowledge and confidence regarding all items in Survey 2 compared to Survey 1. These differences were statistically significant for several items, including knowledge of and confidence with reading research articles that present quantitative findings; formulating research questions and hypotheses; knowledge of identifying appropriate methods for a particular research question; and confidence in discussing the strengths and weaknesses of different methods.
Students’ Self-Assessed Knowledge of and Confidence in Research Skills at the Beginning and the End of the Term
Notes: *p<0.05, **p<0.01, ***p<0.001.
Next, we examined these results in greater detail by separately reviewing progress among students with and without prior research methods training (see the righthand columns in table 2). As expected, those with prior training indicated higher levels of knowledge and confidence in both surveys compared to students without prior training. For instance, the mean level of knowledge for formulating research questions was 3.30 in Survey 1 for students with prior research methods training compared to 2.33 for those without prior training. Among students with a prior research methods background, the mean scores for all knowledge and confidence items were higher in Survey 2 than in Survey 1, which indicates that they felt that they improved their research skills. The lower response rate among this group in Survey 2 (11 of 23; 48% response rate) means that the degrees of freedom are substantially lower for significance tests. Nevertheless, the differences for confidence in formulating research questions and hypotheses, as well as discussing the strengths and weaknesses of different methods, were statistically significant. Thus, among students with prior research methods training, gains were not necessarily in knowledge but rather in confidence regarding skills, which suggests that our approach can improve students’ research methods skills even if they have prior training. However, we must exercise caution in interpreting these findings because the students who participated in Survey 2 may not be representative of all students who had research methods training before enrolling in our course.
The response rate for students without any prior research methods training in Survey 2 was highly acceptable: 14 of 18 students without any research methods background responded to this survey (78% response rate). Moreover, the mean scores for self-assessed knowledge and confidence were higher for all items in Survey 2 than in Survey 1 for this group. More importantly, despite the lower degrees of freedom, we observed statistically significant progress in both self-assessed knowledge and confidence in this group between the two surveys. The mean scores were statistically significant for both knowledge of and confidence with reading research articles and formulating research questions and hypotheses. This group also indicated significantly higher levels of knowledge of identifying appropriate methods for a particular research question and confidence with evaluating the evidence presented in research papers in Survey 2 than in Survey 1. However, this group did not indicate that they had gained significant knowledge or confidence in discussing the strengths and weaknesses of various methods.
These findings indicate that students without any research methods training felt that they gained more from our approach: both knowledge of and confidence in research-related skills increased between the surveys for students without prior training. For both groups, there were no differences in confidence with academic writing and independent study skills between Survey 2 and Survey 1. Thus, students made progress on topics or themes that were discussed explicitly in lectures and tutorials (e.g., choosing appropriate methods). This suggests that the progress observed between the two surveys is not simply attributable to an additional semester of studying.
These findings indicate that students without any research methods training felt that they gained more from our approach: both knowledge of and confidence in research-related skills increased between the surveys for students without prior training.
EVIDENCE OF TEACHING EFFECTIVENESS FROM RESEARCH PROPOSALS
Although the findings from the two student surveys are encouraging, we should approach them with caution. For example, if only hardworking students responded to the survey because they were still taking the course, we might have captured a difference in diligence rather than the effectiveness of the approach. To provide additional supporting evidence on teaching effectiveness, we present data from the students’ research proposals, which was the final assignment of the course. We asked a research assistant (RA) who was unaware of the purpose of the study to code the quality of the main components of the research proposal (for the coding scheme, see online appendix IV).Footnote 4 The proposal quality scores are presented in table 3.
The Quality of Key Elements of Research Design in Students’ Research Proposals
The scores indicate that despite the high levels of heterogeneity in students’ backgrounds, almost all of them demonstrated the core research design skills in their proposal: 97% presented an explanatory research question, and all students provided adequate discussion of the background and the motivation for their research question. On a scale from 0 to 3, students demonstrated satisfactory skills in conducting literature reviews (mean=2.19), developing empirically testable and theory-based hypotheses (mean=2.37), and suggesting appropriate methods and a data-selection strategy (mean=2.04).
The findings from the quality assessment of proposals also align with the results of our surveys. For example, further review of the proposal quality scores indicated that 88% of students included empirically testable hypotheses in their proposal. In fact, in Survey 2, the mean for self-assessed knowledge of formulating research questions was the highest (3.76) along with knowledge of reading research articles, followed by knowledge of formulating research questions (3.72).
The scores presented in table 3 also suggest that discussing the strengths and weaknesses of the research design was the most challenging for students: 74% appropriately discussed the strengths and limitations of their research design. This challenge also was reflected in our surveys. Students’ self-assessed knowledge of discussing the strengths and weaknesses of different methods had the lowest score in Survey 2 (However, students with a prior research methods background indicated more confidence in this skill in Survey 2.) The consistency between the students’ self-evaluation of their knowledge in the surveys and our observations from their proposals provides further confidence that they gained certain research skills as a result of the research design elements integrated into the course. Although we did not explicitly measure this, we had the impression that most proposals engaged in a highly informed discussion of major approaches and theories on the substantive topic. This increases our confidence that learning goals relating to research skills were achieved without detracting from those relating to substantive content.
The consistency between the students’ self-evaluation of their knowledge in the surveys and our observations from their proposals provides further confidence that they gained certain research skills as a result of the research design elements integrated into the course.
CONCLUSION
Developing transferrable skills that will help graduates in their future career is a strategic goal emphasized in many political science curricula (Clark Reference Clark2011; Engbers Reference Engbers2016). Increasing demand for strong analytical skills in the private and public sectors means that graduates of political science programs need skills and confidence in evaluating research quality, formulating research questions, and selecting appropriate methods (Adriaensen, Coremans, and Kerremans Reference Adriaensen, Coremans and Kerremans2014; Engbers Reference Engbers2016; Morehouse et al. Reference Morehouse, Lovecký, Read and Woodman2017). Developing such skills and confidence requires continuous and active student engagement with research (Welch and Panelli Reference Welch and Panelli2003). Our study demonstrates that research methods training can be integrated throughout the curriculum (Knoll Reference Knoll2016; Ryan et al. Reference Ryan, Saunders, Rainsford and Thompson2014) and that students can make significant gains in their research skills even in the absence of dedicated research methods courses. Our findings indicate that students without any research methods background have the most to gain from such an approach in acquiring both knowledge of and confidence in applying research-related skills. We also found that by developing more confidence in their skills, students can make progress in research-related skills even if they already have prior training.
A potential drawback of this approach is that students might find the emphasis on research methods challenging. Some comments in the student evaluations mentioned the challenging nature of adding research design elements to the learning goals.Footnote 5 Nevertheless, this should not discourage instructors from integrating research design elements into content-focused courses. Although students often find research methods and statistics courses challenging, our findings indicate that they are still capable of learning despite the challenges.
ACKNOWLEDGMENTS
The authors thank Ozlem Celik, the editor, and two anonymous reviewers for their valuable feedback and constructive comments, and Clara Faulí Molas for assistance. Any errors or omissions are the sole responsibility of the authors.
DATA AVAILABILITY STATEMENT
Research documentation and data that support the findings of this study are openly available at the PS: Political Science & Politics Harvard Dataverse at https://doi.org/10.7910/DVN/MXJP47.
Supplementary Material
To view supplementary material for this article, please visit http://doi.org/10.1017/S1049096523000483.
CONFLICTS OF INTEREST
The authors declare that there are no ethical issues or conflicts of interest in this research.
Open access
