The Internet and other electronic media have changed the way undergraduate students conduct research. The effects of this technological change on the role of the professor are still not well understood. This article reports on the findings of a recent study that evaluated the scholarly content of student citations in a political science course and tested two interventions designed to improve their quality. The study finds that these students' use of electronic sources was not as poor as some may have assumed, and that the quality of bibliographies improved when in-class instruction was combined with academic penalties. This article reflects on the study's findings, and offers suggestions for how instructors might encourage students to improve the quality of their research.
The Internet and the Role of the Professor
The emergence of the Internet as a research tool has changed the nature of the relationship between professor and librarian in supporting student research. Philip Davis, an academic librarian at Cornell, suggests that this should result in a corresponding shift of responsibility:
Since the mid 1990s, the academic library has lost its control as the sole information resource provider on the college campus and now competes with a multiplicity of resources available over the Internet. Because of this loss of monopoly, professors can no longer solely rely on the library to serve as the intermediary between the student and the universe of information. The networked environment shifts much of this responsibility to the professor. (Davis 2003, 49)
Prior to the popularization of the Internet, it could reasonably be assumed that students conducted most of their research in the campus library. This relieved professors of two concerns. First, professors could leave it to librarians and library collection policies to ensure that the materials students accessed were, for the most part, scholarly. Second, if students needed help evaluating their sources, trained librarians would be close at hand to assist them.
Now, when students conduct research using electronic resources, these assumptions are no longer reasonable. The bulk of the information students encounter on the Internet is not scholarly. Students often retrieve this information from home or other locations rather than from within the library where there are professional librarians available to assist and educate them. Thus, in the Internet age when students are most in need of the critical evaluation skills that librarians have to teach them, instructors must bring the librarians to the students.
But, how can instructors ensure that class time devoted to this purpose is used effectively? Anyone who has tried bringing librarians into the classroom knows that most students seem to assume that librarians have nothing to teach them. Instructors can increase librarians' effectiveness by informing them of the nature of class assignments so that they can focus their presentations on the students' immediate needs. But, is there anything more instructors can do? This article considers whether there would be any benefit to attaching grades to students' application of the skills that librarians teach them. We argue that there is such a benefit.
To make this case we report on a study designed to further and expand upon the work of Philip M. Davis and Suzanne A. Cohen at Cornell University (2001). Our study evaluates the effectiveness of two interventions which, it has been suggested, can improve the scholarly quality of student research: 1) in-class instruction by a librarian in evaluating print and Internet sources (“instruction-only”); and 2) the same inclass instruction plus the adoption of academic penalties for failing to meet minimum standards for scholarly research (“instruction-and-penalty”). When we compared the results of these interventions with those of a control group we found that although instruction-only offered only limited benefits, the instruction-and-penalty intervention proved effective.
Previous Research
Although there has not been a great deal of research into the effects of the Internet and other electronic sources on the quality of student research, some of the findings are of interest. The groundbreaking study by Davis and Cohen (2001) offers valuable insight into the impact of the Internet on undergraduate scholarship. They conducted a comparative citation analysis of bibliographies of undergraduate term papers submitted in 1996 and 1999 for an economics class at Cornell University. Citations classified as book or journal were considered scholarly. Citations classified as magazine or newspaper were considered nonscholarly. The remaining citations were classified as web or other. Journal, magazine, and newspaper articles were classified as such if the source document was available in print, even if students had cited it as having been accessed electronically (311). Over this period, Davis and Cohen found statistically significant increases in the average number of newspaper citations (from 0.9 to 1.9 per bibliography) and web citations (from 1.1 to 2.5) and a decline in the average number of books (from 3.5 to 2.2) (312). Further, when these categories were grouped as scholarly (book and journal) and nonscholarly (newspaper and magazine), they found a statistically significant decline in scholarly citations (from 6.1 to 4.6) (312). This pattern was confirmed when Davis repeated the analysis on papers submitted in 2000 (Davis 2002). Thus, they concluded that student use of the web has contributed to a decline in the scholarly content of citations. Further, the second study suggested that verbal directions given by the professor to improve the quality of research “had little (if any) effect on improving the scholarly component of research papers” (Davis 2002, 59).
In an article published after our project had been initiated, Davis reported on a study conducted in 2001. In this study he found that when students were faced with academic penalties for using less than five “published, scientific (peer-reviewed or university press) articles or pre-prints” the “number of scholarly citations (the culmination of books and journals) returned to 1996 levels” (Davis 2003, 43–44, 47). Our study supports this conclusion.
More recently, in an article published in this journal, Richard P. Barberio reported on an important and revealing study that explores how some political science undergraduates use the Internet to complete research assignments (2004). On the basis of student survey responses, he suggested that students are relying too heavily upon the Internet, comparing their approach to the Internet as a research tool to the strategies problem gamblers take to slot machines. Thus, he cautions, “teaching faculty may be erroneously assuming that the Internet is simply a new arrow in the students' research quiver” (307). Having identified this problem, he prescribes a number of strategies to “help students broaden their palate of research methods and sources.” Among these is the proposal that instructors “create assignments that require a specific amount of citations” from specified types of sources, like books and scholarly journals (310). Although we find that students appear to have more arrows in their quiver than Barberio's survey led him to believe, we also find that his suggestion regarding assignments is right on target.
Method
This study involved political science students at the University of Regina in Saskatchewan, Canada. The University of Regina is an anglophone university similar in organization to public universities in the United States. PSCI 230 is a second-year survey course in Canadian politics covering institutions, political culture, and political behavior in which students are required to write a research paper. The only prerequisite is an introductory course in political science, so classes usually contain a mix of students from political science, administration, and other humanities and social science programs.
As the politics courses were completed in one term we were able to establish a control group consisting of two course sections in the fall semester of 2002 and to test different interventions in each of two sections offered in the winter semester of 2003.1
One professor taught one section in each term and the other sections were taught by two different professors. Although it would have been ideal had the same professor taught all four sections, course content was similar across all sections, the sections used the same text, and all students wrote similar research papers.
The control group sections received no intervention (N = 46). We collected and analyzed the fall sections' research paper bibliographies to serve as a baseline for the interventions conducted in the second term. One section in the winter semester, the “instruction-only” group, received instruction from an academic librarian as well as encouragement from the instructor to include at least five scholarly sources in their research (N = 17). Students were instructed on the importance of evaluating print and electronic sources, distinguishing between scholarly and nonscholarly sources, identifying various types of web pages (based on Alexander and Tate's checklists, 2001), and in criteria for evaluating sources. The other winter section, the “instruction-and-penalty” group, received the same instruction from the librarian but was also required to include a minimum number of scholarly sources in their papers, or receive an academic penalty (N = 21).
The guidelines included in the instruction-and-penalty group's assignment read exactly as follows:
The paper must quote or paraphrase, and thus reference, at least four scholarly sources (e.g. books, academic journals)—while you are encouraged to refer to magazines and newspapers and your text, these will not count toward the four. Papers not meeting this requirement will be penalized at a rate of 3% per source that the paper falls below the four-source requirement.
This guideline was intended to balance two desires: to create a penalty that is likely to improve the quality of the sources students cite, and to keep the guideline simple enough to use that those marking the papers will actually apply it. Although including scholarly Internet sources in the requirement would have more closely reflected our objectives, those who mark papers are unlikely to have the time to evaluate every Internet site students reference, and thus, this requirement would likely go unenforced. The penalty, as designed, then, has two key virtues. On the one hand, it is likely to be enforced because it is easy to apply; it only requires a quick review of the footnotes or endnotes to determine if the required number of books or scholarly journals have been referenced. On the other hand, although it does not address electronic sources directly, it has the potential to indirectly encourage students to improve their quality. It may do so in two ways. First, by requiring students to think about the quality of some of their sources, it may lead them to think about the quality of all of their sources. Second, by requiring them to rely more heavily on scholarly sources, it may allow them to rely less on the Internet, and, thus, to be more selective when they do use it.
We developed a method to analyze bibliographies by modifying Davis and Cohen's approach. We supplemented their typology of book, journal, newspaper, magazine, and other with a separate category for government documents. We also adopted their practice of defining journals as “scholarly periodicals that contain primary research or substantial policy analysis” and magazines as “nonscholarly periodicals that report primarily news, industry information, and events” (2001, 311). As we were interested in the scholarly quality of electronic citations (i.e., information accessed on the Internet or from other electronic sources like proprietary databases), unlike Davis and Cohen we classified citations as electronic if students presented them as such in their bibliographies. If there was no indication that the citation was electronic, we assumed it was print. To facilitate analysis we created a checklist for coding electronic sources into one of eight categories. (For the “Electronic Source Classification Checklist” and further discussion of the rationale behind it, please see Robinson and Schlegl 2003.) Three categories were considered scholarly: electronic-scholarly, electronic-journal, and electronic-government document. Four were considered nonscholarly: electronic-news, electronic-magazine, electronic-other, and electronic-low quality (a web page was considered low quality if it was not clear who was responsible for the page or no contact information was provided besides an email address). One final category— inaccessible—was not included in consideration of scholarly and nonscholarly sources.
Results for each of the three groups (control, instruction-only, and instruction-and-penalty) were compared in a number of ways: average number of each category of citation per bibliography; average percentages of scholarly and nonscholarly citations per bibliography for print, electronic, and overall; and average percentage of low quality electronic citations as a percentage of electronic citations. Further, as it was assumed the interventions would improve citation behavior, one-tailed t-tests were applied to determine the significance of the differences between means for the control group and each intervention group. Finally, analysis of variance (ANOVA) was conducted to confirm differences in means between groups.
Results
As we have reported our analysis of the results of this study elsewhere, we simply summarize the relevant observations here and discuss what they suggest about the role of political science instructors.2
For analysis of the control group please see Robinson and Schlegl 2003. For analysis of the intervention groups please see Robinson and Schlegl 2004.
How Are Students Using the Internet?
First, analysis of the bibliographies from the control group, where students received no instruction and faced no penalties, suggests that although these students' reliance on the Internet is significant, they have not abandoned print sources: in other words, there is more than one arrow in their research quiver. This is reflected in some of the statistics: while the bibliographies included an average of 9.1 sources, the average number of electronic sources cited was 2.5; while 27% of all citations referred to electronic sources, all bibliographies included at least one print source, and 10 of 46 bibliographies did not include any electronic sources (Robinson and Schlegl 2003).
Second, even without in-class instruction the Internet and other electronic sources that these students were citing were not the unscholarly disaster that some reports led us to expect.3
For instance, Gillette and Videon reported that of 36 working URLs cited by students in a composition class, 42% pointed to papers written by other students and a further 8% were suspected to be papers written by students (1998, 191). Similarly, in a study of student use of web sources in a community college English composition class, Grimes and Boening reported that “students were asked if it concerned them that at least one source selected by a fellow student was a personal home page of a sixteen-year-old fan. No one expressed any concern, asserting that such a site was suitable for college research assignments” (2001, 20).
On the one hand, our results suggest that concerns that students use the Internet indiscriminately may be overblown, as only nine out of 113 electronic sources cited (8%) were coded as low quality. On the other hand, fully 29% of the electronic sources these students cited were classified as scholarly. Further, many of the nonscholarly sources they cited were not inappropriate for an academic research paper. In fact, they seem to be accessing information that may have been previously unavailable or difficult to access due to individual libraries' collection policies. Of 51 citations classified as electronic-other, 16 were of interest group web sites and 30 of political party web sites, speeches, press releases, and the home pages of elected officials. Although no one would mistake these for scholarly sources, if used appropriately they certainly have a place in an academic research paper. Thus, if we group these together with the previously identified scholarly electronic sources our study found that 70% of the electronic citations were
not inappropriate for academic research. This compares favorably with the high percentage (84%) of scholarly citations we found for print sources (Robinson and Schlegl
2003, 8–10). Similar relationships between type of source and type of media (print or electronic) were found in the experimental groups (Robinson and Schlegl
2004).
Overall, then, the research behavior of these students was not markedly unscholarly. In the control group, 72% of all sources were considered scholarly and, as noted, many of the nonscholarly sources might well have been appropriate. Still, although our data suggest that reliance on the Internet is not “destroying student research papers” (Rothenberg 1998), reliance on electronic sources is significant and the electronic sources these students accessed did tend to be less scholarly than their print sources. This suggests instructors are right to be concerned about the effect of the Internet on student research behavior.
Percentage of Scholarly Citations in Each Type of Intervention
Print Citation Sources as Percentage of All Citations (Robinson and Schlegl 2004)
Electronic Citation Sources as Percentage of All Citations (Robinson and Schlegl 2004)
Electronic Citations as Percentage of All Citations (Robinson and Schlegl 2004)
Instruction-and-Penalty Improves the Quality of Student Research
Although some improvement was noted in both the instruction-only and instruction-and-penalty groups (that is, either a decline in nonscholarly sources or an increase in scholarly sources), only the instruction-and-penalty intervention had statistically significant effects. For example, although the overall percentage of scholarly sources increased with both interventions, from 72% in the control group to 74% with instruction-only and 86% with instruction-and-penalty, only the latter change was statistically significant (p < 0.05).
Even more striking, but in line with our intentions, although the intervention did not address electronic sources directly, it was associated with an improvement in their quality: nonscholarly electronic citations fell from 71% of all electronic citations in the control group to 50% with instruction-only and 35% with instruction-and-penalty (significant at p < 0.05).
Finally, when numbers for electronic citations as a percentage of all citations are compared, a pattern emerges that resembles Davis's finding that students' reliance on electronic sources declined when they were required to include scholarly sources. Davis observed a decline in web citations as a percentage of all citations from 21% and 22% in 1999 and 2000 when no penalties were applied to 13% in 2001 when they were (Davis 2003, 47). In our study, electronic citations constituted 27% and 29% of all citations for the control and instruction-only groups, and 17% for the instruction-and-penalty group. (The difference in magnitude of the numbers is probably due in part to the fact that our “electronic citations” category captures more citations than Davis's “web” category.) Thus we found strong support for the hypothesis that the instruction-and-penalty intervention can improve the quality of both the print and electronic citations in student bibliographies.
Suggestions: How Can We Influence the Quality of Student Research?
Students' increasing reliance on the Internet for research is transforming professors into the primary intermediaries between students and the universe of information. This study suggests some ideas for how instructors might respond to this change.
First, instructors should resist the temptation to prohibit students from using the Internet to research their papers. Although this would certainly help the small minority of students who use Internet sources indiscriminately, it would do nothing to help all students develop the skills they need to engage critically with the Internet. Further, the Internet offers students access to rich sources of valuable information that they are not finding in their libraries. Our study suggests that students are taking advantage of these sources; instructors should not discourage them.
Second, although this study suggests that student use of the Internet is not disastrously indiscriminate, it does suggest that students are capable of using it much more critically. Since the Internet and other electronic sources enable students to access information without encountering libraries or librarians, professors have a duty to bring student and librarian together on the common ground of our classrooms. As the study indicates, this is a good use of class time.
A final observation is that instructors may need to do more than simply share their classrooms. This study suggests that students appear to respond better to librarian instruction when it is backed up with academic penalties, which only professors can apply. Although our study can shed no light on the reasons for this, we can offer some hypotheses. It may be that students are time or effort optimizers who are only willing to devote a fixed amount of time to researching a paper. If they are required to locate scholarly sources, then they will spend less time locating nonscholarly electronic sources. It could also be that students are “mark maximizers” who respond vigorously to external incentives like avoiding deductions. Our experience in conducting this study does not support this, however. Although we had hoped that, given the very minimal requirements, the penalty would not have to be applied, marks were deducted in seven of 29 cases. Finally, a less cynical possibility is that students respond to what professors indicate is important. Although instructors can tell students that scholarly research is important, and can invite librarians into their classrooms to talk to them about it, nothing speaks more clearly to students about professors' expectations than how they allocate grades. If, as we suspect, each of these hypotheses is true of some of our students, then the instruction-and-penalty intervention seems well-designed to address the problem.
Conclusion
If instructors want to take up the new responsibilities that have emerged with widespread student use of the Internet and other electronic technologies, we may have to go beyond sharing our classrooms with librarians and extend to them some of the authority that comes with our control over grading. The good news, as this study has shown, is that we can do this without significantly increasing the amount of time it takes to grade papers.
The authors would like to thank Philip M. Davis for his helpful replies to our many questions; Pam Mckenzie and Lynne McKechnie, whose encouragement was instrumental in bringing this project to fruition; the Department of Political Science at the University of Regina for allowing us to conduct this research; and the participants in panel H10: “Pedagogical Puzzles and Political Thought” at the 2003 Annual Meeting of the Canadian Political Science Association.
%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
