Influence in the Discipline

Political scientists are evidently concerned with influence in the discipline. For instance, the November 2006 edition of the American Political Science Review (APSR) was solely devoted to the evolution of political science and listed the 20 most-cited articles in the history of the journal. While these 20 articles are undoubtedly very influential, a list of raw counts still does not tell us much about the exchange of knowledge throughout the discipline. And, certainly, a list of raw counts such as the one published in the APSR does not tell us which journal had the most influence on the discipline in 2004 or 2005.²

Citation counts are used in a variety of ways to evaluate the quality of journals. Christenson and Sigelman (1985) utilize citation counts in an early study of journal quality. Additionally, Journal Citation Reports, which is available from the ISI Web of Knowledge, calculates a variety of measures, such as the “impact factor” and “immediacy index,” as well as reporting the raw number of citations for each year. Both the impact factor as well as the immediacy index measure how frequently a journal's “average” article is cited, with the former using “recent” citations (i.e., from the previous two years) and the latter using only citations from the current year. The common problem with measures based upon raw citation counts like this is that a citation is treated as a standard unit regardless of the parties involved. Hence, in terms of impact, there is no difference between Journal A's citation by highly esteemed and well-read Journal B, and A's citation by third-rate Journal C. Yet most political scientists would consider B's citation of A more prestigious. For similar reasons, problems may be caused by journals that self-cite a great deal.

Most work by political scientists that evaluates the relative quality of journals relies on rankings provided by political scientists themselves through surveys. Studies by Giles and Wright (1975) and Giles, Mizell, and Patterson (1989) are examples. Noting that field journals with a narrow readership obtained “surprisingly” high rankings in scholars' subjective evaluations, Garand (1990) pioneered an approach, later used by Crewe and Norris (1991) and Garand and Giles (2003), that includes an additional measure to the subjective quality ranking to account for the discipline-wide familiarity of the journal.³

To repeat, this method is nonetheless subjective. Moreover, surveys are relatively expensive and typically time consuming to dissipate, fill, and assess.

Thus the potential utility of a simple, cheap, systematic method that estimates the contribution of journals to the exchange of knowledge in the discipline. Such a method would compliment current practices, and allow discipline-wide comparison of whether, for example, political scientists' subjective analysis corresponds closely to the actual exchange of ideas among the top journals in the discipline.

Data

We utilize data published by Journal Citation Reports and available online from ISI Web of Knowledge from 2003–2005. The data include total citation counts for 84 political science journals in the given year, as well as a variety of measures based upon these counts. Key to our enterprise, the data record multidirectional citations; that is, they count all citations from and all citations to a journal in each given year.⁴

To keep the below analysis meaningful and tractable, we limit our analysis to the pattern of intellectual exchange among 16 top journals from 2003–2005. We chose journals that scholars believe to be among the field's top journals, as reported by Garand and Giles (2003).⁵

This list provides a nice baseline for comparison while also giving us a list of prestigious journals that are comparable forums for intellectual exchange.⁶ One of the more interesting findings of the recent survey of political scientists by Garand and Giles (2003) is that three of the top nine general journals (American Economic Review, American Sociological Review, and American Journal of Sociology) are not even political science journals (298–9).⁷ They posit that these three journals fare so well because political scientists recognize them as “the flagships of their respective disciplines,” not necessarily because they are familiar or contribute in a meaningful way to intellectual exchange in the discipline. In the following section, we assess this claim as well as some widely held beliefs about journal quality with actual data on intellectual exchange.

Findings

In Table 2 we report the influence scores for each of the journals in our study for 2005. We include the ranking for each individual year in the sample as well as the overall ranking across the three years. Notice that, because the power to influence is relative to other journals, we assign one of the publications as the baseline or reference score of zero. We choose PS: Political Science and Politics (PS) for this purpose in all models. Additionally, in the fifth column we provide the rankings reported by Garand and Giles (2003) for comparison. From the overall results (including all years) in Table 2 we conclude that the American Economic Review (AER) is the most influential journal (of the 16 in the sample), followed by the APSR and World Politics (WP).

Examination of individual year results demonstrates that this pattern is generally stable, although the exact rankings fluctuate slightly. The biggest surprise from the aggregate (and individual year) results is the poor performance of the American Journal of Political Science (AJPS) and JOP relative to the rankings provided by Garand and Giles (2003). Although these two journals are regarded as two of the top three by scholars in the discipline, the results in Table 2 indicate that journals such as WP and International Organization (IO) are more influential. The influence score of Public Opinion Quarterly (POQ) also consistently ranks higher than in the survey performed by Garand and Giles (2003), while the British Journal of Political Science (BJPS) does considerably worse. Political Science Quarterly (PSQ) is also surprisingly influential, although its performance is inconsistent across the three years in the sample.

As noted above, the scores in Table 2 have a direct interpretation outside of the rank order. Consider, for example, the probability that any particular citation involving the APSR and Political Research Quarterly (PRQ) has the APSR influencing PRQ. This is:

which is a near certainty. By contrast, the BJPS and Comparative Political Studies (CPS) are much more evenly matched in terms of influence. Indeed the probability that an intellectual exchange between them has the BJPS influencing the CPS, rather than the other way round, is:

which is close to a coin toss.

An objection here might be the inclusion of journals that are usually considered part of economics or sociology and hence outside of the discipline per se.⁸

In response, we report our model finding just for political science journals in Table 3—it is readily apparent that the rank order of Table 2 is unperturbed by reducing the sample of journals.

A potentially more interesting issue concerns the effect of a journal's origins and general audience in terms of its influence. That is, we could suppose that a journal's influence in political science is in part determined by the discipline it serves. In Table 4 we re-estimate our model making the scores a (linear) function of the discipline that a journal predominately operates within. Hence, we denote the AER as an economics journal, while the American Journal of Sociology (AJS) and American Sociological Review (ASR) are from sociology.

The interpretation of coefficients in Table 4 is not unlike that for a (multinomial) logistic regression, relative to a base category of political science. In particular, economics journals have an estimated advantage in influence of 1.33 over political science journals, while the influence of sociology journals in political science is much smaller (0.21). Both coefficients are statistically significant. This suggests that, for our sample years, for our sample journals, economic titles are much more influential than sociology journals in the discipline of political science, and, in fact, more influential than political science journals broadly construed. This finding is somewhat of a contrast to the argument put forth by Garand and Giles (2003).