Hostname: page-component-7b7567c8c-kwt9d Total loading time: 0 Render date: 2026-11-12T07:14:56.853Z Has data issue: false hasContentIssue false
  • English
  • Français

Ranking Political Science Journals: Reputational andCitational Approaches

Published online by Cambridge University Press:  03 October 2007

Micheal W. Giles and
James C. Garand
Micheal W. Giles
Affiliation:
Emory University
James C. Garand
Affiliation:
Louisiana State University
Rights & Permissions [Opens in a new window]

Extract

Academic journals play a key role in the dissemination of scholarlyknowledge in the social sciences. Hence, publication in journals iscritical evidence of scholarly performance for both individuals andthe departments that they populate. While in the best of worlds eachscholar's performance would be evaluated based on a close reading ofhis/her published journal articles, in the actual practices ofhiring, tenure and promotion review, and departmental evaluationsthis ideal is often honored only in the breach. Instead, evaluatorscommonly base their judgments of the importance and quality ofpublished articles, at least in part, on the journals in which theyappear. The higher the status accorded a journal, the greater theweight attached to publications appearing in it.

Information

Type
THE PROFESSION
Information
PS: Political Science & Politics , Volume 40 , Issue 4 , October 2007, pp. 741 - 751
Copyright
© 2007 The American Political Science Association

Academic journals play a key role in the dissemination of scholarly knowledge in the social sciences. Hence, publication in journals is critical evidence of scholarly performance for both individuals and the departments that they populate. While in the best of worlds each scholar's performance would be evaluated based on a close reading of his/her published journal articles, in the actual practices of hiring, tenure and promotion review, and departmental evaluations this ideal is often honored only in the breach. Instead, evaluators commonly base their judgments of the importance and quality of published articles, at least in part, on the journals in which they appear. The higher the status accorded a journal, the greater the weight attached to publications appearing in it.1

This practice may be justified in terms of the burden arising from assessing a large number of scholars (e.g., a large number of applicants for a position or the large number of faculty involved in a department review) and/or lack of familiarity with scholarship in some of the subfields of a discipline.

Because of the important role that journal status plays in the evaluation of both scholars and the institutions in which they work, social scientists (Giles, Mizell, and Patterson 1989; Garand 1990; Crewe and Norris 1991; Garand and Giles 2003; Hix 2004) have focused considerable attention on measuring the relative status of journals. These studies have typically adopted one of two approaches, citational or reputational. Recent studies adopting the citational approach typically have relied on the Institute for Scientific Information (ISI) Journal Citation Reports and have adopted as a measure of journal status some variant of the ISI Impact score. This score measures the citations a journal receives on average for each article published during a set time-period. Journals with higher impact scores are viewed ipso facto as more influential in scholarly discourse than journals with lower impact scores.

The reputational approach to assessing journals relies on expert opinion. Typically, a sample of members of the appropriate discipline is drawn and asked to rate a list of journals in “terms of the general quality of the articles it publishes” or some similar standard. The respondents are asked only to rate journals with which they are familiar. The status of a journal using the reputational approach, thus, rests on the average professional assessments of those familiar with their contents.

Over the past two decades the citational approach has become the dominant method not only in the rating of journals (c.f. Liebowitz and Palmer 1984; Diamond 1989; Burton and Phimister 1995; Vokurka 1996; Baumgartner and Pieters 2003; Donohue and Fox 2000; DuBois and Reeb 2000; van Dalen and Henkens 2001; Hix 2004; Hantula 2005) but also of scholars (Garfield 1986; Nederhof and van Raan 1993; Bodenhorn and Walstad 2003; Moed 2005) and institutions (Stahl, Leap, and Wei 1988; Kalaitzidakis, Stengos, and Mamuneas 2003; Moed 2005). This reflects in part the increasing availability of citation data through the Web of Science2

and the lower costs compared to the surveys required in the reputational approach. The citational approach is also touted as more “objective” and reliable than the inherently subjective perceptions of experts. The citational approach, however, is not without its pitfalls (Garfield 1996; Adam 2002; Moed 2005; Leimu and Koricheva 2005).3

The citation logroll where scholars enter a pact to cite each others work and drive up their “ratings” may be more an academic legend than a real problem with the citational approach. Moreover, even if it does exist, as long as the participants are spread across journals and departments it should not bias the relative rankings of either.

The creators of the Impact score counsel caution in its use for evaluation purposes, noting that citational practices may vary across disciplines, sub-fields within disciplines, and journals (see also Garfield 1996). Moreover, while the citational approach assumes a positive valence to citation, Hendry (2006, 2) notes that “if the Journal of Economic and Social Tests (JEST) published an execrable paper that attracted a million critical citations as an example of appalling practice, all other papers previously and later published in that journal would suddenly be much more highly ranked.” Unlike publicity, not all citations are a good thing. As discussed below, the demarcation or boundary problem can also be severe for citation studies that examine subsets of journals (Davis 1998).

While citation-based studies are most common in the other social sciences, particularly economics (Liebowitz and Palmer 1984; Diamond 1989; Burton and Phimister 1995), within political science the dominate approach to journal assessment has been reputational (Garand and Giles 2003; Giles, Mizell, and Patterson 1989; Giles and Wright 1975; but see Hix 2004). Given the important role that journal rankings play in the evaluation of individuals and programs and the dominance of the citational approach within other social sciences, a careful assessment of the application of that approach to ranking political science journals is both relevant and timely. The present study does just that. First, we conduct a citation-based assessment of an extensive set of disciplinary journals. Second, we provide a comparison of the results of the citational approach with those of the reputational-based study conducted by Garand and Giles (2003).

Ranking Political Science Journals by Citations

The demarcation of political science journals is a necessary but problematic requirement for both the reputational and citational approaches to ranking journals. No natural boundary neatly separates political science from non-political science journals. However, as a practical matter, studies that rank the journals of political science or those of any discipline must make such a distinction.4

While in defining political science we can resort to the non-exclusionary position that it is whatever political scientists do, this is practically unhelpful in assessing journals. Political scientists may publish in a variety of outlets, including popular media such as the New York Times. Defining any such outlet as a political science journal we think would provide a troubling demarcation criterion and for our purposes would be problematic because of the difficulty of defining the scope of the possible journals to include in any assessment.

One approach to this problem for a citation study is simply to adopt the categories defined by ISI. The ISI includes 79 journals within the category of “political science,” 54 within the category of “international relations,” and 26 within the category of “public administration.” The basis upon which ISI makes these categorizations is not well defined, and as will become clear later in this article some journals in which political scientists publish and consider to be within “political science” fall outside of these categories (see Garand and Giles 2003; Hix 2004).

In the present study, we take advantage of Garand and Giles' reputational survey (2003) to define the population of journals to be included. In selecting the journals to include in their survey, Garand and Giles (2003) began with the list of journals included in earlier surveys (Giles and Wright 1975; Giles, Mizell, and Patterson 1989) and then submitted the proposed list to colleagues in various sub-fields for suggested additions. The result was a list of 115 journals that were included on the survey. Respondents to the survey were also given the opportunity to add journals to the list.5

Respondents to the Garand and Giles survey were drawn from the membership of the American Political Science Association.

Given its process of development and the open-ended provision for respondents to add journals, we feel the list provides a reasonable approximation of the journals that political scientists view as relevant to their research and teaching.6

We do not attempt nor do we think it profitable to demark the boundaries of political science as a discipline. The questions of interests to those who think of themselves as political scientists and who find themselves located in departments of political science often overlap with others who identify with other disciplines (e.g., sociology, law, psychology, etc.) and who are housed administratively in different departments, programs, or even colleges.

In the citation analysis that follows, we delete four journals that, while included in the survey list, clearly are affiliated with other disciplines: American Sociological Review, American Journal of Sociology, Social Forces, and American Economic Review. We also exclude 21 journals for which ISI did not collect citation data.7

The 21 journals include: American Review of Politics, Business and Politics, China Studies, Economics and Politics, European Union Politics, History of Political Thought, International Studies Review, Journal of Developing Areas, Journal of International Affairs, Journal of Legislative Studies, Journal of Policy History, Jurimetrics, Political Analysis, Politics and Policy, Presidential Studies Quarterly, Public Policy, Review of Politics, Slavic Review and Simulation and Gaming. Behavior Science (now Systems Research and Behavioral Science) and Journal of Inter-American Studies and World Affairs (now Latin American Politics and Society) were excluded because confusion over the name changes resulted in a failure to recognize that ISI actually had collected citation data.

A majority of these journals are familiar to less than 10% of the respondents in Garand and Giles' 2001 survey. However, there are notable exceptions, such as Political Analysis, the official journal of the Methodology Section of APSA, familiar to 16% of the respondents despite its status as a relatively new publication at the time of the survey,8

Political Analysis was included in the 2005 Citation Reports but not in those of previous years.

and Presidential Studies Quarterly, a journal familiar to 35% of the respondents to the survey. The limitations of the ISI citation data, thus, reduce somewhat the validity of the boundary claims of our study.9

The ISI database included citation information on 83% (92/111) of the political science journals included in the survey.

With these deletions the analysis focuses on 90 journals recognized by political scientists as disciplinary and for which citation data are available. We refer to these as “political science” journals for efficiency in the discussion that follows. Readers should remember the caveats discussed earlier in the paper when we employ that label.

Citation data were collected for the 90 journals from the ISI Journal Citation Reports for 2003 and 2004. These were the two most recent years for which data were available at the time collection commenced and were close enough to the period in which Garand and Giles (2003) conducted their survey, 2001, to minimize issues of comparability. These data consist of citations by articles published in 2003 and 2004 to articles published in the study journals. For example, the 2004 Journal Citation Reports record citations from articles published in that year in the Journal of Politics (JOP), American Journal of Political Science (AJPS), World Politics, etc. to articles published in previous years in the American Political Science Review (APSR). In the parlance of citation research, these journals send and APSR receives these citations.

The ISI Impact score provided in the Journal Citation Reports is based only on citations to articles published in the preceding two-year period. For example, the 2004 ISI Impact score for the APSR is equal to the number of citations in journal articles published in 2004 to articles published in 2002 and 2003 in the APSR divided by the number of articles published by APSR in that two-year period. Many of the journals included in the study receive relatively few citations in a given year, even when citations from all ISI journals are considered. The number is further reduced when only citations from “political science” journals are included in the analysis. To provide a robust foundation for our analysis we have collected citation data across two years, 2003 and 2004, and have included citations to articles published in the preceding five years. We calculate robust ISI Impact scores by dividing the total number of citations a journal receives by the number of articles published during the combined five-year periods.10

For 2004 only citations in that year that cite articles published in 2003, 2002, 2001, 2000, and 1999 are counted. For 2003, only citations to articles published in 2002, 2001, 2000, 1999, and 1998 are counted. The denominator includes the number of articles published by the cited journal in the six-year period from 1998 to 2003.

Journal Impact using Citations from All ISI Journals

The 90 journals included in the study are listed in Table 1 in the order of their Robust ISI Impact score (RISI) and for convenience in Table 2 by their rank order using various impact scores. The RISI Impact score is the average number of citations an article published in the journal during the 1998 to 2003 time-period received in 2003 and 2004 from all journals reported by ISI. The results using the RISI Impact score are both enlightening and in some ways surprising. The top-ranked journal, International Organization (IO), has an impact score of 6.24. This indicates that each article published in IO in 1998–2003 on average received 6.24 citations in 2003–2004 from journals in the ISI dataset. The median-ranked journal, Studies in Comparative and International Development (SCID), has an impact score of 1.23, indicating that each article it published in the study period received roughly one citation in 2003–2004. For the lowest-ranked journal, Middle Eastern Studies (MES), a published article received on average about one-fifth of a citation or, stated differently, only two in every 10 articles published in MES between 1998 and 2003 were cited by articles published in 2003 and 2004.

Various Impact Scores by Political Science Journal

Journals Ranked by Various Impact Measures

The wide variation in the impact of journals can also be seen in column 2 of Table 1, which presents the value of each journal's RISI Impact score relative to that of IO, the highest-ranking journal.11

Relative RISI Impact = RISI Impact for a journal/RISI for International Organization (6.24).

The median journal, SCID, has only 20% of the impact of IO, and MES only 4%. Stated differently, with all else equal, a scholar would have to publish five articles in SCID or 25 in MES to equal the same citation impact achieved through one publication in IO.

Included among the 30 journals with the highest RISI Impact scores are most of the well-recognized journals of the discipline. Among general journals included in this group are APSR, AJPS, JOP, and British Journal of Political Science (BJPS). Most of the prominent journals of International Relations—International Organization (IO), Journal of Conflict Resolution (JCR), Journal of Peace Research (JPR), and International Studies Quarterly (ISQ)—and comparative politics—World Politics, Comparative Political Studies (CPS), and Comparative Politics (CP)—are also among the top 30.

We think some of the journals included among the top 30 will also surprise political scientists. For example, Political Geography, a journal only familiar to 7% of the respondents to the Garand and Giles (2003, Table 1) survey, is ranked 12th in RISI Impact. American Journal of International Law is ranked 11th and is only familiar to 11% of the respondents. Indeed, of the top 30 journals, 12 (40%) were familiar to fewer than 15% of the survey respondents.

The familiarity of a journal to political scientists says nothing about the quality of the articles it publishes (Garand 1990; Garand and Giles 2003) and is computationally unrelated to the RISI Impact score. Political scientists' low familiarity with many high-impact journals does, however, point back to the demarcation problem. The RISI Impact score reflects the total citations received by a journal from all other journals in the ISI dataset, regardless of substantive classification. The ISI Social Science dataset fully indexes over 1,700 journals across a range of disciplines. While political scientists have identified our 90 journals as publishing work in the discipline, the journals also attract citations from journals in other disciplines. Indeed, the journals vary widely in the proportion of total citations they received during the study period from our list of political science journals. These proportions are presented in column 3 of Table 1.

At the low end, only 5% of the citations received by American Behavioral Scientist in 2003–2004 to articles it published in 1998–2003 were sent from political science journals included in our list. At the high end, over 80% of the citations received by International Interactions (II) and American Politics Research (APR) during the study period were from journals included in our list. Thus, the journals we have demarked as “political science” vary considerably in their centrality to the discursive domain formed by the journals. While survey respondents see all of these journals publishing articles relevant to political science, in terms of the citations they receive many of the journals are more engaged with fields outside of the discipline.

Journal Impact using Citations from Political Science Journals

While the RISI Impact score provides important information about a journal's global impact, restricting the analysis to citations arising from political science journals provides insight into a journal's influence on intellectual discourse within the discipline. The resulting PS Impact scores appear in column 4 and the impact relative to IO, again, the leading journal, in column 5 of Table 1. Since the PS citations are a subset of the RISI citations, the PS impact scores are uniformly lower than the RISI scores. However, there is a pattern of relative stability in the ordering of the PS and RISI Impact scores. The correlation between the two sets of scores is .78.

There are, however, substantial differences in the PS and RISI Impact scores for some of the journals. Eleven of the journals included among the top 30 by the RISI Impact score are not included in that group by the PS Impact score. Those dropped from the top 30 are in italics (RISI) and those added are in bold (PS) in Table 2. Within the top 10, four journals—Journal of Political Economy (JPE), Journal of Law and Economics (JLE), Journal of Law, Economics and Organization (JLEO), and Public Opinion Quarterly (POQ)—were replaced by the Journal of Conflict Resolution, International Studies Quarterly, Journal of Politics, and the Journal of Peace Research. JPE, JLE, and JLEO fell from their positions within the top 10 when measured using RISI Impact to completely out of the top 30 when measured by PS Impact.

The consequence of focusing only on citations from political science journals is clear. Journals with substantive foci that cross disciplinary boundaries generally have their impact diminished when only citations from political science journals are considered. None of the journals that dropped from the top 30 when the PS Impact score was used received more than roughly one-third of its total citations from political science journals. Of those journals added to the top 30, all received a minimum of 45% of their citations from political science journals. These results underline the criticisms noted earlier regarding the sensitivity of citation analysis to the boundaries defined for a study and underscore the need for citation impact scores to be interpreted with caution.

Another approach to assessing the effect of restricting the analysis to citations from political science journals is to compare the relative impact of journals using the RISI (Table 1, column 2) and the PS (Table 1, column 5) impact scores. In both instances, IO has the highest impact score and each journal's relative impact is calculated as a percentage of that of IO. The PS Impact scores suggest a greater dominance by IO and greater differentiation among the journals than seen using the RISI Impact scores. The 30th ranked journal using the RISI Impact score, Post Soviet Affairs, has a relative impact of .29. In contrast, the 30th ranked journal using the PS Impact score, AJIL, has a relative impact of only .18. In the former case approximately three articles in the 30th ranked journal are required to match the impact of one article in IO, while in the latter more than five articles are required. Similarly, Women and Politics, the median-ranked journal using the PS Impact score, has a relative impact of .12, which means that it would take approximately eight articles published in Women and Politics to equal the PS citation impact of one article published in IO. As noted earlier, it would require approximately five publications in SCID, the median-ranked journal using the RISI Impact score, to equal the impact of IO. Not all journals decline relative to IO using the PS Impact score. Notably, the relative PS Impacts of AJPS and International Interactions (II) are at least 10 percentage points higher than their relative RISI impacts.

Journal Impact using Weighted Citations from Other Political Science Journals

While restricting the citations used to evaluate journal impact to political science journals provides additional insight into the status of journals within the discipline, it does not take into consideration two issues: journal self-citations and the relative status of the journals sending the citations. If citations are an indication of the impact of a journal on the field, then self-citations should be omitted in computing a journal's impact. The importance of a journal should turn on how it is referenced by other journals, not by how often its articles in a given year (e.g., 2004) reference articles in the same journal in earlier years (e.g., 1999–2003). In the following analysis, we omit journal self-citations. One journal, Social Science History, drops from the analysis because with self-citations omitted it has no citations from other political science journals in 2003–2004 to articles published in the study period.

The problem of the relative status of the sending journal is somewhat more complex. Both the RISI and the PS Impact scores treat each citation received by a journal as equal, regardless of its source. As several authors have noted (Kalaitzidakis, Stengos, and Mamuneas 2003; Laband and Piette 1994; Liebowitz and Palmer 1984), this is problematic. If journals vary in their status, then citations received from higher-status journals should be given greater weight in assessing the impact of the receiving journals than citations received from lower-status journals. For example, if JOP receives five citations to its articles from articles published in APSR, this indicates greater influence or status for JOP than if those citations were sent by articles published in Polity.

One approach to resolving this issue is to weight the citations received by a journal (e.g., JOP) by the impact score of the sending journal (e.g., APSR or Polity). The complication to this approach is that the impact scores of the sending journals also must be adjusted to reflect the status of the sending journals. Liebowitz and Palmer (1984) propose a resolution to this dilemma, and we adopt it here. Their approach is to begin by weighting the citations by the impact scores of the sending journals, in our case the PS Impact scores. Thus, the start value for Ijt in the formula below is simply the PS Impact score. Once the impact scores for all the journals in the set are adjusted by weighting the citations, these new adjusted scores are substituted (i.e., Iit+1 after the first iteration becomes Ijt in the following iteration) and the process is repeated.

where

i = journal being cited (receiving journal)

j = journal citing articles in journal i (sending journal)

n = number of journals

t = iteration

Ijt = impact score for the jth journal at iteration t

Cij = number of citations from journal j in 2003–2004 to articles published in journal i in 1998–2003

Ai = number of articles published in journal i in 1998–2003

Iit+1 = the impact score for journal i after one iteration.

This iterative process continues until the adjusted impact scores stabilize.

The results of this iterative weighting process with self-citations omitted from the analysis are presented in column 6 of Table 1 and column 3 of Table 2. The weighted PS Impact scores (WPS) stabilized after four iterations.12

We correlated the weighted scores and journal rankings produced by each iteration. The values from the fourth iteration were correlated at .999 with those of the following six iterations estimated.

Since the weighted PS Impact score has no straightforward substantive interpretation, we have only provided the relative version of the measure. IO has the highest weighted PS impact and the weighted impact of all other journals is measured relative to that of IO.

The correlation between the unweighted and the weighted PS impact scores is .94, indicating that a good deal of stability is present across the estimates. However, the weighting reveals even greater hierarchy among the journals. For example, the 30th ranked journal using the weighted impact scores has only 8% of the impact of IO compared to 18% for the 30th ranked journal using the PS Impact scores. The median-ranked journal using WPS Impact, American Journal of International Law (AJIL), has only 3% of the impact of IO. A scholar would need to publish roughly 30 articles in AJIL to have the same citation impact as one article published in IO. Thus, higher-ranking journals within the discipline do not simply receive more citations from other political science journals; they are more likely than lower-ranking journals to receive citations from other higher-ranking journals. Stated differently, the “leading” journals are leading by more than suggested by simply counting citations.

While underlying stability is apparent, the omission of self-citations and the weighting of citations also yield significant shifts in the relative positions of some journals. The ordering of the top 10 journals shifts somewhat but the content experiences only two changes. JOP drops from 8th to 11th, and the European Journal of International Relations from 9th to 13th. II and Conflict Management and Peace Science replace these journals. The latter journals rise from ranks without weighting of 11th and 20th, respectively.

For some journals, the effect of adjusting the PS Impact scores for the impact of the sending journal and omitting self-citation is substantial. Six journals declined 25 positions or more in rank: Journal of Urban Affairs, Urban Affairs Review, Urban Studies, Public Administration Review, Law and Society Review, and Journal of Policy Analysis and Management. In contrast, four journals increase in the rankings 25 positions or more: Political Theory, Rationality and Society, Journal of Strategic Studies, and Studies in American Political Development.

Thus, adjusting impact scores of journals for the impact of the sending journal and omitting self-citations produces substantial changes in the rank ordering of some journals and further suggests that simply focusing on rank order may obscure the extent of differences among political science journals. This analysis also reveals substantial stability in journal ranks. There may be a few surprises among the higher-ranking journals using the weighted PS Impact scores but largely the list includes the usual suspects.

The Prominence of International Studies Journals in the Citational Analysis

Political scientists outside of the field of international studies may find the journals of that subfield disproportionately represented among the highest-ranking journals in the preceding analyses. Among the top 10 journals in PS Impact, arguably seven are international studies journals.13

International Organization, International Security, Journal of Conflict Resolution, International Studies Quarterly, European Journal of International Relations, and Journal of Peace Research. World Politics, more than the other journals, may straddle the subfields of international studies and comparative politics.

For WPS Impact, eight international studies journals ranked in the top 10. As noted previously, one concern with citation-based studies is that disciplines and subfields within disciplines may have different norms and practices regarding the use of citations. Is the dominance of international studies journals in our study reflective of the operation of divergent citation norms and practices?

We are unable to answer this question definitively with the data in hand, but the evidence does suggest that the answer is yes. While we have focused our analysis on the citations that political science journals receive from other political science journals, our data also provide some insight into to the sending (citing) practices associated with the journals. A citation received by one of our journals is also a citation sent by one of our journals.

Table 3 ranks the journals by the average number of citations per article sent to other political science journals in 2003–2004. Only citations to articles published during the study period 1998–2003 are counted.14

For 2004 articles, only citations to articles in 1999–2003 are counted; for 2003 articles, the period is 1998–2002.

The average citation sent per article among all political science journals is approximately 2.12. Six of the journals, highlighted in bold, had average citation rates two standard deviations above the mean. Of these six journals, five are international studies journals and one is a general journal, AJPS. An additional five journals have average citation rates 1.65 standard deviations above the mean. Of these journals, three are international studies journals. In sum, of the 12 journals with citations rates significantly above the mean for political science journals, eight are international studies journals and constitute a significant component of the international studies journals that performed well in terms of PS Impact and WPS Impact.

Journals Ranked by Citations Sent by Political Science Journals to Other Political Science Journals in Articles Published in 2003 and 2004

To explore the implications of this variation in citation rates across our journals, we repeated our impact analysis with citations weighted to equalize the citation rates of the sending journals. To accomplish this weighting, we calculated the average citation rate for all of the journals in 2003–2004,15

(total citations sent by all journals in 2003–2004)/(total articles published by all journals in 2003–2004)

predicted the total citations sent by each journal by multiplying the average citation rate times the number of articles published in each journal in 2003–2004, and then calculated a weight that adjusted actual citations sent by each journal so that the total citations sent by that journal equaled that predicted based on the mean for all journals. The results of this process are presented in Table 4 and may be thought of as the citation impact of journals with citation rates across the journals held constant.16

This process by equalizing the citations sent also treats the journals as if they were all equal contributors. Journals sending 20% of their total citations to political science journals are treated as equal to those sending 80%.

Column 1 of Table 4 lists the top 30 journals based on Citation-Rate-Adjusted Impact (CRPS Impact). Column 2 lists the top 30 journals based on CRPS Impact and weighting for the impact of the sending journal (WCRPS Impact).

Journal Impact with Citation Rates Held Constant

These results are quite revealing. Among the top 10 journals based on PS Impact (Table 2, col. 2) seven are arguably journals of international studies. Among the top 10 journals using CRPS Impact (Table 4, col.1) that number declines to four journals. These results suggest that the number of international studies journals that are ranked highly using unweighted and unadjusted citations (PS Impact) is at least partially a function of the higher citation rates among international studies journals as revealed in Table 3. But that is not the end of the story. Among the top 10 journals ranked using WCRPS Impact the number of international studies journals rises to seven. The process of weighting citations by the impact of the source journal clearly works to the advantage of international studies journals. This result suggests that the probability that an article published in a high-ranking journal will cite to articles in lower-ranking journals is greater among international studies journals than among the journals of other subfields and the general journals of the discipline. Thus, articles in international studies journals not only cite at a higher rate, they appear to employ a different citation pattern.

The citation practices of journals within the subfield of international studies appear to enhance the performance of those journals as measured by citation-based impact measures. Without a network analysis, tracing the higher frequency of citations sent by international studies journals to the higher frequency of citations received by other journals of the subfield, however, this result is not conclusive.17

If in fact such a pattern exists, it has implications for comparative inferences about the citation impact of articles and authors who publish primarily in international studies journals relative to those who do not. The former would be substantially advantaged over the latter.

While other sub-field journals do not stand out so clearly in their performance on the citation impact measures as do the international studies journals, the results reiterate the possibility that variation in citation norms and practices may differ across subfields and bias the results of citation-based studies.

Citation Impact Scores and Reputational Measures of Journal Status

Garand and Giles (2003) asked the respondents to their survey to assign a rating to journals from 0 to 10 based on the respondent's assessment of the general quality of the articles published in the journal. The value of “0” was labeled “poor,” “5” as “adequate,” and “10” as “outstanding.” Respondents were instructed only to rate journals with which they were familiar. From this information the authors both computed a quality rating for each journal and gleaned an indicator of the percent of the respondents familiar with each journal. Following Garand (1990) they combined these two measures to create an impact score (denoted here as GG-Impact) that ranges from 0 to 20. We calculated a relative measure of GG-Impact by taking the impact of each journal as a percent of that of APSR, the top-ranked journal using their measure. The various values are presented in columns 7 and 8 of Table 1 and in the final column of Table 2.

We begin the analysis by correlating the GG-Impact scores and the WPS Impact scores. The correlation between the two scores is .656, which is both statistically and substantively significant. Approximately 40% of the variance in the two measures is shared.18

These results approximate those found by Extejt and Smith (1990) in a comparison of reputation- and citation-based rankings for journals in the behavior sciences and management.

Figure 1 presents a scatter plot of the journals with both impact scores in their relative form with maximum values of 1.00. While comparable in theoretical range (0–1.00), it is evident that the WPS Impact scores demonstrate greater dominance by a subset of journals. IO, which has a value of 1.00 as the highest-ranked journal in WPS Impact, is more distant from the next highest-rated journal on the X-axis than APSR, which has a value of 1.00 as the highest-rated journal in GG-Impact, is distant from the next highest-rated journal on the Y-axis. Moreover, a large number of journals are grouped near the 0 point on the X-axis, indicating very small citation impact relative to that of IO. In contrast, on the Y-axis all of the journals have relative scores of .4 or greater. The lowest-ranked journal on GG-Impact is only 2.5 times lower than APSR. While the correlation and graph indicate substantial variation around the regression line, for only four journals does the GG-Impact score depart more than two standard deviations from those predicted based on the WPS Impact scores. The reputations of the JOP and BJPS are significantly greater than predicted by citations, and the reputational scores for Conflict Management and Peace Science and II are significantly less than predicted based on citations.

Journals Distributed by Garand-Giles and Weighted PS Impact with Regression line and 95% Confidence Interval

While these results indicate that there is a moderately strong relationship between the rankings journals receive based on reputation and citations, they also suggest that the approaches do not render precisely the same results. Some of the lack of fit may reflect data weaknesses with one or both approaches. An obvious problem for the reputational approach is that the evaluation score for many of the journals is based on the judgments of a relatively small number of respondents. Moreover, given that these journals may have lower familiarity because they appeal to a specialized audience, the evaluations may be biased upward.19

Garand and Giles (2003) compensate for this problem somewhat by weighting journal evaluation by familiarity in their impact score.

Likewise, some of the journals included in the study have a relatively small number of citations from other political science journals.20

Eleven journals, including Studies in American Political Development and Polity, received 20 or fewer citations from political science journals between 1998 and 2003.

Given the weighting employed, if one or two of those citations are from journals with high PS Impact scores, the journals' weighted scores will increase dramatically relative to other journals. As a check on the robustness of the results, we repeated the analysis by excluding all journals familiar to fewer than 15% in the Garand and Giles survey and then excluding all journals receiving fewer than 20 citations from other political science journals. In both instances we find that the correlation between GG-Impact and WPS Impact changes only marginally.

Another factor that may suppress the relationship between the citation- and reputation-based measures is the previously noted difference in citation practices between international studies and non-international studies journals. This possibility was assessed by regressing GG-Impact on WPS Impact with 16 international studies journals excluded.21

The following journals were excluded: International Organization, World Politics, Journal of Conflict Resolution, Journal of Peace Research, American Journal of International Law, International Security, International Studies Quarterly, International Affairs, International Interactions, Conflict Management and Peace Science, Journal of Common Market Studies, European Journal of International Relations, Post Soviet Affairs, Studies in Comparative and International Development, Security Studies, and Journal of Strategic Studies.

The excluded group consists of all the journals in our study that are also classified as “international studies” by ISI. Among the remaining 76 journals the correlation between the two measures is substantially stronger, r = .795, than in the full journal set. Indeed, the correlation between the reputation- and the citation-based impact scores is comparable to that between RISI Impact and PS Impact, r =.78. Thus, the citation practices of international studies journals appear to attenuate substantially the relationship between the two measures of impact.

Discussion

Publication in academic journals is a principal means of scholarly communication and, hence, a major consideration in evaluating the records of individual scholars and the standing of academic programs. In the social sciences the status of journals has been measured either by assessing their reputation among scholars or by counting the number of citations that they receive. In political science reputational studies have been predominant. We have developed a citation-based ranking of political science journals and compared it to the standard reputational ranking developed by Garand and Giles (2003). While the efficiency and objectivity of citation-based rankings no doubt will result in increased reliance on this approach, our results provide a cautionary note.

Our citation analysis illustrates that the criticisms generally associated with citation-based assessments are applicable to political science. First, the “demarcation” or boundary problem affects both reputational and citational studies but it is more critical for the latter. Defining the set of “political science” journals for a reputational study may result in the omission of journals that are relevant to the field. However, in addition to not including the journal in the study, for citational studies the citations from that journal to journals within the study will also be omitted, and this may result in lower impact scores for those study journals than they would have otherwise achieved. The declines in the impact scores for interdisciplinary journals as we restricted our analysis to the subset of “political science” journals provide clear evidence of this effect.

Second, variation in citation practices across and within disciplines is another potential pitfall associated with citation studies. Our analysis demonstrates that citation practices clearly differ between international studies and non-international studies journals in political science and suggests that the prominence of international studies among the higher-ranked journals is, in part, a function of that difference. Cross-subfield comparisons within political science based on citational analysis must be done with caution.

More generally, we believe that there is no single “best” measure of journal status in political science. Rather, each of the various measures we examined, citation- and reputation-based, provide useful information about journals. The weight to be given to each depends most importantly on the precise purpose to which it is addressed. Citation-based approaches measure the impact of journals on scholarly discourse by noting the average frequency with which their articles are cited. The RISI Impact score includes citations from an encompassing range of journals and hence provides a global measure of impact. The global impact of a publication may be a critical component of an assessment, particularly in substantive areas that cross disciplinary lines. For example, in evaluating the record of a political scientist specializing in gender issues who published in Signs, it would be relevant to note that the RISI Impact score for that journal is 1.24. Similarly, in evaluating the record of a political scientist who specializes in public policy who published in Public Administration Review (PAR), it would be relevant to note that the RISI Impact score for that journal is 1.72. Both of these journals have comparable global impact to Political Research Quarterly, 1.30, and substantially higher than that of Polity, .36, both general disciplinary journals.

But the reality is that publications in journals outside of political science, with a few notable exceptions, have limited visibility and impact on discourse within the discipline. Thus, in evaluating the status of political science scholars and programs, citations among discipline journals take on primary importance. This is captured by the WPS Impact scores. On this measure Political Research Quarterly ranks substantially higher than Signs or PAR. Indeed, Women and Politics ranks higher in WPS Impact than does Signs. Thus, both RISI Impact and WPS Impact provide useful information about the status of journals. The relative importance and utility of each depends on the purpose for which they are employed.

The citation-based impact scores suggest a far greater inequality among the journals of the discipline than do the reputation-based impact scores of Garand and Giles (2003). In the assessment of a scholarly record that occurs at both the individual and programmatic levels, balancing the quantity and quality of scholarship is frequently an issue. For example, junior faculty often want to know not only how many publications meet the minimum standards for promotion and tenure but if publishing more in lower-ranking journals is a better strategy than publishing less in higher-ranking journals. Based solely on the impact an article has on future work as measured by the citation rankings, the answer is clear: no practically feasible quantity of publications in lower-ranking journals can match the impact of a single publication in a top 10 journal. Publications in most academic journals in political science are stones that fall into the pool of disciplinary discourse without causing a ripple. They are lines on a resume without resonance.

A problem arises, however, when this fact is confused with an assessment of the quality of articles. Citations are a direct measure of impact but measure the quality of an article only indirectly and imperfectly. To the extent that there is a relationship between journal citations and quality it is asymmetrical. High journal impact may provide a reasonable basis for inferring high average article quality for a journal, but low impact does not provide a basis for inferring low quality. This is the message conveyed by GG-Impact, a component of which is based directly on political scientists' assessment of journal quality. On the quality evaluation, respondents only rated a handful of the journals below “5” (adequate) and then only barely so. This may be the equivalent of grade inflation, but we do not think so. Journal reviewers do employ different standards when reviewing a manuscript submitted to a top-ranked journal like APSR, CP, or IO than when evaluating a manuscript for a lower-ranking journal. These generally touch issues such as the generalizabilty of the findings, the methodological sophistication of the analysis, and the substantive and/or theoretical significance of the work. We do not believe that reviewers are willing to accept manuscripts submitted to even low-ranking journals that are flawed in design, execution, or inference. Thus, articles may appear in journals ranked low based on citations for a variety of reasons, but as the GG-Impact measure suggests, a publication in any of the journals in our list is recognition of sound scholarship and should be appropriately credited in the evaluation of scholarly records. In the absence of reputational measures directly assessing perceived journal quality we are concerned that this important distinction between quality and impact might be elided in practices of professional assessment.

Author Bios

Micheal W. Giles is the Goodrich C. White Professor of Political Science at Emory University. In addition to a continuing interest in the sociology of the discipline his research focuses primarily on courts and judicial behavior. His most recent works, “The Supreme Court in American Democracy: Unraveling the Linkages between Public Opinion and Judicial Decision-making,” co-authored with Bethany Blackstone and Richard Vining, and “The Etiology of the Occurrence of En Banc Review in the U.S. Court of Appeals,” co-authored with Virginia Hettinger, Christopher Zorn, and Todd Peppers, are forthcoming respectively in the Journal of Politics and the American Journal of Political Science. He has served the profession in many positions including as president of the Southern Political Science Association, chair of the Law and Courts Section of the American Political Science Association, and editor of the Journal of Politics.

James C. Garand is the Emogine Pliner Distinguished Professor of Political Science at Louisiana State University. He has written extensively in the field of American politics, especially in the areas of legislative politics, voting behavior, public opinion, state politics, public policy, and the sociology of the political science discipline. He is former president of the Southern Political Science Association, former president of the State Politics and Policy Section of the American Political Science Association, and former editor of the American Politics Quarterly. In May 2007 he was named the LSU Distinguished Research Master, the highest career award for scholarly accomplishment given by LSU.

References

Adam, David. 2002. “Citation Analysis: The Counting House.” Nature 415 (February): 726–9.Google Scholar
Baumgartner, Hans, and Rik Pieters. 2003. “The Structural Influence of Marketing Journals: A Citation Analysis of the Discipline and its Sub-Areas over Time.” Journal of Marketing 67 (April): 123–39.Google Scholar
Bodenhorn, Howard, and William Walstad. 2003. “Economic Scholarship at Elite Liberal Arts Colleges: A Citation Analysis with Rankings.” Journal of Economic Education 34 (fall): 341–59.Google Scholar
Burton, M. P., and E. Phimister. 1995. “Core Journals: A Reappraisal of the Diamond List.” Economic Journal 105 (March): 361–73.CrossRefGoogle Scholar
Crewe, Ivor, and Pippa Norris. 1991. “British and American Journal Evaluation: Divergence or Convergence.” PS: Political Science and Politics 24 (September): 524–31.Google Scholar
Davis, John B. 1998. “Problems in Using the Social Sciences Citation Index to Rank Economics Journals.” American Economist 42 (fall): 59–65.CrossRefGoogle Scholar
Diamond, Arthur. 1989. “The Core Journals in Economics.” Current Contents (January): 4–11.Google Scholar
Donohue, J, and J. Fox. 2000. “A Multi-Method Evaluation of Journals in the Decision and Management Sciences by U.S. Academics.” Omega 28 (February): 17–36.CrossRefGoogle Scholar
DuBois, Frank, and David Reeb. 2000. “Ranking International Business Journals.” Journal of International Business Studies 31 (4): 689–704.Google Scholar
Extejt, Marian M., and Jonathan E. Smith. 1990. “The Behavioral Sciences and Management: An Evaluation of Relevant Journals.” Journal of Management 16 (3): 539–51.Google Scholar
Garand, James C. 1990. “An Alternative Interpretation of Recent Political Science Journal Evaluations.” PS: Political Science and Politics 23 (September): 448–51.CrossRefGoogle Scholar
Garand, James C., and Micheal W. Giles. 2003. “Journals in the Discipline: A Report on a New Survey of American Political Scientists.” PS: Political Science and Politics 39 (April): 293–308.Google Scholar
Garfield, Eugene. 1986. “The 250 Most-cited Authors in the Arts and Humanities Citation Index, 1976–1983.” Current Contents 48 (December): 3–10.Google Scholar
Garfield, Eugene. 1996. “How Can Impact Factors be Improved?” British Medical Journal 313: 411–3.CrossRefGoogle Scholar
Giles, Micheal, Francie Mizell, and David Patterson. 1989. “Political Scientists' Journal Evaluation Revisited.” PS: Political Science and Politics 22 (September): 613–7.Google Scholar
Giles, Micheal, and Gerald Wright. 1975. “Political Scientists' Evaluations of Sixty-Three Journals.” PS: Political Science and Politics 8 (summer): 254–7.Google Scholar
Hantula, D. 2005. “The Impact of JOBM: ISI Impact Factor Places the Journal of Organizational Behavior Management Third in Applied Psychology.” Journal of Organizational Behavior Management 25 (3): 1–15.Google Scholar
Hendry, David F. 2006. “Research Assessment Exercises.” Unpublished manuscript.Google Scholar
Hix, Simon. 2004. “A Global Ranking of Political Science Departments.” Political Studies Review 2 (3): 293–313.Google Scholar
Kalaitzidakis, Pantelis, Thanasis Stengos, and Theofanis P. Mamuneas. 2003. “Rankings of Academic Journals and Institutions in Economics.” Journal of the European Economic Association 1 (December): 1346–66.Google Scholar
Laband, David, and Michael J. Piette. 1994. “The Relative Impacts of Economics Journals: 1970–1990.” Journal of Economic Literature 32 (June): 640–66.Google Scholar
Leimu, Roosa, and Julia Koricheva. 2005. “What Determines the Citation Frequency of Ecological Papers?” Trends in Ecology and Evolutions 20 (January): 28–32.CrossRefGoogle Scholar
Liebowitz, Stanley J., and John C. Palmer. 1984. “Assessing the Relative Impacts of Economics Journals.” Journal of Economic Literature 22 (March): 77–88.Google Scholar
Moed, Henk. 2005. Citation Analysis in Research Evaluation. Netherlands: Springer.Google Scholar
Nederhof, A. J., and A. F. J. van Raan. 1993. “A Bibliometric Analysis of Six Economics Research Groups: A Comparison with Peer Review.” Research Policy 22 (5–6): 353–68.Google Scholar
Stahl, Michael, Terry Leap, and Zhu Wei. 1988. “Publication in Leading Management Journals as a Measure of Institutional Research Productivity.” Academy of Management Journal 31 (3): 707–20.Google Scholar
Van Dalen, Hendrik, and Kene Henkens. 2001. “What Makes a Scientific Article Influential? The Case of Demographers.” Scientometrics 50 (January): 455–82.Google Scholar
Vokurka, Robert. 1996. “The Relative Importance of Journals Used in Operations Management Research: A Citation Analysis.” Journal of Operations Management 14 (November): 345–55.Google Scholar
Figure 0

Various Impact Scores by Political Science Journal

Figure 1

Journals Ranked by Various Impact Measures

Figure 2

Journals Ranked by Citations Sent by Political Science Journals to Other Political Science Journals in Articles Published in 2003 and 2004

Figure 3

Journal Impact with Citation Rates Held Constant

Figure 4

Journals Distributed by Garand-Giles and Weighted PS Impact with Regression line and 95% Confidence Interval