Mixed motives

The initial motive behind the creation of citation indexes was the growth in scientific publications, which surpassed the observability of even the most devoted scholarly bibliographers (see Abbott Reference Abbott, Camic, Lamont and Gross2011). At the very beginning of indexing, papers’ cited references were put into the database to offer users the opportunity of identifying additional relevant papers for their particular research interests. Garfield’s ISI appeared on the market of information retrieval at a time when there was a widespread awareness of an increasing inability to cope with the information explosion, a term coined, or at least made popular, at a conference held at the Royal Society in London as early as 1948. The parlance of the exponential growth of publications asked for remedies (see Becker Reference Becker and Sills1968 for a coeval overview) and new devices were established: in 1951, unesco founded the International Bibliography of the Social Sciences, which tried to cover international publication output in an old-fashioned way, initially not even using computers. Garfield’s Citation Index however, which appeared in five printed volumes in 1963 and reported on what had been published during the year 1961 (initially only covering journals from physics and life sciences), was more than just another attempt to serve academia.

The defects of the then existing bibliographical devices asked for “an effective means of disseminating and/or retrieving scientific literature” (Garfield Reference Garfield1963: 289). In 1955 Garfield thought of single papers whose “significance” or “influence” could be measured by the total number of citations made to them. Already at this time Garfield suggested that the data could be used for “sociological evaluations, including personnel and fellowship selection” if one avoids “promiscuous and careless use” (Garfield Reference Garfield1963: 290). In 1963, Garfield and his co-author I. H. Sher declared that they were “interested in certain ‘impact’ factors such as how often a particular paper, author, or journal is cited compared to corresponding average values in a given Citation Index file” (Garfield and Sher Reference Garfield and Sher1963: 199). Analyzing the sheer number of citations brought implausible results, because some journals published more papers than others. Garfield therefore suggested “a more sophisticated approach” to measure the impact by dividing “the number of times a journal is cited by the number of articles that journal has published” (ibid.: 200). The resulting numbers were a better fit for the expectations of the bibliographers.

When first proposed, the aim of the invention was to establish a rationale for the selection of journals for the Citation Indexes. Instead of avoiding ambiguities they became routines: journals became indexed because they contained a critical number of highly cited papers, even though scientists usually referred not to journals but to particular articles.

Different audiences

The main audience for the establishment of the Citation Index was librarians and their acquisition decisions rather than scholars trying to master the growth of scientific information. Whereas scholars would have been interested in locating relevant publications, librarians had to make up their mind with regard to their acquisition budget and wanted to know which journals they should order. The impact factor was first made public in 1975 when ISI started publishing the Journal Citation Reports (JCR) as an annual supplement to the Science Citation Index (SCI). Its immediate success and widespread recognition were accompanied by promises beyond the provision of a system for information retrieval. Sociologists recognized this from the very beginning: “Another technique [to speed up and increase effectiveness of the dissemination of information in science], facilitated by the use of high-speed computers, is the citation index, which enables one to trace the influence of a given paper forward in time; this will apparently be of value to historians and sociologists of science as well as those concerned with the substantive content of materials cited” (Kaplan and Storer Reference Kaplan, Storer and Sills1968: 116). Both the usability of the SCI and the promise to offer more than just another reference work were crucial for the success of Garfield’s one-man-enterprise. After a few years the SCI, the Social Science Citation Index (SSCI), and the Arts and Humanities Citation Index (A&HCI), accompanied by several descendants, such as e.g. the Reference Journal Citation Index, Source Journal Citation Index, Journal Citation Report, Current Contents and New Scientist, established a monopoly for information retrieval and evaluation. From the very beginning, Garfield played with ideas to make use of the SCI to predict Nobel Prize winners (Garfield Reference Garfield1973, Ashton Reference Ashton1978); he provided help for several kinds of decision makers, and through his usage of an evaluative language (significant, important, core, impact, etc.) ISI’s data became more than just data. The IF did not receive much attention during its first three decades;^{Footnote 11} only from the middle of the 1990s onwards did the IF become what it is today: a controversial but highly influential number, trivialized by administrators, and displayed by those who participate in academic impression management.

Experimenting with stored data

Garfield’s admiration for science itself disposed him to experiment with data, which resulted in suggestions for several measurements and indicators. His familiarity with debates in the social sciences prompted him to contribute to the then nascent sociology of science. In 1970 he claimed to be part of “a new breed of sociometrist concerned with the historical, sociological, economic, and behavioral study of science and scientists”, and labeled them “Scientists of Science” (Garfield Reference Garfield and Garfield1977b: 158). Due to his own devotion to statistical calculations and encouraged by leading scientists and exponents from the emerging field of sociology of science, Joshua Lederberg and Robert K. Merton, respectively, Garfield experimented with several measures based on the increasing amount of data at hand (Lederberg Reference Lederberg, Cronin and Atkins2000, Merton Reference Merton and Garfield1979, Merton and Garfield Reference Merton, Garfield and Price1986, Merton Reference Merton, Cronin and Atkins2000). Over the next decades, Garfield and other authors proposed several indicators, calibrated, designed and re-designed them, but in the end only a few of them were accepted. It seems fair to state that all these indicators were the result of a completely inductive procedure. In nearly all cases they arose from ad hoc decisions.

Using stored data from the field of genetics, Garfield and his collaborators found regularities. First, they recognized that the frequency of citations showed a highly skewed distribution. “A small group of 250 journals provided almost half of the 3.85 million references processed for the SCI in 1969” (Garfield Reference Garfield1972, quoted from the reprint in Garfield Reference Garfield1977a: 534). Second, this resulted in the creation of the notion of “core journals”: “No matter what the specialty, a relatively small core of journals will account for as much as 90 % of the significant literature, while attempts to gather 100 % of it will add journals to the core at an exponential rate” (Garfield 1971: 5).

Third, details of the calculation of any IF were yet to be established. Back in 1963, the ratio (“number of citations divided by number of papers published”) was suggested with no restriction to particular time periods. Later on, however, Garfield claimed to have found that a restriction of the sample would not hamper the validity, following the findings of two British authors (Martyn and Gilchrist Reference Martyn and Gilchrist1968) who had used ISI’s data to evaluate British journals. In 1972, Garfield proposed that “the ratio between citations to particular years of a journal and the number of articles published in those years” (Garfield Reference Garfield and Garfield1977c: 270) would work just as well as the computation of all citations. Even more consequential was another decision, which again was first proposed by Martyn and Gilchrist, the British duo mentioned above: Garfield found that “about 25 % of all citations are made to the two-year period prior to the source year chosen”. The resulting “current impact factor” intentionally discounted “the effect of most super-classics” (ibid.: 271).^{Footnote 12} At this time the SCI covered 565 science journals and about one million citations, and these ad hoc limitations were probably not invalidating the results. We will see later that this retrenchment is less plausible for other branches of scholarship, particularly the social sciences and humanities.

Prior to this, one needs to point to a fourth decision which also became part of the definition of the IF business. Garfield and others recognized that particular types of texts, the so-called “reviews”^{Footnote 13} (articles summarizing the literature in trend reports) – which later on became characteristic of publications within the different Annual Reviews – were cited more heavily than others: “Garfield came to recognize that the sentences in review articles are implicit indexing statements and that the process of producing a scientific index could be automated by making these sentences the grist for such an index. This idea eventually led to creation of the SCI” (Bensman Reference Bensman2007: 108).

Today, Thomson Reuters’ computers produce figures that are widely used without looking at the process of their production. According to their help site, the IF as it is computed today is a simple fraction:

However, as always, the devil is in the details. What is the “number of citations”? A simple mind would answer: all references in a particular article published in a particular journal count as citations. For practical reasons providers such as Thomson Reuters’ Web of Science (WoS) might do the calculation only for a selection of journals, but for every journal covered the computation should work well.

Still, the same simple mind would get into even more trouble with regard to the denominator. “The total number of articles published in the two previous years” obviously will not encompass all 247,854 articles (and “reviews”) collected by Sociological Abstracts (SA) for 2009 and 2010, or the 238,955 articles and “reviews” published worldwide during the same period (in Scopus) but only those 8,869 articles and “reviews” covered by WoS for the same two years. But which journals are knighted this way? Again, one has to accept that an enterprise like Thomson Reuters is not disposed to include other journals than those contained in its database, but only if the policy behind the selection is transparent and sound.

In summary one can see that several ad hoc decisions, which made sense for the particular disciplines and journals originally covered in the Citation Index, became transformed into standards. This execution of experimental calculations was a two-sided elitism, favoring science over other academic disciplines, and paying more attention to journals of unquestionable status compared to all other media of scholarly communication. The results of all this could be adjusted and refined because of the observability of the entire field at this point in time. For Garfield and his highly placed academic supporters the calculations made sense, because they fitted into their picture of the American landscape of science, and he rejected other measurements that did not meet this criterion.^{Footnote 15} In later years, when the Citation Index business expanded to other disciplines and more countries and languages, the already established routines and measurements were imposed on others, which lacked the agency to veto or demand adjustments. Like Goethe’s sorcerer’s apprentice, Garfield has raised his voice more than once over the last decades to distance himself from the alleged misuse of his invention, but with as little success as the literary counterpart.^{Footnote 16}

It can be argued with some justification that what happened here was the interaction of the self-fulfilling prophecy mechanism with the emerging monopolistic market position of the ISI. Several authors have highlighted the arbitrariness of the IF, and in the following sections I will provide illustrations for three main aspects: the range of the observation period, the definition of disciplines, and the selection of journals.

Two years are not sufficient, at least not for sociology

The two-year observation period was established as a consequence of inspecting a set of journals from hard science disciplines, and became the standard for all journals covered in the Web of Science (WoS) and the Journal Citation Report (JCR).^{Footnote 17} Today’s journals inherit the burden of these early decisions, but if ISI had studied a different set of disciplines and journals, the result would have been quite different.

One can illustrate this by examining three leading sociology journals (American Sociological Review, British Journal of Sociology and Kölner Zeitschrift für Soziologie und Sozialpsychologie) in order to highlight the practice of sociologists.^{Footnote 18} I coded the list of references from all main articles for two recent issues of each of the three journals. The sample consists of 41 articles with a total of 2,731 references (1,388 of them to articles), or on average 66 bibliographic entries per article. Only 4 % of the citations were self-references, slightly more in the German journal (5.5 %). There is therefore no need to discuss the frequently raised issue of self-citations here in more detail. Half of the references were to journal articles and the other half to any other sort of literature (books, book chapters, newspaper articles, Internet sources, etc.). The 1,388 cited articles appeared in any journal, whereas the calculation of the IF takes into consideration only those journals that are listed in the SSCI. I did not check how many of the cited journals are not in the SSCI^{Footnote 19}, because another limitation left more dramatic traces: only 11.4 % of all citations were directed towards articles which came out in the two previous years.^{Footnote 20} Even if we enlarge the base to all cited sources the percentage remains the same and is still much lower than Garfield’s originally established threshold of 25 % (Garfield Reference Garfield and Garfield1977c: 271). In disciplines like sociology, the practice of citation is obviously very different from science disciplines, and the calculation of the IF therefore affects only about one-tenth of its references. Several of the 41 articles coded did not even cite a single publication within the two years span so crucial for the WoS. The largest share of references goes to pieces published five or more years earlier, with some differences between the three sampled journals.^{Footnote 21} The only defense in favor of SSCI’s routines would be to suggest that this small share of about 11 % might be homogenous enough to produce valid results with regard to the “impact” of publications by sociologists. Given the fact that the IF computed by WoS makes use only of those journals that are covered there the real share of references is even lower; if one assumes that sociologists behave in a similar way to political scientists one could subtract half of the quotations as going to non-SSCI sources.

As the data show, sociologists do not systematically prefer quite recently published articles to older sources. Our debates happen to evolve much more slowly than in hot disciplines such as cell biology. Older papers might finally get the same number of citations, but the discipline’s habitual inertia is not recognized by WoS’s routines. A generalization from a sample of one-tenth most probably produces biased results.

Several critics and a few defenders of the SSCI argue that there is the alternative of a five-year period of observation, and that the JCR also reports this and other measures. If we look at the five-year indicator, in the case of these three sociology journals the data are still demanding: The (cumulated) share of citations to articles goes up to 31 % (or 28 % if one computes all citations < 5 years to all cited sources). But even this percentage is not represented in the calculation of the 5-year IF (because WoS considers only a selection of journals as its source material) and is in all likelihood lower than the magical 25 % Garfield claimed as sufficient for the computation of a valid measure.

This very simple validity check of SSCI routines is in accordance with what every sociologist knows, but that has been lost in an arena where the IF functions as a benchmark. Obviously the 2 years IF is not a sound approximation for sociology^{Footnote 22}, whilst the 5-year IF would probably work for sociology but is not used by its practitioners. At least the editors and publishers of the vast majority of those journals covered by the SSCI jettison their everyday knowledge and use the IF for marketing affairs.

The observation period is not the only shortcoming of the IF computation.

What makes a discipline?

A second particularity which results in arbitrariness is rooted in the definition of disciplines, especially in the social sciences. The criteria for including particular journals in the SSCI and their allocation to subject categories lack comprehension. The SSCI covers 41 different subjects and one cannot detect the underlying rationale for its composition (the SCI covers about 170 subjects). The social sciences are classified according to classical disciplines, such as anthropology, geography, history, law, political science, sociology, etc. but also include special or interdisciplinary fields of research, like ethnic studies, family studies, gerontology, etc. Psychology, for example, is covered in ten different subfields plus psychiatry whereas economics stands alone. In addition one comes across less well known and well-defined specialties, such as ergonomics, planning & development, rehabilitation, substance abuse, transport etc. All in all some 2,700 journals are included in the database (with 1,000 double or multi entries): 305 from economics, 220 in education, and 608 in the ten subfields of psychology. Other fields are as narrow as “history of social sciences” (27)^{Footnote 23} or “social sciences, mathematical methods” (43), but there are also broad categories like “social issue” with very few journals (35), or in the case of “social sciences, interdisciplinary”, with a very large number of journals (84), which is considerably higher than the numbers in more or less established specialized fields, such as “demography” (24), “gerontology” (30), “urban studies” (36), and much less crystalized specialties like “rehabilitation” (64), “transportation” (23) and “substance abuse” (29).

The placement of a particular journal in one of these subjects is contestable. Take for example Poetics which is in sociology, Economy and Society which is listed both in economics and sociology, whereas Minerva is represented in three fields (education and educational research; history and philosophy of science; and social sciences, interdisciplinary; but not in history of social sciences) and New Left Review in two (political science and social sciences, interdisciplinary). Actes de la recherche en sciences sociales is only in “social science, interdisciplinary” whereas Human Relations is there but also in “management”. All these classifications might make sense in some way but they hamper any analysis and comparison of disciplines.

The selection of journals for the SSCI does not correlate with the number of scholarly journals in several subjects. Nor do those selected represent the same share of highest recognized journals in each of the subjects. This can be seen by comparing the range of IFs for particular disciplines. For example, the psychological journal with the highest impact factor is Behavioral and Brain Sciences (21.9) followed by the Annual Review of Psychology (18.3) but the journals on ranks further back than 200 (406 in total) only have an impact factor below 1.8. In Economics the highest ranked journal is the Journal of Economic Literature (7.4), but further back than rank 40 the journals reach impact factors smaller than 1.8, all in all 260. The impact factors of the 76 journals from anthropology range from 4.5 down to 0.076, but the level of 1.8 is exceeded only by the first eleven journals. These few examples suggest that the selection process no longer mirrors the initial rationale for inclusion: to cover only journals with a given reputation, by excluding all those that do not surmount a (pragmatically defined) threshold.^{Footnote 24} Any comparative research for the subfields of the social sciences ends inconclusively. For the 41 social science subunits the (median) impact factor ranges from 1.8 for “psychology, biological” down to 0.3 for “cultural studies” and history.^{Footnote 25}

Whether a particular journal is covered at all should be the consequence of its high reputation, measured by citations. It is fair, however, to remind readers that nowadays the owner of WoS, Thomson Reuters, selects journals not only because of their “reputation” but also for other reasons, as they frankly declare on their website:

The haphazard decision about the inclusion of particular journals in arbitrarily confined pseudo-disciplines (“subject categories”) obviously influences the calculation of the IF, even more emphatically when the numbers (of articles and citations) are small.

European sociological journals in the SSCI

Data from WoS, published in JCR, do not only impress different groups of users but were used for international comparative research as we have seen before. To further demonstrate the inappropriateness of the SSCI for any analysis of national productivity or similar topics let us consider in detail the representation of European sociological journals in it. Since there is no way to establish the phenomenon “European sociology” correctly because of the lack of data at hand, authors have to rely on databases such as CSA Sociological Abstracts (SA), Scopus, WoS and sometimes Ulrich’s Periodicals Directory for their analyses. Whereas ProQuest’s SA covers as many journals as possible^{Footnote 26} – and also other kinds of documents – the two more selective databases from Elsevier and Reuters Thompson also provide statistical data on their holdings. Restricting the search in SA to authors’ affiliation in larger Europe results in a total of about 10,000 articles published in a single year. The SSCI is remarkably more selective and only provides in the subject category “sociology” data for 4,370 articles and 108 “reviews” annually. Scopus covers the broader field of “social sciences” 125,000 articles and reviews (all data are for 2010).^{Footnote 27}

All three databases offer search options to characterize the European sociological landscape in more detail; at the same time these three sources provide very different portraits of this branch of scholarship. The SSCI and its accompanying Journal Citation Reports (JCR) make it possible to sample journals both by discipline and country of publication.^{Footnote 28} 142 journals are listed there as belonging to “sociology” in 2010 and 17 European countries host 59 of them, allegedly.^{Footnote 29} Some journals are classified as belonging to “sociology” only in addition to other fields of research (ranging from “hospitality, leisure, sport and tourism” to more familiar specialties such as anthropology, linguistics, economics, etc.); if one excludes the journals with two, three or even four classifications and restricts the selection to journals whose first and only “subject category” is “sociology”, the number would go down to 36 journals of alleged European provenance.

A case-by-case check brought strange results: In practically all cases the ostensible “country of publication” was indeed the location of the publishing house. For instance the Journal of Sociology is indicated as being located in England only because its publisher Sage is, whereas the editors are located down under and the journal is nothing less than the official journal of The Australian Sociological Association (TASA).

Scrutinizing all journals’ websites revealed that it is impossible to assign a nation state to each of them. Whenever the group of editors assembled scholars from more than one country, I moved them to the “international” group, in sum 37.^{Footnote 30} The alleged “nationality” of the journals changed dramatically in some cases, in particular the United Kingdom and the Netherlands lost many, whereas the number of journals located in the United States went down only slightly.

As we have seen, Thomson Reuters follows a mixed strategy with regard to the selection of journals for its WoS database. The obvious result for sociology is that the 140+ journals belonging to “sociology” surely include the most prominent exponents, but arbitrariness rules below the top 20. The Polish journals Eastern European Countryside (2010 IF 0.053), the Lithuanian Filosofija - Sociologija (2010 IF 0.179) and the Vienna based Innovation - The European Journal of Social Science Research (2010 IF 0.326) should not be considered as belonging to the core of European sociological publications, at least not as long as others of comparable reputation are excluded, or covered in other less visible subject categories.

The increasing inclusion of journals from former Soviet Europe after 1989 in the ssci have skewed the overall picture: 59 European journals are included in the ssci. 17 of them have editors located in the UK, 5 in Germany, 3 in Poland, 2 each in Croatia, France and Spain, and 1 each in Belgium, the Czech Republic, Denmark, Ireland, Lithuania, the Netherlands, Norway, Romania, the Russian Federation, Slovakia, Sweden, and Switzerland. That is, a total of 9 in former Western Europe and 8 from former Soviet Europe, which seems to be in accordance with the number of nation states, but not with regard to population numbers and also not with regard to the number of sociologists. However, there are no data at hand concerning the number of sociologists in Europe. Rough estimates can be drawn from the number of national sociological associations represented either in the European Sociological Association (ESA) and/or in the International Sociological Association (ISA), which is 33. Extrapolating the mean number of sociologists per 100,000 inhabitants from those countries where reliable data are available (n=16), the result would be a minimum of 14,000 sociologists in Europe. This estimate would put Europe on par with the US^{Footnote 31} If we assume that the numbers for sociologists in Europe and the United States are roughly the same and the institutionalization of sociology in both regions shows a similar stage of development, we could, following the clue of Andrew Abbott^{Footnote 32}, expect a comparable number of sociological journals on both sides of the Atlantic. A search in Ulrich’s confirmed this assumption: the 538 journals classified as belonging to sociology are distributed evenly between Europe (244), and North America (US and Canada: 221).^{Footnote 33} Scopus, WoS/JCR and Ulrich’s classify the country of publication primarily by using the location of the publishing house. Therefore many more journals are assigned to the Netherlands and the United Kingdom, while several European countries are missing entirely, e.g. Bulgaria, Greece, and Portugal. Others, like Italy, are definitely under-covered.

A comparison of publication output will result in a less persuasive picture. Scopus offers data for regions from where “citable documents” originate. For the first decade of the 21^st century, 93 % of those were published by journals located in Western Europe and less than 7 % in Eastern Europe. However one should keep in mind that the assignment of nations to journals is highly contestable, as mentioned above. Since there are no indications that sociologists from former Soviet Europe produce much better articles than their counterparts in the old West, one has to suspect that Thomson Reuters follows a marketing strategy rather than a professed scholarly excellence. In order to conquer the market of the transformation countries, this for-profit enterprise was driven to please the presumed purchasers of its service.

The erroneous assignment of countries to journals has serious consequences because practically all authors publishing on the international development of scientific productivity, on changing patterns in the competition between nations or the alleged decline or rise of regions, states, the European Union, etc. build their conclusions on data from WoS.

WoS’ widely used rhetoric of core, recognition, impact, etc. collides with journal selection practices that are anything but persuasive.

The insignificance of the impact factor

The JCR provides several key figures for the journals covered. The IF and the 5-year impact factor are the two most frequently used indicators, and the latter is usually higher than the first. The sociological journal with the highest 5-year IF is the American Sociological Review (2010: 5.8; 2011: 5.7; compared with a 2-year IF of 4.4 in 2011), followed by the American Journal of Sociology (2010: 5.1, 2011: 3.1; 2-year IF in 2011: 3.2), and the Annual Review of Sociology (2010: 5.03, 2011: 5.8; 2-year IF in 2011: 4.4). At rank 8 follows the first European journal, the British Journal of Sociology, with 2.8 in 2010 (2011: 2.1, 2-year IF in 2011: 1.7).

If one computes the mean of the 2-year IF over a longer period of time (2000 to 2011) the results are enlightening: Again, the journal with the highest mean is American Sociological Review with 3.2, followed by Annual Review of Sociology with a mean of 2.9 and the American Journal of Sociology with 2.8, but the next 10 show only very slight differences, ranging from 1.78 to 1.39. Further down the list the differences almost disappear: Rank 14 to 32 is within the range of 1.387 to 1.024. From rank 33 onwards the 12-year-mean is below 1.^{Footnote 34}

In other words, the average article published in one of the 4 highest ranked journals is cited on average 3 times. The next 28 journals’ contributions obtain a resonance equal to 1 citation per year, and the remainder of the 100+ journals covered in the ssci obtain less attention than 1 citation within the 2-year observation period.

As a further illustration of the capriciousness of the ssci’s established routines one could execute the calculation of a slightly different IF. “Officially” it is the fraction of the “cites” in a given year to items published in the two previous years by the number of items published in these two years. The “cites” come from those “citing journals” which are in the ssci and the result is the 2-year IF. Given the slowness of sociologists to pay tribute to papers of others it would be justified if, instead of looking for the immediate preceding two years, one was to go back one additional year. To calculate this more appropriate IF (still for any given year) one would then count the “cites” from year 1 to the years -2 and -3, instead of -1 and -2. The three arbitrarily selected nationally leading journals occupy sufficiently different ranks in the ssci to allow generalizations: Over the 12 year period, ASR is ranked #1, BJS is ranked #13 and the German Kölner Zeitschrift is ranked #67 (out of 148 journals covered). This minor modification results in a much higher IF for ASR, and a steadily higher IF for BJS, but did not improve the IF of the Kölner Zeitschrift, probably because of the overall smaller number of “cites”.^{Footnote 35}

In each year, only a very small share of citations is devoted to papers from the immediate previous year for any of the three journals listed here (and one can generalize this pattern with some justification to all other sociology journals).^{Footnote 36} To illustrate this, we can take the numbers for ASR in 2008 as an example: the official IF divides the sum of the number of times articles published the two previous years 2007 and 2006 were cited (86 and 230) by the number of items published in these two years (42 and 42) which is 3.762. Just by going back one further year in the observation period, the result is much more in accordance with the overall distribution of citations: 230+252/42+41 = 5.807. Because of the completely ad hoc approach of those who created the IF, an adaptation which is in accordance with the overall distribution of data over time should not only be allowed, it should be technically mandatory.

ISI’s competitor Scopus now computes its own “impact factor” (González-Pereira, Guerrero-Bote und Moya-Anegón o. J.), and we do not want to scrutinize it in the same way as the more established indicator. However, we do wish to mention that for the 86 journals covered in both databases, the Pearson correlation between the 5-year impact factor and the journal ranking indicator of the competitor Scopus, SCImago Journal Rank (SJR), is 0.582 (significant at the 0.01 level). We take this as an additional piece of evidence arguing against the use of impact factors as valid measures for scholarly quality, at least not in sociology.

Problems with data reliability

If one checks the JCR of individual journals, one comes across missing data for the 5-year IF, but can find the raw data to calculate it by using what the JCR is offering at each journal’s site. The result is astonishing. For example, for the 5-year IFs: the Journal of Consumer Culture would outstrip the prominent American journals with a 5-year IF of 7.13, and the European Journal of Social Theory with its 3.33 would surpass the BJS. Why does Thomson Reuters omit particular data? The seemingly obvious answer that they excluded the 5-year impact factor because of a lack of data either for the “cites” or the “number of published items” is not true: other journals with similar gaps in their data are listed with their 5-year impact factor, e.g. Eastern European Countryside and Current Perspectives in Social Theory. Most probably Thomson Reuters eliminated the data because their announcement would obviously contradict the expectations of their users: it cannot be the case that a fringe journal such as the Journal of Consumer Culture is doing so much better than the established ASR and AJS. But can we then trust Thomson Reuters with regard to whatever else they deliver?

I used the Social Science Citation Index (ssci) to retrace the numbers for citations and published articles for several journals and the result is remarkable: the JCR data are nearly irreproducible from the source database ssci.^{Footnote 37} I tried to reproduce them manually but encountered difficulties with regard to the search options. The most relevant mismatch is the so-called timespan. The IF calculates the fraction “cites to recent items/number of recent items” for a single year only (“cites in [year]”), but the latter cannot be formulated as a search option in the ssci.^{Footnote 38}

In contrast, a closer look at the data behind the numbers published in the JCR reveals interesting additional information. Let us use the Russian Sotsiologicheskie Issledovaniya (roughly Sociological Studies) as an illustration: This journal is covered by the ssci at least since 2000, its IF is given for 2010 as 0.147 and the 5-year impact factor as 0.135, rank 119 and 93 respectively (the mean IF for the 12-year period is 0.144). Articles and reviews published in 2010 (in journals covered by the ssci) included 64 citations to articles and reviews that were published in Sotsiologicheskie Issledovaniya in the two previous years 2008 and 2009. In the same two years, this journal published an astonishing total of 436 articles and reviews. 64 divided by 436 results in 0.147. Scrolling down on the website where this information can be found, one can click on a link that brings us to the “cited journal data table” with the “number of times articles published in 2010 (in journals below) cited articles published in sotsiol issled+ [this is the acronym used by ISI] (in years below)”. There, one can detect that 47 of the 64 citations appeared in the very same journal. A self-citation rate of 73 % is an enlightening finding, to say the least. In the early years of the citation index business some experts suggested that all self-citations should be excluded from the computation of the impact factor. Later on they established the arbitrary threshold of one-fifth, and found that about 18 % of all journals demonstrate a self-citation rate above this threshold.^{Footnote 39}

The obvious conclusion would be to exclude journals whose self-citation rate is too high (irrespective of where one draws the line) from the reporting, but unlike the cases of those 5-year IFs, which were too high and thus had to be eliminated from the report manually, the distorting numbers located at the bottom of the ranks do not prompt Thomson Reuters to act.

Summarizing the major findings of this study I would like to draw attention to the following shortcomings of the citation indexing business and its flagship, the IF. They can be divided into two groups: deficits intrinsic to the procedures, and shortcomings significant for sociology in particular. The following criticisms do not imply that the whole WoS business is worthless, but that the production and dissemination of professed indicators such as the journal IF are contestable.

1. 1. The IF fraction has been developed inductively by using data from the hard sciences. No rationale for this particular formula has ever been offered, its diffusion is an intended side effect of its stubborn use, and after a while (nearly) everybody accepted it as a valid measure. The predominant inductivism is demonstrated by the regular proposition of new indicators.^{Footnote 40} Above all, the two-year “citation window” is haphazardly chosen and may be applicable only in well-integrated fields of research with a high degree of shared consensus about what research questions will be addressed next.

2. 2. The criteria for what counts as a citable document and which documents are called into play as sources for citations are anything but well-defined. The incongruence between numerator and denominator (a larger share above the line for citations and a smaller below for published items) operates as an invitation to abuse; for instance by publishing editorials (a non-citable type of document) with a huge number of self-citations (because WoS does not apply the same restrictions to the origins of citations).

3. 3. The measurement is highly “reactive”, and increasingly so. Authors, editors, publishers and other interested parties model their own behavior to influence the outcomes of bibliometric measurements. Its objectivity is at risk: supposed citation cartels, slicing reports into “least publishable units”, honorable authorship, and similar social techniques of scholarship are known even to those who never enter the field of scholarly publications. Codes of ethics and comparable provisions prove that deviant behavior exists, but its scope is still contested.^{Footnote 41}

4. 4. Since the WoS originated in the US, its coverage of non-American sources is still disputable, especially in disciplines lacking a well-defined cosmopolitan research agenda. Furthermore the decision about the inclusion of journals is opaque, to say the least. Even the founder of ISI, Eugene Garfield, acknowledges that it is a mixture of what he calls “quantitative and qualitative considerations”:

After the sale of Garfield’s ISI the inclusion policies became even more driven by factors beyond the alleged criteria of measuring the merit of examinees.

In case of the social sciences some additional peculiarities distort the adequacy of the WoS routines:

1. 1. The subdivision of the entire field is unwarranted. Disciplines of comparable status and development, such as economics and psychology, are represented highly unequally; no one could argue that psychology is double the size of economics, and it is also disputable that sociology is only half the size of economics. The broadening of the whole field by including particular applied disciplines and outlandish specialties, such as ergonomics, substance abuse, and transport, leads to the result that the population from where citations might come is autocratically hedged in.

2. 2. Given the lower level of an internationally agreed research agenda in the social sciences, often called fragmentation, the selection of journals is obviously more difficult and contestable at the same time. As a matter of fact, WoS’s selection does not follow any of their alleged principles, but implements the market capture strategies of the present owner, Thomson Reuters. Even according to their own standards the vast majority of covered journals should not be regarded as eligible for inclusion.^{Footnote 42}

3. 3. The scholarly practices of sociologists, and probably also of other social scientists, are not mirrored by the arbitrary routines with regard to the “citation window”. One only needs to browse any of our journal articles’ lists of references to observe that the vast majority of citations is beyond the two-year period crucial for the calculation of the impact factor. This already highly inductive criterion contradicts the routines of disciplines dissimilar to those in the hard sciences and should not be used.

4. 4. The often bemoaned “methodological nationalism”^{Footnote 43} is one more particularity of the social sciences and of sociology in particular. A consequence completely ignored by citation indexing is that sociologists from smaller countries are necessarily hampered in obtaining a comparable amount of recognition via citations. Even those journals that switched to English as their preferred language cannot overcome this restriction. Similar considerations apply for differentiated research specialties: in many cases the number of co-workers is much lower compared to more fancy specialties or areas with stronger international links.

The outcome of several days of data mining is, frankly, that the whole impact factor business proves Karl Marx’s findings on commodity fetishism from a long time ago:

The quite recently established regime of IFs is driven by the business concerns of two international corporations, Thomson Reuters and Elsevier, and accepted as the gold standard in today’s academic market by the newly emerging elite of university administrators and policy makers, using it whenever it fits their impression management strategies.

Table 1

Citation routines in selected sociological journals

Note:

American Sociological Review (ASR), British Journal of Sociology (BJS), and Kölner Zeitschrift für Soziologie und Sozialpsychologie (KZfSS), my calculations.