Bibliometrics are a group of methods used in the field of information science to explore and quantify citation distributions within a field of research, the impact of a journal, published article, researcher or set of researchers. Bibliometrics have a wide application in science and are frequently used to trace relationships among journal citations and to search forward in time for more recent publications which cite a known article. Making sense of the many bibliometric indicators currently available to evaluate research output is a hot topic as they are often used to compare applicants for grants, tenure or promotion. As with all quantitative methods and procedures, there are some fundamental rules or steps that should be followed when comparing research outputs of individual scientists so as to avoid their indiscriminate use which may disadvantage some if used improperly.
You cannot judge a paper by the journal impact factor
Similar to judging a book by its cover, you cannot judge the worth of an article from where it is published. Much has been written on the benefits and limitations of the journal impact factor (JIF) as a measure of quality of academic performance (Reference Garfield1). The JIF is defined as the number of citations to a journal's articles published in the previous 2 years divided by the number of articles by the journal during those 2 years. Thus, the JIF is a poor measure to assess the true merit of a particular article as it is not representative of individual articles, is prone to publication bias (some databases are US centric and favour English language journals), can be influenced by self-citations and is based on a short, 2-year, window (Reference Walter, Bloch, Hunt and Fisher2). Historically, the number of publications, average number of citations and sum of all citations were often used to evaluate the research output of an individual scientist. This method of evaluation did not take into account the large citation differences between articles as one high impact paper could account for the majority of citations for one individual (‘one hit wonder'), while another may have many lowly cited articles of the same number.
The h-index, advantages and disadvantages
In 2005, Jorge Hirsch introduced a new indicator for quantifying the research output of scientists which he named the h-index (Reference Hirsch3). The main advantage of the h-index is that it combines quantity (number of publications) and impact (number of citations) and is relatively easy to calculate once a full list of publications is retrieved using databases such as Web of Science (WoS), Scopus and Google Scholar (Reference Bakkalbasi, Bauer, Glover and Wang4). The h-index is the highest number of single- or co-authored articles by a researcher that have each been cited h or more times. Thus, an h-index of 20 means a researcher has at least 20 articles that have each been cited at least 20 times. However, there are several limitations of the h-index that need to be considered when comparing individual researchers; it is dependent on the career length (age bias), is field dependent (some fields obtain more citations than others), it does not take into account multi-authorships and favours industrious researchers who publish a stream of articles. To overcome some of these limitations, several modifications of the h-index have been proposed [for recent review on how to calculate an h-index and its variants see (Reference Hunt, Cleary and Walter5)]. Getting a realistic h-index for an individual is no easy task due to the difficulty in obtaining a full set of publications for an individual, especially those with common names, name changes or who publish in many fields.
Citation databases
There are a number of available databases to retrieve and track citations as already mentioned above. Selecting more than one database is important to capture the impact of an individual's lifetime publications to ensure that influential articles are not overlooked. One should also use several search features as most databases have many data entry errors (misspellings, wrong years, etc.) that become ‘orphan' to the original master publication. Needless to say, each of the databases used to retrieve citations has its strengths and weaknesses which include different journal sets and time coverage (Reference Bakkalbasi, Bauer, Glover and Wang4,Reference Jacso6).
Combining searches from WoS and Scopus seem to provide the best overall coverage of peer-reviewed material and ensures that researchers with lengthy histories of publications are not disadvantaged using Scopus, as citations prior to 1996 are not reported. As a basic principle, several bibliometric indicators should be used to measure research performance, which includes a full list of highly cited papers, the number for which the researcher was the first or the last (corresponding) author, total number of publications, citations, h-index (from each database) and other variants as required.
Comparing like with like
Differences in the h-index exist between disciplines and speciality areas that need to be considered when assessing the performance of researchers or journal impact (Reference Althouse, West, Bergstrom and Bergstrom7). As pointed out by Hirsch (Reference Hirsch3), scientists who attain an h-index of 40–60 over 20 years show outstanding achievement, while scientists working in non-mainstream areas will typically have lower h values. An h-index of 20 characterises a successful scientist and in some sub-specialties, such as nursing, an h-index of 10–14 indicated an excellent publication record (Reference Hack, Crooks, Plohman and Kepron8). It should also be noted that it normally takes 10–15 years for many articles published to reach their full citation potential (Reference Tsay9) which can be frustrating for those early in their career to attain a high h-index.
A final caveat that should be kept in mind is that no single indicator or number can provide a complete picture of a person's research performance and there are many factors that should be considered in combination when evaluating an individual's academic output (length of career, teaching and clinical loads and grant opportunities). For the reasons stated above, the h-index (or any of its variants) should only be used to compare researchers of a similar age and within the same field of study. It is also important to employ multiple searching methods using more than one database to gather citations and verify authorship of the articles whenever possible.
