There are concerns that there is a lack of recognition for applied disciplines (5), and a bias against clinical research in bibliometric assessments (Reference Narin and Hamilton10;Reference Raymond12;Reference Seglen14). Therefore, it is valuable to know whether publications of importance for improving the health system and its technologies are also highly cited intrascientifically. A way to explore this is to calculate bibliometric indicators for publications related to healthcare technologies that are deemed as very promising by clinical experts. In this study, this is done by assessing the “bibliometric impact” of publications used as sources in a Swedish system for identification and early assessment of new methods in health care developed by the Swedish Council on Technology Assessment in Health Care (SBU): SBU Alert. Furthermore, a general bibliometric analysis of the science base of SBU Alert is conducted. The analysis includes answering the questions of how old the used sources are, where the research was carried out, what countries are over- or underrepresented, in what journals the research was published, and to what subject categories the journals belong.
SBU Alert is an Early Warning System. The Alert program is based at the SBU and is a joint effort by several public organizations in Sweden. The Alert reports are intended to explain new methods and their effects in brief. Manuscripts are prepared in collaboration with appointed experts. The aim of Alert is to promote (i) the efficient introduction of new methods in health care, (ii) the view that the introduction of a new method should be preceded by a scientific assessment, (iii) communication between experts and nonexperts, and (iv) enhance the evidence base for setting priorities in health care. The target group for SBU Alert is primarily decision makers such as politicians, executives, and medical advisers (Reference Carlsson1).
MATERIALS AND METHODS
The bibliometric analysis is based on the citation indices (CI) produced by Thomson Scientific. (Certain data included herein are derived from the 1995–2005 Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index Tagged Data prepared by the Thomson Scientific Inc, Philadelphia, Pennsylvania, USA: Copyright Thomson Scientific Inc® 2006. All rights reserved.) Even though the CI have some important limitations, they still contain unique information that is relevant when conducting a bibliometric analysis.
Average citation rates differ for publications of different types (articles, reviews, letters, and so on), of different age (publication year), and published in journals within different fields (as defined by Thomson Scientific journal subject categories). Bibliometric indicators have been constructed that control for these factors (7;9;13;18). Two indicators were used in this study: item oriented field normalized citation score average (cfz) and item oriented field normalized logarithm-based citation z-score average (cfz) (Reference Lundberg7). In short, the average paper has a cf value of one. A cf value of 2 means that the paper has received two times as many citations as the average paper of the same type, from the same year, published in journals within the same subject category. The world average for cfz[ln] values in turn is 0, with the value 1 meaning that the logarithm of the number of citations received by a publication is 1 SD away from the average of the logarithmic citation rate for publications of the same type, from the same year, within the same field. The two indicators complement each other in so much that the ccf[in] is less sensitive to extreme values as compared to cf. The field categorization in both indicators is based on the ISI (Institute for Scientific Information) Subject Categories of journals produced by Thomson Scientific. Although limited, this is the only classification scheme that is readily available for evaluative purposes across scientific disciplines (Reference Lewison6;8;Reference Ugolini, Casilli and Mela15). In this study, it was set as inclusion criterion that each “normalization group” (publications of a specific type, from a specific year in journals within a specific category) should contain at least 100 publications, and the publications should on average have received at least one citation.
Normalized citation scores were calculated for all articles, letters, and reviews published since 1995 that have been used as sources in SBU Alerts published (or released in new versions) 2001–04. Within the fifty SBU Alerts a total of 953 references were given. To calculate normalized citation scores for the references, these first had to be matched with corresponding references in CI (Table 1). Six duplicate references were removed, then removed were 105 references to items published 1994 or earlier. Third, the 222 references to items not included in CI were removed. The most common reason for an item not to be included was that the scientific journal it was published in was not covered by CI (64 references). Other common reasons not to find items in CI were that they were reports (44), books (20), or Web sites (Reference van Leeuwen16)—item types that are not included in CI. This left 620 publications for which corresponding articles in CI could be identified. Almost all publications were journal articles (89 percent), reviews (7 percent), or letters (0.6 percent). Twenty-two publications were editorials, notes, meeting abstracts, or news items and therefore excluded from further analysis. The matching process left 598 publications for which normalized citation scores could be calculated. In the normalization procedure, two additional publications were removed. One was removed because there were only twenty-five publications in its normalization group (reviews from 2003 in integrative and Complementary medicine). The other publication was removed because the average citation rate for publications in its normalization group (letters from 2003 in general and internal medicine) was only 0.6. The final number of assessed references was thus 596.
Overview of the Process for Identifying Cited Publications and Matching Them with References in CI
CI, citation indices; SBU, Swedish Council on Technology Assessment in Health Care; ISI, Institute for Scientific Information.
To calculate whether a country was over- or underrepresented in terms of the number of coauthored publications being used as sources in SBU Alerts, an expected value was first determined. This was done by counting the number of distinct articles that authors affiliated with organizations in each country have published in journals belonging to fifty-seven subject categories from 1995 to 2004. The categories were selected using the subject categories of the journals in which publications used as sources in SBU Alerts were published. For each country, the expected value was then calculated by dividing the number of articles in the fifty-seven categories with the total for all countries. To compute the actual share of publications for each country, the number of distinct publications that each country has coauthored which are used as sources was divided with the total for all countries. As a last step, the over-/under representation was calculated by dividing the actual share of publications from a country with the expected share. To identify systematic reviews the titles of all references were manually examined for the phrases “systematic review” or “meta-analysis.”
RESULTS
The publications used as sources in SBU Alert are highly cited compared to publications of the same type, from the same year, or published in journals within the same field. On average, they are cited eight times more than the world average (cf = 7.79). The average logarithmic citation rate is 1.7 SD above the world average (![$\bar c_{fz[\ln ]}\,{=}\,1.69$](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054823092-0741:S0266462307080099:S0266462307080099_inline6.gif?pub-status=live){kind=small}
). The high bibliometric score is consistent over the whole assessed time period, with the highest value seen for publications used as sources in SBU Alerts published in 2003 (![$\bar c_f\,{=}\,10.3;\,\bar{c}_{fz[\ln ]}\,{=}\,1.9$](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054823092-0741:S0266462307080099:S0266462307080099_inline7.gif?pub-status=live){kind=small}
), and the lowest for SBU Alerts published in 2002 (![$\bar c_f\,{=}\,4.7;\,c_{fz[\ln ]} \,{=}\,1.4$](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054823092-0741:S0266462307080099:S0266462307080099_inline8.gif?pub-status=live){kind=small}
). Only 12 percent of all articles had received fewer citations than the world average within their field.
The sensitivity (or recall) of using positive normalized citation score to identify articles used as sources in SBU Alert is 0.88. Seventy-three percent of the publications had a logarithmic citation rate more than 1 SD above the world average. The distribution of publications over classes of ![$\bar c_{fz[\ln ]}$](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054823092-0741:S0266462307080099:S0266462307080099_inline9.gif?pub-status=live){kind=small}
is shown in Figure 1. Only two of the fifty SBU Alerts used publications as sources that on average have received fewer citations than the world average (Supplementary Table 1, which can be viewed online at http://www.journals.cambridge.org/jid_thc). No SBU Alert used sources with average logarithmic citation rate below the world average.
If the publications were grouped by the subject categories of the journals in which they where published, general and internal medicine was the most common category, followed by surgery, cardiac, and cardiovascular system, peripheral vascular diseases, and clinical neurology. In all categories with at least ten publications, the average citation rate was above the world average. In total, 216 journals belonging to 57 different categories were used as sources in SBU Alerts 2001–04.
The New England Journal of Medicine was the most frequently used source for information, followed by Circulation and the Lancet (Table 2). There was a considerable range in the impact factors of the journals.
Journals in CI, Being Cited at Least Five Times in SBU Alert Reports 2001–04
CI, citation indices; SBU, Swedish Council on Technology Assessment in Health Care; IF, Impact Factor 2005.
Distribution of publications over classes of cfz[ln].
Coauthors of the publications cited in SBU Alerts 2001–04 most frequently had an affiliation in the United States (31 percent) or the United Kingdom (12 percent). The figure for the United States was in line with the expected (0.9), while the United Kingdom was overrepresented by 1.6 times. The highest overrepresentation, according to the method used, was seen for Sweden (5.1 times more than expected), Denmark (2.4), and the Netherlands (2.3). Japan (0.2) and Taiwan (0.2) were the most underrepresented countries, of countries with at least one publications used as source in SBU Alerts. It could be noted that researchers affiliated with organizations in India and Russia had not been coauthors of any publications used as sources.
Twenty-two systematic reviews were identified, which represent 2.3 percent of the total number (953) of unique references found in the SBU Alerts. A serendipitous result was noted when looking at the grading of evidence in the SBU Alerts. In forty-six of the fifty Alerts (92 percent), the evidence for cost-effectiveness was graded as nonexisting or poor!
DISCUSSION
One way to assess the value of clinical research is to explore whether it is subsequently recommended for implementation into practice. This can to some extent be done by assessing the sources used in vehicles for identification and early assessment of new methods in health care, such as SBU Alert and other health technology assessment systems. Even though some sources might recommend that a technology is not put into practice or not yet, the vast majority of papers in SBU Alert could at least be considered as having some impact on policy makers. The publications that are used as sources here are cited eight times more often than the world average. This finding is a very high value considering that in assessments of research groups of similar size (∼50 publications per year) an average citation rate of three times the world average signals excellence (Reference van Raan17). There was no a priori theory on whether publications used as sources in a system for identification and early assessment of new methods in health care also should be highly cited, but intuitively this is to be expected because high-quality publications often are highly cited, and it is reasonable to believe that publications serving as basis for technologies judged as promising and important also are of high quality. It has been argued that there is bias against more clinically oriented research in bibliometric assessments based on the CI produced by Thomson Scientific (Reference Narin and Hamilton10;Reference Raymond12;Reference Seglen14). The bias would exist because the CI are based on citations given in scientific journals, and not on documents more closely related to patient care, such as guidelines or systems for identification of emerging technologies in health care. The results of this study contribute to this discussion. It is possible that the use of field normalized bibliometric indicators (used in this study) reduces some of the bias against clinical research earlier shown to exist when using, for example, impact factors.
Articles in a few high-impact general medical journals often served as the scientific basis for SBU Alert. However, several of the articles were published in field-specific journals with low-impact factors (Table 2). This finding argues against uncritical use of impact factors in assessment of clinical research. Impact factors may hide the very variations they are believed to display, and are also much dependent on the composition of document types in a journal and the subject category it covers (Reference Seglen14).
Most articles used in SBU Alerts were published by authors based in the United States or the United Kingdom. Publications from Sweden and from countries with similar health systems, such as Denmark and the Netherlands, were overrepresented as science base in SBU Alert. This finding is natural because local adaptation often is necessary when turning research into practice (Reference Øvretveit11). It is also in line with a study on the scientific basis of British clinical guidelines, which showed UK papers to be represented two and a half time more than expected (Reference Grant, Cottrell, Cluzeau and Fawcett3).
Evaluations based on data in the CI have some important limitations. For example, does CI not cover all journals, and are biased toward English language journals? In this study, 8 percent of the sources (published 1995–2004) used in SBU Alert were articles published in journals not covered by the CI. It is not unreasonable to believe that some researchers may publish considerable parts of their work in such journals. It is also worth noting that only a few HTA reports are published as journal articles (such as the UK Health Technology Assessment series). The results also show that the journals outside the scope of CI play an important role in medical research. This strengthens the argument that evaluative bibliometrics is less applicable in fields where internationally oriented scientific journals are not the main carrier of science communication (Reference van Leeuwen16).
The technologies and methods being reported in SBU Alert still are at an early stage. Thus, the results with few systematic reviews being used in SBU Alert are more natural than when analyzing the evidence base of clinical guidelines (Reference Chalmers2;Reference Grant, Cottrell, Fawcett and Cluzeau4). The finding that evidence for cost-effectiveness is most often lacking at the time of assessing a technology in SBU Alert reconfirms the early phase of the technology diffusion.
POLICY IMPLICATIONS
This bibliometric analysis showed that the health technology publications deemed promising by clinical experts were also highly cited within the scientific community, underlining the usefulness of the method. The policy implications of this study are twofold. First, bibliometric indicators may be utilized in research assessments, even in more applied (or clinical) areas. Second, even though Alerts are usually written quite early in the stage of the technology's development, Alerts sources are very highly cited, which suggests that decision makers are getting scientifically sound advice.
CONTACT INFORMATION
Jonas Lundberg, PhD (lundbergphd@gmail.com), Researcher, Mats Brommels, MD, PhD (mats.brommels@ki.se), Professor, Medical Management Centre, John Skår, PhD (john.skar@ki.se), Professor, Medical Management Centre, Karolinska Institutet, Berzelius väg 3, 17177 Stockholm, Sweden
Göran Tomson, MD, PhD (goran.tomson@ki.se), Professor, Division of International Health Academic Institution: Karolinska Institutet, Nobels väg 9, 17177 Stockholm, Sweden
The authors acknowledge the very valuable feedback given by editors and anonymous reviewers in the preparation of this manuscript.
