It is increasingly important for research commissioning organizations to demonstrate the value of their research investment. This is principally due to three reasons: the need to justify existing levels of research, the need to meet accountability requirements, and to inform and improve the research commissioning processes (Reference Hanney, Buxton, Green, Coulson and Raftery1). Measuring the use and impact of funded research is highly complex, however. Changing policy, healthcare, and research environments and a limited understanding of the mechanisms of knowledge transference from published research to clinical practice or policy generates significant challenges for any assessment of research use (Reference Druss and Marcus2;Reference Ward, House and Hamer3).
Various models of uptake of research evidence have been proposed, the most frequently adopted being the “Payback” model developed by the Health Economics Research Group at Brunel University (Reference Buxton and Hanney4). This model has been particularly influential due to its wide application and recognition that measures should go beyond assessing academic knowledge transfer to include the uptake of research evidence in other areas, such as informing policy, changing health practice, and broader health economic impact.
Specific tools have been developed to assess the use of research findings. Bibliometric analyses, for example, have been used to assess the academic knowledge transfer of research evidence through an assessment of publication counts, journal impact factors and citation rates. Druss and Marcus (Reference Druss and Marcus2), for example, explored the publication output of National Institute for Health (NIH) grants, reporting an average of 7.6 publications per grant. More sophisticated bibliometric analysis has been developed by Hanney et al. (Reference Hanney, Frame and Grant5), where the number of times a study was cited was assessed as well as the nature of citation (e.g., whether the cited study supports the conclusions of the article) and the relative importance of the citation (e.g., whether the cited study is peripheral or essential to the article).
Postal, telephone, and online surveys have frequently been used to examine the uptake of research evidence. These have often been completed by Principle Investigators of studies to self-report evidence of research impact, such as publications, conference presentations and guideline uptake (Reference Hanney, Buxton, Green, Coulson and Raftery1). Hailey (Reference Hailey6) used questionnaires to identify evidence of uptake of evidence from rapid reviews conducted by INAHTA agencies.
Another commonly applied approach for assessing research use are qualitative interviews. These are often conducted with Principle Investigators and end users of research projects to elicit information on the nature of uptake of research evidence. Campbell et al. (Reference Campbell, Redman and Jorm7), for example, interviewed policy makers from the New South Wales Department of Health Policy Development Committee and researchers to explore the use of research in policy, current practice in research dissemination, and the level of communication between researchers and policy makers. The study reported that policy makers rarely used research to inform policy agendas or to evaluate the impact of policy (Reference Campbell, Redman and Jorm7). Ritter (Reference Ritter8) similarly explored the use of research evidence in drug policy. She conducted group interviews with drug policy makers in health and policing and found that an expert and technical reports were the most frequent source of obtaining research evidence.
Whereas these approaches have generated important evidence on the nature and scale of use of research evidence, few studies have explored the uptake of research internationally. This is significant as the use of research evidence in informing policy and practice decision making in other countries is an important means of reducing potential waste in research commissioning. Evidence has shown, for example, that duplication in HTA commissioning activity can occur where the findings from completed studies are not used internationally (Reference Turner, Adams, Cook, Price and Milne9).
The NIHR HTA Programme previously undertook a preliminary assessment of international uptake of findings from research commissioned by the Programme by examining citations using ISI Web of Knowledge and Google Scholar and analyzing Webtrend data of visits to the HTA Programme Web site. This revealed that the United States was the country accessing and downloading reports the most after the United Kingdom (Reference Hanney, Buxton, Green, Coulson and Raftery1). This study presents findings from a study that builds on this earlier research and seeks to explore the merits of different methods for assessing the international use of HTA reports funded by the NIHR HTA Programme.
METHODS
The study adopted an exploratory design, using three core methods. The exploratory approach was appropriate given that little is known about international use. Multiple methods were selected as research use is typically multi-dimensional and hence different methods were used to explore different areas of use. The design of the project was informed by the Payback model of research impact (Reference Buxton and Hanney4). This model was selected due to its wide application in assessing the impact of research funding programs and its recognition that measures should go beyond assessing academic knowledge transfer to include other areas, such as informing policy and changing health practice.
The study focused on three areas of use. These were selected to explore a range of examples of research uptake, including academic use, HTA use, and Internet use. Academic use was identified through bibliometric and citation analysis of the top ten most cited HTA reports from research funded by the NIHR HTA Programme. Citation analysis of reports was possible as HTA reports that are funded by the NIHR HTA Programme are published in the journal series Health Technology Assessment. The ten most cited reports were selected on the assumption that they would be more likely to reveal evidence of international uptake of research findings and thus provide good examples through which the bibliometric approach could be examined. The list of the top ten cited reports was created in January 2011 using Web of Science and Medline (Ovid). Analysis included: the total number of publications arising from HTA reports; the impact factor of journals in which papers were published; the total number of citations; the total number of countries hosting journals in which reports are cited.
Internet use was assessed using Webtrends software to determine the top ten most downloaded HTA reports from January 1, 2004, to June 30, 2010. Webtrend reports were used to determine the proportion of international visits for each report.
International NIHR HTA use was explored by examining the Center for Reviews and Dissemination (CRD) HTA database. This database is produced in collaboration with the International Network of Agencies for Health Technology Assessment (INAHTA) and contains completed technology assessments. A specific search was made on the subject areas of the ten most cited NIHR HTA reports. Methodology reports were excluded due to the challenges of searching the CRD HTA database in broad subject areas such as “guidelines” or “qualitative research.” Consequently the CRD HTA database was searched in the broad thematic areas of asthma, eczema, Down's syndrome, and obesity. All reports identified in the CRD HTA database in these thematic areas were examined to see if the selected NIHR HTA Programme funded reports had been cited in other non-UK HTA reports.
All types of NIHR HTA reports were included in the study, including primary studies (typically randomized controlled trials), evidence syntheses (mostly systematic reviews), methodology reports (typically reviews of methodological issues such as qualitative research design), and TARs (Technology Assessment Reports, which identify, assess, and synthesize research evidence from several healthcare interventions, providing estimates of relative effectiveness and cost-effectiveness of a range of interventions). All report types were included to enable comparisons to be made when applying tools for assessing research use across different types of study.
RESULTS
Bibliometric Analyses
Table 1 shows the top ten cited NIHR HTA reports identified in January 2011. The table reveals all but one report had in excess of 100 citations. The top cited report was Grimshaw et al. (2004), who had in excess of 400 citations. It was noted that the majority of the most highly cited reports were methodological reports.
Top Ten Cited HTA Reports Identified via Web of Science and Medline (Ovid), January 2011
*HTA reports selected for additional analysis of international HTA report uptake through the CRD HTA database.
HTA, health technology assessment; CRD, Center for Reviews and Dissemination.
Bibliometric analyses were conducted on the top cited NIHR HTA reports. Book chapters, editorials, press releases, and conference abstracts were excluded from the analysis. This revealed that the most cited HTA report also had the highest number of publications of the top ten reports with eight peer reviewed publications. The average number of published journal papers for the top ten HTA reports was two publications. The average journal impact factor for those journals that had an impact factor was 5.22.
Table 2 illustrates the international distribution of editorial bases for journals in which papers citing the ten most cited reports are published. This provides a basic indicator for the level of international uptake of findings through peer reviewed publications. Table 2 reveals that 41 percent of the 549 citing journals are based in United States. This is followed by England with 35.5 percent of journals, other European countries with 15.7 percent of journals and other (non-European) countries with 7.8 percent of journals. It is notable in this regard that the majority of citations of the most cited HTA reports were from countries outside England. This analysis reveals the potential for bibliometric data to identify international academic knowledge transfer.
Countries of Editorial Bases of Journals in which Papers Citing the Ten Most Cited NIHR HTA Reports Are Published
NIHR HTA, National Institute of Health Research Health Technology Assessment.
Webtrend Analyses
Table 3 shows the top ten downloaded HTA reports for the period January 1, 2004, to June 30, 2010, and compares the proportion of United Kingdom to international visits. Visits are defined as the number of times a whole report is accessed by means of the Internet as opposed to the number of times each Web page related to a specific project is accessed. This revealed that all but one report had over 50 percent of visits from outside the United Kingdom. For three reports—Chen et al. (Reference Chen, Jobanputra and Barton10), Dinnes et al. (Reference Dinnes, Deeks and Kunst11), and Hobbs et al. (Reference Hobbs, Thorpe and Delaney12)—in excess of 80 percent of visits were outside the United Kingdom. Whereas it is not possible to generalize levels of impact across all HTA reports, certain observations can be made from the data. It is interesting to note, for example, that the two lowest internationally visited reports were methodological reviews—Murphy et al. (Reference Murphy, Dingwall, Greatbatch, Parker and Watson13) and McColl et al. (Reference McColl, Jacoby and Thomas14)—whereas evidence syntheses had higher levels of international visits. Webtrend analyses did therefore identify an international dimension in evidence uptake through Internet use.
International Visits of Top Ten Downloaded NIHR HTA Reports: January 1, 2004 – June 30, 2010
NIHR HTA = National Institute of Health Research Health Technology Assessment.
HTA Report Analysis
Five NIHR HTA Programme funded primary research and evidence syntheses reports identified from the ten most cited reports identified in January 2011 were examined for citation by another international HTA agency listed on the CRD database (Table 1). The CRD database revealed that one report selected was not cited by another HTA report. The four remaining reports were cited collectively twenty-eight times in other HTA reports, eighteen of which were by international HTA agencies. Canada was the country citing NIHR HTA funded research reports the most with five citations. Avenell et al. (Reference Avenell, Broom and Brown15) was the report cited most internationally with ten international HTA agencies citing the publication. Three HTA reports were identified from New Zealand agencies citing NIHR HTA reports. Two HTA outputs citing NIHR HTA reports were identified from HTA agencies in each of the following countries: Germany, Sweden, and the United States.
Analyzing the CRD HTA database was therefore useful in revealing evidence of international uptake in other international HTA agencies.
DISCUSSION
This study has revealed that methods often used for research impact assessment are useful in generating interesting data on international uptake of HTA findings. There are limitations to this study, however, that should be considered when assessing the implications of the research findings. It is important to note that not all Payback domains of research uptake were selected for assessment. Certain domains, such as the influence of HTA findings on health policy or practice, were not supported in this study due to issues of feasibility. Another limitation to the study was the use of U.S.-based databases for assessing citations, which may have generated a U.S. bias in the citation analysis of reports.
A further limitation was the fact that HTA reports were selected where international use was more likely to be detected to enable methodological value to be assessed. This was appropriate as the study aimed to explore the appropriateness of strategies for detecting international use, although it did mean that it was not possible to generalize the level of international use across the HTA program. Similarly, the study assumed that international uptake would be detected from highly downloaded or cited NIHR HTA reports. Certainly evidence of international use was found in the reports selected for analysis and thus served as good examples to assess the methods for exploring international use. However, it is recognized that it cannot be assumed that the most cited or downloaded reports would necessarily be the most internationally used. Consequently, it is not possible to infer that the reports selected for this study would be the most used internationally in the NIHR HTA Programme.
This study contributes to the literature on the uptake of research evidence by demonstrating the potential application of certain approaches in assessing international use. All methods used in this study have been applied previously in studies exploring the impact and uptake of findings from funded health research (Reference Hanney, Buxton, Green, Coulson and Raftery1;Reference Seif and Trope16). However, these studies typically used these methods to explore aspects of use within the country in which the research was conducted, rather than assessing international uptake. The international use of HTA findings and research evidence more broadly is an important concern both for enhancing research impact and in reducing waste during research commissioning by minimizing duplication (Reference Turner, Adams, Cook, Price and Milne9).
Each of the methods assessed in this study revealed useful information concerning international uptake. Bibliometric analyses revealed evidence of international transfer in the uptake of evidence from HTA reports, most notably through citations where a U.S. dominance of editorial bases of journals in which papers cited NIHR HTA reports was identified. This supported evidence reported previously on the international uptake of NIHR HTA funded research through citation analysis (Reference Hanney, Buxton, Green, Coulson and Raftery1). Webtrends and HTA report citations similarly provided evidence of the transfer of HTA findings internationally. Citation analysis also revealed a relatively high proportion of methodological reports that were highly cited, which potentially indicates their usefulness in informing the methodologies of knowledge producers and others.
There were limitations to each approach to assessing international uptake of HTA report findings, however. The limitation of the bibliometric approach is that it provides an indicator of international uptake without exploring the nature of use. As Wallin (Reference Wallin17) and Hanney et al. (Reference Hanney, Frame and Grant5) commented, citation analysis does not necessary equate with quality and the context in which citation occurs is not revealed. Webtrends similarly are helpful in indicating levels of international visits, but do not with any rigor identify who visited the HTA report Web pages or the nature of the reports’ subsequent use. The same limitation applies to the citation of NIHR HTA reports in reports from other international HTA agencies where the nature of citation or the level of influence is not assessed.
One observation was that the methods used for assessing research use were very sensitive to the timeframe over which the analysis was conducted. It is important, therefore, that start and end dates for citation analyses and Webtrend analyses are selected with care as these will have an influence on the relative scale of research use. A further observation is that the NIHR HTA Programme is unusual in having a journal series in which final reports are published, thus enabling bibliometric, Webtrend, and CRD database assessments of international use to be made. Opportunities to undertake this analysis are more limited for those agencies that do not publish full access reports. In addition, the international uptake of research findings is limited further for those HTA agencies that do not publish findings in English.
Further areas of research are indicated from this study. As the study explored only certain areas of international use, it is recommended that further research is conducted exploring the international influence of HTA findings on health service delivery, design, and policy. In addition, as the methods used in this study generated indicators of international uptake of findings, it is recommended that further research is conducted that uses a case-study approach to explore the nature of HTA use.
The analysis of editorial bases of journals in which peer reviewed publications cited NIHR HTA reports did provide an indicator of international knowledge transfer. However, it is recommended that further research is conducted applying more developed measures for international academic use, such as exploring the country of first author of the citing study. Finally, it is recommended that research is undertaken that explores the mechanisms by which international uptake of findings is supported. Findings from this research would yield important information that can be used to enhance the international uptake of research evidence.
CONCLUSION
Assessing international use is an important consideration when exploring the uptake of HTA evidence. Methods commonly used in identifying research impact, such as bibliometrics and Webtrends, are helpful in generating evidence of international use. HTA agencies should consider the international uptake of findings from research they commission and / or undertake. Techniques identified in this study for assessing international use should be considered by HTA agencies where feasible.
CONTACT INFORMATION
Dr. David Wright, Senior Research Fellow, National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre (NETSCC), University of Southampton, Alpha House, Enterprise Road, Southampton SO16 7NS
Prof Ruairidh Milne, Director of the Wessex Institute and Head of NETSCC, Wessex Institute, University of Southampton, Alpha House, Enterprise Road Southampton SO16 7NS
Alison Price, Information Scientist, National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre (NETSCC), University of Southampton, Alpha House, Enterprise Road, Southampton SO16 7NS
Nicola Tose, Senior Programme Manager, National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre (NETSCC), University of Southampton, Alpha House, Enterprise Road, Southampton SO16 7NS
CONFLICTS OF INTEREST
The authors report they have no potential conflicts of interest.
