The purpose of health technology assessments (HTAs) is well established as a form of input used in decision making in the area of health technology. The format and content of HTAs vary, however, because HTAs are generally defined by their purpose, not their methods (9;14), and because the methods that are used are applied in the varying local contexts of different countries (6;7). Despite this finding, pressure is mounting to bring about a degree of conformity in HTAs performed in different countries as a means of increasing comparability and avoiding duplication of work (3;4;18). Examples of international collaboration include the checklist of HTA methods and content developed by International Network of Agencies for Health Technology Assessment (INAHTA) to specify minimum requirements for HTAs (9), the EUR-ASSESS project (2), and the ECHTA/ECAHI project (11).
One aspect of HTAs on which there is no general agreement is the presence and extent of recommendations. Authorities in some countries demand that specific recommendations are included in HTAs, whereas others such as the Canadian authorities do not want recommendations. Still others ask for conclusions based on the evidence but do not want these formulated as recommendations. These differences in preferences have been present since the early days of HTAs (20) and are restated in both the EUR-ASSESS project and in the INAHTA Checklist, where the need for clear conclusions is expressed but the issue of recommendations is left up to the individual country (8;9;14). There are multiple reasons behind the differing views on recommendations. Some argue for the inclusion of recommendations on the grounds that the most knowledgeable people in the field provide them based on the evidence and that they conveniently summarize the results of an HTA and ready them for use in decision making. Others argue against recommendations, preferring to leave it up to decision makers to interpret the evidence and formulate conclusions and recommendations to accommodate a given political context, country-specific or regional conditions, or other non–evidence-based circumstances. Still other arguments focus on the differences between the way in which HTA producers and HTA users reason or formulate their views, that is, on the gap between science and policy making. In light of these differences of opinion from country to country, one might wonder what the implications are of including or not including recommendations in HTAs, if indeed there are any.
No such empirical study on possible explanatory variables and pssible consequences of including recommendations in HTAs exists in the literature. Studies that do analyze recommendations in HTAs deal with the frequency of such recommendations. Four comparative studies (6;7;17;22) and five single-country studies conducted in Canada and Australia (10;13;16;19;21;23) have shown that the frequency of recommendations differs from country to country, varying between 15 percent and 50 percent. Other studies deal with policy recommendations at different levels (national, regional, hospital) or with recommendations conveying different signs (positive, neutral, negative). Yet none of these uses statistical analysis to shed light on the possible implications of including or not including recommendations in HTAs. Most of the studies discuss the arguments and rationale for including recommendations or not and the close connection between HTAs and policy making but at a rather general level (1;15;17).
METHODS AND DATA
The findings presented in this study stem from an earlier, broader study of the content and applications of HTAs worldwide. The methods are described elsewhere (6). In short, the study was designed as a structured literature review of 433 HTAs published by eleven HTA institutions or agencies in nine countries between 1989 and 2002. All the institutions were national and non-profit, and they all had a fairly large experience in producing HTAs. All were full HTAs, written in English or one of the Scandinavian languages, and available online or by written request.
The present study focuses on the recommendations in the HTAs included in the study. The HTAs were categorized into four groups: three for HTAs with different types of recommendations and one for HTAs with no recommendation. A Policy recommendation group was defined for HTAs that included an explicit statement concerning recommended policy changes in light of the results of the HTA. A research recommendation group was defined for HTAs that included an explicit statement that further research was recommended. An Other recommendations group was defined for HTAs containing any explicit statement that did not recommend a particular policy change or further research but did recommend another kind of action, such as the need for further clarification. Finally, a No recommendation group was defined for HTAs that did not contain any explicit statement of recommended action on the basis of the HTA. The assessment of recommendations in the HTAs was carried out very strictly, meaning that the recommendation had to be precise and obvious, leaving with no doubt about its actual presence. If not, it was coded as No recommendation.
The presence or absence of recommendations in the international sample of 433 HTAs was statistical analyzed for possible explanatory variables using logistic regression. We conducted two different analyses: one for policy recommendations and one for research recommendations. The third and fourth groups, Other recommendations and No recommendation, were not analyzed separately: Other recommendations because there were only very few observations (2 percent of all observations corresponding to 9 observations) and No recommendations because the results of this are the reverse of the results for Policy recommendations and Research recommendations and cannot add new information. Furthermore, Other recommendations is probably the least interesting group of recommendations.
Table 1 shows the characteristics of the HTAs that were used as covariates in the logistic regressions analysis. We also included four additional covariates in the analysis. Each of these represented one HTA aspect: the clinical aspect, the economic aspect, the organizational aspect, or the patient aspect (6). The clinical aspect was included as a score between 0 and 6, where an HTA scored one point for the inclusion of each of the following elements in the assessment: efficacy, safety, effectiveness, other outcome measures, indications for use of the technology, and the relevant target population. The mean score in the sample was 4.06 (SD=1.14). The economic aspect was included as a score between 0 and 5 and measured how many of the following items an HTA considered: efficiency, cost, cost-effectiveness, cost-utility, and/or cost-benefit. The mean economic score was 1.26 (SD=1.17). The organizational aspect was included as a score between 0 and 7 and measured the number of the following items included in an HTA: diffusion, centralization/decent, utilization, accessibility, skills/routines, education/training of personnel, and other organizational parameters. The mean score on the organizational aspect was 1.23 (SD=1.55). The patient aspect was included as a score between 0 and 5, where one point was given for the consideration of each of the following items: social impact, ethics, acceptability, psychological reactions, and other patient parameters. The mean patient score in the sample was 1.04 (SD=1.36).
For each group of recommendations, we analyzed using two models—a full model containing all covariates, followed by a reduced model, containing only those variables that were significantly related to recommendations in the full model (at a 10 percent level of significance). The reduced model was estimated using backward selection, that is, successively excluding the covariate with the highest test probability (p value) for the hypothesis that the coefficient of the covariate equalled zero until all p values were less than 0.10.
RESULTS
Table 2 shows the results of the logistic regression analysis divided into the two groups of recommendations.
Policy Recommendations
Type of assessor, type of technology, and inclusion of a comparison technology had no significant impact on the likelihood of stating policy recommendations in the 433 HTAs. In the reduced model, HTAs published between 2000 and 2002 had a lower likelihood of stating policy recommendations than HTAs from the other periods of publication, a likelihood that was 64 percent lower (1 minus 0.36) for 2000–2002 compared with 1989–1999. The most powerful indicator of the presence of policy recommendations in an HTA was the country of origin. Canada was used as the reference, and the results show Australia, Denmark, The Netherlands, Norway, Sweden, and Great Britain as significantly different from Canada, in both the full and the reduced model. For Australia, the likelihood of recommendations in HTAs was forty-four times higher than for Canada; for Great Britain it was approximately twenty-five times higher, and so forth. The Scandinavian countries were 6 to 10 times more likely than Canada to include policy recommendations in their HTAs, whereas HTAs from New Zealand and the United States did not differ significantly from the Canadian HTAs. The scope of the HTA reports, represented by number of aspects and number of dimensions in the aspects assessed, shows that the aspects of technology and patient did not have any significant impact on policy recommendations but that the economic and organizational aspects did, although this impact was mild compared with the country impact. However, the more comprehensively these two elements were assessed in the HTAs, the higher the likelihood of policy recommendations (between 1.22 and 1.43 times), depending on which of the two models was used.
Research Recommendations
For research recommendations, the type of assessor was a significant explanatory variable in both the full model and the reduced model. Compared with HTAs written by personnel at the HTA institutions themselves, HTAs written by external partners and in collaboration with external partners significantly raised the likelihood of including research recommendations in HTAs by 3.3 to 8.7 times. The type of technology assessed and the use of a comparative technology did not have a significant impact on the likelihood of stating research recommendations in the HTAs However, the 1997–1999 publication period was associated with a 2.13 times higher likelihood of stating research recommendations in HTAs, compared with the other periods.
The country of origin did have a significant impact on the presence of research recommendations, with Great Britain as the most striking example. In the reduced model but not in the full model, all countries but the United States and Norway showed a significantly higher likelihood of including recommendations for further research than Canada did. The likelihood for Great Britain was 156 to 187 times higher than for Canada.
The scope of HTA reports also had a certain impact on the likelihood of including research recommendations. Likelihood was increased when the technology or the patient aspect was analyzed more comprehensively (by factors of around 40 percent), and it was reduced when the economic aspect was analyzed comprehensively (by a factor of 33 percent). The scope of the organizational aspect did not have an impact on likelihood of including research recommendations in HTAs.
DISCUSSION
The results of this study show that the inclusion of policy and research recommendations in HTAs differs strongly from country to country. The content and scope of HTAs have some impact on recommendations. Extensive assessment of the economic aspect and the organizational aspect increase the likelihood of including policy recommendations. Extensive assessment of the technology and the patient aspects increases the likelihood of including research recommendations, whereas the economic aspect is negatively related to research recommendations. The most striking result is that the use of external partners for assessment increases the likelihood of including research recommendations in HTAs but not policy recommendations.
The strong association between country of origin and inclusion of recommendations confirms the sovereignty of the individual countries in formulating principles of HTAs and shows that there is no unambiguous best practice in the area. Almost all countries show a higher likelihood of including policy and/or research recommendations than Canada does. This finding is probably because the mandate of the Canadian national HTA agency Canadian Coordinating Office for Health Technology Assessment is to provide information but not to make recommendations (19). New Zealand and the United States are the only countries that do not differ significantly from Canada, which means that HTAs from these two countries do not include recommendations to a great extent. Great Britain, on the other hand, differs greatly from Canada, although to a greater extent for research recommendations than for policy recommendations.
The results for the four different HTA aspects shows that the more comprehensive and complete an assessment of the economic and organizational aspects are in terms of the number of elements assessed, the greater the likelihood that the HTA will refer to specific policy that concerns the technology in question. Further research is more often recommended when the technology and the patient aspects are assessed extensively, and this suggests a lack of knowledge, not necessarily on these two aspects, but sometimes also on other aspects as well. The opposite is the case for the economic aspect in relation to research recommendations; the more extensive an assessment is in terms of the number of elements assessed, the lower the likelihood that research recommendations will be included. This finding indicates that the more completely the economic aspect is assessed, the lower the likelihood that a need for further research will be expressed.
The inclusion of policy recommendations was not determined by the type of assessor—internal, external, or a combination of both—but research recommendations were. When external assessors are involved in an HTA, the frequency of stating research recommendations rises. It might be because outsourced HTAs concern technologies that are less studied in the literature or due to the assessors' own incentives. The latter explanation would depend on who the external assessors are. If they are researchers, academic merit might be an incentive for recommending further research. The data cannot confirm any of these possibilities, as the group of external assessors was not divided into subgroups. A Danish study might shed some light on this question, inasmuch as 60 percent of applicants for grants for conducting HTAs on behalf of the Danish National HTA agency came from clinical departments and general practices (12). Furthermore, a recent study shows a significant relation between external assessors and use of assessment methods such as meta-analysis, surveys, economic evaluations, and randomized clinical trials (4). This clearly shows that researchers have an interest in and actually conduct HTAs. On the other hand, one has to remember even though the purpose of an HTA is to serve as an input in decision making, recommending further research is a relevant finding, because it points to areas in which knowledge is lacking (3).
The results show a declining tendency to state policy recommendations over time, and a higher tendency to include research recommendations in 1997 to 1999, compared with the other time periods. This might be explained by shifts in the objectives of and underlying policy questions posed in the HTAs and in the policies of the HTA commissioners, assuming that outputs from HTAs correspond to the phrased policy question. If these questions have changed from being more policy oriented to more research oriented, the relative distribution of policy and research recommendations might have changed in the same direction. Another explanation might be related to the assessors of HTAs. A previous study showed a major shift from internal assessors in the early time period to external assessors from 1997 to 1999 and then toward joints assessments in the latest period (5). These results correspond very neatly if we assume that researchers are more inclined to recommend further research, whereas HTA agency staff are more inclined to state policy recommendations. In this study, the type of technology assessed and the question of whether comparative technology was used did not have any consequences for policy or research recommendations.
Some restrictions apply to this study. First, the study only included a selection of HTA institutions and agencies worldwide, omitting all regional and local institutions and all reports not written in English or the Scandinavian languages. The sample cannot be seen as a total sample of HTAs worldwide. Furthermore, the data are based on a structured literature review of the actual HTA reports, and as no internationally uniform style or methodology for such reports exists, it might be claimed that the conclusions cannot be objective. The risk of subjectivity is reduced, however, by the reviewing process, as one person reviewed all the reports and, thus, can be assumed to have applied the same rationale and logic to all the HTAs.
It should also be noted that the term “recommendation” is used in a very broad sense, with no differentiation of the content of the recommendations other than the basic types listed above. Whether a recommendation is phrased positively, negatively, or somewhere in-between is not visible in this study, which has only considered the presence or absence of recommendations. The analysis, thus, only deals with recommendations in the HTAs in general terms and does not provide detailed information on the specific content of the recommendations.
Finally, this study analyses a limited number of possibly related variables concerning the inclusion of recommendations in HTAs, and other variables might also have an impact, as a variety of factors can be seen as determinants for policy questions and recommendations in HTAs. We limited ourselves to those we found most interesting. Others might suggest different variables.
POLICY IMPLICATIONS
The policy implications of this study are mostly relevant for the HTA institutions and agencies, decision makers, and funding authorities, inasmuch as the recommendations they receive from HTAs are influenced by the choices they make in advance. Outsourcing HTAs to external partners results in a greater likelihood of research recommendations, and the scope and content of HTAs has an impact on the type of recommendations Most strikingly, however, the individual countries have their own preferences, and these preferences amount to a very strong predictor.
CONTACT INFORMATION
Eva Draborg, PhD (eud@sam.sdu.dk), Assistant Professor, Christian Kronborg Andersen, PhD (cka@sam.sdu.dk), Associate Professor, Institute of Public Health—Health Economics, University of Southern Denmark, J.B. Winslowvej 9, DK-5000 Odense C, Denmark
