Health technology assessment (HTA) has been used as input in decision-making processes in the health care sector for more than 25 years. The HTA method has experienced a growing dissemination worldwide, spreading from North America to other continents, and has spread to both national and regional institutions, to private non-profit, private for-profit institutions, and to universities and other research areas (1;3;14;22;27).
Originally, HTA was introduced as a method for producing evidence on the basis of synthesizing existing evidence, primarily in the form of literature reviews, perhaps supplemented with primary research for country-specific data (6;14;24;25). Studies confirm literature reviews as the most prevalent assessment method in HTA, but also show that HTAs include a wide range of other assessment methods (7;12;15;19;23;30).
Over time, there have been no major developments in the HTA methodology per se in an international context. However, there seems to have been a shift from HTAs that include economic evaluations toward HTAs that include randomized controlled trials (RCTs) and modeling/evaluation studies. That is, the fraction of HTAs that use economic evaluation techniques as an assessment method has decreased, whereas the fraction of HTAs that use RCT and modeling/evaluation studies as assessment methods has increased over time as shown by Draborg and Gyrd-Hansen (7). Compared with a Canadian study, this finding might seem surprising, in as much as the Canadian study showed the opposite results, namely an increase in the number of HTAs with economic evaluation included (20;21;23).
Furthermore, there has been a shift in the type of assessed technology and in assessors during the period from 1989 to 2002 (7). Whereas the fraction of HTAs that assess procedures has been relatively constant over time, there has been a shift from devices to pharmaceuticals during this period. Additionally, the relative fraction of HTAs produced by personnel at the national HTA institutions has decreased greatly toward external partners alone or in cooperation with the HTA institutions' own personnel.
The objective of this study is to analyze statistically a possible relationship between the methods of assessment used in practical HTAs and the type of assessed technology, type of assessors, and year of publication of the HTA reports. No such analysis has appeared previously in the literature, where most studies are of a descriptive nature, analyze the parameters separately, and are mostly restricted to fewer HTA institutions and certainly to fewer countries. Nor is there any mention of this type of analysis in the methodological HTA literature (17;29;30), and it is only sporadically touched on in the International Network of Agencies for Health Technology Assessment (INAHTA) Checklist for HTAs (13).
Three studies precede the current study and have an international focus but are of a descriptive character (7;8;28). Of international studies concerning comparison of HTAs between countries, García-Altés et al. confined their study population to four countries and presented descriptive results (11) and Perry et al. (26), Perry and Thamer (27), and Mears et al. (19) relied on international surveys among a variety of HTA institutions.
Lehoux et al. (16) did a literature review restricted to 187 Canadian HTA documents from 1995 to 2001 but included other types of HTA documents too. Furthermore, they applied a different grouping of type of technologies not comparable to the one presented in this study. Another Canadian study reviewed 117 HTAs from four agencies and used a comparable grouping of technologies but also presented their results in a descriptive way (20;21;23).
METHODS AND DATA
We used data from a review of 433 HTA reports published in 1989–2002. The methods for retrieving these reports have been described in detail elsewhere (7;8). Briefly, a structured literature review of HTA reports published from 11 HTA institutions in nine countries was performed. The HTA institutions were all national, non-profit, and non-commercial and had a fairly large level of experience in the field of producing HTAs. The institutions were traced from member lists from INAHTA and International Society of Technology Assessment in Health Care (ISTAHC). HTA reports were included in the review if they constituted a full HTA in accordance with the individual institution's own characterization and by the first author's judgment of the reports. Furthermore, the HTA reports had to be written in English or one of the Scandinavian languages and had to be available from the HTA institution's Web page or by written contact with the institution.
As type of assessor, we distinguished between the following three groups: first, HTAs performed exclusively by the national HTA institutions own personnel; second, HTAs outsourced to external assessors, that is, assessors not employed at a national HTA institution; and third, HTAs performed by both national HTA institution employees and external assessors.
As the type of technology, we distinguished between procedures, devices, and pharmaceuticals as defined by Banta and Luce (2). A procedure was defined as “a combination of provider skills or abilities with drugs, devices, or both”; a device as “any physical item, excluding drugs, used in health care”; and pharmaceuticals as “any chemical or biological substance that may be applied to, ingested by, or injected into humans”.
Assessment methods were coded according to the individual description in the reports, and they were grouped in seven general methods of assessment: literature review, meta-analysis, survey, RCT, economic evaluation, modeling/evaluation, and other methods.
The assessment method literature review contains all assessments for which some kind of literature review was used in a report, both systematic and less-formalized reviews, such as narrative reviews of the subject in question. Meta-analysis was judged to be present when the results of a formal meta-analysis were presented and were designated as such in a report together with at least a description of data and methods of analysis. Surveys were coded as present in a report if they were described together with data and survey methods used. Both interviews and questionnaires were included in this method of assessment as long as they included new data and quantitative analysis. Randomized controlled trials were judged to be present in a report when this term was stated and the data, procedure of randomizing, and method of analysis were presented in the reports. The assessment method economic evaluation contains all assessments that were labeled as economic evaluations by the authors and includes types of evaluations, such as cost-minimization, cost-effectiveness, cost-utility, or cost-benefit analysis. The term modeling/evaluation included assessment methods based on different kinds of modeling techniques, for example, decision analysis, simulation models, and so on, and of less-structured evaluation studies not included in the categories of surveys and RCTs. The subjects of modeling or evaluation studies were not confined to the clinical parameters but could concern other parameters too, for example, economic parameters. The last method of assessment, other method was coded as being present when an HTA contained an analysis not covered by one of the other six assessment methods. Among these were formal methods of synthesis, such as consensus conferences and Delphi techniques, and assessment methods, such as case reports, expert opinion, and so on.
The year of publication was categorized into four groups: HTAs published 1989–93, 1994–96, 1997–99, and 2000–02. Table 1 shows the characteristics of the 433 HTAs included in the review.
Statistical Analysis
Multivariate logistic regression was used to calculate odd ratios (ORs) and 95 percent confidence intervals (95 percent CIs) of the possible determinants of applying a specific assessment method in an HTA. That is, for each assessment method, we estimated a logistic regression model, with the individual assessment method as the dependent variable. As explanatory variables, we used type of assessor(s), year of publication, and type of technology. Table 2 shows the characteristics of the assessment methods applied in the HTA.
The explanatory variables were included into the logistic regression model as dummy variables. To avoid perfect collinearity between the explanatory variable, we defined a group of reference HTAs as HTAs published in 1989–93 that considered a procedure and were performed by national HTA institution employees only. Thus, the odds ratios expressed the additional likelihood of an HTA to include the assessment method under consideration compared with the reference HTAs, for example, the additional likelihood of including a literature review if the HTA considered a device compared with HTAs that considered a procedure.
RESULTS
All HTAs that included an RCT as the assessment method were performed by external assessors in 1999–2002 and mostly considered procedures (Table 2). Therefore, there was not enough variation in the explanatory variables to estimate a logistic regression model. Table 3 shows the results from the logistic regressions analyses on the remaining six assessment methods.
In terms of the ability of the regression model to correctly predict the inclusion of an assessment method in an HTA report on the basis of the explanatory variables of the model, the goodness-of-fit was good. The proportion of correctly predicted observations ranged between 81 percent and 97 percent.
Assessor
The logistic regressions indicated that HTAs outsourced to external parties increased the likelihood of applying meta-analysis, survey, or economic evaluation as assessment methods, whereas external assessors were less likely to include literature reviews and other methods than assessors within a national HTA institution (Table 3). For literature reviews, an OR of 0.12 (95 percent CI, 0.03–0.46) indicated that HTAs outsourced to external assessors had a lower likelihood of applying a literature review as a method for assessing the technology under consideration than HTAs performed by national HTA institution assessors. The percentage reduction in the likelihood of including a literature review if the HTA was outsourced to an external assessor can be estimated as 1 minus the odds ratio. Thus, the likelihood of including a literature review was reduced by 88 percent (1–0.12) if an external assessor performed the HTA (95 percent CI, 54%–97%).
For the meta-analysis method, the parameter estimates for external assessors as well as the parameter for both external and national HTA institution assessors were both significantly related to the inclusion of meta-analysis. That is, the likelihood of including a meta-analysis was 5.43 times higher if the HTA was performed by external assessors than if the HTA was performed by assessors within a national HTA institution (95 percent CI, 1.24–23.9). The confidence interval did not include 1, which indicated that the effect of external assessors was significantly different from that of assessors within the national HTA institution. Furthermore, HTAs performed by both assessors from national HTA institutions and external assessors had a 7.58 (95 percent CI, 1.50–38.2) times higher likelihood of including meta-analyses than HTAs performed by national HTA institution assessors only. Overall, these results suggest that HTAs that involved external assessors increased the likelihood of applying meta-analysis as assessment method compared with HTAs that involved internal assessors only.
For survey and economic evaluation, the likelihood of applying these methods was 5.76 (95 percent CI, 2.11–15.8) and 8.11 (95 percent CI, 2.96–22.2) times higher for HTAs performed by external assessors than those performed by internal assessors. The assessment method modeling/evaluation showed no statistically significant association with the type of assessors. Assessment methods other than the six above had a lower likelihood of being included in an HTA if it was performed by external assessors compared with HTAs performed by personnel at the national HTA institution (OR, 0.18; 95 percent CI, 0.07–0.44).
Year of Publication
The year of publication was significantly related to the inclusion of economic evaluation in an HTA only. HTAs published between 1997 and 2002 were less likely to include an economic evaluation than HTAs published between 1989 and 1993, and between 1994 and 1996. Conversely, HTAs published in 1989–93 were more likely to include economic evaluations than HTAs published later. Comparison of HTAs published in 1989–96 with HTAs published in 1997–2002 indicated that the former had a 5.46 (95 percent CI, 2.01–14.82) higher likelihood of including an economic evaluation than HTAs published in 1997–2002. None of the other assessment methods was statistically significantly related to the year of publication of the HTA reports.
Type of Technology
The type of technology in focus influenced the inclusion of surveys and economic evaluation and the use of other assessment methods in an HTA. HTAs assessing a pharmaceutical lowered the likelihood of including a survey in the assessment by 71 percent (1–0.29) and lowered the likelihood of including other methods by 86 percent (1–0.14), whereas this increased the likelihood of including an economic evaluation by 2.56 (95 percent CI, 1.39–4.71) compared with assessing a procedure. The other assessment methods, literature review, meta-analysis, and modeling/evaluation, were not statistically related to the type of assessed technology, and assessing devices compared with procedures did not show any significant relations with assessment methods.
The general results of this study show that the outsourcing of HTA reports to external partners is associated with a higher likelihood of using assessment methods, such as meta-analysis, surveys, economic evaluations, and RCTs; and with a lower likelihood of using assessment methods, such as literature review and other methods. The only statistically significant relation between year of publication and assessment methods is found in economic evaluations, and assessing pharmaceuticals is statistically related to economic evaluations, surveys, and other methods.
DISCUSSION
The results of the present study give an interesting picture of the association of three potentially explanatory variables with the choice of assessment methods in 433 HTA reports from nine countries during the period from 1989 to 2002. To the authors' knowledge, these results are the first ever to go beyond the mere descriptive characterization of definitions of health technology assessments and contents of HTA reports.
Looking first at the time perspective in the results, no major developments across assessment methods during the four time periods appear. The only significant association is the number of economic evaluations in this sample of HTAs, which, strikingly, decreases relatively from around 1997 compared with the earlier time periods. This trend is surprising in the light of methodological developments in economic evaluations in general, which have taken place with considerable speed during the past 20 years. Furthermore, when looking into the arguments behind HTA, the economic restraints on the health care sector are very often mentioned as an argument, but this is obviously not manifested in the practical HTAs (4;10;16;18).
None of the other assessment methods shows significant development during the study period, which means that, during the time period of this study, the choice of assessment methods in HTA in general is relative constant. This result confirms the results of an earlier and more-descriptive study of the 433 HTAs (7), even though there is increasing acceptance of the advantageousness of multiple assessment methods in HTAs (1;9;10;14;22).
Results of the previous studies show procedures as the most often assessed type of technology (8;19), even though there is no total agreement on this finding between studies (16), and a shift from devices to pharmaceuticals, with procedures at a more constant fraction during the study period (7). The present study supplements these results by analyzing the possible association between these types of technology and the six different methods of assessment. When procedures are used as reference in the estimations, the results show that there is no relation between assessing a device and the choice of assessment methods in HTAs. Pharmaceuticals, on the other hand, are related to certain assessment methods in increasing the likelihood of applying economic evaluation in HTA and decreasing the likelihood of applying surveys or other methods. A possible explanation for the association between pharmaceuticals and economic evaluations might be the introduction of guidelines for reimbursement for pharmaceuticals based on economic evaluations in different countries starting from the mid-1990s. The association between pharmaceuticals and surveys might be explained by differences in the relevant populations or samples in an HTA given the type of technology. Assessing the relative cost and consequences of pharmaceuticals is most often related to a certain group of patients, whereas surveys are more often related to larger groups of patients or even populations and might concern more general topics than those assessed in connection to pharmaceuticals.
Probably the most interesting results of this study might be the association between outsourcing HTAs to external partners and the choice of assessment methods. Use of external partners alone as assessors has significant implications for the majority of assessment methods.
First, it is worth mentioning that all RCTs in the study were performed by external partners alone. Furthermore, using external partners as assessors suggests a shift from literature reviews and other methods to meta-analysis, surveys, and economic evaluation. This finding could be described as a shift from more or less unstructured/unformalized assessment methods to more structured assessment methods. One explanation might be that external partners, such as employees at research institutes compared with personnel employed at an HTA institution, have greater interest in academic merit and, therefore, apply the more structured assessment methods to a greater extent. This possibility is one of the arguments mentioned by Cookson and Maynard (5) and Maynard and McDaid (18). Another explanation might be, turning the argument around, that when HTA institutes want an HTA with the more structured/formalized assessment methods, such as meta-analysis, surveys, and economic evaluations, they outsource the task assessment to external partners. It is worth mentioning that this argument only holds if the assessment methods of literature reviews and other methods are considered academically inferior to the meta-analysis, surveys, and economic evaluations.
Another way of describing this trend might be that outsourcing an HTA to external partners is unnecessary when enough, valid, and relevant evidence exists in the literature; otherwise, when evidence is missing or insufficient, external partners are more relevant as assessors in an HTA.
However, some restrictions apply to the study, which have to be taken into account. The sample is labeled a full sample in that it included all full HTAs from the selected institutions, but we only selected national HTA institutions with a certain level of experience. This strategy meant leaving out regional or other locally defined HTA institutions. Furthermore, the sample is restricted on the basis of language, as only HTAs written in English or one of the Scandinavian languages were included.
The results of the study are presented as overall measures of all the HTAs in the sample, which suggests no considerations for institution- or country-specific aspects influencing the HTAs, such as certain types of assessment methods, type of technologies, or the choice of assessors. The results of the current study present the general picture of 433 international HTAs from the period 1989 to 2002 but not variations or differences between those institutions and countries.
CONTACT INFORMATION
Eva Draborg, PhD (eud@sam.sdu.dk), Assistant Professor, Christian Kronborg Andersen, PhD (cka@sam.sdu.dk), Associate Professor, Institute of Public Health–Health Economics, University of Southern Denmark, J. B. Winsløvsvej 9B, First floor, 5000 Odense C, Denmark
Eva Draborg is financially supported by the Danish Centre for Evaluation and Health Technology Assessment.
