Mini-HTA (health technology assessment) is defined as a form or checklist with several questions about the prerequisites for and consequences of using health technology. The mini-HTA forms used in Denmark typically include twenty to thirty questions grouped according to the four HTA perspectives: technology, patient, organization, and economy. The purpose of the form is to provide a brief two- to five-page basis for decisions about the introduction of a specific new health technology or a specified change in the indication for the use of existing technology (3). Mini-HTA is intended to be a flexible and dynamic tool adaptable to local conditions and the current requirements of decision makers, for example, facilitating local and regional budget, planning and priority processes. Where the problem or the application extends beyond a specific local context, however, the mini-HTA cannot replace a full-size HTA (Reference Ehlers, Vestergaard and Kidholm4).
In Denmark, mini-HTA was described and evaluated in a large national project in 2005 funded by the Danish Center for Evaluation and Health Technology Assessment where an example of the mini-HTA form also can be found (3).
Since the publication of the national mini-HTA project, the use of mini-HTA in the healthcare sector in Denmark has increased. At the hospital level, mini-HTA is used by the hospital and clinical department managers as input to decisions about introduction of new health technologies. At the national level, the Association of Danish Regions (the hospital owners) applies mini-HTA when they request regions and hospitals to submit annual reports on new treatments to be implemented in the following year. This information is used by the Association of Danish Regions in the negotiations with the National Government about the healthcare budget in the annual budget agreements.
The National Board of Health also started using mini-HTA as part of the documentation for the regulation of activity at public hospitals. When hospitals apply to the National Board of Health for permission to start new highly specialized treatments, they are now encouraged to submit a mini-HTA describing the clinical evidence and the organizational and economic consequences. Clinical departments with highly specialized treatments also have to submit an annual report to the National Board of Health describing their treatment activity, research, quality, capacity, etc. and their use of mini-HTA. Finally, mini-HTA is requested by the National Board of Health when hospitals apply for changes in the DRG values or groups, for example, in connection with the introduction of new treatments.
The strength of mini-HTA is that it can be undertaken locally by the clinical staffs who are considering using the technology and it can be produced within a limited amount of time (3). A study at Rigshospitalet in Copenhagen (based on interviews with twenty-six clinical and administrative managers) revealed that 65 percent of the respondents took between 2 and 15 hours to produce a mini-MTV (Reference Folkersen and Pedersen6). This estimate does not include time for literature search and literature review.
The question is, however, whether the clinical staff at large university hospitals, who are considering introducing new health technologies, are able to produce mini-HTA with a sufficient level of quality of the information? The objective of this study is to assess the quality of the information included in mini-HTA used at Danish hospitals and to discuss the consequences of this to decision making. The method used and the results are described below.
METHODS
The quality of a HTA report can be examined by use of checklists as the one presented by in the INAHTA initiative and described by Hailey (Reference Hailey8). This commonly used checklist includes seventeen questions which address several minimum standards that should apply to HTA reports.
The checklist includes five questions about preliminary information (authors, context, conflict of interest, etc.), three questions about why the assessment was undertaken (questions addressed, scope, description of technology), three questions about how the assessment was undertaken (sources of information, selection, interpretation), two questions about the results (presentation, interpretation) and four questions about the implications (conclusion, discussion, legal aspects, further actions).
The seventeen questions are supplemented in the checklist by several subquestions, but to facilitate a reasonable assessment of each mini-HTA in this study only the seventeen main questions were applied. With respect to four of the questions, however, additional sub-questions are added reflecting subjects that are of special interest to hospital managers based on our experience from Danish hospitals. The questions are presented in the tables in the section with the results below.
Each of the mini-HTA reports were assessed on the thirty variables and data were recorded and analyzed in STATA.
Data
To study the quality of mini-HTAs, 52 mini-HTAs from Danish hospitals in 2008 were identified. These mini-HTAs were part of the Association of Danish Regions’ annual collection of reports on new treatments which the hospitals are planning to introduce. Reporting a treatment or technology is considered relevant by the Association if no hospital has implemented the technology before, if the indication criteria for treatment have changed or if the cost per patient has increased significantly. In 2008, 185 reports were submitted including 172 mini-HTAs. Of these 172 mini-HTAs, 38 described new treatments using new health technology and 14 described experimental treatments. In total fifty-two mini-HTAs. The remaining mini-HTAs described new applications or increased activity of already implemented health technologies and were not considered relevant for this study.
The fifty-two mini-HTAs were included in the analysis because it was expected that the quality of the information in the assessments was at the highest level. This expectation was based on the assumption that the medical and administrative staff at the hospitals will produce a more complete description of the evidence for a new treatment, when the technology is new and the decision of implementation has not been made. This is in contrast to mini-HTAs describing existing treatments which are part of the current activity at hospitals and where the decision to use the technology has been taken some years ago.
The fifty-two mini-HTAs of new health technologies constitute 30 percent of the mini-HTA reports in 2008. The remaining reports describe new applications for existing health technologies (47 percent), existing treatments with increasing number of patient (10 percent), a general increase in the expenditures because of cost increases for an existing technology (4 percent) and technologies which have been assessed previously in a mini-HTA or HTA report by another hospital or region (9 percent). All 172 mini-HTAs are included in the national mini-HTA database at www.minimtv.dk which is available for the staff at all public hospitals, the five Danish regions, and the National Board of Health. The database is created by Odense University Hospital and the Association of Danish Regions in collaboration and includes 306 HTA-reports from the hospitals.
RESULTS
The types of technologies and medical specialties assessed in the fifty-two mini-HTAs are presented in Table 1. The majority (77 percent) are describing the consequences of new pharmaceuticals. Examples are Nilotinib for chronic myeloid leukemia, maintenance therapy with Rituximab for patients with recurring and refractory follicular lymphomas and Avastin for ovaries cancer. The most frequent medical specialties are hematology and oncology. Five assessments concern new surgical procedures, for example, minimal invasive surgery for atrial fibrillation.
Types of Technology and Medical Specialty Assessed in the 52 Mini-HTAs
Preliminary Information
The content of the fifty-two mini-HTAs with regard to preliminary information and why the assessment was prepared is described in Table 2. The table shows that mini-HTAs generally lack information about the author and who to contact for further information. The assessments also lack information about potential conflicts of interest and whether the report has been externally reviewed.
Questions about Preliminary Information and Why the Assessment Was Undertaken (N = 52)
Table 2 also shows that 42 percent of the mini-HTAs include a short summary in nontechnical language, reflecting that a summary is a required part of the local versions of the mini-HTA form at some but not all hospitals.
With regard to the question being addressed in the mini-HTA and the context of the assessment, this is not stated explicitly in any of the reports. However, it is implicit that mini-HTA is an assessment of a defined technology compared with the technology used so far at the specific hospital, and that the assessment has been produced by the staff at a specific hospital. The scope of the assessment (question 7), for example, the attributes of the technology being addressed, is always described, because the mini-HTA consists of twenty to thirty questions trying to cover all potential attributes of a technology. Most of the mini-HTAs (98 percent) include some form of description of the technology being assessed (question 8) and the comparator, which in these fifty-two mini-HTAs often is conservative treatment (status quo) or no treatment. In summary, Table 2 reveals that even though mini-HTA potentially includes all aspects of a technology, there is still large variation in how well the requirements of the checklist with regards to preliminary information, context and purpose of the assessments are met.
How Is the Assessment Undertaken?
None of the mini-HTAs include information about the literature search strategy used, see Table 3. This reflects the fact that the mini-HTA used by Danish hospitals does generally not include questions about search strategies. However, 96 percent of the mini-HTAs state that a systematic literature search has been performed and is the basis of the description of the technology. Similarly, the level of evidence in the documentation of the clinical effect of the treatment is described in 75 percent of the mini-HTAs and a short list of references is presented in 94 percent of the cases. On the other hand the sources and bases of the cost data are rarely described.
Questions about How the Assessment Was Undertaken (N = 52)
Question 10 about the process of selecting material for the assessment is difficult to answer in general. Instead the question is divided into four additional questions in Table 3. The table shows that the types of cost elements included in the calculations are described in 88 percent of the assessment. The estimated costs always include the resources used within the specific clinical department. However, only 29 percent include costs outside the department (e.g., cost of anesthesia and radiological procedures in assessments of new types of surgery) and only 8 percent include costs outside the hospital (e.g., use of general practitioner and nursing home).
Approximately one-third of the mini-HTAs include some kind of interpretation of the selected data (question 11) on which the assessment is based. Often this is in the form of a short description or comments to the results of the studies in the references, the assumptions on which the interpretations are based or additional information about the clinical quality of the specific technology based on data from national clinical databases.
RESULTS AND IMPLICATIONS
Question 12, about the presentation of the results in the checklist, is divided into five sub-questions in Table 4 reflecting the four HTA perspectives in the Danish HTA model. With regard to results on the effectiveness, only 25 percent of the mini-HTAs include a quantitative estimate of the size of the effect. Instead the effectiveness is described as for example ”. . . a positive impact on mortality and morbidity' or ”. . . a significant improvement in quality of life' with no quantitative measure of the size of the effect. The patient perspective including aspects of health related quality of life is described in only 25 percent of the assessments.
Questions about the Results and Implications of the Assessment (N = 52)
The organizational consequences inside the clinical department are generally included in the assessment (81 percent), but the consequences beyond the departments’ own routines and organizational structure are described only in 48 percent of the cases. This is related to the fact that the description of the economic consequences outside the department also is lacking in most cases. Almost all mini-HTAs (92 percent) include estimates of the cost of using the technology at the specific hospital or the regional level. Costs are mostly presented as the total costs with information about the cost elements included and sometimes the cost per patient but with very little information about unit costs and marginal analysis.
The 52 mini-HTAs do not include an explicit statement of the interpretation of the results (question 13). This could be caused by the fact that a clinical department only submits a mini-HTA to the Association of Danish Regions if the management of the clinical department views the technology as an improvement to the patients and the hospital.
The findings of the assessment are discussed (question 14) in 40 percent of the mini-HTAs. Often the discussion is about the uncertainty of the results with regard to, for example, the number of patients per year or the unit cost of the treatment per patient. Medico-legal implications (question 15) are not included in any of the fifty-two mini-HTAs considered in this study.
The existing versions of the mini-HTA form do not generally include a question about the conclusion of the assessment (question 16) and it is therefore rarely clearly stated. However, as mentioned above, the conclusion in the mini-HTAs submitted by the hospitals are implicitly that the technology is considered as a preferable improvement based on the existing evidence. Suggestions for further action are mentioned in 4 percent of the assessments, for example, requests for new clinical studies of the effectiveness of the treatment.
DISCUSSION
This is the first study to investigate the quality of mini-HTAs produced in the Danish healthcare sector. We assessed fifty-two mini-HTAs corresponding to all reports on planned new health technologies and experimental treatments in Danish Hospitals in 2008. The mini-HTAs were collected by the Association of Danish Regions and all fifty-two mini-HTA have been used as input in the budget negotiations with the Danish Government in the spring 2008.
The critical assessment of fifty-two mini-HTAs from Danish hospitals shows that the quality of the HTAs is insufficient on certain dimensions. With regard to, for example, the checklists points about the question being addressed in the HTA and the conclusion of the HTA, the insufficient or missing information reflects that mini-HTAs are always assessing a new technology by comparison with the existing technology at the specific hospital, and the mini-HTA is only produced and submitted if the hospital considers the new technology beneficial to implement. However, this is implicit and not stated explicitly in the HTA reports.
The mini-HTAs generally include short descriptions of the assessed technology and the comparator, but information about the selection and interpretation of the clinical literature and other data is often missing. The level of evidence for the clinical effects and the main references are generally included, but only 25 percent of the mini-HTAs include a quantitative estimate of the size of the clinical effects. With regard to the organizational and economic aspects 81 percent of the mini-HTAs include information on organizational consequences inside the clinical department and 92 percent includes a cost estimate.
A main problem with mini-HTA is that the size of the clinical effects of the technology on the patients’ health is not presented in quantitative terms in 75 percent of the assessments. Instead the effect is described in very broad terms such as”. . . a positive impact on mortality and morbidity is expected”. If decision makers in public hospitals are trying to maximize the health effect of the scarce resources available when choosing between different new health technologies, this information is fundamental and must be present in the assessment. This quality flaw may have severe consequences on the policy impact of the mini-HTA report either because the reports are disregarded as relevant inputs for decision making or because erroneous decisions are made on the basis of the mini-HTAs.
The quality of a HTA is high if it gives a precise, comprehensive and evidence based description of the consequences of introducing a health technology. A study of the quality of HTA could therefore be carried out by comparison with the actual consequences of the technology after the implementation (Reference Lafortune, Farand and Mondou9). Until a gold standard for quality assessment is refined and the proper research effort in assessing the impact of mini-HTA is carried out, we choose to assess quality as the availability and quality of information in the mini-HTAs based on the checklist presented by in the INAHTA initiative (Reference Hailey8). Other checklists for assessment of the quality of HTA reports has been published by (Reference Busse, Orvain and Velasco1) including forty-four questions about basic information, methodology, context, data, technical description, results, etc. A large number of questions and aspects of HTA rapports are similar in both lists and we considered the list by Hailey (Reference Hailey8) to be the most appropriate given the concept of mini-HTA.
The findings show poor quality of the mini-HTAs but the use of the INAHTA checklist may exaggerate the quality problems because several aspects may not apply to mini-HTA reports because they are produced for a rather specific purpose for the specific hospital. However, even with this reservation we still believe that there is a need for quality assurance of mini-HTAs to improve the accuracy of the information without harming the timeliness and the limited use of resources in the production of the reports. We also believe that there is a need for a revision of the forms or check lists applied in various hospitals because some relevant information about the HTAs seem to be completely missing in the reports.
An obvious solution to the insufficient quality of the assessment of clinical effectiveness in mini-HTAs could be to use evidence synthesized in other HTA reports produced by the national or international HTA institutions, which have the scientific knowledge and resources to produce HTA reports of high quality. However, the number of new health technologies implemented at the hospitals each year is much higher than the capacity of national HTA units as stated elsewhere (Reference Elston and Stein5;Reference Oliver, Mossialos and Robinson12). Most of these institutions have only resources to examine a small proportion of the many new technologies, let alone the many thousands of existing technologies. As described above the hospitals submitted 185 HTA reports to the Association of Danish Regions in 2008. In 2006, the hospitals submitted seventy-eight mini-HTAs about forty-six different health technologies. In June 2006, a comparison was made with the HTA projects produced by the national Danish HTA unit DACEHTA (13). The result was that fourteen national HTA reports were identified in which the same technologies were assessed with regard to the same patient group as in the forty-six mini-HTAs. Thus, national HTA reports covered 30 percent of the treatments reported in the mini-HTA carried out in the same year. This illustrates the need for production of some form of basis for decision making at the hospital level as previously demonstrated in other countries (Reference Maddern10;Reference McGregor and Brophy11).
From both a clinician's and policy maker's perspective there is a need for accurate, relevant, timely and accessible HTA reports, but these objectives may be in conflict. Timeliness, relevance and, obviously, accuracy are of high importance to clinicians because they find themselves in a position where they have to consider and make recommendations on introduction of new and expensive treatments (Reference Chantler2). Mini-HTA can be an alternative to full scale HTA reports, not only providing timely and relevant information but also integrating the recommendations into practice because the mini-HTAs are deeply rooted in the local institutional and organizational context (Reference Gagnon, Sanchez and Pons7).
POLICY CONCLUSIONS
We consider our results to be an argument for an increased focus on quality assurance of the mini-HTAs produced at the hospital or regional level rather than only relying on national scientifically high quality HTA reports. At two hospitals in Denmark, Rigshospitalet in Copenhagen and Odense University Hospital, mini-HTAs are systematically being reviewed by health economists or local HTA units and this has lead to a higher level of quality of the assessments compared with the other hospitals. It is also possible to create an improved awareness of and interest in HTA in the clinical departments by offering courses in mini-HTA and HTA conferences to the clinical staff. The new national mini-HTA database is also introduced with the objective of improving the quality of mini-HTA in practice.
This investigation of fifty-two mini-HTAs is the first study of the quality of mini-HTA as a basis for decision making in practice. The study illustrates some of the problems with the quality of the information included in this kind of assessment, but the results must be completed with more in-depth studies of the impact of mini-HTA following, for example, the framework suggested in (Reference Lafortune, Farand and Mondou9). One possibility for further research is to make comparisons between mini-HTA and full HTAs similar to a recent study (Reference Watt, Cameron and Sturm14) comparing rapid HTA with full HTA.
CONTACT INFORMATION
Kristian Kidholm, MSc, PhD (kristian.kidholm@ouh.regionssyddanmark.dk), Health Technology Assessement Consultant, Department of Research and Health Technology Assessement, Odense University Hospital, Kloevervaenget 8, Odense, 5000, Denmark
Lars Ehlers, MSc, PhD (lars.ehlers@stab.rm.dk), Associate Professor, Institute of Public Health, University of Aarhus, Vennelyst Boulevard 6, DK-8000 Aarhus C, Denmark; Senior Researcher, Health Technology Assessement & Health Services Research, Center of Public Health, Central Region of Denmark, Olof Palmes Alle 17, DK-8200 Århus N, Denmark
Lisa Korsbek, MA, PhD (lisa.korsbek@ouh.regionsyddanmark.dk), Documentalist, Department of Research and Health Technology Assessement, Odense University Hospital and Svendborg Hospital, Kloevervænget 8, DK-5000 Odense C, Denmark
Rolf Kjærby, MSc (rolf.kjaerby@ouh.fyns-amt.dk), Administrative officer, Centre for National Clinical Databases, South, Odense University Hospital, Kloevervaenget 8, Odense, 5000, Denmark
Mickael Beck, MSc, PhD (Mickael.beck@ouh.regionssyddanmark.dk), Professor, Department of Health Economics, University of Southern Denmark, J.B. Winsløwsvej 9, 1, Odense, 5000, Denmark; Head of Department, Department of Research and Health Technology Assessement, Odense University Hospital, Kloevervaenget 8, Odense, 5000, Denmark
