Studies from Australia, Canada and Europe have shown that production of HTA within health-care institutions under certain circumstances can increase the use of clinical and economic evidence in decision-making (Reference Ehlers, Vestergaard and Kidholm2;Reference Gallego, Gool and Kelleher4;Reference Lettieri, Masella and Nocco10;Reference McGregor and Brophy11;Reference Nielsen, Funch and Kristensen14). Examples are reported from The McGill University Health Centre in Canada (Reference McGregor and Brophy11), The Southern Health, Melbourne, Australia (Reference Gallego, Gool and Kelleher4), a sample of Italian hospitals (Reference Lettieri, Masella and Nocco10), and the Odense University Hospital, Denmark (Reference Kidholm, Ehlers and Korsbek8). Thus, locally-produced HTA can be rapid, timely, and systematically included in priority setting and budgeting (Reference Nielsen, Funch and Kristensen14).
Today, managers in many health-care institutions around the world face an opportunity to implement new tools and processes for the local production and use of HTA. Managers in health care generally agree that priority setting should be clear, fair, consistent, and based on best available evidence (Reference Gallego, Gool and Kelleher4;Reference Mitton and Donaldson13). Nevertheless, decisions in “real life” are often value-based, ad hoc, and driven by experts and/or cost considerations (Reference Vuorenkoski, Toiviainen and Hemminki18). One possible explanation for this paradox could be that only a very limited number of “success stories” exist. These stories are not randomized, controlled trials or other comparative studies of improvements in patient outcomes and cost savings that would not otherwise have occurred. Furthermore, they do not tell “what made the deal”, they do not indicate why there are so few examples, and there are (as far as we know) no published reports on unsuccessful experiments or failures to implement local HTA.
In Denmark, the mini-HTA tool has been used in the Danish public hospitals for several years with some success (Reference Ehlers, Vestergaard and Kidholm2;Reference Folkersen and Pedersen3;Reference Kidholm, Ehlers and Korsbek8), and in May 2008 a new mini-HTA form designed especially for the Danish municipalities was released by the Danish National Board of Health (17).
The Danish health-care sector predominantly consists of public hospitals and private general practitioners financed and administrated by the regions. In 2007, a major structural reform created larger regions and reduced the number of municipalities (from fourteen to five regions and from 270 to 98 municipalities). At the same time, the health-care responsibilities for elderly care, social psychiatry, prevention and health promotion, rehabilitation, and other types of care that are not directly related to hospital inpatient care were transferred from the regional to the local (i.e., municipality) level. The intention was to give the municipalities a stronger role in prevention and rehabilitation and provide incentives for municipalities to step up their efforts by introducing municipal co-payments for hospital services (Reference Strandberg-Larsen, Nielsen and Vallgårda16). This transference of the health-care responsibilities also created a need for new decision support systems in the municipalities. Addressing this need, the Danish National Board of Health took the initiative to provide the municipalities with new managerial tools such as mini-HTA for municipalities and the Health Impact Assessment tool (Reference Gulis, Gry and Kræmer6;Reference Kristensen, Sigmund and Kristensen9;17).
The new mini-HTA tool for municipalities was essentially the same as the mini-HTA for hospitals (see, e.g., Ehlers et al. (Reference Ehlers, Vestergaard and Kidholm2) or Kidholm et al. (Reference Kidholm, Ehlers and Korsbek8)), but the “translation” of the existing mini-HTA form for hospitals was performed by a project organization within the National Board of Health in collaboration with Local Government Denmark (i.e., the interest group and member authority of Danish municipalities). The tool consists of a form/checklist with twenty-nine questions in four categories (technology (effort), patient (citizen), organization, and economy) concerning the prerequisites for and consequences of using (new) health technology and a written guideline for “how to use the form.” The mini-HTA form is available as a Supplementary Form 1, which can be viewed online at www.journals.cambridge.org/thc2012033. In short, answering the twenty-nine questions should provide a brief, written (2–5 pages) evidence-based basis for decisions performed by local employees or HTA advisors.
Today (2012), mini-HTA is still not being used regularly in the Danish municipalities. This article analyzes the attitudes toward mini-HTA among the participants in five voluntary workshops on mini-HTA in the municipality held shortly after the introduction of the new HTA tool. Furthermore, the attitudes and perceptions about the barriers and possible ways of overcoming these are used to hypothesize why mini-HTA has not yet been the success in the municipalities, and what may be the main differences between the successful implementation in the Danish hospital sector and the failure in the municipal sector. We believe our study points at barriers for the uptake of mini-HTA that may be generalizable to other forms of local HTA in other decentralized healthcare systems.
METHODS
The Questionnaire About Attitudes Toward Mini-HTA
In the period from May 2008 to February 2009, five voluntary workshops/seminars on mini-HTA in the Danish municipalities were held in five cities (Helsingør, Copenhagen, Aabenraa, Kolding, Hjørring). At each seminar the participants were given a general introduction to HTA and the Danish mini-HTA tool. After the introduction (approximately 1 hr), a short questionnaire was handed out to the participants, and the participants were given time to answer the questions.
The questionnaire contained the following questions:
• What do you think mini-HTA could be used for in the municipality?
• What do you think are the main barriers for using mini-HTA in the municipality?
• What do you think could make it easier to implement mini-HTA in the municipality?
Furthermore, the respondents were asked to provide background information about their professional position, and they had the opportunity to give other comments. All responses were anonymous.
Analyses
In the analysis all participants were assumed to represent the group of people who were interested in mini-HTA as a tool or appointed by their superiors in the municipality to participate in the workshop. Answers were analyzed qualitatively and quantitatively using binary coding and statistical examination of patterns. R-factor analysis was used to validate a combination of responses that looked meaningful at first glance. Here, we wished to identify factors that would summarize the information on attitudes (latent traits) toward the introduction of mini-HTA in the municipalities. For that purpose we first identified eleven topics mentioned in the open-ended answers to all the questionnaires. These eleven topics were sufficient to represent the information in the answers to all the questionnaires. We then examined, for each questionnaire, which of the eleven topics that were mentioned and which were not. The analysis was performed on those response patterns.
R-factor analysis is a technique developed with the purpose of explaining patterns within a correlation matrix of manifest variables by introducing latent variables (Reference Hair, Black, Babin and Anderson7). The idea is that a subset of variables may show considerable pairwise correlations due to their common association with a single variable known as a factor. The analysis comprises a determination of the number of factors, and based on this number an assessment of the association between the individual factor and the manifest variables related to that factor (called factor loadings) is made. Squared factor loadings indicate what percentage of the variance in the manifest variable is explained by a factor. The sum of squared loadings for a factor is called eigenvalue and must be above 1, that is, the factor accounts for more variance than a single manifest variable (Reference Hair, Black, Babin and Anderson7).
Inspection of the factor loadings associated with each factor is necessary to interpret/name the factors. The interpretation of the factors should generally rely on strong conceptual foundation, for example, prior research or theory, and it is an iterative process aided by factor rotation, inspection of factor loadings, interpretation and respecification (Reference Hair, Black, Babin and Anderson7).
One of the main motivations for use of the exploratory factor analysis is the desire to represent a large part of the information content in the original correlation matrix by a few factors. Originally, the technique was developed with the purpose of analyzing continuous variables stemming from a normal distribution. In our case, clearly the variables are neither continuous nor normally distributed. Several solutions to this problem have been advanced (Reference Hair, Black, Babin and Anderson7;Reference Mislevy12). Here, we decided to address the problem directly by replacing the (Pearson) correlation matrix (which is typically the input to the factor analysis) by the tetrachoric correlation matrix. The latter is designed to measure correlation among binary variables. With this replacement we can continue in the usual manner and use the procedures already available in most statistical programs. All analyses were performed in Stata version 11 (StataCorp LP, College Station, TX).
RESULTS
A total of ninety-three participated in five courses in mini-HTA. Seventy-five (81 percent) of the respondents worked in a municipality, and of these were forty-five managers and thirty employees. Seven (8 percent) worked as health professionals in either a private or state-owned health-care institution. Eight (9 percent) were employed in a consultancy firm or interest organization, two worked in health-care administration at regional or state level, and one was unknown. Questionnaires were received from all participants. One was blank.
What do You Think Mini-HTA Could be Used for in the Municipality?
Ninety-two (99 percent) of the respondents gave positive examples of what they believed mini-HTA could be used for in the municipality. There were no negative/critical comments. All ninety-two respondents had several answers.
Seventy-one respondents (76 percent) mentioned applications of mini-HTA that could be classified as general in the sense that they resembled any other praise of HTA as input to decision-making. This included statements like input for decision-making, decision support, input for prioritization, preassessment of new initiatives, to use evidence more systematically in the municipality, to qualify decisions, and a systematical approach. These were comments that the respondents could have learned at the workshop.
Fifty-six respondents (60 percent) mentioned more specific applications that concerned their own workplace. This included statements such as assessing new physical training programs used by the rehabilitation unit as template instead of the usual template/project evaluation form, to strengthen the cooperation within the county between academic and white-color workers etc. These were comments that were not covered by the introduction to mini-HTA given at the workshop.
Thirty-eight respondents (38 percent) mentioned both general and specific applications of mini-HTA.
What do You Think Are the Main Barriers for Using Mini-HTA in the Municipality?
Eighty-one (87 percent) of the respondents addressed the issue concerning main barriers.
Sixty-three respondents (68 percent) mentioned the lack of available time or resources needed to conduct mini-HTAs. The time issue was most often expressed only as a short statement without further explanation, however, some respondents added nuances meaning “it takes time, but we don't have time”.
Twenty-nine (31 percent) mentioned the lack of skills or competences inside the organization. This was explained as a lack of skills, competences, know-how, or routine in working with HTA, a lack of relevant support functions, a lack of experts in the municipality who know about literature search and assessment of evidence, and a lack of relevant education among decision-makers and employees.
Twenty-six (27 percent) mentioned the lack of knowledge about HTA. Some respondents just stated this briefly, while others specifically added the lack of knowledge among decision-makers.
Forty-three (46 percent) mentioned what could be classified as “system barriers”. System barriers at the political level were expressed as populism, lack of accountability, political reasons, the political agenda, or other traditions for decision-making. System barriers at the institutional level included asymmetric interests, that decision-makers adopt a local perspective, a lack of expectations that produced mini-HTA reports would be used in decision-making, that people will see it as control, that HTA represents “cold hands” (i.e., not contact with patients), the wish for freedom of choice in methods, and a lack of willingness among professionals to be confronted with “hard evidence”.
Finally, seven respondents (8 percent) expressed some kind of doubt that mini-HTA was even relevant for the municipality. Three of these respondents worked in private organizations (consultants), one in a region, and one in the Danish Welfare Ministry, that is, there were only three respondents from the municipalities.
What do You Think Could Make it Easier to Implement Mini-HTA in the Municipality?
Ninety-one (98 percent) of the respondents addressed the issue concerning ease of implementation.
Forty-three (46 percent) mentioned the need for a decision whether or not mini-HTA should be used in the municipality. The need for a decision was stated differently, for example, that leaders must “show the way”, that leaders should want to implement HTA or find it relevant, that managers and politicians must prioritize HTA, prioritize time for HTA, demand documentation/evidence, and accept mini-HTA as a managerial tool. Some respondents expressed a need for a decision to use mini-HTA in specific circumstances, and others a need for convincing managers and politicians that HTA is a “good solution”.
Forty-three (46 percent) mentioned dissemination of information about mini-HTA. This included brief statements such as more information, knowledge, inspiration, as well as more elaborate explanations to spread the knowledge about HTA and mini-HTA.
Thirty-two (34 percent) mentioned the need for more education. That included courses such as the one the respondents attended, assistance from consultancies, possibilities for support inside the organization when doing HTA, training of key employees, and education of politicians, managers, and employees.
Forty-two (45 percent) gave further comments or recommendations. This included information campaigns targeted at stakeholders, dissemination/communication/dialogue with politicians and managers, using the human resource department to spread information and knowhow, using examples to show the potential financial advantages for the municipality, provide access to a common database for mini-HTAs (like the Danish mini-HTA database for hospitals), publication of HTAs/mini-HTAs relevant for Danish municipalities, key persons to “drive” the implementation process, inter-organizational meetings and “demystification of HTA” or to get it down to a level where “we all understand it”.
Patterns Among Respondents
The analysis of patterns revealed that respondents who focused on system barriers were more likely to mention the need for a political/managerial decision to use mini-HTA (p = 0.0001).
Respondents who mentioned the lack of competences to do mini-HTA were more likely to mention the need for more education (p = 0.004).
There was no differences among managers and employees in the sample except that managers more often mentioned specific applications of the mini-HTA (p = 0.001).
Factor Analysis
Table 1 shows the topics mentioned by the respondents. The topics are elicited from free replies to the questions in the questionnaire and reformulated by the authors as auxiliary questions to perform a factor analysis. Table 2 holds the results from the factor analysis.
Translation of the Topics Mentioned by the Respondents Into an Auxiliary Questionnaire
Results From the Factor Analysis
Note. Factor loadings less than +/− 0.6 have not been printed. Eigenvalues for the first four factors were 2.74, 1.95, 1.66, and 1.43, respectively. Communality refers to the amount of variance in the particular variable that is not explained by the four factors.
Based on the eigenvalue criterion the number of factors should be four. This means for example that the variance of the variable “Dissemination of information about the mini-HTA is needed to implement the mini-HTA” was not sufficiently well explained by the four-factor solution and the variable was therefore dropped for further analysis. This kind of decision is common when carrying out an exploratory factor analysis.
For the first factor there was a positive association between the factor and the mentioning of topic 1 and 7, and a negative association between the factor and mentioning of topic 2. For this reason factor one was labeled “HTA's are generally good”.
There was a positive association between factor number two and topics 6 and 8. Therefore, we named factor two “Involvement by management is a critical factor”.
We found positive factor loadings between factor three and the topics 4, 10, and 11. Factor three was, therefore, labeled “Medium-term resources are critical factors”.
Finally, for factor four we saw a positive factor loading associated with topic 3 and a negative factor loading associated with topic 5. Factor four was labeled “Short-term resources are critical factors”. Higher values of this factor were associated with answering “Yes” to the first item related to this factor and “No” to the second item.
DISCUSSION
This article studies why the mini-HTA tool has not yet been adopted in the Danish counties despite of the push strategy by the Danish National Board of Health and the seemingly general acceptance of the tool among respondents. We used factor analysis to answer this question assuming that correlations among answers could reveal latent attitudes and barriers that were not identifiable by single questions. The factor-analysis approach used in this study should therefore be considered as an exploratory rather than confirmatory analysis. It does, however, point at several different factors that constitutes barriers at the individual and organizational level for the adoption of mini-HTA which may be similar to barriers for local HTA in other countries with decentralized health-care systems.
Barrier 1
HTAs are generally (too) good. At first glance all respondents seemed to agree that mini-HTA could be a valuable new tool for the municipality. This was obvious in the answers to the first question in the questionnaire where all respondents gave positive examples of what mini-HTA could be used for. Although many of these examples could be taken from a textbook in HTA, they revealed a general acceptance of the HTA-principles among many of the respondents. This is in line with other studies showing a seemingly general acceptance of HTA-principles (Reference Gallego, Gool and Kelleher4). On the other hand the factor analysis showed an association between the mentioning of general applications of HTA in the first question and later expressions of doubts that mini-HTA was relevant for the municipality. One hypothesis could, therefore, be that although respondents may find HTA-principles applicable they may also find HTA a bit “over the target” or “too ambitious” for regular use in the municipality. Only seven respondents mentioned this critique of mini-HTA in the discussion of barriers however, but if we had included a specific question about the relevance of mini-HTA, more respondents may have expressed the same doubts.
Barrier 2
The significant latent factor named “Involvement by management is a critical factor” points at the importance of a managerial/political decision to implement and systematically use mini-HTA in the municipality to overcome the political and institutional barriers. A possible interpretation of this finding is that HTA-researchers must engage more directly with decision-makers and convince them to make a decision to use mini-HTA if the tool is to be used systematically. This is in accordance with the advocacy coalition perspective (Reference Gibson, Lin and Gibson5) that simply improving the dissemination of information from the HTA-world to decision-makers will not ensure the implementation of tools for “evidence-based policy” in the municipalities.
The strategy of the Danish National Board of Health was mainly to disseminate knowledge about mini-HTA and at the same time open the door for the Local Government Denmark to help designing the information campaign. The Danish National Board of Health has only an advisory role toward the Danish municipalities, that is, it cannot enforce a decision to use mini-HTA upon the local politicians and managers. Compared with the successful implementation of mini-HTA in the Danish hospital sector, this lack of authority may be the single most important explanation of why mini-HTA has not yet been implemented in the Danish municipalities despite the effort from the Danish National Board of Health.
The implementation of mini-HTA in the hospital sector was led by the regional HTA-unit in Aarhus (Reference Ehlers, Vestergaard and Kidholm2). This HTA-unit is a part of the regional hospital organizational structure in the Central Region of Denmark with a direct line responsibility (and direct access) to the regional director of health care. Thus, the successful implementation of mini-HTA in the Danish hospital sector can be explained partly by the support from the regional director of health care, but perhaps more important the ability to put mini-HTA on the decision agenda at a meeting with all the regional health-care directors where a decision to collect information from all hospitals every year for the annual budgeting negotiations with the Government was suggested to be carried out with mini-HTA as tool (1). The decision to demand and collect a mini-HTA for every new health technology in Danish hospitals was thus preceded by another decision to strengthen the collection of information for the annual budget negotiations, and the idea of using the mini-HTA tool for this matter was “pushed” by the HTA-unit at the regional level. The HTA-principles or the tool itself was not necessarily the decisive factor. Noticeably, other examples of “successes” with the mini-HTA tool in the Danish hospitals also entail a set-up of new local decision processes. At the University of Copenhagen, Denmark (which was a state hospital at the time the mini-HTA program was implemented) political approval of new treatments and health technologies was required by Danish law. A special procedure for the systematic presentation and formal approval based on mini-HTA forms was set-up for that purpose (Reference Folkersen and Pedersen3). At Odense University Hospital the hospital management introduced several initiatives to emphasize EBM and HTA, and one of these initiatives was that every application for extra budget funding should be accompanied by a mini-HTA synthesizing the clinical evidence for the specific proposal. Thus, the decisions to produce and use mini-HTA in decision-making were in fact (or at least to some extent) solutions to local management problems or “natural extensions” of the managers’ existing agendas.
Barrier 3
The significant factor labeled “Medium-term resources are critical factors” indicates several barriers that may be very difficult to deal with in the short term. The lack of education among managers and employees, lack of skills to search databases and understand evidence, lack of IT and support facilities etc. will need to be changed over time. The importance of these barriers has been shown in other studies as well (Reference Vuorenkoski, Toiviainen and Hemminki18).
Barrier 4
The factor “Short-term resources are critical factors” on the other hand points at a lesser problem of time constraints and lack of knowledge. These minor problems could be dealt with in the short term. Noticeably, these short-term barriers were the ones that the national information campaign addressed.
Compared with the results from the questionnaire about mini-HTA in the Danish hospital sector (Reference Ehlers, Vestergaard and Kidholm2) it is interesting to see differences in what the respondents in this study do not mention. They generally do not praise the standardization that mini-HTAs offer, and they do not express any worries that the quality of mini-HTAs may not be “good enough”, and they actually do not mention the need for more evidence in decision-making today. These issues were all central in the debate regarding the introduction of mini-HTA in the hospital sector (Reference Folkersen and Pedersen3;Reference Sigmund15). The health-care responsibilities for rehabilitation and prevention services were still very new to managers and employees in the Danish municipalities at the time of the questionnaire, and these issues may be more relevant as time passes. However, it seems reasonable to hypothesize that managers’ attitudes toward evidence and reluctance to change local decision processes may be a central explanation of the poor uptake of mini-HTA in the Danish municipalities.
Recommendation
Mini-HTA (and other local HTA tools) needs to be considered as more than just a “recipe” for making a sound report. Guidelines should be extended to include tailored descriptions of ways to reform and/or set up a new local decision process that includes clear responsibilities for producers and assessors, an application process, an assessment committee or secretary, a communication strategy, as well as teaching and learning about HTA-principles.
There is a need for more knowledge on how (and to what extent) the inclusion of local production of HTA in decisions at local institutions can contribute to improvements in patient outcomes and cost-effectiveness. Such empirical findings from comparative studies may be much more convincing to managers than general and theoretical arguments.
National information campaigns to support the uptake of HTA tools in local healthcare institutions should be supplemented with a strategy to secure local political/managerial willingness to use HTA and the removal of short-/medium-term barriers.
CONCLUSION
A national information campaign to support the uptake of HTA tools in local health-care institutions was insufficient to ensure implementation of mini-HTA in the Danish municipalities. The implementation of local HTA should be seen not just as a question of how to increase the use of evidence in decision-making, but rather as a matter of reforming local decision processes.
CONTACT INFORMATION
Both authors report they have no potential conflicts of interest.
Lars Ehlers, Professor, PhD, Aalborg University, Danish Center of Healthcare Improvements (DCHI), Faculty of Social Sciences and Faculty of Health Sciences, Aalborg, Denmark
Morten Berg Jensen, Associate Professor, PhD, Business and Social Sciences, Aarhus University, Department of Economics and Business, Aarhus, Denmark
CONFLICTS OF INTEREST
Both authors report they have no potential conflicts of interest.
