Health technology assessment (HTA) should serve as a bridge between science and policy, helping to translate evidence into practice. HTA agencies are providing products, documents, research activities, and scientific publications that are considered worthwhile by healthcare stakeholders and managers in the decision-making process (Reference Gabbay and Walley8). This process usually takes place at the healthcare system level, where policy decisions are made, but also at the patient level where the innovations of healthcare markets are introduced and where decisions are made about whether to adopt new medical technology in clinical practice.
In recent years, the impressive development of technologies in health care has been accompanied by an increase in the demand for services due to patient and provider expectations, and subsequently by rising costs (Reference Jonsson and Banta16;21). In Spain, the services of the healthcare system are provided and financed mainly by the public sector. Decision makers must plan and provide for the supply of technology in agreement with concrete objectives, based on the population served and the quality and effectiveness of the health services offered. Concerns about the cost of health care, increasing patient and provider expectations, pressure on limited resources, and budget constraints lead to a difficult balance between planning, providing, and receiving services.
Over the past 30 years, HTA has become established as a means of informing certain decisions and policies about the adoption and use of technology in health care (Reference Jonsson and Banta16). HTA is defined as a structured and systematic analysis based on the scientific evidence, the aim of which is to provide information for policy making. HTA can serve as a bridge between the sources of knowledge and the levels of decision, offering decision makers the best synthesis of scientific evidence and the added value of the analysis of related economic, social, ethical, and legal issues (Reference Battista and Hodge4;Reference Jonsson and Banta16).
Useful, relevant, and evidence-based information provided through HTA on a timely basis will allow decision makers to have enough elements to decide on the future use of new technology and to promote the rational use of healthcare services (Reference Jonsson and Banta16). The knowledge thus generated can contribute to healthcare management at the three levels of decision making: macro (health policies), meso (institutional rules), and micro (clinical management) (Reference Battista, Lance, Lehoux and Regnier5).
The products of HTA agencies vary depending on the main focus of the evaluation: this focus may be on specific aspects of a technology such as efficacy, safety, effectiveness, or costs, or on broader, system-level activities such as health services research, the implementation of information systems, or the assessment of organizational models. At each level of decision making, the information needed and the products required from HTA agencies and organizations may be different and may change according to the context and environment, which will determine the relationship between HTA users and producers in the decision-making process (Reference Battista, Lance, Lehoux and Regnier5;Reference Hivon, Lehoux, Denis and Tailliez13).
The ability of HTA organizations to improve and participate in the decision-making process depends on the development of strategies to tailor the scientific evidence to the context for target stakeholders and specific situations (Reference Battista and Hodge4;Reference Rossen and Gabbay24). For the purpose of meeting this challenge, it is crucial to understand what the needs and requirements of policy makers are to deal more specifically with the concerns of HTA users (Reference Henshall, Koch and von Below12;Reference Hivon, Lehoux, Denis and Tailliez13).
Several studies have reviewed and evaluated the production of organizations involved in HTA using different approaches. Draborg et al. (Reference Draborg, Gryd-Hansen, Bo Poulsen and Horder7) reviewed 443 HTA reports to explore the practical application of HTA definitions and dimensions to the local context. Mears et al. (Reference Mears, Taylor, Littlejohns and Dillon18) reported a worldwide survey of 50 HTA institutions to capture a profile of HTA production: the types and methodological aspects of HTAs, methods of priority setting, dissemination strategies, and the relationship between HTA and policy making. Another study explored the diversification of the Canadian HTA agencies’ portfolios to respond to the information needs of users (Reference Lehoux, Tailliez, Denis and Hivon17). There are also a limited number of experiences from the point of view of HTA users, to clarify their perceptions and practices of HTA (Reference Hivon, Lehoux, Denis and Tailliez13), and to engage decision makers who could play a relevant role in the decision-making process through the establishment of HTA networks (Reference Henshall, Koch and von Below12).
Since 1981, as part of the political process of decentralization in Spain, healthcare “competencies” have been progressively taken over by the seventeen Autonomous Communities and, in this context, six HTA agencies have been set up in various parts of the country. In 2003, a new Unit of Health Technology Assessment (UETS) was created within the Laín Entralgo Agency, with the purpose of providing high-quality and evidence-based information for decision making in the Regional Health Authority of Madrid.
One of the first activities of our HTA Unit was to design a study to identify the needs and requirements of the healthcare decision makers in our region, with two main objectives. The first aim was to explore these needs and requirements at the policy level of our Regional Health Care System and in a clinical context within hospitals and primary care centers. Second, we aimed to use the information obtained to guide the strategic development of the portfolio and future activities of the HTA Unit according to the needs and concerns of potential HTA users. A subsidiary objective was to begin to inform potential users about the types of products and services that could be offered by the HTA Unit in support of policy and practice decisions in our environment.
Identifying key products and services as defined by potential users and beneficiaries of UETS, that is, the managers and decision makers of the healthcare system of the Community of Madrid, is essential to appropriately diversify the HTA portfolio and adapt our production and assessments to the needs of different stakeholders and decision makers.
METHODS
Study Design
We conducted a Delphi study to use the information obtained about HTA needs from decision makers to guide the strategic development of the portfolio of the HTA Unit of the Community of Madrid.
The Delphi technique is a qualitative research methodology to explore the opinions, agreements, and disagreements of the participants in the process. This technique, used widely in health services research to reach consensus, allows the convening of a large group of experts while avoiding the effect of “domination by personalities” frequent in the group encounter. It is often found during the Delphi process that the range of the answers will decrease and that the group will converge toward a consensus position (Reference Helmer11;Reference Jones and Hunter15).
Before the Delphi study, we searched multiple Web sites for products and services offered by leading HTA agencies and institutions at national and international level to develop the questionnaires used in the study. We performed a review of published literature (papers, reports, official publications, and so on) HTA activities and projects and strategic development plans of the relevant institutions in the HTA field (1–3;14;19;20;25,26).
The Delphi technique we used proceeded in two consecutive rounds, between April and June of 2003. Before the first round, the research group designed and validated the questionnaire and selected the panel of participants from experts of the Community of Madrid. The questionnaire was distributed to the expert panel by post and electronic mail. After completion and return of the first-round questionnaires, the answers were analyzed. The second-round questionnaire incorporated the evaluation of the products and services from the first round. Panelists answered the second survey knowing the broad outlines of all other participants’ responses to the first one, with the opportunity to contrast and change their score in view of the group's response. Only panelists who responded to the first round were invited to participate in the second one.
Participants in the Delphi Study
Relevant decision makers (managers, medical directors, and general directors) who represented the potential user groups of HTA products and services were invited to participate in the study. We focused on four different settings: public hospitals, private hospitals, primary care centers, and the Regional Ministry of Health. The initial panel included eighty-seven experts from twenty-one public hospitals, eleven care primary centers, six private hospitals, and eight Departments of the Regional Ministry of Health.
Products and Services Offered by HTA Agencies
We considered that the types of products and services that could be offered by our new agency would be similar to those that are offered by many of the member agencies comprising the International Network of Agencies for Health Technology Assessment (INAHTA). Therefore, the products and services included for this study were identified from the portfolio offered by twenty-six INAHTA agencies from fifteen different countries (Table 1).
Health Technology Assessment Agencies Portfolio Review in 2003
We selected twenty-two products and services that included four types of products classified in seven subcategories, from traditional products such as HTA reports, systematic reviews and economic evaluations, to more innovative products such as rapid reviews, the monitoring of emerging technologies, and clinical practice guideline (CPG) evaluation (Table 2). The list of products also included institutional tools, products to support the incorporation of technology in healthcare centers, and the development of individual research projects as well as the dissemination of HTA information and publications.
Products and Services Offered by HTA Agencies and Organizations
Note. This table is also available online at http://www.journals.cambridge.org/jid_thc.
HTA, health technology assessment; HT, health technology.
Questionnaires
The Delphi study comprised two questionnaires. The first-round questionnaire included semistructured questions to evaluate twenty-two possible products provided by an HTA Unit. The questionnaire used a 5-point Likert scale consisting of “no interest,” “low interest,” “moderate interest,” “high interest,” and “essential.” There was also an open-ended question to prioritize the five products considered most important from the point of view of each organization.
The evaluation of the products ranked in the first survey was incorporated into the second questionnaire with the same twenty-two close-ended questions included in the first-round questionnaire. For each second-round question, there was the panelist's own first-round answer as well as the group's response (expressed as a percentage of all responses at each position on the Likert scale), to contrast both sets of responses and to assist in reaching consensus on the relative importance of HTA products. There was also a specific final question to prioritize the services and products previously rated, in the same way as the first questionnaire, although in this case, respondents were not provided either with copies of their own first-round priority list or any information regarding the expressed priorities of the rest of the panelists.
Analysis
Descriptive analyses included results both globally and stratified by four participant settings, after the second round. Participants’ preferences were summarized according to their interest level in the HTA products and services, aggregated in three categories: low interest (no interest and low interest), moderate interest and high interest (high interest and essential). Due to the descriptive aim of the study, statistical analysis of global and group differences was not performed. Analyses were carried out with an SPSS software package.
The relevance of HTA products was examined through two rankings, one based on preferences and the other on prioritization. Regarding preferences, questions were weighted according to the answer of each participant scored as follows: no interest, 0.1; low interest, 0.2; moderate interest, 0.3; high interest, 0.4; and essential, 0.5. This ranking of the sum of weighted preference scores identifies products with the greatest absolute importance: the higher the total score a question achieves, the more important products are for panelists.
The prioritization of services was analyzed according to the relative priority given to each potential product or service by each participant and was weighted as follows: first position, 0.5; second position, 0.4; third position, 0.3; fourth position, 0.2; and fifth position, 0.1. Although the weighting system for the preference and prioritization ranking is arbitrarily constructed, it preserves a rational order of importance.
RESULTS
Global Analysis
The overall response rate was 83.9 percent. Seventy-three of eighty-seven potential panelists participated in both rounds of the Delphi study. In three settings, the response rate was above 85 percent, reaching 100 percent among panelists from primary care centers and representatives of the Regional Ministry of Health (Table 3).
Global Response Rate Stratified by Setting in the Community of Madrid
Figure 1 shows the interest of the participants in the selected HTA products and services. Ten of the twenty-two potential products were rated as of high interest by more than 80 percent of all decision makers independently of the setting.
Ranking of preferences according to the assessment of each participant highlighted the top ten products and services common to all panelists (Table 4). Prioritization ranking was estimated through responses to the open-ended priority question, in an attempt to identify the most relevant HTA products across all organizations (Table 5).
Ranking of Preferences
Note. This table is also available online at http://www.journals.cambridge.org/jid_thc.
a Preference score = ∑(# no interest * 0.1) + (# low interest * 0.2) + (# moderate interest *0.3) + (# high interest * 0.4) + (# essential * 0.5).
HTA, health technology assessment; CPGs, clinical practice guidelines.
Ranking of Priorities
a Priority score = ∑(# 1st position * 0.5) + (# 2nd position * 0.4) + (# 3rd position * 0.3) + (# 4th position * 0.2) + (# 5th position * 0.1).
b When equal scores appear, the number of first positions are taken into account to reorder the ranking.
HTA, health technology assessment; CPG, clinical practice guideline.
Stratified Analysis
Preferences for HTA products and services were stratified by type of organization to identify the similarities and differences in the assessment of the top ten preferred products. A comparative analysis across different settings identified common elements and notable differences among the four types of organization.
As shown in Table 6, two products were assessed with an average level of interest greater than 80 percent for all organizations: the monitoring of emerging technologies and systematic reviews. Rapid reviews and studies of the appropriate use of health technologies obtained the second highest degree of agreement, being considered of high interest by more than 95 percent of respondents from three of the four settings.
Preferences Stratified by Organization
Note. Dark gray areas are where respondents from every setting showed a high degree of interest.
a Percentage of respondents from each setting considering each product as “high interest.”
HTA, health technology assessment; CPG, clinical practice guideline.
The most significant divergences were found with drug and medical/surgical procedure assessments, considered of high interest in public healthcare settings (public hospitals and primary care), in sharp contrast to the much lower rating given to these products by private hospitals and health departments.
Overall interest level for HTA services and products. HTA, health technology assessment; CPG, clinical practice guideline; HT, health technology.
Although all settings showed interest in healthcare information system assessment, private hospitals and health departments considered this essential, likely reflecting that organizational management is a priority in these settings.
Table 7 presents the relationship between the products and services considered by more than 80 percent of participants as being of high interest and the “macro/micro” HTA perspective of each type of organization. Products were classified in different sections from classic and newer HTA products requested by clinical centers from a “micro” level perspective, to institutional tools and services centered on the diffusion and impact of HTA, at the “meso” and “macro” levels.
Products and Services Evaluated by Organizations: Different HTA Perspectives
Note. The product “CPG Implementation Assessment” was not rated as high interest by more than 80 percent of respondents from any setting and, hence, does not appear in the table. Empty cell indicate that products and services were not rated as high interest by more than 80 percent of respondents. Data given are the percentage of respondents from each setting considering each product of high interest.
HTA, health technology assessment; CPG, clinical practice guideline; HT, health technology.
In our study, there were differences in the types of HTA products and services required by the various types of organization. A “micro” HTA perspective was more common in public healthcare settings, where classic products such as HTA reports, systematic reviews, economic assessments, and drug and medical/surgical procedure assessments were considered of highest interest, as well as more innovative products focused on their own centers such as rapid reviews, CPG appraisals, appropriate use of health technologies, monitoring of emerging health technologies, and feasibility studies for health technology incorporation.
Departments of the Regional Ministry of Health and private hospitals, in addition to the requirement for classic and more innovative HTA products, shared a more “macro” HTA perspective of the products needed, including in their preferences such products as organizational model and information system assessments, HTA research projects, elaboration and diffusion of HTA periodical publications, educational interventions, and HTA institutional relationships.
Ranking of priorities stratified by setting explored the priorities given by different organizations to the ten most relevant products from the open-ended question, highlighting the priority position for HTA products in each organization. There was a heterogeneous assessment of products and services from the point of view of each organization (Table 8). There are only four products common to the priority ranking of all organizations (economic evaluations, HTA reports, monitoring of emerging health technology, and systematic reviews). Of these, economic evaluations and systematic reviews are the only two products included in one of the top five positions for at least three types of organization. It should be remembered that private hospitals and Health Departments had a low relative weight in the global ranking of priorities; this is reflected in the fact that some of different services and products prioritized by them are not included in Table 8.
Top 10 Global Priorities Stratified by Organization
Note. Empty cells indicate that the products and services in the global ranking did not figure among the 10 top priorities in every setting.
HTA, health technology assessment; CPG, clinical practice guideline; HT, health technology.
DISCUSSION
Our study explores the needs and concerns of potential HTA users and relevant stakeholders from different levels of decision making in our regional healthcare system in the specific context of the establishment of a new HTA Unit in the Community of Madrid. Knowing the needs, expectations of, and demands on HTA from healthcare decision makers is the first step in the strategic development of a new HTA portfolio. The institutional context in which an HTA agency is established defines and influences its production and the types of assessments it delivers. For a better impact on policy, HTA agencies try to design and develop their products according to the demands and needs of decision makers, future users of HTA products and those responsible for priority setting (Reference Lehoux, Tailliez, Denis and Hivon17;Reference Oliver, Mossialos and Robinson22).
The high study participation rate of 84 percent supports the representativeness of the results in our regional context, involving participants from different levels of the decision-making process. There was considerable interest in the HTA products being offered by a new HTA Unit. The different professionals who participated in the study reflected the importance that HTA must have in decision making in their daily practice. The high interest in HTA products is an encouraging initial point for a new HTA Unit. Improving HTA uptake is related to the engagement of relevant decision makers at all levels before beginning projects and research (Reference Gagnon, Sánchez and Pons9;Reference Henshall, Koch and von Below12).
After selection of potential products and services offered by HTA institutions, our study showed the interest and preferences of decision makers for classic HTA products and services such as HTA reports, systematic reviews, and economic evaluations. However, it is notable that global preferences also included four products not always routinely offered within an HTA agency's portfolio, such as rapid reviews, appropriate use of HTA, monitoring of emerging health technologies and CPG appraisals. The demand for these products could be due to the practical approach they represent (Reference Hivon, Lehoux, Denis and Tailliez13). Rapid reviews are an example of provision of a brief, readable, and immediately usable response to guide policy that decision makers regard as a basic and desirable product, but which is not always delivered by agencies (17 percent in the study by Lehoux et al.) (Reference Lehoux, Tailliez, Denis and Hivon17), perhaps due to the effort required to produce such reports in a rigorous, systematic, and, above all, timely manner.
We also explored the importance of HTA products through a prioritization ranking that reveals a somewhat different picture. Prioritization is an assessment of the usefulness of HTA products within each respondent's own organization, with a more practical focus. Hence, feasibility studies for the incorporation of health technology in centers and the agency's collaboration in the development of CPGs were considered as priorities. Managers of hospitals and primary care centers, the two groups with most participants, gave much more importance to local context.
Different visions of HTA were found between public healthcare settings (public hospitals and primary care) and the Regional Ministry of Health and private hospitals. Although the “micro/macro” classification of HTA products used in this part of the analysis has been inferred by us, the apparent differences in perspective do make intuitive sense and relate to the observation often highlighted in the literature that a clinical perspective considering benefits for individual patients, which is at the center of healthcare delivery, is different from a more managerial or epidemiological perspective that typically considers health benefits at population level (Reference Gagnon, Sánchez and Pons9;Reference Rossen and Gabbay24).
There are some limitations to the present study. First, the HTA portfolio review did not include an exhaustive list of HTA agencies, although all HTA agencies with significant experience were searched to retrieve their production. Second, there was a relatively low response rate among participants from private hospitals, perhaps because they did not perceive any direct benefits to their collaboration in an HTA survey of this nature. Third, product preferences and priorities in this study are specific to our regional healthcare context and may not be generalizable to other regions and countries with a different HTA orientation, vision, or mandate. However, the necessities and demands of decision makers, independent of context, are almost always focused on the same concerns: the timing, readability, reliability, and accessibility of HTA information (Reference Henshall, Koch and von Below12;Reference Hivon, Lehoux, Denis and Tailliez13;Reference Lehoux, Tailliez, Denis and Hivon17).
The development of an HTA agency's portfolio is a challenge that requires a process of selection and prioritization of four or five basic products based on available resources and needs, while considering the opportunity costs. In our case, and based on the results of this Delphi study, we designed our initial HTA portfolio including HTA reports, rapid reviews, systematic reviews, and monitoring the use of emerging technologies.
In a second stage, we diversified our production in response to the specific interests of decision makers and users other than those within our direct institutional context (the Regional Ministry of Health) where, inevitably, much of our priority workload is determined, in the same way as with similarly funded HTA agencies (Reference Lehoux, Tailliez, Denis and Hivon17). Educational and training interventions were included as a way to disseminate actively the knowledge generated by the Unit. Health service research using appropriate use assessment of medical and surgical procedures (RAND method) (Reference Brook, Chassin and Fink6) was another innovation introduced (Reference Quintana, Escobar and Aróstegui23).
Since 2005, we have been working on the development of CPGs (e.g., for the management of knee arthrosis in primary care) (Reference Gracia San Roman and Calcerrada Díaz-Santos10), with other agencies at the national level and with the participation of healthcare professionals and patients, which is crucial for successful implementation in their centers (Reference Gagnon, Sánchez and Pons9). We are also starting to cover other areas and topics incorporating a broader HTA dimension including economic, organizational, ethical, and social aspects, a task that is more difficult to achieve in the HTA field (Reference Draborg, Gryd-Hansen, Bo Poulsen and Horder7;Reference Lehoux, Tailliez, Denis and Hivon17).
There are different demands and needs for each type of organization, but with important common elements that can allow us to prioritize the products and services offered, the development of ongoing projects, and even the role and profile of the professionals in an HTA Unit. Identifying relevant decision makers and tailoring products for specific market segments while incorporating common requirements can enable optimization of the portfolio of an HTA Unit.
There appears to be a compelling need to increase the perceived importance within our environment of “macro” HTA services, perhaps by more active dissemination and communication strategies concerning the advantages of integral approaches that can add value and have an impact in local settings. Collaboration and coordination among agencies, both nationally and internationally, is well understood within the HTA community as an efficient way to strengthen common projects and combine efforts by bringing together decision makers from European and other international healthcare systems so that best practices and experiences can be shared (Reference Henshall, Koch and von Below12;Reference Lehoux, Tailliez, Denis and Hivon17). This is not, however, an approach that is immediately understood by decision makers outside the HTA community to have direct benefits in their own local context.
Within our environment, the Health Care System of the Community of Madrid, we concluded that this study allows us to focus the HTA portfolio and adapt our production and assessments to the needs of managers and decision makers from different levels. Contextualizing HTA at regional, national, and international levels, with active feedback between users and producers, could achieve a better HTA impact in policy and decision making.
CONTACT INFORMATION
Elena Andradas, MD, MPH, MHTA (elena.andradas@salud.madrid.org) Health Research, Director, Área de Investigación y Estudios Sanitarios (Health Research and Health Studies), Juan-Antonio Blasco, MD, MPH, MHTA (juan.blascoa@salud.madrid.org), Health Technology Ass-essment Unit Director, Beatriz Valentín, MD, MPH (beatriz.valentin@salud.madrid.org), Researcher, María-José López-Pedraza, MD, MPH (mjose.lopezpedraza@salud.madrid.org), Researcher, Francisco-Javier Gracia, MD, MPH (javier.gracia@salud.madrid.org), Researcher, Health Technology Assessment Unit (UETS), Agencia Laín Entralgo, Gran Vía 27, 28013 Madrid, Spain
The authors thank Martin Jordan for his support in the review of this manuscript. We are also grateful to the managers and directors of the primary care centers and hospitals and the representatives of the Regional Ministry of Health who participated in our Delphi study.
