Medical devices produce many benefits in patient care, yet represent a major factor in the continuing rise in healthcare costs (12 percent of all French healthcare expenses). To help policy makers and funders make informed decisions, health technology assessment (HTA) has become an indispensable tool worldwide (Reference Drummond, Schwartz and Jönsson1). In European countries, HTA is generally performed by independent public agencies, as is the case in France where HTA activities are managed by the French national authority for health (Haute Autorité de Santé, HAS) (Reference Orvain, Xerri and Matillon2). The HAS provides recommendations for reimbursement by the French Health Insurance of medical devices for individual use.
Nevertheless, only a small proportion of devices used in hospitals are actually assessed by the HAS. In France, the cost of most in-hospital medical devices for individual use is covered by diagnosis-related group (DRG) funding. Among them, only implantable medical devices (IMDs) are currently evaluated by the HAS (Reference Huot, Decullier, Maes-Beny and Chapuis3). A report in 2010 by the French national audit office for social affairs and health (Inspection Générale des Affaires Sociales, IGAS) highlighted the inadequacy of resources at the HAS to assess all non-IMDs in regard to their number and diversity. Furthermore, little clinical evidence of effectiveness can be found for most medical devices currently available on the European market, because manufacturers are not required to demonstrate this to obtain CE marking (Reference Storz-Pfennig, Schmedders and Dettloff4). Consequently, neither clinical effectiveness nor cost efficiency of many potentially innovative although costly nonimplantable devices purchased by French hospitals has been demonstrated.
To offset the national evaluation inadequacy, university hospitals (UHs) have set up local organizations to assess medical devices not reimbursed by the French Health Insurance. None, however, provide any feedback of their assessments to the HAS (Reference Martelli, Billaux, Borget, Pineau, Prognon and van den Brink5). Despite the IGAS report suggesting combining regional (UH) and national (HAS) resources in a French HTA network in which the HAS would determine the methodology to be applied for assessing the devices, hospital-based HTA activities have not developed consistently in all UHs, and there remains no common methodology.
A hospital-based HTA consists in contextualizing the HTA according to the needs of that particular hospital (Reference Gagnon6). This approach takes into account the local organizational context in which the health technology is adopted, such as available comparators (treatment currently used locally) or available resources (Reference Gagnon, Desmartis, Poder and Witteman7). Many initiatives have emerged worldwide in recent decades with regard to hospital-based HTA, yet their real impact remains unclear (Reference Gagnon6;Reference Catananti, Cicchetti and Marchetti8;Reference Ehlers, Vestergaard and Kidholm9). Among the four models proposed by the Health Technology Assessment international (HTAi) in 2008, use of the mini-HTA is increasing across Europe (Reference Gagnon6;Reference Fure, Ormstad and Vang10;Reference Grundström, Friberg and Medin11). According to the HTA glossary, a mini-HTA can be defined as “a report that includes a comprehensive systematic literature review, or a systematic review of high level evidence, evaluating the safety and effectiveness of a technology, but that does not include an analysis of the cost-effectiveness of the technology.”
Generally, a decision support tool (a checklist or a form) is used to conduct the rapid assessment (Reference Ehlers, Vestergaard and Kidholm9). It appears to be a useful tool for hospitals unable to devote the time required to undertake a comprehensive HTA such as that performed by national HTA agencies (Reference Grundström, Friberg and Medin11). In 2005, the Danish Centre for Evaluation and Health Technology Assessment (DACEHTA) proposed a national mini-HTA form divided into four domains (technology, patient, organization, and economy) including twenty-three questions (twenty-six questions taking into account the introduction) (Reference Vestergaard, Ehlers, Kidholm, Holt, Bonnevie and Jensen12). Since its release, the use of the mini-HTA in the Danish healthcare sector has increased and become, in certain cases, compulsory for funding innovative health technologies (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck13). In addition, this national mini-HTA form has enabled the standardization of practices in Denmark and, in turn, the centralization of information in a national mini-HTA database (Reference Kidholm, Korsbek, Kjærby, Jeppesen and Bech14).
Based on this experience, the Danish mini-HTA model would appear a potential solution promoting a common HTA methodology among French UHs and improved feedback to the HAS. Whether such a mini-HTA would be suitable for the French UH setting remains to be evaluated, as little is known about hospital-based HTA activities in France (Reference Martelli, Billaux, Borget, Pineau, Prognon and van den Brink5). Although such initiatives may already exist locally, it is currently undetermined whether the topics evaluated in French UHs are comparable with those assessed in the mini-HTA approach. The objective of our study was, therefore, to explore the topics covered in French UHs to evaluate medical devices for individual use and to compare these with those assessed in mini-HTAs.
MATERIALS AND METHODS
Systematic Review on Mini-HTAs
Although the mini-HTA was invented in Denmark and the Danish national model can be considered the original model, many other mini-HTA-like models are currently used worldwide (Reference Vestergaard, Ehlers, Kidholm, Holt, Bonnevie and Jensen12). To not limit our research to the Danish mini-HTA, data from an international survey of existing mini-HTA systems performed in 2010 by the Norwegian Knowledge Centre for the Health Service (NOKC) was used as a starting point (Reference Ormstad, Graff and Norderhaug15). This was completed by a systematic review carried out on Medline, Embase, Health Technology Assessment database (Centre for Review and Dissemination, University of York), and Google Scholar to identify articles reporting the use of mini-HTAs and decision support-like models. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed to report the systematic review.
Databases were screened independently by two researchers (N.M. and C.D.) for publications from January 1990 to December 2013. The search terms used are presented in the study protocol (Supplementary Table 1). The search strategy was first developed for Medline and then adapted for other databases. To be included, an article needed to report the use of a decision support tool (form or checklist) relating to medical devices, which is still in use and was designed in a local or regional setting. Relevant references were reviewed in full and the name and country of the institution proposing the tool extracted along with the “domain” of the HTA (technology, organization, etc.) and related “topics” (effectiveness, safety, net cost, innovativeness, etc.). The terms “domain” and “topic” were used in reference to the HTA Core Model of the European network for Health Technology Assessment (EUnetHTA). Web sites of organizations identified in the systematic search were also consulted, as were congress papers to complete the data retrieved (gray literature).
Decision Support Tools Retrieved from the Systematic Literature Review
Note. The source of each decision support tool is presented in Supplementary Table 2.
Survey of 18 French University Hospitals
To explore hospital-based HTA initiatives adopted in French UHs, a survey was conducted from October 2012 to April 2013. This survey has been detailed in a previous study (Reference Martelli, Billaux, Borget, Pineau, Prognon and van den Brink5). To achieve data saturation in qualitative research using semi-structured interviews, fifteen to twenty interviews are considered as being sufficient (Reference Ghiglione and Matalon16). In this line, semi-structured interviews were performed to collect data from eighteen UHs selected at random. Participants were asked to indicate the types of information (topics) they considered in their evaluation of new medical devices. From this survey, three major types of hospital-based HTA processes were identified in French UHs: medical device committees, innovation committees, and “pharmacy & management” processes. HTA units had been implemented to support the medical device committee and the innovation committee for technology assessment in several UHs. Finally, the use of standardized HTA forms in the context of multidisciplinary committees was noted. For further investigation, all decision support tools and related documents in use at the time of the survey were also collected.
To provide an accurate overview of the topics evaluated in the UHs, data from both sources (survey and forms collected) were combined by means of data triangulation. The number of times each topic was cited in both sources was counted and compared with the occurrence. This approach helped identify the most frequent topics evaluated in the French UH setting.
Coding Topics of Decision Support Tools
To avoid issues with variability in terms denoting similar topics, topics from all decision tools identified in the literature and in the survey were coded. Such a measure has been previously applied to decision criteria for resource allocation in healthcare (Reference Guindo, Wagner and Baltussen17). The coding was performed in two steps. First, terms were assigned to one of the four domains (technology, patient, organization, and economy) of the Danish mini-HTA. Then, related topics referring to the same concept were grouped under a single topic code using Berelson's coding rules. For example, terms such as “adverse effects,” “side-effects,” “safety,” or “adverse events” were grouped under a single topic code called “SAFETY.” The coding was performed independently by two researchers (N.M. and C.D.). In cases of discordant coding, the discrepancies were discussed until a consensus was reached. The intercoding and intracoding reliabilities were assessed by calculating the Cohen kappa coefficient (K). Intercoding reliability compared the coding given by the two researchers and intracoding reliability compared the coding given by the same researcher on two occasions, 1 month apart. K values were assessed using the Landis & Koch's scale (Reference Landis and Koch18).
Comparing the Coded Topics Using Jaccard Similarity Coefficients
Jaccard similarity coefficients were then calculated to compare the Danish mini-HTA model with all decision support tools retrieved from the literature review and from the French UH survey. The aim of this approach was to determine the degree of similarity between topics evaluated in the documents retrieved. This was possible after achieving uniformity of topics by the coding process. This method is widely used in ecological research investigations to compare the presence/absence of species at several sites, as well as in genetics to compare gene sequences (Reference Real and Vargas19). The Jaccard coefficient consists of calculating the ratio between the size of the intersection of the sets and the size of their union. Considering the two models X (Danish mini-HTA model) and Y (decision support tool from the literature review or from the French UH survey), each model has topics which can either be absent and coded “0” or present and coded “1.” Thus, groupings of topics from X and Y can be as follows: M 11 represents the total number of topics having a value of 1 in both X and Y; M 01 represents the total number of topics absent in X but present in Y; and M 10 represents the total number of topics present in X but absent in Y. The Jaccard coefficient J is given as J = M 11 / (M 11 + M 01 + M 10) where 0 ≤ J ≤ 1. Jaccard coefficients were calculated using Microsoft Office Excel® 2010.
Statistical Analyses
For the different data sets retrieved, the distributions of the topics among the four Danish mini-HTA domains were calculated and compared using the χ2 test. Data were analyzed using the R software version 2.14.1 (R Foundation for Statistical Computing, Vienna, Austria, 2011). Values were considered statistically significant for p < 0.05.
RESULTS
Search Results on Mini-HTAs
The search strategy yielded 176 references (Figure 1), leaving 118 for screening after excluding duplicates. At this stage, we excluded a further thirty-one not related to hospital-based HTA, forty that did not deal with medical devices and thirty not mentioning a decision support tool. The selection criteria were thus met by seventeen references, from which we identified twenty-one decision support tools. Searches in the gray literature revealed a further two tools. However, data were sufficient to completely define the content of only 16 decision support tools (Table 1; see Supplementary Table 2 for the detailed sources).
PRISMA flow-chart.
We extracted 452 topics from the sixteen tools and grouped these under a list of 166 single topic codes. Κ reached 0.83 and 0.98 for intercoding and intracoding reliability, respectively (both for NM and CD). Κ values between 0.81 and 1 indicate an “almost perfect agreement” according to Landis & Koch's scale (Reference Landis and Koch18). The number of topics per decision support tool ranged from eight to fifty-nine with a mean of twenty-eight. Among the sixteen complete forms/checklists, we calculated Jaccard coefficients between the Danish mini-HTA model and the other decision support tools retrieved. According to the number of domains, the number of topics and the similarity coefficient, we differentiated two groups of decision support tool: group 1 showing similarities to the Danish mini-HTA model; and group 2 more generally based on a different concept (Table 1). For group 1, the mean Jaccard coefficient reached 62 percent similarity with the Danish mini-HTA model, while it reached only 27 percent similarity for group 2. In addition, the 452 topics distributed over the four respective Danish mini-HTA domains (technology, organization, economy, and patient) as follows: 108 (47 percent), 55 (24 percent), 49 (21 percent) and 20 (8 percent) for group 1; and 89 (40 percent), 72 (33 percent), 45 (20 percent) and 14 (6 percent) for group 2. The two groups showed no difference in terms of distribution (χ²ddl 3 = 5.02; p = 0.17).
Survey Results
We identified 282 topics cited during the participants’ interviews. After coding using the list of 166 single topic codes, the number of single coded topics was 66. In addition, we collected twelve decision support tools from which we extracted 292 topics, coded into 112 single topics. All topics cited during the participants’ interviews were found in the decision support tools collected. However, we noted that the participants mentioned only a subset of the topics extracted from their own decision support tool (averaging 39 percent). The distribution of the 574 topics cited and extracted from the decision support tools according to the four Danish mini-HTA domains was as follows: 271 (47 percent) for “technology,” 208 (36 percent) for “economy,” 84 (15 percent) for “organization,” and 11 (2 percent) for “patient.” The distribution from group 1 and from the UH survey was compared and found to be significantly different (χ2 ddl 3 = 39.049; p < 0.05).
Triangulation of the data from the survey and from the decision support tools collected in the UHs helped us to identify the thirteen most frequent topics evaluated in the French UHs (Table 2).
Comparison of the Most Frequent Topics Evaluated in the 18 Surveyed French UHs and Those from the Danish Mini-HTA
Note. Common aspects in bold.
Among these, five of the investigated topics were identical to those used in the Danish mini-HTA model: potential indications of the technology, newness of the technology in comparison to usual practice, need to conduct a literature review, previous experience with the technology, and predicted impact on hospital activity over several years. The comparative analysis revealed that organizational impacts in French UHs were not as thoroughly investigated compared with using the Danish mini-HTA model. For example, effects on other departments within the hospital or cooperation with other hospitals were not taken into account. In addition, some topics from the Danish mini-HTA are not applicable to devices for individual use. For example, how it is accommodated into the physical setting is clearly adapted to large technologies, such as an MRI scanner.
The two models showed some similarities in the way in which the “technology domain” was investigated. Indeed, some topics showed high similarity such as “clinical benefits” for French UHs and “effect on diagnosis, treatment, care, rehabilitation and prevention” for the Danish mini-HTA, although we considered this latter topic to be broader. Surprisingly, topics relating to safety (“risks, adverse events and other adverse events”) were not one of the most frequent topics evaluated in French UHs. This is one of the relevant topics defined in the Danish mini-HTA (Table 2).
Topics relating to the economic impact showed considerable differences, except for the “predicted impact on the hospital activity over several years.” French UHs focused on the reimbursement/additional payment availability whereas the mini-HTA models provided a more global approach based on additional or saved costs.
Finally, for the “patient domain,” no common topics were found among the French UHs surveyed. Ethical or social issues relating to the use of a new medical device did not appear to be a major concern for the UHs surveyed.
DISCUSSION
Main Findings and their Meaning
Our findings show that the way new devices for individual use are assessed in French UHs has little in common with the mini-HTAs applied worldwide and especially with the Danish mini-HTA. Indeed, only five topics among the thirteen most evaluated in French UHs showed similarities with those of the Danish mini-HTA.
A first explanation for the slight concordance is that, in French UHs, devices for individual use and for large equipment are not considered in the same procedure, notably due to their funding being completely different (Reference Martelli and van den Brink20). In addition, stakeholders involved in the two processes are not necessarily the same, and their expectations differ with regard to the information required for decision making. As mini-HTAs have been designed to assess not only devices for individual use but also for large equipment and drugs (Reference Ehlers, Vestergaard and Kidholm9;Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck13), some topics in the “organization domain” of the Danish mini-HTA are not relevant with regard to devices for individual use. Regardless of the type of product, we also noted fewer topics in the “organization domain” of the French UH evaluations compared with the mini-HTA-like tools from the literature. Considerations regarding the organizational impact of new medical devices were rarely included in the hospital-based HTA of the French UHs surveyed.
Topics from the technology and economy domains were, on the other hand, very developed in the French UHs. These findings are in accordance with those of a recent study based on interviews of fifty-three hospital managers from nine European countries. They show that information concerning clinical effectiveness and economic aspects were considered the most relevant for decision making to investment in new treatments (Reference Kidholm, Ølholm and Birk-Olsen21). Our results also showed some similarities between the French UHs and the Danish mini-HTA within the “technology domain.” This is not surprising because topics within this domain seem less dependent on the local setting than those from other domains (Reference Nielsen, Funch and Kristensen22). Furthermore, we found the economic information evaluated by the French UHs to be highly focused on reimbursement. This focused vision of the economic impact was also highlighted in the information deemed by hospital managers as necessary in their decision-making process (Reference Kidholm, Ølholm and Birk-Olsen21).
Finally, we noticed that ethical, social, or legal concerns from within the “patient domain” were not seen as major topics in French UHs. Very few of the UHs surveyed actually investigated these aspects, meaning no common ground could be found. Ølholm et al. also reported that information on these aspects did not seem to be of interest to hospital decision makers (Reference Ølholm, Kidholm, Birk-Olsen and Christensen23).
Research and Policy Implications
The present study has important research and policy implications both for building a French HTA network and for promoting a hospital-based HTA culture in France. First, a mini-HTA model cannot be introduced into French UHs without first making important adaptations. Thus, to be useful within a French setting, separate HTA tools are needed to assess medical devices for individual use and for large equipment. We also noted that French UHs rarely investigate organizational impacts when introducing new medical devices. Yet one study recently found that the assessment of organizational impacts is essential for innovative devices (Reference Roussel, Carbonneil and Audry24). These aspects should be properly considered when developing a common HTA tool for French UHs.
Information on patient preferences and ethical issues should also be more systematically considered in French hospital-based HTAs. The importance of introducing the patient's perspective into the local HTA process is now widely recognized (Reference Gagnon, Desmartis and Gagnon25). The development of a common HTA tool for French UHs represents a unique opportunity for the promotion of these aspects. The “patient domain,” one of the four main categories of the Danish mini-HTA, could easily be used as a reference (Reference Ølholm, Kidholm, Birk-Olsen and Christensen23). Currently, risks and adverse events would appear to be insufficiently evaluated in French hospital-based HTAs. These aspects should be carefully taken into account when developing a common HTA tool for French UHs.
Finally, an operational French HTA network will need better cooperation with other European UHs actively involved in hospital-based HTA. This was the aim of the Adopting Hospital Based Health Technology Assessment (AdHopHTA) project, which has recently disseminated tools for hospital-based HTA. However, because France has not been directly involved in the AdHopHTA project to build a common European framework for hospital-based HTAs, there is a risk that the procedures and tools developed through this project need adapting before they can be used in a French setting.
Strengths and Weaknesses of the Study
To our knowledge, this is the first study to investigate the use of HTA decision support tools nationally and locally in France. The survey was not conducted in all French UHs; however, the eighteen hospitals surveyed together represented around 70 percent of patient admissions to UHs in 2011. The results presented here are complementary to those already shown in our previous study (Reference Martelli, Billaux, Borget, Pineau, Prognon and van den Brink5) and contribute to strengthening our knowledge on hospital-based HTA initiatives in France and, more generally, in Europe. The present study also provides an original approach for comparing topics evaluated in local HTAs by using Jaccard coefficients. We found this method less subjective than the sole use of a descriptive approach.
However, some limitations in this study have been identified. First, the survey we conducted was qualitative and not designed for statistical purposes. Consequently, the results only aim to provide an overview of topics evaluated in the context of French local HTA, rather than to precisely describe all possible considerations. As stated above, we found a difference between the topics identified in the survey and in the decision support tools. This can be explained by the fact that the participants only identified relevant topics from their own perspective. Data triangulation helped us to reduce this potential reporting bias. The Jaccard coefficients must be interpreted with great care and cannot be regarded as decisive evidence. Finally, we found a high level of variation in the terminology used in the decision support tools reviewed. Therefore, our classification and coding topics were limited by the subjective interpretation of the terms. Nevertheless, these steps were performed independently by two researchers to minimize errors and closely followed Berelson's coding rules.
CONCLUSION
A mini-HTA-like decision support tool could be a solution to boost coordination and diffusion of the hospital-based HTA culture in France. Nevertheless, this model cannot be immediately implemented in France without first being adapted to the French context.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317000393
Supplementary Table 2: https://doi.org/10.1017/S0266462317000393
CONFLICTS OF INTEREST
The authors declare that there are no conflicts of interest.
