Health technology assessment (HTA) is a multiscientific and interdisciplinary activity aimed at assessing the evidence supporting the setting of priorities and decision making in relation to prevention, diagnostics, treatment, and rehabilitation in a healthcare system. HTA historically has been intended for decisions mainly at macro level, such as regulatory and reimbursement decisions (Reference Kristensen and Sigmund1). However, an effective HTA system should be an integrated environment where interaction occurs at all levels (international, national, regional, local). Hospital-based HTA (HB-HTA; local level) is expanding internationally due to the emergence of new health technologies and budget restrictions within hospitals (Reference Cicchetti, Sampietro-Colom, Martin, Marchetti, Sampietro-Colom and Martin2). It is becoming increasingly important to support hospital directorates with making sound evidence-based decisions (Reference Grenon, Pinget and Wasserfallen3). HB-HTA constitutes an approach that is increasingly used for this purpose. Several hospitals from different countries, thus, have established HB-HTA units (Reference Cicchetti, Sampietro-Colom, Martin, Marchetti, Sampietro-Colom and Martin2;Reference Gagnon4;Reference Cicchetti, Marchetti, Iacopino, D'Amico and Sampietro-Colom5).
HB-HTA units constantly produce reports (Reference Cicchetti, Sampietro-Colom, Martin, Marchetti, Sampietro-Colom and Martin2;Reference Kidholm, Ølholm and Birk-Olsen6;Reference Saaid7) that often come in the form of a mini-HTA, rapid review or full HTA that reflect the hospital context and take into account the information needs of hospital decision makers (Reference Sampietro-Colom, Lach and Cicchetti8). Even though decision makers from other hospitals could highly benefit from these studies and prior assessments (Reference Granados9), many of the existing HB-HTA reports are currently not being shared at all or not consistently in a single database.
A database for the structured collection of HB-HTA reports could help the dissemination and collaboration between hospitals with or without HB-HTA units. For this purpose, the European Union funded project called AdHopHTA created a database in which HB-HTA information (particularly HB-HTA reports) can be stored (Reference Granados9). It was expected that such database could potentially prevent waste of resources by avoiding duplicative work, thus saving time and resources (Reference Gagnon4). Moreover, hospital managers and/or physicians that have been provided more scientific research are backed up with more evidence and this could lead to improved patient care (Reference Gagnon4).
This study explores to what extent stakeholders (e.g., hospital managers, researchers, consultants, and physicians) are willing to share HB-HTA reports produced in their institutions, with the focus on the use of a database, and to analyze what the main opinions are on the collection and dissemination of HB-HTA reports. Furthermore, we aimed to identify the main restrictions and barriers but also the main values and usefulness of a joint effort for the collection of such reports in a database. Moreover, this research explores what professionals consider suitable criteria that should be applied to the database and their opinion on the AdHopHTA database, which is of importance because this might lead to the improvement of the AdHopHTA database or any other (new) database.
METHODS
We conducted a cross-sectional study among HB-HTA experts. The target population was identified as people who are directly or indirectly involved in producing or using HB-HTA reports, mainly members in the emailing list of the HB-HTA interest group at the Health Technology Assessment International (HTAi) society. This interest group consists out of developers and users of HTA reports in the hospital setting such as hospital managers and researchers. The study consisted of a survey that was emailed to the target population with the method of convenience sampling (Reference Etikan, Abubakar and Alkassim10), meaning that, in this study, emails were only sent to the population with easily accessible email addresses and that there was no random component.
The questionnaire contained out of seventeen open-ended and fifteen close-ended questions, which were divided in three sections. The questions were prepared by conducting a short narrative literature review, research team meetings, and after the development of the first draft the receiving of input and validation from two external HTA experts (n = 2) who were not involved in this project. These external experts mainly checked for the face validity and provided feedback to the suggested questions in the survey. Moreover, the questionnaire was pilot-tested and validated among internal HB-HTA researchers (n = 3) to see if the questionnaire was understandable, feasible, and whether it covered all relevant subjects.
The three-sectioned questionnaire was constructed with the help of the revised evidence-based practice decision-making model (Reference Satterfield, Spring and Brownson11) and a self-made conceptual framework. The first section was designed with the purpose of identifying socio-demographic characteristics to check whether the respondents were within the scope of our preset population of HTA and HB-HTA experts. The second section was designed to gain insight in the background of the participants and in the existing methods of sharing and collecting HB-HTA reports. Therefore, questions in this section explored whether respondents already share and/or collect HB-HTA reports, their methods, and to investigate their knowledge on HB-HTA and the AdHopHTA database. For example, the following two questions were asked in the first two sections: “What is your current job function (e.g. researcher, clinician or hospital-manager)?” and “Does your organization share these HB-HTA reports externally?”. The third section was the main part of the questionnaire and explored the main research questions of this study. Therefore, this section consisted out of questions regarding their opinion on the values, usefulness, and barriers of collecting and disseminating HB-HTA reports, as well as questions on the criteria of a database, willingness to share HB-HTA reports, and opinion on the existing AdHopHTA database. For example, the following question was asked: “What are according to you the barriers for sharing and collecting HB-HTA reports?”.
Moreover, the research team has ensured that the survey is understandable and provided the surveyed persons a definition list and certain sections could for only be filled out by persons in the position of sharing HB-HTA reports. The questionnaire is available as a supplementary material in this article.
An initial email describing the survey and its purpose accompanied by the questionnaire was sent to over 200 HB-HTA experts attached to an email in English. Names and contact information of potential experts were mainly gathered from the HTAi HB-HTA Interest Group's contact list. Other contacts related to the field of HB-HTA were retrieved directly from published papers, policy documents, and other existing HB-HTA contact lists, including participants to the latest HTA 102: Introduction to Hospital Based HTA workshops performed during an HTAi meeting. Contacts were, therefore, often from different organizations but could also be from the same HB-HTA organization. Two reminders were sent to the contacts who did not respond to the initial email.
Data derived from close-ended questions were analyzed using descriptive statistics. Responses to open-ended questions were analyzed using the method of content analysis, which enabled the researcher to analyze the data in a structured and unbiased way. In more detail, this study used the content analysis method named inductive qualitative data analysis. This method can be used when, first, an extensive and varied raw text data have to be condensed into a briefer summary format. Second, when from the raw data clear links have to be derived between the research objectives and the summaries findings. Third, when a theory has to be developed about the underlying structure of experiences or process which are evident in the text (Reference Thomas12). These three purposes are all in line with those of the study; therefore, inductive coding was used for the analysis.
The analysis of this study was performed according to the five steps of inductive coding (preparation, close reading, creation of categories, analysis of overlapping coding and uncoded text, and refinement). Moreover, none of the data findings have been paraphrased. To prevent bias from the surveys, the transcripts were kept confidential. The steps of inductive coding were performed with two researchers when deemed necessary and when there was doubt of, for example, the creation of a category it was discussed until consensus was reached. Moreover, factual statements from the surveys were checked when possible and this also shows the truthfulness of the survey results and this study. In addition, some close-ended questions were used to support the interpretation of the open-ended questions. The results of these questions were analyzed by calculating average scores.
RESULTS
The data were collected between April 19 and May 22 in 2016. There were 255 emails sent to persons working in HB-HTA or HTA related activities in this period, of which fifty-eight did not work due to permission or security reasons according to an automatic response from their moderator or because of outdated email addresses. From the remaining population of 197 persons, there were thirty-six respondents who answered the questions from the survey of which one participant was excluded because he was currently not involved in any HB-HTA activities and was unable to answer the questions. Therefore, the response rate of the survey was 18.3 percent.
Characteristics of Respondents
Fourteen respondents (40.0 percent) were heads, managers, or directors in HTA or HB-HTA departments, and twenty-one respondents (60.0 percent) were researchers, physicians, or consultants in the field of HTA or HB-HTA. In addition, from these thirty-five respondents, there were twenty-three respondents (77.1 percent) working in a local setting, which means that they were working in a hospital or a local healthcare setting, and twenty-seven respondents (77.1 percent) were currently working directly with HB-HTA. Other respondents were working for instance in national HTA organizations but have previously worked in HB-HTA or were knowledgeable about HB-HTA. Moreover, participants from eighteen different countries were obtained, see Table 1.
Description of Respondents by Country, Job Function, HB-HTA Awareness, and Work Setting
a When a respondent answered both local and, for example, national, it will still be marked as local here.
Existing Methods of Sharing and Collecting HB-HTA Reports
In total, 65.8 percent of the participants responded confirmative to the question “Does your organization share HB-HTA reports externally?”. Six different methods of sharing HB-HTA reports by the respondents were identified (see Table 2). The most common used methods to share HB-HTA reports were internal databases or publications on Web pages of for instance hospitals or national authorities (e.g., www.sst.dk > planning) where the concerning HB-HTA units were based (54.5 percent) and was especially prevalent in countries such as Canada, Denmark, and Sweden. The second most used method is the collection of HB-HTA reports through networking and direct requests (40.9 percent). This means that information is often only disseminated within local or national networks among, for instance, hospitals. Example countries in which this method was mentioned are Spain, Switzerland, and Canada. Third, 27.3 percent of the respondents mentioned that they shared their HB-HTA reports externally through common databases. Examples of countries using this method of sharing are Canada, Denmark, Sweden, and the United States. The mentioned databases are: EuroScan, INAHTA, CRD, CADTH, and PubMed. Last, the three other methods of sharing HB-HTA reports are the use of scientific journals (22.7 percent), the newly developed AdHopHTA database (18.2 percent), newsletters (4.5 percent), workshops (4.5 percent), and conferences (9.1 percent).
Current Methods of Sharing HB-HTA Reports Externally
a A respondent can use one, more than one, or no method.
Values, Usefulness, and Barriers of Collecting and Disseminating HB-HTA Reports
In the close-ended questions, every respondent agreed with the second statement of section 3, which states, “The collection of hospital-based health technology assessment (HB-HTA) information is useful.” Furthermore, in the open-ended questions, there were in total seven different reasons distinguished for values and reasons for usefulness combined and fourteen for barriers. The categories that were identified during the analysis can be found in Table 3.
Values, Usefulness, and Barriers of the Collection and Dissemination of HB-HTA Reports According to the Respondents, a Categorization Table Through Content Analysis
The most frequent answer to the question on values and usefulness is that the collection and dissemination of HB-HTA reports will reduce duplicative work in this field (54.3 percent). The second most mentioned value is that the collection helps with further HB-HTA research (42.9 percent), followed by the fact that previous research will help with the decision and policy making for other hospitals (40.0 percent). Other answers can be found in Table 3.
The most frequent mentioned barrier is the problem of not being able or willing to share confidential information (45.7 percent). This often refers to confidential information regarding costs but some respondents also mention legal, ethical, political, and patient information issues. The second most mentioned barrier for the collection and sharing HB-HTA reports is that of the investment of time and money to maintain and update a good working database (40.0 percent). Barriers not mentioned in the Table but brought up by the respondents are low availability of internet, bureaucracy, unknown reports, increase of inertia on the part of some hospitals, decentralization of production, ensuring the right use and intention of reports, and difficulties finding the relevant reports.
Criteria Database, Willingness to Share Reports, and Opinion on the AdHopHTA Database
The main criteria to establish and maintain a database of HB-HTA reports according to the respondents can be divided into six main different categories (see Table 4). The most frequented used criteria are those of having arranged organizational issues and factors regarding funding, management, the collection and the spread of HB-HTA information, and educating stakeholders in the field of HB-HTA (40 percent). The second most mentioned criterion for a database according to the respondents is that the HB-HTA reports that would be submitted to such a database, will have to be of clear defined high quality (37.1 percent).
Categorization Table Through Content Analysis in Three Areas: (Reference Kristensen and Sigmund1) Criteria According to the Respondents to Establish, Implement, Maintain, and Improve a Database; (Reference Cicchetti, Sampietro-Colom, Martin, Marchetti, Sampietro-Colom and Martin2) What Information the Respondents Are Willing to Share; (Reference Grenon, Pinget and Wasserfallen3) Opinion of the Respondents on the Amount of Information in the AdHopHTA Database
a Only relevant for the respondents in the position of sharing HB-HTA reports (n = 24).
In total, 62.9 percent of the respondents agreed, 17.1 percent had no opinion, and 20 percent disagreed to the statement that “sharing HB-HTA reports conducted by my organization would not be a problem”. For the open question of “what information people are willing to share”, only the respondents that were in the position of sharing HB-HTA reports were included (n = 24). The information respondents are willing to share can be divided into three categories.
The last question in the survey asked directly about the opinion of the respondents on the AdHopHTA database. Those who had an opinion (74.3 percent) were to be differentiated into five categories. The most mentioned criterion was that there is a need for more explicit information (25.7 percent).
DISCUSSION
This study shows that experts in the area of HB-HTA from different countries in general agree that collection and dissemination of HB-HTA reports is useful and valuable. It seems that there are opportunities and supporting arguments to help solving the underreporting of HB-HTA reports and, thus, the waste of knowledge in this area (Reference Gagnon4). Moreover, this study shows that many experts have already established networks and HB-HTA unit's databases to share reports internally or on a small scale. This suggests that there is a demand among researchers and policy makers for these reports and also that there is a willingness to act upon this demand.
However, there are some barriers that prevent some potential contributors from sharing reports in an open manner. Most important barriers include confidentiality, time and money to establish, maintain and update a usable database, and applicability problems (e.g., differences in context, economics, methods, setting, and framework). It is, therefore, the question whether the value and need to share and collect reports is greater than the barriers or whether these barriers can be overcome. Moreover, right now HB-HTA reports are not shared in a central database. This might be due to some of the barriers (e.g., confidential information and intellectual property issues) or due to the lack of availability of such a database.
By looking to the results of this study it can be argued that the willingness is present for most persons in the position of sharing HB-HTA reports but that a well-established cooperation and resources are not yet available. It could be of added value if there was more research conducted on this topic among, for example, hospital managers who are not involved in HB-HTA or HTA and to explore whether they also think that the collection of these reports is valuable. This could enlarge the use of the submitted reports in a database extensively.
Furthermore, this could potentially lead to more financing and wider recognition of the usefulness of HB-HTA research, as some of the experts mention in the survey. This is important because it will enable hospital managers to make informed decisions and this will improve the quality of care (Reference Kidholm, Ølholm and Birk-Olsen6). Moreover, the number of unpublished HB-HTA reports that could contribute to the informed decision making by hospital managers all over the world is unclear. It seems that only few HB-HTA studies are actually published, but further research should be conducted to give a more precise number because this would show the potential benefit of improving and maintaining a database. However, this study already shows that there are multiple hospital managers that mention that they only share certain reports through for instance direct contact.
One possible reason of why there are not yet many HB-HTA reports submitted to public databases is that there is not a right platform for the possible contributors to share these reports. However, this problem has also been recognized by the European Commission (EC). Therefore, the EC has proposed regulation that would address several problems with the current situation, of which standard IT tools (e.g., central database) is an important aspect (13).
Important Barriers
The most frequently mentioned barrier is the inability or unwillingness to share confidential information included in some of the HB-HTA reports. In other words, hospital managers more often than researchers are reluctant about promoting openness and transparency because this could put a hospital in a vulnerable position (Reference Burns14). Specifically, the transparency about the quality of health care and price is largely missing in the health care of some countries (Reference Burns14). However, it is argued that it is favorable that hospitals should be transparent and that “confidential” information should be open to the public because this will indeed indicate the weak points of a hospital and will, therefore, enable improvement (Reference Burns14). This could in turn improve the hospital and the quality of care because it will show the comparisons and differences with their competitors (Reference Burns14). Building on this development, patients are likely to develop more confidence in the health care given in their country and it in turn would promote more competitiveness between hospitals (Reference Goozner15).
The second most frequently mentioned barrier is the need of time and resource investment to establish, maintain, and update a database in a consistent manner. In other words, as one of the respondents mentioned, “there will have to be a serious plan to finance and manage such an activity to ensure sustainability and development”. Thus, there will have to be financing and plans to manage such an operation as maintaining a database. It would probably be beneficial if an organization would actively promote the database to ensure that it would be used by as many stakeholders as possible. The third barrier to be considered is the applicability problems of reports. HB-HTA reports contain hospital specific information and it is, therefore, in the concern of some of the experts whether the reports are transferable and applicable for other hospitals or contexts.
However, as stated in the AdHopHTA handbook (Reference Granados9), HB-HTA reports should clearly state their goal and scope and should present analysis that are performed systematically using good practice methods and appropriate tools. This way, it could be adopted by other hospitals according to the AdHopHTA handbook (Reference Sampietro-Colom, Lach and Cicchetti8). Moreover, the AdHopHTA organization has produced a checklist for assessing the quality of HB-HTA reports and one of the three guiding principles is regarding methods, tools, and transferability (Reference Sampietro-Colom, Lach and Cicchetti8). It is argued that the generalizability and transferability of data in HTAs is increasingly relevant as health care becomes more globalized (Reference Stephens, Handke and Doshi16), and an international database for HB-HTA reports would again stress the need of the application of standards in the conduct of HTA to increase the comparability of results.
Criteria and Design of a Database
This study shows that there is room for improvement regarding the criteria and design of a database. The AdHopHTA database is relatively new (created around 2014) and other databases specifically for HB-HTA do not exist (Reference Sampietro-Colom, Lach and Cicchetti8). Most respondents mention that the AdHopHTA does not contain enough information and also many other respondents have not formed an opinion yet. Suggested changes are that entire reports are shared instead of only abstracts of reports. Therefore, the amount of information given in the AdHopHTA database is according to some potential contributors not enough and it is advisable to change this to full reports where possible. The observations of the respondents are accurate because, for the database, you only need to submit an abstract. Entire reports can then be retrieved by contacting the contact person of that abstract. Therefore, the findings of this study show that it is recommended to share the reports more openly and to for instance create checklists and templates to ensure comparable and complete information.
Furthermore, experts involved in this study mention several criteria for a database which could be used for the improvement of the AdHopHTA database or establishment of another database. Clearly defined high quality criteria for reports, reliable and easy to use search tools, and contact information from the authors are some of the criteria mentioned by the respondents. In addition, some respondents of this study pointed out that it is the question of whether a separate database is necessary to collect HB-HTA reports or not.
Limitations
This study has some limitations in its study design. First, the study population gives an overview of opinions within a large variety of countries. However, it may underrepresent Germany, the Netherlands, and the United Kingdom where HTA is already well established, at least at the macro level. Moreover, given the limited sample size, it was not possible to go into detail about the problems within each country. More studies should be done within countries to find for instance the barriers, willingness, and opinions to collect and share HB-HTA reports. It would then become clearer what the contributing factors are, such as the amount of presence of HB-HTA, health systems, and cultural differences, and how this affects the willingness to share reports. This would probably lead to more inclusion of respondents because specific barriers can be dealt with, ensuring greater cooperation and participation in these countries.
Second, this study has described the main barriers for the collection and dissemination of HB-HTA reports. Now the barriers have been identified in this explorative research, it would be useful to have a more quantitative research that goes more into depth about the possible solutions and what the stakeholders think of those solutions. Especially, the development of a template for encoding basic information about each HB-HTA report would be interesting. This research could find out how many extra reports this would deliver to a database if these solutions were to be put into practice.
Third, because the HB-HTA experts are located in many countries, a survey was used. The response rate of the survey (18.3 percent) was somewhat low. Literature shows that response rates to email surveys have decreased since the late 1980s. Now, email response rates may approximate 25 percent to 30 percent (Reference Fincham17). We hypothesize that the survey rate we obtained in this study was below this approximation because it was sent to an external population, the relatively long length of the survey, and the very technical subject. Moreover, the data obtained in this research is relatively superficial due to the properties of a survey. It could, therefore, be beneficial if there was more in-depth research to the opinions of leading HB-HTA stakeholders on the barriers and solutions of sharing HB-HTA reports and the design of a database by conducing for instance open-ended interviews. However, the current study was meant to be explorative and remains, therefore, valid although more research will have to be conducted.
Last, this study focused on people with experience in HB-HTA and are, therefore, perhaps more likely to be in favor of the dissemination and collection of HB-HTA reports. It would be of much added value if another research would focus on for instance hospital managers who have not yet been in contact with the concept of HB-HTA or to conduct research in countries which have not (yet) implemented any HB-HTA activities.
In conclusion, this study suggests that there are opportunities and interests for sharing HB-HTA reports and to have a database to collect HB-HTA reports from many different countries and that many respondents are willing to share their HB-HTA reports. However, there is need to overcome the barriers of collecting and sharing HB-HTA reports, to meet some criteria for the database, and to make sure that people in the right position submit their reports. More research will have to be conducted to examine whether a central database could be used by an international community as opposed to the multiplicity of various systems to which HB-HTA reports are currently disseminated.
SUPPLEMENTARY MATERIAL
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462318000570.
Supplementary File 1: https://doi.org/10.1017/S0266462318000570
CONFLICTS OF INTEREST
There was no funding for the making of this article and the authors declare they have no competing interests.
