Hepatocellular carcinoma (HCC), the predominant form of liver cancer, is one of the most common and lethal malignancies worldwide. The majority of patients with HCC are diagnosed in the advanced stages of presentation due to the relative paucity of symptoms in the early stages. As a consequence and due to the multifocal and advanced stage of disease at time of diagnosis, potentially curative treatment for HCC is not feasible in 80 percent of patients (Reference Zhu23). Current treatments for HCC include surgical resection, liver transplantation, and ablation therapy. Inoperable or unresectable tumors are treated with transcatheter arterial chemoembolization (TACE). Unlike other cancers, the effectiveness of chemotherapy is limited, however, newer agents are emerging and expected to have an impact on treatment outcomes (Reference Lord, Suddle and Ross10). Several technologies that are emerging in this field have been identified by horizon scanning systems (HSSs) (see for example Reference 1).
HORIZON SCANNING
For many years, health technology assessment (HTA) agencies have been interested in early identification and monitoring of new and emerging technologies. However, in practice, HTA focuses on established technologies, identifying emerging and new technologies before they are licensed or launched; prioritizing, assessing, and disseminating the information to decision makers is the function of HSSs. According to Douw et al., the aim of HSS is to assist control and “rationalize the adoption and diffusion of new technologies in health care practice” (Reference Douw, Vondeling, Eskildsen and Simpson5). Horizon scanning usually involves regular searches of different information sources, such as medical news, clinical literature, as well as contacts with industry, researchers, and clinical experts. Technologies identified by horizon scanning systems might be monitored for assessment in the future or prioritized for early assessment (Reference Stafinski, Topfer, Zakariasen and Menon18). For example, in the United Kingdom new pharmaceuticals identified by the National Horizon Scanning Centre (NHSC) are reviewed by the National Institute for Health and Clinical Excellence (NICE) to determine whether appraisal or guidance for clinical practice is needed (Reference Simpson, Packer and Carlsson17). Furthermore, the International Information Network on New and Emerging Health Technologies (EuroScan) has currently seventeen members based in fourteen countries. This includes the Australia and New Zealand Horizon Scanning Network (ANZHSN), the Canadian Agency for Drugs and Technologies in Health (CADTH), the Swedish Council on Health Technology Assessment (SBU), and the French Committee for the Evaluation and diffusion of Innovative Technologies (CEDIT), just to name a few.
In an exploration of the principal sources of information for identifying emerging healthcare technologies, Robert et al. found that medical experts were recognized as an important source of information (Reference Robert, Gabbay and Stevens13). Stafinski et al. have described clinicians as important “sentinels” that not only serve as practitioners and patient advocates but innovators, developing and applying new technologies. The author's study, which involved surgeons in a horizon scanning workshop, concluded that they are an accessible and important source of information for identifying emerging technologies for HTA (Reference Stafinski, Topfer, Zakariasen and Menon18). The Euroscan toolkit for the identification and assessment of new and emerging technologies also lists “experts” as a source of information (Reference Simpson, Hiller and Gutierrez-Ibarluzea16). A survey of horizon scanning systems conducted by Douw and Vondeling found that the “filtering process” (i.e., the step before the actual selection of technologies) could be improved. At the time this study was conducted, two HSSs used a form with predefined questions and asked individual experts or groups of clinical experts about the time horizon of the introduction and the likely impact of the technology in the healthcare system (Reference Douw and Vondeling4). By selecting inappropriate technologies, there is a waste of scarce analytical resources. New methods could be used to enhance this process and make it explicit and systematic. Consequently, this study used for the first time a type of discrete choice experiment called best-worst scaling (BWS) to explore clinicians’ views on emerging technologies with respect to their expected impact on HCC outcomes in the next 5 to 10 years.
METHODS
Identifying the Emerging Technologies
A qualitative study and a targeted literature review that included databases of peer-reviewed literature CINAHL, EMBASE, Medline, and PubMed with the following terms ((“liver”[MeSH Terms] OR “liver”[All Fields]) AND (“neoplasms”[MeSH Terms] OR “neoplasms”[All Fields] OR “cancer”[All Fields]) AND (“technology”[MeSH Terms] OR “technology”[All Fields] OR “technologies”[All Fields]) AND (“2006/01/01”[PDat]: “2010/10/01”[PDat]) allowed the identification of emerging health technologies. Respondents for the qualitative study were purposively sampled in eleven different countries: Australia (n = 1), China (n = 1), France (n = 1), Germany (n = 3), Italy (n = 1), Japan (n = 2), Spain (n = 1), South Korea (n = 2), Taiwan (n = 1), Turkey (n = 3), and the United States (n = 4); to constitute a geographical and professionally diverse sample of clinical experts in liver cancer and related disease prevention, detection, and management. Potential respondents were identified from peer reviewed publications and lay media, presentations at liver disease conferences, leadership roles in national societies, centers, or advocacy organizations, government agencies, and recommendations from advisory groups and other respondents. This ensured that the emerging technologies chosen were grounded in the experts’ experience. Detailed description of this qualitative study is provided elsewhere (Reference Bridges, Gallego and Blauvelt2).
Eleven health technologies were identified by means of the literature review and the qualitative study:
• Adjuvant/Neo-adjuvant therapies: Therapy used in combination or after surgical resection of HCC.
• Biopsy-free HCC diagnostics: Advanced imaging or other noninvasive diagnostics or scanning techniques to precisely confirm the diagnosis of benign or malignant tumors.
• Early detection of HCC: Development of a range of technologies to track methylation of hepatitis viruses or other liver disease to be able to better predict the most probable time of HCC development, for example, before actual radiologic confirmation of a tumor.
• Genetic/genomic biomarkers: Gene array analysis and identification of genomic biomarkers that will be useful in predicting risk of HCC and/or likely responsiveness to specific therapies.
• HCV vaccination: Therapeutic vaccines for the treatment of hepatitis C virus.
• Immunomodulation; Ability of the immune system to identify and destroy tumor cells and to elicit a long-lasting memory of this interaction.
• Improved surgical techniques: Advances in surgical techniques that reduce the need for liver transplantation and improve outcomes following surgical interventions in HCC.
• Interventional radiology: Improved radiology treatment or therapy in HCC tumor management.
• Molecular targeted therapy: Treatments that work by interfering with specific molecules involved in tumor growth and progression.
• Stem cell therapy: The advent of personalized treatment and other applications of stem cell research applied to liver regeneration, particularly in cases of resection or ablation of the liver due to HCC.
• Transplant technology: Advances in transfer of liver / liver tissue from one person to another, including transportation and maintenance of viable livers for transplantation.
The technologies encompassed diagnosis, treatment, and management of HCC. Some of the technologies described were not novel, for example neoadjuvant/adjuvant therapies are currently available. However, their effectiveness is limited hence the respondents expect “new” and effective neoadjuvant/adjuvant to be available and have an impact in the next 5 to 10 years.
THE EXPERIMENT
Clinicians’ views on the impact of emerging health technologies were explored using best-worst scaling (BWS), administered as part of a cross sectional survey of health professionals involved in liver cancer and related disease prevention, detection, and management. BWS is rooted in random theory and described as a compromise between discrete choice experiments and ranking scales (Reference Louviere and Flynn11). BWS assumes that respondents can easily choose items that are extremes (best and worst, most and least, smallest and largest) in a set of three of more items. In this study, the items are emerging health technologies which were presented in choice sets. To construct these sets, a balanced incomplete block design (BIBD) was used (Reference Street and Street19). A BIBD is a well-studied experimental design which gives fixed set sizes and has various desirable statistical properties, such as equal occurrence and co-occurrence of items across all comparison sets. Unlike fractional factorial experimental designs from the 2k family, BIBDs produce comparison sets of a fixed size for a given number of items (Reference Louviere and Flynn11). This design ensured that each task contained five technologies, each one appeared five times and as a pair to another given technology five times. Respondents were presented the eleven sets (one at a time) and asked to identify those technologies that will have the most and least likely impact on HCC outcomes and control within 5 to 10 years (see Figure 1). In this study, the assumption was made that when participants are presented with five technologies they chose most first (choice 1). From the four remaining technologies, they then chose least (choice 2). This is known as sequential best-worst.
Example of best-worst choice task.
Participants
Ten countries were selected in three different regions (i) Asia Pacific: China, Japan, South Korea, and Taiwan (ii) Europe: Italy, France, Spain, Germany, and Turkey; and (iii) North America: United States. Criteria for country selection included: liver cancer had been identified as a national priority and/or liver cancer guidelines have been formulated, as well as geographical dispersion. Respondents were selected if they were involved in HCC prevention, diagnosis, treatment and care, active clinicians at major medical institutions (including cancer and other liver disease centers); active members in national and international liver associations and/or published extensively in peer review journals. Exclusion criteria included: not board certified, certified for less than 1 year, practicing medicine for less than 3 years, living/practicing in country for less than 3 years. Respondents who participated in the qualitative study were also excluded.
Sample Size
The current theory of sampling for discrete choice experiments such as best-worst scaling does not directly address the issue of minimum sample size requirements in terms of reliability of the parameter estimates produced in the design. The final sample size required is based upon characteristics of the design itself such as the number of attributes included, the number of choice scenarios presented. Researchers may wish to rely on sample size estimation methods for discrete choice models, such that provided by Kessels et al. (Reference Kessels, Jones, Goos and Vandebroek14) for the general case of best only choices. It is worth noting, however, that sample size requirements for Case 1 best-worst choice models will be less than sample sizes for best only choices as BWS choices provide extra choice data for any given sample size.
Data Collection
The survey was conducted between September 2010 and January 2011. Potential respondents (N = 323) were informed about the study by means of email or mail, and received a follow-up telephone call or email if a response was not received within 2 weeks. The email or mail invitation was sent in English and the country's official language. Potential respondents were sent a maximum of four reminders; if no response was received they were noted as a “no response.” Upon a positive response, the interviewer scheduled an appointment for administration of the survey instrument in person or by telephone.
Data Analysis
Three measures of priority scores were used: most minus least scores, square root estimates, and conditional logistic regression.
Most and Least Scores
The “most minus least scores” also known as the scores were calculated as per Finn and Louviere (Reference Finn and Louviere7), by considering the number of times an emerging technology was chosen as most and the number of times it was chosen as least across all choice sets and respondents. The total number of times it was picked as least was then subtracted from the number of times it was picked most. This gives the initial ranking of all eleven emerging technologies from most to least.
Square Root Estimates
To account for the number of times the technology was available the most and least scores were then divided by the availability of each emerging technology (the number of times it appears across the design in this case 5 times the sample size (i.e., number of respondents who completed the survey) 120 (5 × 120 = 600). Then the square root of the scores were calculated by dividing the number of times each technology was chosen as most by the number of times it was chosen as least and taking the square root of the resulting number. The natural log of the square root was then calculated.
Conditional Logistic Regression
Each chosen emerging technology was expanded to the 2K-1 = 9 (five best and four worst) available in each choice set (profile). The outcome variable, choice, was coded equal to one whether a best attribute level or a worst attribute level is chosen and equal to zero for the remaining (nonchosen) emerging technologies for a particular profile (choice set) and individual. The independent variables were then coded with a sign change for all observations pertinent to the least choice data, to reflect the reciprocal relationship between most and least probabilities. A conditional logistic regression with choice as the binary outcome (0, 1) was then conducted. To overcome the long identified problem of the dummy variable, effects coding (i.e., treat the outcomes as attribute levels of a single attribute) was used.
Ethics
The study was deemed exempt from further human subject's consideration from the Johns Hopkins Bloomberg School of Public Health's Institutional Review Board (IRB). All participants were informed about the study and its potential risks and benefits. Participation in the study was voluntary and respondents could end participation at any time and, to ensure objectivity, respondents were not reimbursed for participation.
RESULTS
A total of 323 invitations were sent to eligible respondents; 159 responded saying they would participate, of these 15 did not meet the inclusion criteria and 24 returned an incomplete survey. In total, 120 international liver disease and liver cancer clinicians’ fully completed the survey, response rate was 37 percent (Figure 2 describes the study recruitment). Respondents included hepatologists (40 percent), oncologists (22 percent), radiologists (13 percent), surgeons (18 percent), and other experts (7 percent) involved in hepatocellular carcinoma (62 percent), hepatitis (16 percent), transplantation (13 percent), and metastatic liver cancer (9 percent). Respondents self-identified as having local/regional (31 percent) national (39 percent) or international (30 percent) influence in liver cancer control. Even though the aim was not to describe regional differences, analysis by region (Asia, Europe, United States) of background characteristics showed no differences amongst main area of involvement of participants (p = .082) or major area of focus in medicine (p = .335). There was statistical significant difference in terms of self-identified level of involvement in liver cancer control (p = .021), participants in the United States perceived they have more involvement at the international level compared with respondents in Europe and Asia (see Table 1).
Study recruitment.
Clinician's Background Characteristics by Region (N = 120)
Note. *This included: pathologist, immunologist, researchers. Chi-square.
Despite the cross-cultural differences and languages in which this survey was conducted, respondents noted during the interviews that the best-worst task was easy to complete and not labor intensive. The methods used to calculate priority scores (i) best/worst scores, (ii) square root and natural log best/worst scores, and (iii) conditional logit were consistent and produced the same ranking (see Table 2, and Supplementary Figures 1 and 2, which can be viewed online at www.journals.cambridge.org/thc2012040). According to the best/worst scores, it could be seen that all emerging technologies were chosen at least once as best or worst across the design (i.e., there were no zeros). The technologies with the highest score were molecular targeted therapy (310) and early detection of HCC (228). Improved surgical techniques (-192) and biopsy free HCC diagnostics (-227) had the lowest scores. When scores were determined by region (Asia, Europe, United States), in all instances molecular targeted therapy came first followed by earlier detection of HCC.
Estimate of Subjective Priority Scores for Emerging Technologies Using Three Different Methods (N = 120)
Note. Sqrt, square root; SE, standard error; ln, natural log.
The square root best/worst ratio results, which are ratio scaled estimates, allows us to make statements such as “molecular targeted therapy (7.9) is approximately eight times as important as biopsy free HCC diagnostics (0.3)” or “if asked on several occasions, the average clinician would pick molecular targeted therapy approximately eight times as often as biopsy free HCC diagnostics. Conditional logistic results were consistent with previous priority scores. The standard errors from this regression are informative in identifying heterogeneity (i.e., inter-respondent variation): larger standard errors suggest greater disagreement among respondents. According to the results, stem cell therapy, HCV vaccination, and genetic/genomic are the emerging technologies with the most preference heterogeneity).
Technologies that clinicians consider will have the “most” impact included: molecular targeted therapy, early detection of HCC, genetic/genomic biomarkers, HCV vaccination, adjuvant/neo-adjuvant therapies. Immunomodulation, transplant technology, stem cell therapy, improved surgical techniques, and biopsy free HCC diagnostics were seen as the “least” important emerging technologies. The BWS ranking results were divided into three groups (high, medium, low scores). These were compared against the number of clinical trials reported in the literature review. The first group with the highest scores encompassed: molecular targeted therapies, technologies used in early detection of HCC, and genetic/genomic biomarkers. Consistent with the high scores, the literature review found that several clinical trials are currently under way exploring the effectiveness of different molecular targeted therapies (Reference Wysocki21). A noninvasive method proposed for the assessment of liver fibrosis in patients with chronic liver diseases, transient elastography, has been having promising results and could be used for early detection of HCC. Some of these trials were due to be completed in 2011 (Reference Lord, Suddle and Ross10). Biomarkers may help improve the current staging systems and facilitate the treatment decision making according to some authors (Reference Freidlin and Korn8).
A second group included HCV vaccination, interventional radiology, and adjuvant/neoadjuvant therapies. Currently inoperable or unresectable tumors are treated with transcatheter arterial chemoembolization (TACE-an interventional radiology procedure); according to the literature, there are ongoing trials investigating advances in TACE in combination with molecular targeted therapies and other pharmaceuticals. Trials are also under way exploring the use of adjuvant treatment in the prevention and recurrence of HCC either after local ablation or surgical intervention (Reference Ye, Takayama, Geschwind, Marrero and Bronowicki22). A review by Torresi et al. described recent progress in the advancement of a vaccine to treat HCV perhaps in combination with interferon (Reference Torresi, Johnson and Wedemeyer20).
The third group with the lowest scores and those perceived as having the least impact were immunomodulation, transplant technology, stem cell therapy, surgical techniques, and biopsy free HCC diagnostics. Surgical resection, liver transplantation, and ablation therapy are current treatments for HCC (Reference Lord, Suddle and Ross10). New advances in these areas are limited according to the literature. Immunomodulation and stem cell therapies are in the pipeline for treatment of cancer, however, its impact could be beyond the time frame considered (5 to 10 years). Biopsy free HCC diagnostics was the technology with the lowest score therefore considered as the one to have the least impact in HCC outcomes in the next 5 to 10 years. Current HCC treatment is guided by staging of the disease which means biopsies are still needed (Reference Bruix and Sherman3). However, the literature identifies that there is room for improvement in current technologies use for HCC diagnosis (Reference Shariff, Cox and Gomaa15). Furthermore, as noted by Peng et al., “Many candidates have been identified but few have been successfully validated and made an impact clinically” (Reference Peng, Wang, Geng and Zhang12).
DISCUSSION
As noted by Shariff et al., the recognition of HCC as an important “global health issue” means that changes in the incidence, diagnosis and treatment of HCC are certain to take place over the next few years (Reference Shariff, Cox and Gomaa15). Consistent with the study results, recent reviews have identified key areas of advancement in HCC including neoadjuvant or adjuvant therapies, chemoprevention after resection, the use of genomic- and proteonomics-based technologies, and molecular targeted therapies (Reference Lord, Suddle and Ross10;Reference Ye, Takayama, Geschwind, Marrero and Bronowicki22).
Ibargoyen-Roteta et al. have suggested that early assessment of emerging technologies can aid the decision-making process and could promote the adoption of beneficial technologies (Reference Ibargoyen-Roteta, Gutierrez-Ibarluzea, Benguria-Arrate, Galnares-Cordero and Asua9). According to Douw and Vondeling (Reference Douw and Vondeling4) horizon scanning systems are meant to identify innovations likely to have a significant impact. However, the survey of thirteen members of the EuroScan (Reference Douw and Vondeling4) found that some early warning systems have “missed important technologies” and in some cases the process has been described as subjective. To date studies have used qualitative methods such a Delphi, and semi-structured interviews to elicit clinicians’ views on the impact of emerging technologies; furthermore, Douw et al. asked clinical experts to “vote” (Reference Douw, Vondeling and Oortwijn6;Reference Stafinski, Topfer, Zakariasen and Menon18). To overcome the limitations of the methods previously used, this study used a quantitative method BWS to prioritize emerging technologies likely to impact hepatocellular carcinoma (HCC) outcomes in the next 5 to 10 years.
Emerging health technologies could offer improved patient outcomes while producing pressure on healthcare budgets, one of the key challenges for healthcare policy makers is to decide which technologies to fund. The information provided by this stated preference method could be used by policy makers as a part of the horizon scanning process, allowing them to filter or prioritize innovations in the field likely to have a significant impact, enabling appropriate decision making (such as resource allocation), and adoption as well as identification of further research requirements.
This research also demonstrates the value of including clinicians’ preferences as a source of data in horizon scanning, but such methods could be used to incorporate the opinions of a broad array of stakeholders, including those in patient/disease advocacy and public policy. Further research could explore the differences amongst countries or stakeholders. This is the first application of BWS as a horizon scanning tool to prioritize emerging technologies. Other applications of BWS in healthcare policy could include: ranking for health priorities, stakeholder involvement, and health needs assessment.
STRENGTHS AND LIMITATIONS
The difficulties of engaging stakeholders and groups of experts are acknowledged. However, discrete choice methods like best-worst scaling could be potentially better and more generalizable methods for the inclusion of stakeholders and experts, compared with other methods such as Delphi, interviews, or voting. These activities, however, will need to be supported by horizon scanning agencies which could build the infrastructure, capacity, and expertise on these prioritizing methods to facilitate the involvement of a larger set of stakeholders and experts.
Multiple methods of analysis were used to demonstrate the robustness of the results. As previously noted all scores produced the same ranking which highlights the validity of the BWS approach. While very complicated methods such as conditional logistic regression can be used to model the underlying preferences of the respondents, simplified methods (i.e., scores) which are less technical demanding can be used and obtain consistent results.
This study has some limitations. First, it is unknown if clinicians’ considered the financial, organizational, social, and or ethical, aspects of these technologies which are important in a horizon scanning system. Nevertheless, this method could be used to involve other stakeholders and ask them to consider these aspects. The most appropriate method to identify new technologies is still a challenge; however, the focus of this study is on the use of best-worst scaling for prioritizing technologies. Other more traditional methods for the identification of technologies might be more suitable and will need to be considered. The technologies were broad (diagnosis, treatment, and control of HCC) and the definition could have been ambivalent, for example some people might have interpreted HCV vaccination as preventative. However, all efforts were made to ensure consistency, all respondents were briefed in the same way by trained interviewers.
Even though the response rate might appear low (37 percent), compared with similar studies the response rate is rather high, especially considering that that no incentives were given for participation. Information about the reasons why some clinicians did not complete the survey is not available. However, potential respondents had to meet the same inclusion criteria to be invited so they have similar characteristics to the nonrespondents. Finally, because respondents were based in different countries, their values, cultures, and healthcare priorities will differ. Nevertheless, the aim of the study was not to explore these differences but to demonstrate that BWS could be an important research tool to facilitate horizon scanning and HTA more broadly.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1
Supplementary Figure 2
CONTACT INFORMATION
Gisselle Gallego, BPharm, PhD, Postdoctoral Fellow, John F.P. Bridges, PhD, Assistant Professor, Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
Terry Flynn, PhD, Senior Research Fellow, Centre for the study of choice (Censoc), University of Technology Sydney, Sydney, Australia
Barri M. Blauvelt, MBA, Adjunct Faculty, Institute for Global Health, University of Massachusetts, Amherst, Massachusetts; CEO, Innovara, Inc., Hadley, Massachusetts
Louis W. Niessen, MD, MsC, PhD, Associate Professor, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland; Professor Public Health Economics & Public Health Modelling School of Health, Health Policy and Practice, University of East Anglia, United Kingdom; University of Cambridge, Cambridge, United Kingdom, Director, Centre for Chronic Diseases ICDDR, Bangladesh
CONFLICT OF INTEREST
Gisselle Gallego, John Bridges, and Louis W. Niessen report their institution has received a grant from Innovara; Gallego and Bridges have received support for travel to meetings for the study from Innovara/Bristol-Myers Squibb; and Bridges has received consulting fees from Innovara. The other authors report they have no potential conflicts of interest.
