As the importance of innovative technology expands in the healthcare sector, new practices and organizations are constantly evolving. New technology enables the refinement and personalization of existing healthcare practices, which have the potential to prevent grave diseases and save more lives. Although the technological revolution within health care shows great potential, not all innovations serve their purposes (Reference Strønen, Hoholm, Kværner and Støme1). Documenting the effects of healthcare innovation is, therefore, essential in prioritizing adequate technology implementation. Unlike the product cycle of pharmaceuticals, where the timeframe from development to implementation can take several years, technology-enabled and organizational innovation in the healthcare sector move at a much faster pace (Reference Tarricone, Torbica and Drummond2). The methods for value assessment and priority settings, therefore, need to be adapted to a faster product cycle with a greater diversity of products.
Over the past few decades, validated methodology such as health technology assessment (HTA) has contributed to sound decision making worldwide. HTA is defined as an interdisciplinary process for synthesizing information regarding medical, social, economic, and ethical issues related to the introduction of new health technology (Reference Kristensen, Lampe, Chase, Lee-Robin, Wild and Moharra3). Although HTA methods and approaches have been subject to significant improvements over time, there are several challenges in the field of health technology assessment (Reference Husereau, Henshall, Sampietro-Colom and Thomas4). HTA is deemed a robust method for technology in later phases of national implementation.
In its current form, it continues to lack the incentive to promote innovation, include local considerations for decision making at an institutional level, and express the value of dynamic interactions with private businesses. This challenges HTA in showing the whole value chain to promote value-based health care. Hospital-based HTA (HB-HTA) is an approach adapted to inform decision makers at different levels in a hospital setting and ensure acceptability at a local level. This includes processes and methods used to produce HTA reports in and for hospitals (Reference Sampietro-Colom, Lach, Cicchetti, Kidholm, Pasternack and Fure5). Although this assessment and management tool successfully addresses decision making at an institutional level, more research is necessary to identify sustainable innovative ideas and products in the healthcare system (Reference Nielsen, Funch and Kristensen6). In promoting innovation in the healthcare sector, research should be dedicated to methods and approaches for early assessment to allocate public support effectively and produce the most beneficial technology for society.
The international network EuroScan, a collaborative network for information exchange on important emerging new drugs, devices, procedures, programs, and settings in health care, is currently evaluating the consequences of early technology assessments on the diffusion of new technologies in the healthcare sector. An article from the network states that early awareness is increasingly becoming an important component in decision making, implementation, and the spread of new health technology (Reference Packer, Simpson and de Almeida7).
Although limited, an increasing number of reports on the methods of early assessment can be found in the literature. Many of these studies take an industry perspective, emphasizing market entry and reimbursement (Reference Hartz and John8). Both individual studies and review papers broach the subject of early assessment of medical technology (9;10). Fasterholdt et al. (Reference Fasterholdt, Krahn, Kidholm, Tderstræde and Møller Pedersen10) provide an overview of early assessment of medical technology and discuss which models hold the most promise for hospital decision makers. However, early decision support for service innovation in a healthcare setting is less embodied in the literature. A service innovation can consist of both a technology-enabled reorganization of the health supply or simply an organizational innovation. A mobile application for the registration of blood sugar levels for diabetic patients can change patient pathways and create a new service, which is an example of a technology-enabled service innovation. However, reorganizing the health supply such that a healthcare worker measures blood sugar levels at the patients’ homes would also be a service innovation in terms of an organizational innovation.
The aim of the present study was to identify methods and discuss the analytical approaches applied to the early assessment of innovation in a healthcare setting, with a particular focus on technology-enabled and organizational innovations. The characteristics of the analytical approaches applied will be discussed according to the stage of innovation.
Methods
A knowledge synthesis based on a structured search and thematic analysis was conducted to identify early assessment methods used to evaluate innovation in the healthcare sector. This review attempts to summarize existing studies on a specific topic to improve understanding and identify research gaps to define future research. The knowledge synthesis also seeks to address broader topics, where a diversity of study methodologies and designs exist and synthesize the findings narratively.
Search Structure
The review was structured according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) (11). The review of the articles was accomplished in two consecutive screenings. (i) Articles reporting on the early assessment of innovation in the health sector (articles were excluded if they did not report on assessment in the healthcare sector, for instance, if the evaluation only took place in the industry) and (ii) articles reporting on methods or practices for the early assessment of health innovations (articles were excluded if they did not report on the early assessment of technology-enabled or organizational innovation). Detailed inclusion and exclusion criteria are shown in Table 1.
Inclusion and Exclusion Criteria
Identifying the Research Issues
Based on the health technology assessment (HTA) definition of the International Network of Agencies for Health Technology Assessment, “early assessment of medical devices” can be defined as the early examination of the medical, economic, social, and ethical implications of the medical device to determine the potential of incremental value in health care (12). The research aim was to identify methods for the assessment of early assessment of technology-enabled and organizational innovation in a healthcare setting and discuss the analytical approaches applied according to the stage of innovation (development, introduction, and early diffusion).
Identifying Relevant Studies and Study Selection
A literature search was conducted in January and February 2017 of the major medical reference databases (MEDLINE Ovid and Embase Ovid). Due to the limited amount of literature on this topic, we did not set a limit on the publication date. The protocol, search strategy, and literature search were elaborated and undertaken in collaboration with a librarian with vast experience in knowledge-based synthesis.
The search was accomplished using a combination of controlled vocabulary (medical subject headings and Emtree terms) and text words. The search strategy for MEDLINE was built using the MeSH term “Technology Assessment, Biomedical” and synonyms and near-synonyms thereof combined with the text words “early,” “pilot,” “novel,” or “first-stage” or “first-phase” or “horizon.” This search component was then combined with search terms covering various methods and theories using MeSH terms such as “Decision Support Techniques” OR “Cost Benefit Analysis” OR “Risk Assessment” and text word equivalents. The MEDLINE search strategy was translated and adjusted and then conducted in Embase.
A similar search was conducted in Cochrane Library using the keywords “Technology Assessment, Biomedical” combined with the text words “early, pilot, novel,”.. Due to the novelty of the topic and that Cochrane Library is a database for systematic reviews, the search resulted in significantly fewer outcomes. The complete search is visible in the Supplementary Material.
Although the search identified literature from the EuroScan network, much of this literature was excluded as it mainly concerned horizon scanning and early alert systems which is not subject of this review.
Table 1 shows the final inclusion and exclusion criteria agreed to by the review team. References from each database search were imported into database-specific folders in EndNote version X7 and duplicates were eliminated. Abstracts were first assessed by LNS using the selection criteria listed in Table 1 and then each of the full-text articles was appraised independently by two reviewers (L.N.S. and K.J.K.). Disagreements were solved by means of discussions or referred to a third author (K.K.).
Charting the Data and Collating, Summarizing, and Reporting the Results
The data were initially extracted by L.N.S. and then discussed with K.J.K. A framework based on the assessed literature was agreed upon and core themes to answer the research issue were identified. When there was a disagreement among the authors as to the appropriate theme, the article was discussed until agreement was achieved. Bibliographic data and study content were collected and analyzed using templates developed iteratively with feedback from the other authors (K.K. and T.M.).
Data Collection: Framework and Themes
The following categories of data were extracted.
Stage of Innovation
Based on how Ijzerman and Steuten (Reference Ijzerman and Steuten13) distinguished early HTA in different stages, this review divided the innovation process into the following three stages: the developmental stage, the introduction stage, and the early diffusion stage. The developmental stage is when an innovation is in a concept phase and is not yet piloted. The introduction stage is when the innovation is undergoing the first pilot. A pilot study is normally a small test with a few patients in which the innovation is tested. Finally, the early diffusion stage is when the pilot is transferred or extended to other populations or locations.
Type of Innovation (Technology-enabled or Organizational Innovation)
Type of Study (Theoretical or Empirical)
Study Design
Type of analysis
The identified articles were distinguished in strategic, economic, and clinical analysis based on the purpose of the analysis and not on the analytical approach used, as one analytical approach can be used for different purposes.
Methods (Qualitative or Quantitative)
Main Target Audience
An attempt was made to identify whether the assessment targeted the following audience groups: decision makers on implementation, patients/users, healthcare personnel, or innovators.
Results
Literature Retrieval
Figure 1 is a flow chart of the literature selection process. In total, the literature search yielded 1,064 papers and 373 duplicates that were excluded. Upon reviewing the 691 abstracts, 638 papers were excluded on the basis of the following criteria: not used in the healthcare sector, not an early assessment of technology-enabled or organizational innovation, and language not in English, Norwegian, or Danish. After the first exclusion of abstracts, fifty-three articles were included in full text. Based on relevance, an additional twenty-one papers were excluded. A total of thirty-two articles met the inclusion criteria, while a further seven articles were included based on screening of their reference lists.
Overview of the inclusion process.
Table 2 describes the data extracted from the included studies and summarizes the analyses of the early assessment models. Twelve studies presented the early assessment of technology that was still in the developmental stage. Fourteen studies assessed technology in the introduction stage. Thirteen papers were included in the early diffusion stage. Twenty studies presented early assessments of medical devices, while only seven studies dealt with organizational innovation alone. Twelve studies evaluated both medical devices and the consequential organizational innovation. Table 2 shows that the included articles consisted of twenty empirical studies and nineteen theoretical studies.
Description of the Data and Data Analysis
The main target audience of the evaluation was based on the authors’ interpretation of who was likely to benefit the most from the assessment. A majority of the articles addressed decision makers on implementation as the main target audience of the assessment, equaling 36 percent of the included studies (14/39). Eleven studies targeted innovators as the main target audience, resulting in 28 percent of the included studies (11/39). A total of 26 percent of the studies targeted healthcare providers as the main audience (10/39). Only 10 percent of the studies targeted patient/users as the main audience (4/39).
Analysis of Early Assessment Models Identified: Variation in Methods Depending on Stage of Innovation
This section describes the type of analyses identified based on the innovation stage, the analytical framework used to guide the study. The methods for early data collection and assessment were categorized as qualitative (n = 15), quantitative (n = 12), and mixed method (n = 12). The studies were categorized as strategic, economic, or clinical analyses or a combination. This categorization was based on the purpose of the assessment in terms of outcomes. For example, an analysis was deemed strategic if its core outcome was to determine the acceptance rate of a technology to plan future implementation, or deemed economic if the core outcome was to determine socioeconomic value through a Markov model.
Developmental Stage
A majority of the articles presented a combination of strategic, economic, and clinical analyses (n = 6). Two studies were categorized as economic analyses, one as strategic and one as clinical. One study combined strategic and clinical analyses and one combined economic and clinical analyses. The empirical articles used analytical approaches that reflected the amount of data available and the intention of the assessment in each stage. The methods applied in the developmental stage stressed the need to generate more data. Quantitative simulations based on scenario drafting applied qualitative data from stakeholder insights, expert opinions, and focus groups to provide insights into the reality of an innovation (Reference Steuten14–Reference Brear17). The theoretical studies in the developmental stage recommended more complex quantitative models such as Markov modeling, Bayesian modeling, and clinical simulations, as well as strategic models such as political, economic, socio-cultural, and technological (PEST) and strengths, weaknesses, opportunities, and threats (SWOT) analyses (13;18–20). Although these analytical approaches are applicable with scarce data, they are more resource intensive than the methods applied by the empirical articles.
Introduction Stage
In this stage, the greater part of the studies focused on strategic analysis (n = 4). Two studies consisted of economic analyses, three consisted of clinical analyses, and three consisted of the combination of all three analyses. Two studies had a combination of clinical and strategic analysis. In the empirical studies, this stage was characterized by a mixture of preliminary data collection and estimates. Quantitative and qualitative methods for assessment and data collection such as closed questionnaires, focus groups, and semi-structured interviews were frequently used to both capture the impact for the users and facilitate the innovation process (17;21–24). Literature reviews also provided insight when a small amount of data was available (25;26). The theoretical studies highlighted case studies with subsequent economic modeling as an applicable approach to collect and analyze data (27;28).
Early Diffusion Stage
This stage showed a prevalence of studies including all three analyses (n = 6). One study had an economic analysis, three had strategic analyses, one combined strategic and economic analyses, and one combined strategic and clinical analyses. The empirical studies placed greater emphasis on quantitative cost-effectiveness models, implementation and diffusion scenarios, and the logistics associated with the intervention (29;30). However, among the theoretical studies, the importance of qualitative approaches to data collection such as expert opinion and stakeholder analysis were highlighted (31;32). Table 3 is a descriptive table on the identified analytical approaches.
Description of Analytical Approaches
Stakeholder Involvement for Data Generation in Early Health Technology Assessment
In the developmental stage, simulations based on stakeholder analysis and expert interviews were used to understand the effect of innovation on the target population, organization, and society. In the introduction stage, stakeholder analysis provided additional data to scenarios for simulations on the adaption and development of innovation. In the early diffusion stage, the analytical approaches placed greater emphasis on implementation and dissemination scenarios.
An early innovation stage is characterized by a small amount of data and high uncertainty. Stakeholder insight was, however, highlighted to assess the potential benefit of health innovation (19;27;31;33). Harris-Roxas and Harris (Reference Harris-Roxas and Harris27) found that stakeholders’ views regarding potential benefits are central for assisting the assessment of an innovation and also for the prioritization of effects. Such data can potentially ease adoption and diffusion through steering the intervention to achieve value-based innovation (Reference Kummer, Schafer and Todorova34). This suggests that the innovation should be assessed in the context where it will be used to disclose how it is adopted and used in the real world.
Stakeholders can provide data on the underlying logic of an innovation to help understand changes in outcomes in the target population at an organizational level. Such data can provide valuable information on the potential suitability of the innovation (Reference Esposito, Taylor and Gold29). Stakeholder data can be applied in scenario analysis to provide necessary outcome overviews and direct and accelerate the procurement process (Reference Gantner-Bar, Meier, Kolominsky-Rabas, Djanatliev, Metzger and Voigt35). Through integrating qualitative scenarios from the perspective of stakeholders and experts into a cost-effective model, the potential value of the innovation can be estimated in an early phase (Reference Retel, Joore, Linn, Rutgers and van Harten22). Retel et al. (Reference Retel, Joore, Linn, Rutgers and van Harten22) developed a framework for the assessment of technology still in development by means of scenario drafting to determine the effects, costs, and cost-effectiveness of possible future diffusion patterns.
Evidence Gaps and Uncertainty in Early Economic Modeling
Economic modeling of the trade-off between further technological development and the value of investing more research appears largely in the developmental and introduction stage. The studies which contained economic analyses in the early diffusion stage were used to steer the implementation and facilitate proper investments.
It is believed that early economic analysis of an innovation's likely cost-effectiveness can help steer the implementation and restrain resource-inefficient technologies (Reference Hartz and John36). Numerous attempts to fill evidence gaps in early economic models were detected in the literature. Expert elicitations using scenario drafting can provide qualitative and quantitative data to fill the evidence gaps in early health technology assessment (15;23). Potential economic consequences can be estimated to forecast the effects of healthcare innovations already at the early research and concept phase to prevent ineffective investments (14;24;35).
Scenario drafting can also be useful for identifying critical factors that may affect the speed of adoption (Reference Joosten, Retel, Coupe, van den Heuvel and van Harten37). To account for the dynamic characteristics of an early innovation, future technological development, organizational change, and learning curves should be incorporated into the models (38;39). Studies pointed to the use of sensitivity analysis to deal with uncertainty in the interpretation of results and to test the impact of different implementation strategies when the technology is still dynamic (14;18;24;26;38;40). Constructive technology assessment that takes into account the learning curve seems to be appropriate in the early assessment of technologies that are still dynamic (19;22;41–43). Modeling based on sophisticated mathematic techniques such as Bayesian modeling or Markov modeling can also play an important part in early decision support and provide incentive for data collection before implementation. Use of such models in early economic modeling can help determine which efficacy and clinical performance has to be attained for different cost outcomes (19;36;39;44).
Uncertainty is an issue in all decisions; information is valuable because it reduces the expected cost of uncertainty surrounding decisions. Value of information (VOI) analysis recognizes the option to postpone the adoption or development of the technology and investing in more research to reduce uncertainty. Waiting may, however, result in health benefits forgone, and developing before conducting research may also reduce uncertainty (13;36;44). Real option analysis (ROA) can be useful for establishing the trade-off between development and research (13;19;20;44).
Clinical Efficacy in Trials with a Small Amount of Data
Articles containing clinical analyses were primarily found in the developmental stage. Assessing clinical efficacy in early stages can be challenging. Randomized clinical trials (RCTs) have long been considered the gold standard in assessing clinical outcomes. However, RCTs can have limitations, especially for evaluations of early stage interventions (Reference Chang and McLean45). RCTs require a large amount of data and, therefore, consume time and resources. The difficulty of blinding was also evident in the literature on the assessment of technology-enabled and organizational innovation. The literature, however, pointed out some applicable methods. Clinical trial simulations based on prior clinical outcomes can supply information otherwise unavailable in early stages (13;36;46;47). Input data for clinical simulations can also consist of expert opinions or a structured literature search on clinical outcomes (14;35;38;48). Clinical trials performed in a controlled laboratory setting, such as bench studies, were also highlighted in the literature to reduce uncertainty regarding the efficacy of clinical outcomes (Reference Markiewicz, van Til and Ijzerman18).
User Involvement
Involvement of potential users of an innovative technology in the early stages could make assessments more relevant and acceptable (Reference Gagnon, Candas, Desmartis, Gagnon, La Roche and Rhainds49). Although users or patients should be an important part of a stakeholder analysis, this is not always the case. Stakeholders are all the affected parties of an innovation, for example, an innovator, decision maker at the hospital or municipality, purchase unit, etc. A user is the one who directly uses the innovation. In this review, only 10 percent (4/39) of the studies targeted patients or users as the main target audience of the analysis. Early analysis and modeling of outcomes from user involvement in early assessment helps prevent failures and can accelerate implementation (Reference Beuscart-Zephir, Watbled, Carpentier, Degroisse and Alao16). Gollamudi at al. (Reference Gollamudi, Topol and Wineinger50) addressed the significance of user and health data collected through mobile devices. Such data allow individuals the opportunity to make informed health decisions and provide researchers and decision makers the opportunity to assess innovative health technology in real time. Smartphone-enabled health technologies provide a novel source of data for qualitative and quantitative analysis purposes.
Discussion
The purpose of this knowledge synthesis was to identify methods for early assessment of innovation in a healthcare setting and discuss the analytical approaches applied according to the stage of innovation. As illustrated in the Results section, several different methods for early assessment of innovation were found, and the majority of the articles included a combination of strategic, clinical, and economic analyses with qualitative and quantitative analyses. However, no article validated the specific methods used for early assessment against a technology assessment completed in later phases with additional data. In the earlier innovation stages, the methods focused on identifying available data sources, while in later stages various simulation and analysis methods were used in new ways to increase the impact of the scarce availability of data. However, the involvement of stakeholders was considered a prominent data source in every stage.
Challenges Concerning Early Assessment of Health Innovations
The present study has identified empirical and theoretical approaches for the early assessment of innovations in a healthcare setting. Although contributions have been made to the development of new methodology, the choice of method may lead to different outcomes as no universal method was found. Markiewicz et al. (Reference Markiewicz, van Til and Ijzerman18) argued that there is a lack of evidence on how effective the methods are and that there is a need to develop an agreed-upon method for early assessment. This coincides with the perception by Hartz and John (Reference Hartz and John36) on the use of early economic data, which is scarcely used in decision making on public policy. Bridges reported the need for new health financing mechanisms to ensure the implementation of valuable innovation (Reference Bridges33). However, it has been argued that evaluations are rarely seen as an integral part of implementation, thus resources are not usually dedicated to evaluation (Reference Brear17).
A further challenge stressed in the literature is the scarce evidence available in an early innovation stage (19;29;36). Small data sets lack the power to control for variables that could explain the observed effect, and short investigation periods make it difficult to identify changes in outcome. Efforts have been made to deal with uncertainty and lack of data through applying more complex mathematical models. However, Craig et al. (Reference Craig, Carr, Hutton, Glanville, Iglesias and Sims39) argued that these models suffer from the precision required for data input. Such potential sources of data could be challenging to acquire at an early stage. Furthermore, the authors highlighted that these models can be difficult to apply without in-depth knowledge of economic modeling.
Scenario analysis built on expert elicitation has been used to acquire data on potential outcomes in early assessment. However, there are concerns regarding the loss of information that may occur in scenario analysis, as a scenario does not cover all outcomes in a real-world system (22;34). The same is true for expert elicitation as different approaches were used, which may lead to varying results (Reference Kip, Steuten, Koffijberg, Ijzerman and Kusters24). Different studies included in the present review have also stressed the need for the integration of patient or user perspectives or preferences in early assessment (13;18;37). Bartelmes et al. (Reference Bartelmes, Neumann, Luhmann, Schonermark and Hagen19) suggested that early assessment of health innovation cannot replace a comprehensive HTA, but rather form a preceding step in a multi-staged HTA process.
Limitations
This knowledge synthesis may not have identified all published studies on the early assessment of health innovation, in particular the grey literature. Despite attempts to adjust the search strategy to several different terms previously used in the literature to describe similar methodologies, other terms may also exist. Although three comprehensive health databases were included in the search (MEDLINE, Embase, and Cochrane Library), searching other databases may have included additional published studies. Our search included only studies in English, Norwegian, and Danish, although only English terms were used in the search. Furthermore, no consultations from stakeholders or experts were included in this review. Finally, although the method was systematically followed by the reviewers, each reviewer subjectively included studies based on the study criteria. The classification and interpretation of the results were also subject to reviewer bias.
Further Research
Although this knowledge synthesis has identified several different methods applied in early assessment, no single method can be highlighted as prominent relative to the robustness of the results or the frequency of use. More research is, therefore, needed to systematically validate the methods suggested in this review with the aim of finding a standardized recommendation for methodology concerning early health technology assessment. An empirical test of the precision of the early assessment method needs to be competed in practice. Research should be dedicated to enhance the precision of methods that deal with lack of data and uncertainty. Such research may suggest combining existing methods to address risks from more perspectives or/and profit from the elevated availability of data sources in an increasingly digitalized world. This was also emphasized by Ijzerman et al. in a recent study of early HTA where observational studies and big data were highlighted as data sources that would allow more detailed analysis in early HTA (Reference Ijzerman, Koffijberg, Fenwick and Krahn51).
In conclusion, existing health technology assessment is considered a robust method to support decisions in later phases when the technology is well tested in clinical environments and a large amount of data is collected. Research on altering and adopting these methods to earlier phases of decision making is emerging in the literature. This knowledge synthesis has shown that the use of methods has a tendency to change according to the stage of innovation. Stakeholder analysis was highlighted in this review as a prominent method of collecting data in the three innovation stages. This applies particularly in the earliest stage of innovation, the developmental stage, as the introduction and early diffusion stage involves greater availability of data and the use of more complex methods and models.
Barriers to the identified methods have also been discussed as all of the innovation stages suffer from lack of data and much uncertainty. Early assessment may address clinical value and risk but due to short investigation periods, it is challenging to obtain concluding evidence. Although user or patient involvement in the early phases of innovation is recommended in the literature, there is a shortage of studies in this review that effectively involves them. More research is required to promote innovation and dynamic interaction between health institutions and industry through the use of HTA. Although this review has identified applicable approaches for early assessment in different innovation stages, research is required regarding the value of the available data and methods and tools to enhance interactions between different parties at different stages of innovation.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462318003719
Supplementary Appendix 1: https://doi.org/10.1017/S0266462318003719
Acknowledgements
The research for this study was financially supported by the Norwegian Research Council, grant no. 237766/O30.
Conflicts of Interest
The authors declare no conflicts of interest.
Ethical approval
The authors have nothing to declare. No ethical approval was required for this study.
