Introduction
The recent pandemic has accelerated awareness of the beneficial role of digital health technology (DHT) in providing continuity of healthcare at home balanced against the substantial investment required for its optimal and ongoing use. As health services increasingly make investment decisions on DHTs for managing the health needs of people with chronic disease, performing a DHT-specific comprehensive Health Technology Assessments (HTA) is crucial in ensuring a systematic and multidisciplinary approach (Reference O'Rourke, Oortwijn and Schuller1) to assessing value-for-money.
Growth in development and demand for DHT interventions that manage chronic disease at home has led to a steady increase in peer-reviewed primary research studies. However, it is unknown whether this research covers the content required for a DHT-specific comprehensive HTA. Systematic reviews on the adequacy of evidence generation (published up to 2015) for HTAs found that less than half of electronic/mobile health HTA reports considered organizational or social domains. Very few considered the technology, safety, ethical, and legal domains (Reference Vukovic, Favaretti, Ricciardi and de Waure2). Mobile health economic evaluations varied significantly in reporting quality, costing strategies, and length of follow-up periods (Reference Iribarren, Cato, Falzon and Stone3). For home monitoring DHTs, economic evaluations varied greatly in the types of equipment and the types of tasks for health care staff that were included in the costs (Reference Kidholm and Kristensen4). More recently, Forsyth et al. (Reference Forsyth, Chase, Roberts, Armitage and Farmer5) found over half the peer-reviewed studies on DHTs for self-management of Type 2 diabetes failed the NICE framework effectiveness standards due to poor trial design or reporting: absence of comparator group; no justification of sample size; no measurable improvement in condition-related outcomes; lack of statistical analysis.
DHT-specific evaluation frameworks used in HTA, such as the NICE Evidence Standards Framework for DHTs (6), are maturing. In a prior systematic review (Reference Von Huben, Howell, Howard, Carrello and Norris7) (Figure 1), we conducted an extensive search of international peer-reviewed and gray literature to identify evaluation frameworks specific to DHTs that manage chronic disease at home. We compiled a comprehensive list of the most frequently recommended content across a nine domain HTA framework based on the EUNetHTA Core Model (8). The nine domains to be covered in an HTA report cover the current health problem, the technology, safety, clinical effectiveness, costs and economic evaluation, ethical, social, organizational, and legal aspects. We identified fifty-seven DHT-specific content items, for example, cyber safety/security, and fourteen content items common to all technologies but essential for a comprehensive DHT HTA.
Process diagram for identifying content for a DHT-specific and comprehensive HTA.
The current systematic review summarizes current trends in primary research on DHTs that manage chronic disease at home, particularly the coverage of previously identified (Reference Von Huben, Howell, Howard, Carrello and Norris7) (Figure 1) content recommended for a DHT-specific comprehensive HTA. As in our previous review, our focus is on DHTs for use at home for active monitoring or treatment as defined by the NICE framework (6), namely remote monitoring via implants or wearable devices, and web-based treatment programs. NICE classifies DHTs providing these functions into the highest evidence tier (Tier C) as they present the highest potential risk to the user. They are also strictly regulated as Medical Device Software (MDSW) under the new European Union (EU) Medical Devices Regulation (MDR) (9).
Methods
This systematic review was registered with PROSPERO (#CRD42021224833) and is reported in accordance with the preferred reporting items for systematic reviews and meta-analysis (PRISMA 2020) guidelines (Reference Page, McKenzie, Bossuyt, Boutron, Hoffmann and Mulrow10).
Information Sources and Search Strategy
Given the focus of this review is current trends in DHT primary research, Medline, Embase, Econlit, CINAHL, and the Cochrane Library were searched from 1 January 2015 to 20 March 2020 using keywords related to HTA domains (safety, effectiveness, costs, and economic evaluation) and DHT. The full search strategy is presented in Supplementary Table S1.
Inclusion Criteria
Eligible for inclusion were peer-reviewed journal articles examining the comparative safety, effectiveness, cost, or cost-effectiveness of a DHT intervention used by a patient at home to “actively monitor” or “treat” the risk factors, symptoms, or common comorbidities (e.g., depression) of a diagnosed non-communicable chronic disease. Chronic disease is defined as any long-lasting disease with persistent effects (11), for example, diabetes, cardiovascular disease. NICE defines “active monitoring” as the automatic recording and transmission of patient data to health services to inform clinical management decisions, and “treat” as providing treatment for a diagnosed condition.
Exclusion Criteria
DHTs solely targeting populations diagnosed with a chronic mental or behavioral disorder were excluded given the more heterogeneous nature of these diseases and populations. Studies for DHTs that were not MDSW or that did not “actively monitor” or “treat” a diagnosed chronic non-communicable disease population at home were excluded. Studies not published in English were also excluded.
Study Selection
All authors participated in the title and abstract screening. Full-text screening was undertaken by AvH, with ten percent of full texts reviewed independently by JC and conflicts resolved by MH.
Data Extraction
Data extraction elements included year of publication, country/region, chronic disease population targeted, technology function (active monitoring/treatment), technology type (e.g., mobile or Web site applications “Apps”, SMS “text messages”), study objectives (clinical effectiveness/non-clinical impacts/cost analysis/economic evaluation), study type, age group (child/adult), sample size, characteristics (of intervention, comparator, and patients), duration (of intervention and follow-up), primary/key secondary outcomes, declared or apparent conflicts of interest, the inclusion of disabled and rural/remote participants, use of a DHT-specific framework such as CONSORT E-HEALTH or MAST, number of languages provided, and information on exclusions based on digital literacy.
Data extraction was conducted by AvH and checked by JC.
Coverage Assessment
DHT studies were assessed for coverage of the most frequently recommended content across a nine domain DHT-specific HTA. The assessment also included all relevant papers referenced in the included studies to ensure the review covered DHT design, feasibility, efficacy/accuracy, effectiveness, economic evaluation, or implementation testing.
As discussed, the recommended HTA content items were identified in a prior systematic review (see Figure 1). The content items are structured in two lists: 1. DHT-specific content, 2. Content common to all technologies but essential for DHTs. The content lists were tested and refined over multiple samples of DHT studies, with AvH assessing coverage and MH, SN, and KH providing feedback. This process resulted in modifications of content items (provided in Supplementary Table S2) for greater clarity and applicability to primary research.
A coverage rating scale was also developed and refined over multiple samples of DHT studies. We extended the ratings of “Yes”, “Partly” and “No” of Vukovic et al. (Reference Vukovic, Favaretti, Ricciardi and de Waure2) at the HTA domain level, into more granular ratings at the content item level such as “Not covered” (item is relevant to the study scope, but was not mentioned), “Poor” (item mentioned in limitations of current study/for future research), “Fair” (defined for each content item), and “Good” (defined for each content item). “NA” (not applicable to intervention) and “Not reported” (not relevant to the scope of the study) were provided for specific content items. Defining ratings at the content item level (Supplementary Table S2) assisted with rating consistency over the larger sample.
The final coverage assessment was conducted by AvH on DHT intervention studies targeting a population with cardiovascular disease, diabetes, or both. Ten percent of these studies were independently rated by JC. Discordance in ratings before discussion resulted from differing interpretations of words in the content items rather than differing use of the rating scale. All discordance was resolved by clarifying keywords.
Synthesis of Results
For current research trends, the included studies were summarized over the data extraction elements to identify the most/least common study characteristics. For the coverage assessment, the proportion of studies in each rating category for each content item was calculated.
As the focus of this review was a coverage assessment of previously defined content items, the risk of bias assessment was not relevant.
Results
Study Characteristics
The search identified 11,824 records (Supplementary Figure S1). Removing duplicates, protocols, and reference types that were not published papers produced 6,676 records for title and abstract screening, of which 6,454 did not meet the inclusion criteria. Full-text reviews of 222 papers identified 201 reports (see Supplementary Table S3 for paper references) of 178 DHT intervention studies published between 1 January 2015 and 20 March 2020.
Table 1 summarizes the included study characteristics. The studies are predominantly in high/middle-income countries in Europe/North America. Thirty-eight percent of DHT interventions targeted cardiovascular disease populations, sixteen percent diabetes, and nine percent two or more chronic diseases. Seven percent of DHTs were designed for children or adolescents.
Characteristics of included papers and studies
a Includes active monitoring as a component of the intervention, but the intervention may also include treatment components.
b Includes no active monitoring component in the intervention.
c Study type: II: Randomized Controlled Trial RCT, III-1: A pseudorandomized controlled trial (i.e., alternate allocation or some other method), III-2: A comparative study with concurrent controls: Non-randomized, experimental trial, Cohort study, Case–control study, Interrupted time series with a control group, III-3: A comparative study without concurrent controls: Historical control study, Two or more single-arm study, Interrupted time series without a parallel control group, IV: Case series with either post-test or pre-test/post-test outcomes.
Ninety-four percent of studies included an effectiveness trial within the search period, but fifty-nine percent had yet to conduct a cost analysis or economic evaluation. Eleven percent had examined changes in health service utilization without costing. Seventy-eight percent of studies conducted randomized controlled trials (RCTs) for effectiveness, and a further sixteen percent conducted a comparative trial with concurrent controls. Median sample sizes were 170 (IQR: 90–350) participants, median duration six (IQR: three-twelve) months, and follow-up six (IQR: four-twelve) months.
Fifty-seven percent of DHTs provided an active monitoring component, mostly via standalone telemonitoring devices. The remaining DHT interventions provided treatment without active monitoring, primarily via mobile or web-based applications.
Thirty-four percent of studies were funded by pharma/biotechnology/health insurance companies. Half the studies did not explicitly exclude people with mental or physical disabilities. In thirty-five percent of studies, there was enough information to understand the level to which participants were excluded based on their digital literacy. Only six percent stated they provided an intervention with two or more languages, and only eight percent reported involving rural/remote participants in testing. A digital-specific framework such as CONSORT E-HEALTH (Reference Eysenbach12) or MAST (Reference Kidholm, Ekeland, Jensen, Rasmussen, Pedersen and Bowes13) was referenced in only five percent of studies.
Coverage Assessment
The coverage assessment was undertaken for DHT interventions targeting cardiovascular disease, diabetes, or both (112 studies, sixty-three percent of all included studies). DHTs for chronic disease management have been pioneered in these disease populations, so this sample is most likely representative of DHT research practice in other chronic disease populations. Less than half of CVD/diabetes studies covered DHT-specific content in all but the health problem domain (Figure 2 and Supplementary Table S4). Coverage of content common to all technologies but essential for DHTs was greater than fifty percent in all but effectiveness and ethical domains (Figure 3 and Supplementary Table S5). Coverage assessment is summarized by the HTA domain below.
Digital health technology (DHT)-specific content items for a health technology assessment (HTA). Percentage of included studies attaining each coverage rating.
Content items common to all technologies but essential for a digital health technology (DHT) health technology assessment (HTA). Percentage of included studies attaining each coverage rating.
Domain 1: Health Problem and Current Use of the Technology (CUR)
Because this was the best-covered category (Figures 2 and 3), we describe the percentage of studies attaining the “good” category to highlight emerging good practices. Almost all DHT studies rely on the participant to pay for data usage costs, and many require a personal mobile phone or computer. Only four percent of studies examined whether patients would pay, or estimated costs, for data usage fees and the cost of personal technology required to use the DHT. None addressed whether the health service or patient should pay for data usage fees or provide the personal technology. Three percent of studies discussed how the DHT was designed to overcome utilization limitations, such as available platforms, languages, connectivity, and digital literacy. The NICE requirement to collect ongoing DHT usage data was found for one-third of interventions. Almost one-third explained the comparative advantage of inputs and outputs of the DHT, but only six percent detailed the algorithms/engine logic well enough to understand its limitations/advantages over other DHTs. Twenty-one percent explained the DHTs people with the condition already had available to them (Figure 3).
Domain 2: Description and Technical Characteristics of Technology (TEC)
As this domain was moderately well covered, we focus on “good” category studies. Discussion of how the DHT was designed to minimize investments in technology required to run the DHT in the health service was detailed in three percent. Four percent discussed privacy/cyber safety/digital literacy training for patients and staff. Mention of more than one DHT feature for overcoming technical barriers, such as interoperability, data extraction, or visualization, was found in eight percent (Figure 2). Although most DHTs were tested within the health system (Figure 3), small sample sizes in all but sixteen percent limit the evidence that the DHT could cater to the expected patient population. Over forty percent of DHT studies indicated the technology was mature with no significant future development anticipated.
Domain 3: Safety (SAF)
This DHT-specific domain was poorly covered, so we focus on studies attaining a “fair” rating. Controls for cybersafety and cybersecurity, such as compliance with privacy and data security legislation, were covered in less than one-fifth of the studies. Only one study reported that users were given the DHT owner's contact information and information on how their data were collected and protected. Without screenshots/archived DHTs, as recommended in CONSORT E-HEALTH, we could not investigate this further. Only six studies mentioned processes for correctly identifying users within the DHT (a cybersafety control).
In terms of interoperability, less than one-fifth could demonstrate a process to support the creation and maintenance of accurate healthcare records that could be integrated with health system databases. In terms of algorithm risk, only ten percent disclosed enough detail to understand the limitations of the data used, algorithms deployed, output validation, or how the algorithms control the clinical decision-making process (an essential control for learning or complex algorithms).
One-fifth of the studies discussed the technical reliability and stability of the DHT. There were few references to prior technical reliability trials, and only four percent addressed updates or continuity management. However, over forty percent discussed the process of identifying and responding to a patient's acute deterioration (Figure 3).
Domain 4: Clinical Effectiveness (EFF)
As effectiveness was not well covered, we focus on “fair” studies. Three-quarters of DHT studies employed RCTs for effectiveness. Methods to achieve at least single blinding were mentioned in sixty percent of these. Online adherence or use was reported in over forty percent of studies. Whether changes were made in the DHT during the trial, control groups were restricted in DHT use, or biases arose from implicit exclusions based on digital literacy or embedded data collection were more difficult to determine given that there was little use of CONSORT E-HEALTH.
Reliable Information Content and Use of Appropriate Behavior Change Techniques
For these NICE framework requirements, less than half the DHTs providing health information referenced a reliable source or development of content by health professionals at the DHT development stage. Only two studies evidenced a process to keep this information up-to-date. Of the sixty percent of DHTs that aimed to promote behavior change, less than half referenced a peer-reviewed behavior change theory relevant to the purpose of the DHT.
External Validity/Generalizability
Six percent of studies reported including participants in rural or remote areas; ten percent reported disabled participants, and sixteen percent reported participants with limited prior use of digital technology. However, very little subgroup analysis was provided.
Patient Satisfaction
There was no evidence of patient involvement (patient surveys/focus groups/usability and feasibility testing) in the design of almost three-quarters of DHTs. Although twenty-seven percent had evidence of patient satisfaction data being collected and analyzed in the effectiveness trial, no studies demonstrated ongoing collection/extraction of these data.
Domain 5: Costs and Economic Evaluation (ECO)
We discuss “fair” and “good” rating results for this domain for better practice discrimination. Of the forty-three studies that produced a cost analysis/economic evaluation, twelve studies estimated the costs to support the running of the DHT service (fair), and four estimated the costs to provide it at a scale for health system use (good). Eleven acknowledged a change in fixed costs for scaling up the DHT (fair), but only three estimated this cost function (good).
For all rated studies, DHT-specific outcomes such as self-management benefits or better-connected healthcare professionals were reported in almost one-third (fair), with seventeen percent using validated measures (good). Seven percent considered start-up times and the realistic use of DHT functions (fair), but only three percent incorporated this into an economic evaluation (good).
Domain 6: Ethical Analysis (ETH)
Similar to safety, the ethics domain contains many DHT-specific controls to promote cybersafety and provide safeguards when the patient is remote from the clinician. As this domain was not covered well, we focus on “fair” rated studies. A description of a secure process for data transmissions, especially alerts about a patient's health, was reported in only fifteen percent of studies. No study discussed protecting patient data from commercial use. The user was informed of the data collected by the DHT and its intended use in four percent of studies. Only three percent named all parties that hold personal data collected by the DHT. Only one study indicated that users would be informed of the potential risks of data sharing when using the DHT. No study stated that the DHT is regularly audited for transmissions with third parties.
Twelve percent of studies noted patient feedback on the DHT promoting a false sense of security or creating harm from accessing data without someone to interpret it. Managing incidental findings from testing done by the DHT was discussed in only two of thirty-two applicable studies. Discussion of the DHT design using simple, understandable language, or collection of patient feedback on this, was found in twelve percent of studies.
Autonomy
For DHTs targeting behavior change, controls to limit the DHT's influence on a person's behavior for purposes other than those stated or how the range of options was chosen so the user could make independent decisions were not discussed. Only one study stated that potential conflicts of interest (e.g., funding, promotion) were disclosed to DHT users. For DHTs providing health information, eight percent provided concise information for the user on how the DHT content was selected or who was responsible for the content.
Justice & Equity
Descriptions of how the DHT overcame access barriers for patients with a lack of economic resources, poor IT skills, disabilities, or low digital health literacy were found in one-fifth of the studies. Seven studies justified the choice of languages provided, discussed language as a limitation on use, or provided many languages. Unless the DHT was explicitly targeted towards hard-to-reach patients (thirteen percent), for example, patients in low-socioeconomic areas or low-income countries, there was no evidence of how effective the DHT would be for these populations.
Domain 7: Organizational Aspects (ORG)
“Good” studies provided qualitative or quantitative evidence on how staff work methods and interactions with patients changed (four percent). A “fair” discussion of changes to electronic communication, information reporting systems, face-to-face consultations, and staff communication required for the DHT to operate was found in thirteen percent of studies. Implementation studies are rare but provided better coverage of required changes and recommendations for enablers of DHT uptake (nine percent). Evidence of a relevant healthcare expert's involvement in the design, development, testing, or sign-off of the DHT (fair) was only found in one-third of the studies.
Domain 8: Patients and Social Aspects (SOC)
Twenty percent of studies gave qualitative or quantitative feedback on increases in connectivity between patients and healthcare providers (fair). Five percent reported qualitative or quantitative analysis on rural and remote participants (good). Only four percent stated that users were provided with expected direct and data usage costs, an important enabler of treatment adherence (good).
Domain 9: Legal Aspects (LEG)
This was the least covered domain, with only one study clarifying the parties responsible for medical advice, monitoring or reviewing patient data, and who owned the DHT-related data. No study discussed potential litigation risks, insurance, or professional registration consequences to healthcare practitioners using or recommending the DHT.
Discussion
Current Research Trends
The growth in effectiveness studies of chronic disease DHTs over the last 5 years is encouraging, particularly with the majority being RCTs employing practices to overcome methodological problems associated with DHTs, for example, single blinding and choice of a comparator reflecting standard care. However, small sample sizes, short trial durations, and short follow-up periods limit the ability to detect treatment effects, determine the optimal treatment dose, and estimate the persistence of effects. Lack of inclusion of populations from low-income countries, settings where telecommunication infrastructure/connectivity is poor (e.g., rural and remote communities), and exclusion of people who do not speak the primary language or own the required personal technology, limits the generalizability of these studies. As most studies had yet to conduct a cost or economic analysis, cost-effectiveness compared to alternate interventions remains largely unknown.
Coverage Assessment
Close examination of the included CVD/diabetes DHT studies revealed that content coverage in technical, safety, ethical, and legal domains remains low, as was found by Vukovic et al. (Reference Vukovic, Favaretti, Ricciardi and de Waure2) in HTA reports to 2016. Although DHT-specific controls for cybersecurity, cybersafety, technical reliability, and stability exist across multiple domains, they are mainly concentrated in safety and ethical analysis domains. These domains were not well covered despite being significant areas of risk to the user.
In terms of effectiveness, the NICE framework standards of ensuring reliable and accurate health information and best practice behavior change techniques were only evidenced in a minority of studies providing these services. The lack of evidence for ongoing controls to keep health information up-to-date is concerning. Three-quarters of studies could not provide evidence of patient involvement in the DHT design, which is a critical failure for technologies designed for patients to use at home. In the organizational aspects domain, two-thirds could not evidence a relevant health care expert's role in the design, development, testing, or sign-off of the DHT, a key enabler for DHT uptake.
In terms of economic evaluations, similar to Kidholm (Reference Kidholm and Kristensen4), we found that the inclusion of costs was variable. Most studies only included the cost of the equipment for the patient, not the costs for the equipment required to run the DHT service or downstream costs of changes in health outcomes resulting from the DHT. The fixed costs of providing the DHT in the health system and at scale (licensing, platforms, hardware, security) can escalate rapidly from costs involved in a clinical trial. These costs should be estimated and included. Even though most DHT trials assume the patient will pay data usage fees and bring their own device, at a minimum, a sensitivity analysis including these costs should be reported.
Existing DHT-specific frameworks and a phased research approach with improved referencing to prior work could be employed immediately to improve the quality of trial design and reporting to meet the needs of HTA. Coverage of at least six of the thirteen DHT-specific effectiveness items plus four additional items over technical, safety, and ethics domains, could be achieved by designing and reporting effectiveness studies in compliance with CONSORT E-HEALTH (Reference Eysenbach12), a reporting standard available since 2011. This reporting standard should not be limited to RCTs as many items are relevant to other comparative study designs. A phased research approach should, at a minimum, include a review of existing DHTs available to the target population, design and initial testing with target patients and relevant health professionals, efficacy/accuracy testing, and safety testing for technical reliability, stability, cybersecurity, and cybersafety, before clinical effectiveness trials. This prior work should be referenced or reported in clinical effectiveness publications. Finally, economic evaluations should be performed considering increases in costs for operating the DHT service in the health system at the expected scale.
Limitations
Our findings are limited to the information reported in the included peer-reviewed journal papers, referenced papers, and supplementary materials. No attempt was made to contact the authors for additional information. While using the seventy-one content items recommended for a DHT-specific comprehensive HTA promotes a thorough investigation, we recognize that these items’ number and equal weighting are not efficient for regular use. Refining these lists into more practical companion materials for performing or assessing HTAs is warranted.
Conclusion
Although primary research in DHTs that manage chronic disease at home is steadily increasing, it is not covering the content required for a DHT-specific comprehensive HTA, particularly in the critical areas of cybersafety, cybersecurity, technical reliability, stability, and patient satisfaction. The inability to conduct such an HTA will likely result in suboptimal decisions in the investment of health service budgets. Measures to increase the quality of trial design and reporting using existing tools and DHT-specific frameworks are required.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321001665.
Acknowledgments
We thank Ms. Bernie Carr, Academic Liaison Librarian, Fisher Library, University of Sydney, for assisting with the search strategy.
Funding
AvH is supported by an Australian Government Research Scholarship and Postgraduate Scholarship in Health Economics (Patient-Centered Care and Outcomes in Chronic Disease) from the University of Sydney, School of Public Health. JC is supported by a Postgraduate Research Scholarship from the Australian Prevention Partnership Centre (TAPPC). MH is funded by an Australian Government National Health and Medical Research Council program grant.
Conflict of Interest
The authors would like to declare that they have no conflicts of interest.
