The Internet and computerization are constantly changing our world, our means of communication and the ways health care is provided. Telemedicine is defined as “the use of telecommunication technologies to provide medical information and services” and the term e-Health was further introduced to include other interactive technologies and applications for improving patient's quality of life, facilitating and improving quality of work for doctors and nurses, enhancing efficiency and productivity of the health services (Reference Ahern, Kreslake, Phalen and Bock1). According to the World Health Organization (WHO), e-Health is the application of information and communication technologies across the wide range of activities that are carried out in the health care, from diagnosis to follow-up. This definition suggests the use of information tools to support and promote the prevention, diagnosis, treatment, and monitoring of diseases and also the management of health and lifestyles, tailored to individual patients and consumers (Reference Ahern, Kreslake, Phalen and Bock1;Reference Silber2).
The m-Health, defined by the WHO as “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices”(Reference Saleh, Mosa, Yoo and Sheets3;4), is a component of e-Health. The m-Health technologies can be used for a range of functions, from both the healthcare professional's side (i.e., clinical decision support systems and data collection), and the patient's side (i.e., support for the lifestyle change and chronic disease management) (Reference Free, Phillips, Felix, Galli, Patel and Edwards5;6). The number of m-Health apps, downloads, and users almost doubles every year (Reference Cortez7). According to some recent estimates (8), around 97,000 m-Health apps were available in 2013 across multiple platforms on the global e-market. There are some predictions that by 2018 there could be around 1.7 billion m-Health users worldwide (Reference Cortez, Cohen and Kesselheim9).
The use of e-Health and m-Health technologies can address some of the challenges when providing accessible, cost-effective, and high-quality healthcare services in developed and also in the developing countries (4;10). Nowadays healthcare systems in Europe and around the globe are facing new challenges, such as ageing populations and increased budgetary pressures, and e-Health and m-Health technologies could confront these challenges by providing a more patient-focused health care, emphasizing prevention while also improving the efficiency of health care such as reducing the unnecessary consultations (6).
Appraisal is crucial for the integration of e-Health/m-Health applications into the healthcare systems and their further sustainability. Evaluations of e-Health/m-Health technologies are often criticized for the poor quality of research design and the lack of consensus on the appropriate evaluation methodology (Reference Ahern, Kreslake, Phalen and Bock1;Reference May, Mort, Williams, Mair and Gask11). Furthermore, the assessment of e-Health/m-Health should take into consideration patient's perspective and roles. In fact, outcomes related to the use of e-Health/m-Health depend also on patient's knowledge, attitudes and behaviors toward them (Reference Hofstede, de Bie, van Wijngaarden and Heijmans12;Reference Currie, Philip and Roberts13). In this light, health technology assessment (HTA) could offer a sound methodological basis for these evaluations.
Being a multidisciplinary approach, the HTA aims to produce a scientific evidence about the efficacy, effectiveness, cost-effectiveness of health technologies, as well as organizational, ethical, legal, and social implications of their use (Reference Leys14). HTA also allows making process evaluations because of the involvement of all stakeholders and the use of participatory and qualitative methods. HTA can be relevant to e-Health/m-Health applications because they can be considered as subcategories of health technologies (Reference Gagnon and Scott15;Reference Pandor, Thokala and Gomersall16). This is the starting point from which e-Health and m-Health could be evaluated: a concrete evidence on which policy makers, administrators, public health practitioners, physicians, and other users can base their decisions, both at the very beginning of their use and at the end of their life cycle (4;Reference Kreps17).
The aim of our study was to perform a systematic review of the literature to evaluate HTA reports on e-Health/m-Health technologies and to describe their characteristics by analyzing transparency, consistency, and thoroughness of the reports using the International Network of Agencies for Health Technology Assessment (INAHTA) checklist and checking the presence of the domains suggested by the European network for Health Technology Assessment (EUnetHTA) HTA Core Model.
METHODS
Literature Search
To identify HTA reports that have evaluated e-Health/m-Health technologies, we performed a systematic literature search in accord with the recommendations from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Reference Moher, Liberati, Tetzlaff, Altman and Group18). Two independent reviewers (V.V. and C.d.W.) searched the PubMed, ISI Web of Science, and University of York - Centre for Reviews and Dissemination (CRD)-electronic databases for relevant reports on e-Health/m-Health technologies, published up until April 1, 2016, in English or Italian. The York CRD database is linked to the Cochrane's HTA database and uses information obtained from members of the INAHTA and other HTA organizations.
Each database was systematically screened by using specific search query as a combination of subject headings and text words. The following search strategy was used for searching the PubMed database: (e-health OR ehealth OR electronic health OR m-health OR mhealth OR mobile health OR telehealth OR digital health OR digital medicine OR telemedicine) AND (health technology assessment OR HTA OR technology assessment). Other databases were searched using the appropriately modified initial PubMed search query (detailed search strategy is available upon request).
Selection of Publications
Databases were searched to identify HTA reports that evaluated interventions involving the use of e-Health/m-Health technologies. Documents that were self-defined by the authors as being HTA reports were selected. Cross-linking of the studies retrieved from different databases was performed to remove duplicates. Two reviewers (V.V. and C.d.W.) independently screened titles and abstracts of the retrieved reports. Initial selection was based on the information provided within the abstract/summary, as agreed upon among reviewers. Full texts of the potentially eligible reports were subsequently retrieved for closer inspection and assessed independently by the two reviewers, and any differences in opinion were resolved through consensus for the final inclusion.
Data Collection and Analysis
Data on the first author's name, year of publication, country, HTA agency (if any), type of report (as defined by authors), study technology, and medical field of application were extracted. Two reviewers (V.V. and C.d.W.) independently conducted all the data extraction and the report evaluations, and any disagreements were resolved through the discussion or in consultation with other co-authors. The widely accepted INAHTA checklist was used to evaluate transparency and consistency of the included reports. This checklist contains fourteen main items concerning the information that should be included in every HTA report—basic information, as well as the details on various steps of the HTA process.
The following items were evaluated: appropriate contact, authors’ information, conflict of interest, external review provided, nontechnical summary provided, policy question addressed, research questions addressed, scope specified, finding discussed, conclusion stated, suggestions for further actions stated. Items 9–11, namely description of health technology provided, details of sources and literature search, details of assessment and interpretation (data appraisal, legal implications, economic analysis, ethical implications, social implications, and other) were not considered because they overlap with the EUnetHTA core model used for the evaluation of thoroughness. Detailed instructions for using the checklist are provided elsewhere (Reference Hailey19).
We also assessed thoroughness of the HTA reports by checking the presence of evaluation of domains suggested by the HTA Core Model released by the EUnetHTA. The HTA Core Model is a methodological framework for producing and sharing HTA information. It defines the content elements to be examined in an HTA and provides a consistent structure for the production of HTA reports. The Core Model consists of nine domains to be addressed in every HTA report: health problem and current use of the technology, description and technical characteristics of technology, safety, effectiveness, costs and economic evaluation, ethical analysis, organizational, social, and legal aspects. Accuracy, the 10th domain, was also checked, as recommended by EUnetHTA when evaluating diagnostic technologies.
The “accuracy” domain was included in the evaluation of thoroughness because we expected to find reports taking into consideration telemedicine applications aimed at diagnosing, classifying or monitoring patients. Indeed, we considered important to investigate if their accuracy was assessed. A subgroup analysis of the thoroughness, according to the EUnetHTA core model, was performed on full reports only, because they were expected to be more comprehensive. Within each domain, we evaluated presence of the topics foreseen by the HTA Core Model and if some of them were present but not all, we classified it as partly. Afterward, we confronted the results between reviewers and discussed them to find the common final result. Detailed explanations of the EUnetHTA core model domains are described elsewhere (20;21). A narrative review was further used for describing the results.
RESULTS
Characteristics of the Included Studies
Initial search of the PubMed, ISI Web of Science, and York CRD online databases yielded a total number of 2,249 documents. After removing duplicates and reading abstracts and titles, forty-three full texts of HTA reports were assessed for further eligibility. By not fulfilling the inclusion criteria, fifteen full texts were excluded, leaving twenty-eight eligible HTA reports to be finally included in the review. Figure 1 shows the entire process of literature search and study selection. Our search covered a wide time interval, incorporating studies published from 1999 (Reference Simpson, Doze, Urness, Hailey and Jacobs22) to the most recent one from 2015 (23). The majority of the reports were conducted in Canada (n = 14; 50 percent), while other locations included four reports from Australia (14.3 percent), three from Italy (10.7 percent) and from the United Kingdom (10.7 percent), and the rest from Austria, Belgium, Finland, and the United States.
Flowchart depicting literature search and study selection.
Ten reports were classified as full HTA reports (35.7 percent), five were Horizon Scanning (17.9 percent), three Rapid HTA reports (10.7 percent), while the remaining included evidence base assessment, evidence briefing, and participatory report. The twenty-eight included reports evaluated technologies from several fields of medicine, mostly cardiology (21.4 percent), psychiatry (21.4 percent), and neurology (10.7 percent). Other areas were oncology, ophthalmology, chronic diseases, dermatology, pathology and cytology, pulmonology, dermatology, and intensive care. Detailed characteristics of the included HTA reports are presented in Table 1.
Characteristics of the Included HTA Reports
HTA, health technology assessment.
Transparency and Consistency of Reports According to the INAHTA Checklist
The INAHTA checklist contains brief description of several important domains related to the HTA reports. Details are summarized and presented in Table 2. All of the included HTA reports clearly defined the scope of research (100 percent), while items like author details (85.7 percent), summary (82.1 percent), discussed findings (82.1 percent), and conclusion (89.3 percent) were present in more than 80 percent of the evaluated HTA reports. Appropriate contacts were listed in 75 percent, but research questions were clearly stated in only 53.6 percent of the HTA reports included in our study. Less than 50 percent of reports tackled the remaining items; in particular the policy question was clearly defined in only 32.1 percent of reports and conflict of interest and peer review were reported in 36 percent.
INAHTA Items Checklist Evaluation of Included HTA Reports
Thoroughness of Reports According to the EUnetHTA HTA Core Model
With respect to thoroughness, around 70 percent of the reports dealt with the effectiveness, and the costs and economic evaluation. More than 50 percent precisely described the health problem while the organizational and social aspects were tackled in 43 percent and 39 percent of the reports, respectively. The remaining domains, description of technology, safety, accuracy, and ethical aspects, were evaluated in very few reports, with legal aspects of the investigated technology being present in only one HTA report. As expected, almost all reports assessed telemedicine applications aimed at diagnosing, classifying or monitoring patients but only 30 percent addressed the accuracy of technology. Only one report did not tackle a technology for diagnostic or monitoring purpose: in fact, Sullivan and Hiller (Reference Sullivan and Hiller24) considered an SMS service in the improvement of outpatient attendance. Further details of the evaluation are presented in Table 3. The subgroup analysis showed that ethical and legal aspects were often not considered, also in the full reports. Similarly, full reports reported in less than 50 percent of cases the description of the technology, its safety and social implications. On the contrary, they always addressed the economic implications (Supplementary Table 1).
EUnetHTA HTA Core Model Domains Addressed by Included HTA Reports
DISCUSSION
This review identified available HTA reports assessing e-Health/m-Health technologies and described their main characteristics using two well-known tools, the checklist for HTA reports by INAHTA and the HTA Core Model by EUnetHTA to assess their transparency and thoroughness. To the best of our knowledge, this study is the first attempt to evaluate HTA reports on e-Health/m-Health using these tools. We included a total of twenty-eight reports, published between 1999 and 2015, mostly conducted in high-income countries (western Europe and North America) and addressing different fields of medicine (cardiology, psychiatry, neurology, etc.). Several types of reports were included, the majority was represented by full HTA reports, but there were also some Horizon Scanning and Rapid HTA reports. The INAHTA checklist items, that is, appropriate contacts, author details, summary, research questions, and scope of research were present in more than 50 percent of the evaluated reports while policy question, conflict of interest, and peer review were reported only in few reports.
As for the EUnetHTA Core Model domains, health problem, effectiveness, and costs and economic evaluation have been evaluated in the majority of reports, unlike organizational, social aspects, and legal aspects that were mostly not taken into account.
E-Health/m-Health represents a dynamically developing field in medicine, with the potential to revolutionize the way health care will be delivered in the future through better planning, reducing unnecessary consultations, and more prepared professionals. As a result, patients could stay healthier and the resources could be further used (6;25). Even though first evaluations of these technologies have been promising, there is a reasonable probability that the full potential will not be achieved unless the implementation of e-Health/m-Health into the healthcare system is being encouraged, after a thorough and valid scientific evaluation (Reference Ahern, Kreslake, Phalen and Bock1;25;Reference Willem, Buijink, Visser, Marshall, Willem and Buijink26).
HTA has often been addressed as “the bridge between evidence and policy-making” and can play an important role in the healthcare decision-making process. To do it, the assessment needs to be relied on robust methods and to encompass several important criteria to bring accountability, transparency, and legitimacy in the decision-making process. Thus, HTA should provide a solid description of the technical, economic, clinical, legal, ethical, social, and organizational aspects related to the use of a health technology (Reference Gagnon and Scott15;Reference Sorenson, Drummond and Kanavos27). Current evidence on e-Health/m-Health is disseminated in diverse forms, from the peer reviewed literature, white and green papers, reports, to the presentations and web-blogs. The evidence is thus heterogeneous in quality and completeness and lacks in comparability and standardization (Reference Ahern, Kreslake, Phalen and Bock1;Reference May, Mort, Williams, Mair and Gask11).
In this study, we used INAHTA checklist and EUnetHTA Core Model as methods for the evaluation of transparency, consistency, and thoroughness of the available HTA reports because they are highly acknowledged by the scientific community and allow dealing with the important aspects of HTA. In fact, the INAHTA checklist serves to encourage a consistent and transparent approach to an HTA (28) while the HTA Core Model is intended to define and standardize evaluation elements that are essential for a good HTA (20;21). Our analysis showed that several domains were under-represented in the available HTA reports on e-Health/m-Health technologies, especially ethical, organizational, social, and legal aspects.
Ethical analysis appraises the ethical and moral questions brought by the technology itself and issues that might arise with its implementation or even when it is not being implemented. It plays a significant role in shaping the specific background in which health technology is being used (20). Nevertheless, some specific problems may emerge at ethical level when using e-Health/m-Health technologies, namely the use of data and privacy, informed consent, dependence on technology, self-management of health, as well as the technology gap (between those who have the technology and skills to use it and those who are marginalized due to the lack of technology or knowledge) (29). All these issues need to be addressed before implementing e-Health/m-Health technologies.
The organizational domain should answer what kind of resources have to be used when implementing a health technology, and what changes or consequences in the organization might be further induced by the health technology itself. Focus here is specifically on the ways for delivering and monitoring of e-Health/m-Health (20).
The social domain, on the other hand, should take patient as the point figure and focus on diverse social areas where patients’ life take place (hospitals, ambulance, everyday life, homes, workplace, leisure, etc.) and in what way health technology may change, positively or negatively, their roles and positions as family members, citizens, employees, etc. (20). With respect to e-Health/m-Health technologies, it is particularly important to acknowledge disadvantaged and vulnerable groups (elderly, disabled, poor people, etc.) that might not be competent enough for the effective use of these technologies (29). Patients, caregivers, or individuals’ perspectives can provide unique insight when taking into account their previous experiences, attitudes, habits, values, motivations, and expectations about the technology, which further brings more complexity when evaluating e-Health/m-Health technologies. Also they may experience different feelings with regard to the use of technology, that is, hope, fear, uncertainty, which should also be addressed (30).
Eventually, the legal aspects are especially important to be evaluated, because there is still a lack of firm legal framework on the implementation of e-Health/m-Health technologies. The current European legislation often remains unclear (Reference Callens31). In 2015, the American Food and Drug Administration (FDA) published a guideline on how to regulate medical apps (32) and have declared that these apps should have an assured quality, be scientifically solid, and cost-effective. In brief, legal approach in the HTA should investigate the compliance of the health technology and of its use with the valid legal regulations.
We hypothesize that these domains got less attention in HTA reports on e-Health/m-Health technologies because it might be hard to assess them in this specific field. Further on, there is still no universally accepted methodology for the assessment of e-Health/m-Health interventions, which consequently makes the quality of available studies markedly variable. Many evaluation frameworks, models, and guidelines have been suggested (Reference Gagnon and Scott15;Reference Hailey, Ohinmaa and Roine33–Reference Kidholm, Ekeland and Jensen35), but none of them is widely accepted. Also the multidisciplinary nature of e-Health/m-Health (combination of health care and technology), and the speed and dynamic of innovations in this area might have contributed to this phenomenon (Reference Silber2;Reference Agarwal, Lefevre and Lee36).
Among available frameworks to evaluate telemedicine applications, there is the Model for Assessment of Telemedicine applications (MAST) (Reference Kidholm, Ekeland and Jensen35). The MAST is extensively based on EUnetHTA Core Model. Only differences are that the first two domains of the EUnetHTA Core Model are put together in one domain as well as ethical, social, and legal aspects; furthermore, MAST includes a domain on patients’ perception. MAST is a three steps approach supporting decision makers from the very beginning to the end of the decision process. In fact, alongside the multidisciplinary assessment of the technology through the evaluation of domains, MAST proposes to have a first step of preceding considerations to determine whether it is relevant to carry out an assessment and a final step regarding the evaluation of transferability of results.
Because of its broader contents and aims, and in the light of our goal to only assess the transparency and thoroughness of HTA reports on e-Health/m-Health, we have opted for the application of the classic EUnetHTA Core Model which focuses more on contents of the evaluation. Nonetheless, independently by the evaluation framework, there is a need for strengthening and, to some extent, developing methods for assessing e-Health/m-Health also in the light of changing paradigms in the field of HTA (Reference Husereau, Henshall, Sampietro-Colom and Thomas37). In fact, because of the specific features of e-Health/m-Health, early scientific dialogue and the engagement of all relevant stakeholders have to be considered a milestone to tailor these technologies on patients and health systems and to assess their affordability and value.
Our work has several strengths and limitations. The comprehensiveness of research focused on various e-Health/m-Health technologies and the attempt to provide a thorough evaluation of the available evidence are definitely the strengths of this study. In fact, our search covered a wide time interval and a variety of settings including different fields of medicine where e-Health/m-Health technologies are intended to be implemented.
Some limitations are also present due to several factors. First, an important limitation of our work is that not all reports included were indicated as full HTA, in fact only ten reports were marked as full HTA reports, five were Horizon Scanning, three Rapid HTA reports while the remaining included evidence base assessment, evidence briefing, and participatory report. The resulting heterogeneity of included reports might have influenced the outcomes of the evaluation and thus caution should be placed when interpreting the results. The subgroup analysis may help having a more straightforward insight in the evaluation of full reports. Second, our search strategy was limited by English and Italian language and the search query was constructed using general term, thus we might have missed some reports that did not use these terms to define the evaluated technologies and/or reports published in other languages. Third, the evaluation of transparency and thoroughness may be affected by a bias because of its subjective inherent nature. Furthermore, the HTA Core Model used to assess the thoroughness of HTA reports is a general methodological framework and it is possible that HTA reports on e-Health/m-Health technologies require some specific kind of recommendations/models that deserves further investigations. Finally, all of the published reports were conducted in high-income countries and this makes difficult the translation of results to other countries and healthcare systems.
CONCLUSIONS AND RECOMMENDATIONS
E-Health/m-Health technologies are increasingly used in medicine and there is growing evidence about their evaluation also in the field of HTA. Nevertheless, our review showed that available HTA reports on e-Health/m-Health technologies are heterogeneous in terms of transparency and thoroughness. Several reports failed to tackle the relevant assessment elements, especially ethical, social, and organizational implications. There is a need for strengthening and standardizing the evaluation methods used for these technologies, especially considering the speed and dynamic of innovations in this area to overcome barriers to the full implementation of relevant e-Health/m-Health technologies in the health care systems.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317004512
