The European Commission has expressed strong support for telemedicine as a solution to the challenges faced by healthcare systems with aging populations, increasing numbers of patients with chronic conditions and decreasing supply of human resources (3).
Telemedicine is here defined as the delivery of healthcare services through the use of information and communication technologies in a situation where the actors are at different locations. The term telemedicine application refers to the overall intervention or service and not just to the telemedicine device used as part of the service.
The EC has pointed out that despite the benefits and maturity of the technology, telemedicine services are still not widespread in Europe. Small scale services, not integrated into local healthcare systems, dominate the scene.
Similarly, several literature reviews on the effectiveness of telemedicine demonstrate lack of evidence. Examples include a systematic review identifying 1,300 papers making claims about telemedicine outcomes. Assessment of these showed that only 46 reported effects on clinical outcomes (Reference Hailey, Roine and Ohinmaa12). Another review analyzed the suitability of telemedicine as an alternative to face-to-face care, concluding that establishing systems for patient care using telecommunications technologies is feasible; the studies, however, provided inconclusive results regarding clinical benefits and outcomes (Reference Currell, Urquhart and Wainwright6).
The quality of telemedicine studies has been questioned in reviews and there is debate about appropriate research methodologies and techniques. One review found that economic analysis of telemedicine had not yet met accepted standards (23); others suggested that few existing studies are well-designed (Reference Hersh, Helfand and Wallace13;Reference Hersch, Hickam, Severance, Dana, Krages and Helfand14). Finally, one review called for the development of an “evaluation template” (Reference Barlow2).
Against this background, the EC decided to support the development of guidelines for consistent assessment of the outcomes of telemedicine (3). In 2009, the EC initiated the MethoTelemed project, aiming to provide a structured framework for assessing the effectiveness and contribution to quality of care provided by telemedicine applications. The framework was to be based on the users’ need for information for decision making on whether or not to use new telemedicine applications. Simultaneously the framework was to be based on a systematic review of relevant scientific literature.
Therefore, as part of the development of MAST, a systematic literature review of reviews on impacts (all types of outcomes included in the reviews) of telemedicine services with the purpose of identifying the “state of the art” was carried out by Ekeland et al. (Reference Ekeland, Bowes and Flottorp10).
Interventions included all e-health interventions; information and communication technologies for communication in health care; Internet-based interventions for diagnosis and treatments; and social care if an important part of health care and in collaboration with health care for patients with chronic conditions. Each potentially relevant systematic review was assessed in full text using a revised check list from EPOC (Cochrane Effective Practice and Organization of Care Group) to assess quality. Qualitative analysis of the included reviews was informed by principles of realist review.
Reviews from 2005 and onward were included, and seventeen data sources and databases of literature were searched without language restrictions. The literature review was structured according to the PRISMA checklist. The main results from the review of 80 systematic reviews were that despite the large number of studies on the effects of telemedicine, high quality evidence to inform policy decisions is still lacking. Large studies with rigorous designs are needed to improve evidence and stronger focus is needed on economic analysis and patients' perception as well as an understanding of telemedicine as complex development processes.
The purpose of this article is to present the Model for ASsessment of Telemedicine applications (MAST), developed in the MethoTelemed study. The presentation includes description of the methods used, the objective of the model, the elements included and relationships to other models and frameworks.
METHODS
The MAST model was developed through user and stakeholder workshops and on the basis of a systematic literature review (Reference Ekeland, Bowes and Flottorp10) as described above.
To assess the users’ and stakeholders’ needs for information on the content and effects of telemedicine applications in decision making, two workshops were held in 2009 at DG Information Society and Media in Brussels. Supplementary File 1, available at www.journals.cambridge.org/thc2012006, includes a list of all participating stakeholders in both workshops.
At the first workshop, twenty participants represented key stakeholder groups (service providers, users, and policy directors). They were presented with results from the literature review and introduced to the EUnetHTA Core Model (Reference Lampe, Mäkelä and Garrido15;Reference Lampe, Anttila and Pasternack16).
The participants were divided into two groups and asked to apply their knowledge and experiences, while setting aside their usual professional roles. They were asked to play the roles of members of the decision-making body of a health organization which had been approached by a doctor with a proposal for a telemedicine solution. The participants’ job was to determine the type, amount and quality of information needed to decide whether or not to implement the service. The participants were also asked to evaluate the EUnetHTA model and the usability of the nine domains with regard to telemedicine represented by two cases presented at the workshop (see Supplementary File 2, available at www.journals.cambridge.org/thc2012007).
The facilitators used “brown paper” sessions for structuring the group discussions, described in, for example (Reference Arlbjørn1). Each participant wrote examples of information needed on self-stick notes. These notes were then placed on nine sheets of brown paper representing each domain in the EUnetHTA Core Model. Next, a facilitator helped the group analyze the results and regroup the notes. Finally the group agreed on the results, and the outcomes were presented in a plenary session.
At the second workshop a new group of eighteen stakeholders discussed and validated the results from the first workshop. As background they had the results from the previous workshop and the literature review. The draft model was described and the participants were divided into two groups discussing the model and the content of the suggested domains and recommended methods for data collection. Finally, results were presented in a plenary session where suggestions for improvements were reviewed.
During the workshops, two members of the team took notes capturing the discussion and oral comments of the participants. Also, all the notes and papers from the brown paper sessions were collected at the end of each workshop. Subsequently, the team used the notes and papers to prepare a summary and collectively synthesize the results. The results were presented in a report for each workshop, which was forwarded to the respective participants for comment and correction before being finalized and used by the team for the next stage of work.
Workshop results were discussed by the project team (the authors of this article) during video conference meetings, face-to-face meetings and meetings with EC representatives. The results of the literature review were also used, as the articles identified formed the basis for describing the content of each domain and for selection of topics and outcome measures.
RESULTS
Results from the Workshops
In the first workshop, the EUnetHTA Core Model was accepted as a reasonable starting point, albeit needing some adjustments. First, it was suggested that an assessment should start with strategic consideration of the organizational level (local, regional, national) at which the assessment should be carried out. Second, the model should in addition to the EUnetHTA Model focus on specific aspects of telemedicine including: Economic sustainability for the institution involved; Patients' perceptions of the telemedicine application; Safety: Risk of harms, loss of data, network problems, data safety, etc.; Effects on workflow and co-operation between primary and secondary care; and Ethical and legal aspects of telemedicine.
Examples of ethical aspects discussed included equality of patient access to telemedicine services, for example, for frail patients with low socio-economic status, cognitive impairment, living in geographically remote areas; differences in patients' ability to use telemedicine services; patients' dignity in relation to being monitored constantly in their own environment; as well as the patient's right to refuse treatment which used telemedicine.
Finally, the transferability of results from small scale assessments of telemedicine was discussed and highlighted as an important subject in new assessments.
At the second “validation workshop” the draft model was discussed. Main comments were as follows: The purpose of the model should be clearer; Description of the purpose of the application should be part of the preceding considerations; All domains needed further development and examples of outcome measures; and Potential outcomes for the relatives of the patients should be included.
The possibility of using the domains in the EUnetHTA Core Model was also discussed. However, the proposed seven domains were considered relevant and sufficient by the stakeholders in assessment of telemedicine.
The Objective of the Assessment
Based on the workshops, the project team concluded that if the objective of an assessment of telemedicine applications is to describe effectiveness and contribution to quality of care and to produce a basis for decision making, then the relevant assessment framework fulfilling this objective is a multidisciplinary process which summarizes and evaluates information about the medical, social, economic, and ethical issues related to the use of telemedicine in a systematic, unbiased, robust manner.
Key concepts in this statement of principle are “multidisciplinary” and “systematic, unbiased and robust.” The first concept suggests that the assessments should include all important outcomes of the applications for patients, clinicians, healthcare institutions and society in general. The others suggest that assessments should be based on scientific studies and methods and on scientific criteria for quality of evidence. This statement of principle is based on the definition of HTA in the EUnetHTA project.
The Elements in MAST
Figure 1 presents the three elements in MAST. The assessment must start with preceding considerations to determine whether it is relevant for an institution to carry out the assessment.
The elements in MAST.
After the preceding considerations, the multidisciplinary assessment is carried out to describe and assess the different outcomes of the telemedicine application. As shown in Figure 1, the outcomes and description of patients and the telemedicine application can be divided into seven domains based on the EUnetHTA Core Model and results from the stakeholder workshops. The domains are described in Table 1. Finally, in relation to the description of the outcomes, an assessment should also be made of the transferability of the results. Issues to consider are described below.
The Domains in MAST
Note that the term “assessment model” prescribes a structure of aspects or outcomes of telemedicine applications which should be included in a certain order (with the preceding considerations before the multidisciplinary assessment) in the assessment.
Step 1: Preceding Considerations
Preceding considerations start by determining the aim of the telemedicine application and the relevant alternatives. This initial description is used to get a first view of the characteristics of the patients, the primary outcomes and whether comparison should be made with usual care, an upgraded system or a different technology.
Second, the following conditions should be considered to identify barriers and issues that must be addressed before a multidisciplinary assessment of the outcomes of a telemedicine solution should be made:
Legislation. Before the introduction of a telemedicine service, a provider must assess whether its implementation accords with national and regional legislation. This may include inter alia legislation regulating medical care provision; accreditation systems for care providers and liability for care provision.
Reimbursement. Reimbursement is the principle by which national or regional health authorities’ and insurance bodies pay, for example, hospitals for their services. Many countries apply variations of the DRG-system (Diagnosis Related Groups), where hospitals are paid fully or partially for each discharge performed. In some cases, telemedicine does not change the DRG-tariff of a service, but in other cases the DRG-tariff is reduced. Some telemedicine services have no DRG-tariff.
This can have a significant impact on the result of the business case in the economic analysis of telemedicine applications and should be considered before an assessment.
Maturity. The development of telemedicine applications takes time. Therefore evaluation of telemedicine applications must consider their maturity or degree of development over time (Reference Simera, Moher and Hirst23).
A review by Taylor (Reference Taylor22) suggests that evaluations of telemedicine should first try to establish its safety, that is, whether its use may result in immediate risk or harm to patients. Only then should evaluations of feasibility or practicality be carried out. Finally, an evaluation of the effectiveness of telemedicine can be done to determine whether the application is worthwhile. Thus, only in this phase of the development of the application, studies of the outcomes can be carried out summatively.
Similarly, Drummond et al. (Reference Drummond, Griffin and Tarricone7) describe frequent modifications of new devices as a general problem in the design of the economic evaluation of medical devices. If the development of a new telemedicine application is not in a substantial “steady-state” period, an evaluation based on a randomized controlled trial (RCT) is particularly problematic.
Before an assessment of the outcomes of telemedicine is initiated it is therefore important to determine whether the application is mature, that is, whether the application has been used on a sufficiently large number of patients, such as in a cohort study, for most problems to have been addressed.
If the application is still being developed and needs improvement, an assessment based on MAST should not be started. Instead other kinds of assessments should be carried out, such as formative studies conducted during the development of a service, often more than once, for in-house staff of the service with the intention of improving the service (Reference Scriven20).
Number of Patients. Implementation of telemedicine often involves large investments in equipment and integration with other information systems. Often it is necessary to educate clinical staff and change the organization of work. Thus, the fixed costs of implementing telemedicine can be substantial. It is, therefore, important that assessment of telemedicine applications includes a sufficient number of patients to inform the economic analysis, thereby making it possible to approximate the estimated costs to the cost in real-life use of the technology. This means for example that if a hospital does not have enough relevant patients per year, cooperation with other hospitals may be necessary. Notice that modeling approaches (e.g., a Markow model) are also an option, but this requires information on the cost variables to include and their sensitivity toward changes in number of patients, which is often not available.
Step 2: Multidisciplinary Assessment
As illustrated in Figure 1, multidisciplinary assessment involves assessment of the outcomes of the specific telemedicine application compared with one or more comparators, where the outcomes are divided into seven domains.
In Table 1 the definition and topics of each domain are described. Topics are defined as issues for consideration within the domain.
When choosing between different study designs and methods for data collection within each domain, the general principle is that designs and methods must follow the state of the art research methods within the domains and produce valid and reliable estimates of the outcomes of the telemedicine application. However, it should be noted that telemedicine is a complex intervention with interacting components, and this may necessitate adjustment of the design of new studies (Reference Craig, Dieppe, Macintyre, Mitchie, Nazareth and Mark5). Reporting of results should also follow general guidelines for reporting health research (Reference Scriven21) such as guidelines for reporting of RCTs (4) and economic evaluation (Reference Drummond, Manca and Sculpher8).
The choice of outcome measures should be based on careful consideration of the aim of the specific application, the patient group and the organization using the application.
Step 3: Assessment of Transferability
The transferability of results from studies of healthcare technologies from one setting to another is a general problem, as pointed out in the description of the EUnetHTA Core model. Special attention should be paid to this in assessments of telemedicine applications.
One reason is that implementation of telemedicine in healthcare systems is generally a process which affects the organization. To reach the full potential of telemedicine, adjustments must often be made in the distribution of tasks between different healthcare professions (task shifting) and in communication between professionals. Similarly, interoperability—the degree of integration with other clinical or administrative systems—is crucial to realization of the benefits for patients and healthcare institutions
A second reason is that the number of patients using a telemedicine application often determines the costs per patient, with some positive economics of scale. A third problem is that technical infrastructures often vary between countries and this can create substantial differences in the possibilities and costs per patient of introducing the same service to different countries.
Therefore, new studies based on MAST should include information that could be relevant for others in an assessment of the transferability of the results. For example, estimation of costs of using telemedicine should be followed by an assessment of how the cost per patient is expected to vary with the number of patients, for example, by estimation of a cost-function.
Similarly, if an assessment is based on systematic literature review, any potential problems with validity and reliability of the studies included must be described. A practical tool for combining results from different studies is meta-analysis (Reference Egger, Smith and Altman9). This approach includes statistical methods for handling differences between the interventions being studied.
If the studies included have been conducted in other countries, other issues to consider include the possibilities for cross border transfer of results and whether differences in, for example, legislation, reimbursement or organization of the healthcare sector prevent transfer of the results. The EUnetHTA project has developed an “adaptation toolkit” with several questions that can be answered to assess the transferability of results from other studies within five domains (11).
DISCUSSION
MAST provides a structure for future assessment of telemedicine applications, adapted to the users’ and stakeholders’ need for information in decision making and based on the EUnetHTA core model and a systematic literature review.
Assessment of the opinion of stakeholders on a complex issue like the need for information as a basis for decision making is difficult and therefore a qualitative approach like “brown paper” sessions was used. However, the use of a small sample size can be criticized and it would have been interesting to combine the qualitative approach with a more quantitative collection of information from each stakeholder, for example, by use of questionnaires.
Other models and frameworks for assessment of similar health technologies can be found in the literature. The relation between MAST and important examples of other models are discussed below.
EUnetHTA
The EUnetHTA Core Model is a general framework intended to facilitate international collaboration and sharing of results in HTAs, see Lampe et al. (Reference Lampe, Mäkelä and Garrido15;Reference Lampe, Anttila and Pasternack16). The model includes nine domains and describes several topics and issues within each domain.
To a high degree MAST is based on HTA as described in the EUnetHTA project. In addition to the support for this approach from the stakeholders at the workshops, the main reasons for using this as the starting point are as follows: HTA also aims to produce a basis for decision making; HTA is a familiar concept to stakeholders in the EU, national health authorities, industry, and health professionals; and MAST can be a starting point for a future EUnetHTA model for telemedicine and thereby prepare the way for one common model for assessment of telemedicine across the EU.
MAST deviates from the Core Model by placing “Current use of the technology” and “Description and technical characteristics of technology” in one domain. Similarly ethical, social, and legal aspects are combined in the seventh domain. Finally, the patients' perception of the technology is a separate domain as requested at the stakeholder workshop.
The reasons for combining these five domains into two domains in MAST are, first, a wish from the stakeholders to simplify the model. Second, the description of the health problem and the technology (including the current use) constitute a common basis for the assessment of the effects of the technology. The ethical, social, and legal aspects of telemedicine include broader societal aspects of telemedicine and the information basis is typically from experts or legal documents and not from data collection at patient level. The division of outcomes into domains can always be discussed, but the seven domains in MAST were chosen on the basis of the stakeholders requests at the workshops. The ongoing empirical test of MAST (see below) will show the general usability of the model in practice.
INAHTA Approach
As part of an INAHTA project Ohinmaa et al. (Reference Ohinmaa, Hailey and Roine18) describes an approach to assessment of telemedicine, which includes a business case and subsequent evaluation of technical aspects, effectiveness, user assessment, economic evaluation and sensitivity analysis. Thus, this approach has a special focus on the business case. Generally the topics mentioned are included in the MAST domains. The problems with generalizability of results from local studies are also included in both approaches.
National Telehealth Outcome Indicator Project
This Canadian project aimed to produce national guidance on specific outcome indicators for evaluating telehealth applications. The guidance, described by Scott et al. (Reference Scott, McCarthy and Jennett19), was based on a systematic literature review and a workshop with national experts and describes the development of 34 approved outcome indicators divided into four categories: quality, access, acceptability and costs. Although the division of outcomes is different, several outcomes are included in the domains in MAST. The outcome “access,” constitutes a theme in itself in the guidance, whereas in MAST an improvement in patients' access to treatment can be included as a clinical effect, an economic effect caused by increase in the number of users, a reduction in transportation costs or increased patient satisfaction.
Empirical Test of MAST
Empirical tests of the usability of MAST are needed to ensure its usefulness. An empirical test will be carried out in 2010–13 in the Renewing Health project initiated by the European Commission, see www.renewinghealth.eu.
In this project, assessment will be made of telemedicine applications for patients with diabetes, heart failure, and COPD in nine EU countries. Based on the project an evaluation will be made of the model and necessary adjustments. Criteria for success are that an assessment based on MAST can be carried out within a reasonable time frame and resources and that stakeholders find the results sufficient for decision making.
CONCLUSION
The overall aim of the MethoTelemed project was to provide a structured framework for assessing the effectiveness and contribution to quality of care of telemedicine applications, based on need for information by users and stakeholders in decision making. On the basis of the EUnetHTA Core Model we have produced a draft model.
We hope that the model will form the basis for future decisions on whether or not to implement telemedicine services in healthcare systems and that MAST can help decision makers in choosing the most efficient technologies to be used in the most cost-effective way.
Similarly, private and public manufacturers of telemedicine can use MAST as a structure for data collection and description of the outcomes of their applications for patients, hospitals etc. They can also find help in the MAST manual, for example, examples of outcome measures within each domain and methods for data collection (17). Several practical guides for data collection are also available, see www.renewinghealth.eu.
SUPPLEMENTARY MATERIAL
Supplementary File 1 www.journals.cambridge.org/thc2012006
Supplementary Files 2 www.journals.cambridge.org/thc2012007
CONFLICT OF INTEREST
The project was funded by the EU under SMART 2008/0064 as part of the MethoTelemed study. Kristian Kidholm, Anne Ekeland, Lise Kvistgaard Jensen, Janne Rasmussen, Claus Duedal Pedersen, Alison Bowes, and Signe Flottorp have received grants and travel support to their institutes from Commission Information Society & Media DG; Alison Bowes has also received grant and travel support to her institute from the University Hospital of North Norway, and Mickael Bech reports having no potential conflicts of interest.
CONTACT INFORMATION
Kristian Kidholm, PhD, HTA Consultant, Department of Research and HTA, Odense University Hospital, Odense, Denmark
Anne Granstrøm Ekeland, PhD, Special Consultant and Researcher, Norwegian Center for Integrated Care and Telemedicine, Tromsø, Norway
Lise Kvistgaard Jensen, MSc, Coordinator, Department of Research and HTA, Janne Rasmussen, MA, Project Manager, Claus Duedal Pedersen, MSc, Chief Consultant, Center for Clinical Innovation, Odense University Hospital, Odense, Denmark
Alison Bowes, BA, PhD, Professor, School of Applied Social Science, University of Stirling, Stirling, Scotland, United Kingdom
Signe Agnes Flottorp, MD, PhD, Professor, Norwegian Knowledge Centre for the Health Services, Oslo, Norway
Mickael Bech, PhD, Professor, Institute of Public Health, University of Southern Denmark, Odense, Denmark
