Internationally, electronic health (eHealth) technologies such as mobile health (mHealth) applications (apps), electronic health records (EHR), and telemedicine are increasingly being used in healthcare (1–3). Globally, apps with a diagnostic or therapeutic purpose are generally known as mobile medical applications (MMA) and have been growing in popularity over the past few years (Reference Byambasuren, Sanders, Beller and Glasziou4–Reference Laranjo, Lau, Oldenburg, Gabarron, O'Neill and Chan7). MMAs are now being used and recommended by healthcare practitioners within clinical consultations around the world (Reference Byambasuren, Sanders, Beller and Glasziou4–Reference Laranjo, Lau, Oldenburg, Gabarron, O'Neill and Chan7). However, currently, healthcare practitioners are not being reimbursed in universal healthcare systems for using MMAs even though the software is being recommended and utilised (Reference Moshi, Tooher and Merlin8). This raises the question of how governments in universal healthcare systems would conduct a health technology assessment (HTA) on this dynamic technology in order to inform their reimbursement decision making (1–3).
MMAs pose different harms and risks than traditional medical devices or eHealth technologies. Therefore, evaluations of MMAs require consideration of different dimensions than those used in a typical medical services evaluation (Reference Moshi, Tooher and Merlin8–12). Unlike other technologies, MMAs have a fast life cycle; are available on non-specialised off-the-shelf devices; are available on various operating systems (e.g. Android or iOS) and operating platforms (e.g. smartphone, smartwatches, tablets); have software updates; can provide real-time post-market performance data or “real world data”; and connect to various databases and data systems via the internet; and have other networking capabilities (9–12). Due to the diverse and large nature of the growing eHealth industry, this study uses Australia and the country's tax-funded health system as a case study.
In Australia, government reimbursement of clinical encounters and prescribed interventions is available to healthcare practitioners (e.g. general practitioners [GP], medical specialists, allied health workers, dentists, optometrist) and patients through the national health insurance scheme called the Medicare Benefits Schedule (MBS) (13;14). For a medical service or intervention to be reimbursable through the MBS, it has to first be approved by the Therapeutic Goods Administration (TGA), and then undergo a HTA to ensure it is safe, effective and cost-effective (5;6;14). After a medical service has undergone an HTA conducted by an independent body – under the guidance of the Australian Medical Service Advisory Committee (MSAC) – and approved by the Federal Minister of Health, it is given an item number and listed on the MBS. Patients are reimbursed by the government for paying for the service (13;14). Currently, there is no mechanism for MMAs to be reimbursed through the MBS, although other eHealth technologies, such as telemedicine, are reimbursed (15–17).
To date, no method has been developed for evaluating MMAs as part of a complete HTA evaluation (Reference Moshi, Tooher and Merlin8). Therefore, the aim of this study was to develop evaluation criteria (a module) that could be used to adapt current HTA frameworks for the assessment of MMAs.
Methodology
The development of the module was based on findings from two of our previous studies (8;18) in addition to findings from a series of interviews. We developed this module by synthesising the results of the two previously mentioned studies with insights generated from nine in-depth semi-structured interviews with a range of stakeholders (healthcare practitioners, app developers, and policymakers) (8;18).
To summarise, our first study on this topic was a methodological systematic review which assessed the suitability of existing MMA evaluation frameworks for HTA purposes (Reference Moshi, Parsons, Tooher and Merlin18). Our second study presented a policy analysis and exemplar case studies which evaluated the regulation of MMAs in the jurisdictions of International Medical Device Regulator's Forum (IMDRF) members to determine if current regulatory methods implemented in Australia and other jurisdictions properly assessed the challenges posed by the software (Reference Moshi, Tooher and Merlin8). Our third study (reported here) sought stakeholder views on possible pathways and impediments to MMA reimbursement in Australia. The findings from these interviews were integrated with the findings from our two preceding studies to inform the creation of the HTA evaluation module.
Since the complete methodologies utilised in the first two studies are available in their respective publications, only the methods relating to the interviews, as well as the creation and testing of the module, are described below (8;18).
In-Depth Interviews
It is important to note that these interviews were aimed at supporting and informing the creation of the MMA evaluation module, alongside the two preceding studies. The interviews were not a standalone qualitative study. For this reason, an abbreviated method was adopted.
Recruitment
The stakeholders were purposively recruited by email (between April and December 2017) from those who have experience with MMAs in their professional careers. Potential participants were identified by scoping done by the first author (MM), as well as through contacts of the second (RT) and third (TM) authors. Passive snowballing via email was then used to identify further participants. All participants were emailed the participant information sheet in English prior to agreeing to the interview. The participants joined the study voluntarily and were free to withdraw prior to this publication. Adopting an information power approach, participants were only recruited if we identified that they could substantially add to the range of views about this topic (see Supplementary material A for more information) (Reference Malterud, Siersma and Guassora19–Reference Morse21). Patients were not interviewed as they are the end-users of the proposed modified HTA process.
Data Collection
The data were collected using nine semi-structured in-depth interviews between September and December 2017. The participants included four healthcare practitioners (two GPs, a nutritionist, a physiotherapist), three policy makers, and two developers of health apps. Three different interview schedules were used, one for each type of stakeholder. The interviews were conducted by the first author (MM) in-person or via teleconference. The interviews were recorded using an audio-recorder. Verbal and written consent was sought before the start of each interview. In the case of in-person interviews, written consent was obtained before the interview whereas, with teleconferences, a completed written consent form was returned via email before the start of each interview. The participants were not remunerated for their involvement in the study.
Data Analysis
The interviews were transcribed verbatim by a professional transcriber and were analysed through thematic analysis (Braun & Clarke) using the epistemology of pragmatism (see Supplementary material A for more information) (Reference Braun and Clarke22). The assistive software NVivo 11 (QSR International Pty Ltd) was used by the first author (MM) to aid in the analysis (23). The coding and analysis were checked by the second author (RT). Data source triangulation was achieved through the use of multiple participants with different expertise from five Australian jurisdictions (Reference Carter, Bryant-Lukosius, DiCenso, Blythe and Neville24–Reference Rossmman and Rallis26). Quotations from participants are used below to illustrate the findings.
Ethical Considerations
The stakeholder interviews (from the larger research project) were approved (H-2017-039) by the University of Adelaide Human Research Ethics Committee (HREC) Low Risk Human Research Ethics Review Group.
To minimise any risk of reputational damage to the participants through disclosure of information about clinical or regulatory practices, we took extra steps to ensure that participants or their organisations could not be identified from any data extracts used as quotation(s), or from any example(s) included in this written publication.
Development of MMA Evaluation Module
The module was developed by synthesising the results from two previous studies with the findings from the stakeholder interviews (8;18). This was achieved by the identification of the key policy changes and assessment criteria needed to enable the appraisal of MMAs (8;18). For example, one of our studies identified the key considerations for conducting an HTA on MMAs for reimbursement and decision-making purposes (Table 1) (Reference Moshi, Tooher and Merlin8). The second study identified critical regulatory considerations for MMAs which are not addressed by the Australian regulatory authority (TGA) or by other IMDRF regulatory members (Table 1) (Reference Moshi, Parsons, Tooher and Merlin18). Thus, on the basis of these studies, and informed by the insights generated from the interviews with stakeholders (Table 1), the information was integrated and a module developed.
Information that was used to inform the development of the module
HTA, Health technology assessment; MMA, Mobile medical application; MSAC, Medical services advisory committee.
a Accuracy: Closeness of the quantity's true value to its measured quantity.
b Analytical validity: The MMAs ability to reliably and accurately produce the intentional output from the input data
c Precision: Under unchanged conditions the degree to which the recurrent measurements generate the same result (i.e. reproducibility, repeatability)
Testing of the MMA Evaluation Module
The MMA evaluation module was tested by applying it to the current HTA evaluation frameworks used in Australia by the Federal Department of Health to appraise medical services for public funding decisions (13;14). These frameworks included the MSAC technical guidelines for both therapeutic and investigative medical services (27).
Results
Development of MMA Evaluation Module
The complete evaluation module for MMAs is detailed in Table 2. The module is evidence-based and all of the domains are mandatory, with the exception of the social domain. We included items that could fall under the jurisdiction of a regulatory authority because one of our previous studies found that Australian and other international regulators do not comprehensively assess MMAs (Reference Moshi, Parsons, Tooher and Merlin18). In Australia, regulatory approval is the first step to being eligible for reimbursement. Regulatory and reimbursement evaluations are independent of each other and typically focus on different aspects of the technology. However, if the criteria used by the regulatory authority to evaluate the MMAs do not cover all of the relevant concepts and/or items, then these must be addressed in the HTA supporting the reimbursement decision.
A module which can be used in addition to existing core HTA domains to adapt HTA and/or reimbursement evaluation frameworks to assess MMAs
APHRA, Australian Practitioner Regulation Agency; GP, General practitioner; HTA, Health technology assessment; MMA, Mobile medical applications.
Description and Technical Characteristics
The module included items which identify and review the compatibility of the operating system (OS) and operating platform for the MMA. These items include identifying what OS (i.e. Android, iOS, etc.) and the MMA is compatible with, as well as classifying what platforms (i.e. smartphone, tablet, smartwatch) the MMA can be run on.
Current use of the Technology
Numerous items relating to the current use of the technology were included in the module. The first were items related to the intended purpose(s) of the technology, in terms of whether the MMA is aimed at informing, diagnosing, and/or treating a medical condition (8;18). Items to review the MMA input, algorithm, and output were also included in the module (Reference Moshi, Parsons, Tooher and Merlin18).
Effectiveness
Multiple technology specific items were added in the module to ensure the proper appraisal of an MMA's clinical effectiveness (Table 1). Previous research had found that the technical evolution and dynamic nature of an MMA should be considered. To address this, as well as the concerns raised in the stakeholder interviews, items were included to evaluate software changes (updates), information security (cybersecurity), communication and display, and connectivity. Items addressing regulatory concerns were also added – including analytical validity (reliability) of the software, software accuracy (constant error), software precision (variable error), and software configuration. Furthermore, our previous research indicated that comparative safety, effectiveness, and cost-effectiveness should be assessed, so a comparator should be considered for MMA evaluations, for example, clinical evaluation of a patient without the assistance of an app (or usual care) (8;18).
Safety
Only one item was added to address safety concerns. This item was focused on evaluating the risk of misinformation in the app and how this could affect healthcare decision making.
Cost-Effectiveness
Module considerations relating to cost-effectiveness were minimal and were made within a fee-for-service healthcare paradigm. These items include a consideration of the applicability of the operating systems and platforms of the MMA being evaluated as well as the various versions of the MMA. The outright cost of the MMA and/or any possible in-app purchases should be considered as part of the unit cost of the technology. Changes to subsequent patient care as a consequence of the MMA would be costed as part of the normal HTA process, along with incremental changes in health outcomes due to the use of the app.
Organisational Aspects
To address the data fidelity concerns that arose during the interviews, organisational concerns were included in the module (Table 1). An example of a data fidelity concern identified was the concept of digital health literacy for both practitioners and patients. Healthcare practitioners were particularly concerned that the efficacy and safety of MMA might depend on the fidelity of data entered by the user (i.e. patient) into the app and, further, that patient self-management might be compromised by a lack of knowledge about how to interpret the app's output.
“I'd still want to verify the data that has been inputted into it by the user I suppose, I guess my preference to sort of use my clinical judgement perhaps isn't so much about the quality and strength of that app but more so the quality and completeness of the raw data that the clients inputting.”
Healthcare practitioner
Thus, the organisational aspects that were incorporated into the module addressed data fidelity issues related to practitioners and patients as well as the overall system informatics and usage. The organisational items included: any training or education that was needed by users (i.e. practitioners or patients); professional accreditation for medical practitioners; whether adopting the MMA will alter the current utilisation of services (i.e. workload, work force, compliance, etc.); whether the use of the MMA will change treatment location (i.e. home-based, rural, remote, hospital, clinic, etc.); as well as an assessment of whether the MMA can interact with the health informatics systems (e.g. Procura, Electronic Protocols Application Software [EPAS]) used in hospital, surgeries, and allied health clinics.
Ethical Aspects
The ethical aspects added to the module included equity (i.e. disability, language availability, age, literacy, socio-economic status, etc.), patient confidentiality, and patient privacy, as these were raised in our previous research, and also highlighted in the stakeholder interviews (8;18). Privacy is a major concern due to cybersecurity risks and the fact that some companies that manage or produce MMAs do sell consumer data without consumers' knowledge (8;18;28). Additional ethical considerations include equity of access to the technology (i.e. due to the cost of platform, in-app purchases, cost of app, geographic location, internet availability, etc.) and any potential conflicts of interest related to the app developer, app owner, third party sponsors, and funding sources (8;18;29;30).
Legal Aspects
Legal aspects were included in response to concerns raised in the stakeholder interviews. The interviews reflected that there are unclear lines of responsibility with respect to jurisdictional oversight and practitioner accountability for MMAs. The ownership of the health data produced by an MMA is a significant stakeholder concern as it affects patients' rights to data privacy and confidentiality. For example, if the app data are owned by the company that developed the MMA, would they have the right to sell the patients' (or users') private medical data? Stakeholders were also concerned about the impact of app use on the relationship between clinicians and their patients, including the clinician's duty of care, and whether this would affect professional indemnity. The stakeholders suggested that bringing MMAs into the standard technology regulatory framework would help to address these concerns.
“there is an issue around who actually owns the data and monitors the data and is responsible for contacting the patient and/or their GP, if there is a marker that says you know they might have had an episode”
Policymaker
To address these concerns, items were added to review how MMA use and/or recommendations could affect personal insurance (i.e. professional indemnity, life, health, income, etc.), professional registrations (i.e. for healthcare practitioners), and risk of litigation. Furthermore, additional items were included to provide clarity around which party (i.e. manufacturer, app developer, medical practitioner, etc.) owns: the data produced by the MMA; the medical advice produced by the app; as well as monitoring and reviewing the patient data. Legal concerns partially overlap with the Ethical domain above. This overlap occurs as it addresses privacy concerns through seeking clarification around ownership (i.e. patients, medical practices, companies, etc.) of data produced by MMAs.
Social Aspects
Our previous research found that currently the evaluation of social issues are not considered an integral part of an MMA's evaluation (8;18). Therefore, items included in the module that consider how MMAs could affect social issues are largely consistent with approaches used generally in HTA. These considerations included: whether the MMA can affect a care giver or family; the impact on patient's autonomy; and the patient's relationship with their healthcare practitioners.
Reappraisal
This newly added HTA domain was created to address MMA-specific challenges. The domain is aimed at evaluating post-market data (including real-world data) as part of a HTA, in response to concerns on this topic arising in our previous studies as well as in the stakeholder interviews (8;18). The stakeholders' concerns were due to a MMA's fast lifecycle and the rapid technological evolution of digital health technologies.
“The technologies are evolving fast and the policy just doesn't have a chance to catch up”
Policymaker
Stakeholders were particularly concerned about how post-market data and software updates might lead to changes which effectively create a new app or function that was not the subject of the original evaluation upon which the regulatory and reimbursement approval was based.
Items were added to allow the post-market surveillance of MMAs, in terms of data monitoring, post-market data collection (including real-world data) and utilisation (i.e. how the data could be used to modify the MMA) (8;18).
Testing the HTA Evaluation Module
The MSAC technical guidelines were selected to test the HTA evaluation module, as these guidelines are used by the Australian Federal Department of Health to determine if both therapeutic and investigative medical services should be reimbursed through the national universal health system (14;31). These guidelines are just a few of a variety used in Australia to assess health interventions for reimbursement purposes (14;31).
These Guidelines were chosen as stakeholders, particularly the healthcare practitioners, expressed trust in the current processes used to evaluate medical services for reimbursement purposes, and would continue to trust it if it was used to assess MMA for public funding eligibility.
“So I think, if an app was listed on the MBS I think that would for me would indicate to me that oh well I would assume that there had been…it would be evidence based, there had been quality check done on I guess the content and the process”
Healthcare practitioner
A range of adaptions were introduced to the MSAC guidelines from the MMA evaluation module (Table 2). Sections A through F of the original MSAC guidelines were modified to assess technology-specific and pre-market (regulatory) evaluative concerns. While, two new sections – G and H – were added to incorporate additional technology specific items as well as any additional HTA information deemed relevant (27). All proposed adaptations made to the Australian MSAC guidelines to ensure the proper assessment of MMAs are summarised in Table 3. A copy of the adapted MSAC guidelines, which describe a method for evaluating MMAs for public funding decisions within the Australian health system, is available from the authors upon request.
Module adaption made to the Australian Government's MSAC technical guidelines
MMA, Mobile medical application
a Only applies to therapeutic MMAs
b Only applies to investigative MMAs
c Accuracy (measurement uncertainty): Closeness of the quantity's true value to its measured quantity.
d Analytical validity: The MMAs ability to reliably and accurately produce the intentional output from the input data
e Precision: Under unchanged conditions the degree to which the recurrent measurements generate the same result (i.e. reproducibility, repeatability)
Discussion
Many of the stakeholders' concerns about MMA use in clinical practice pivot on the trustworthiness of the apps, the evidence-base underpinning them, and the regulatory and evaluative processes that support their use. Building stakeholder trust in the system for evaluating apps will strongly encourage integration of MMAs into the healthcare system and services. The module that we have developed attempts to address concerns about MMAs and increase trust in MMAs through a thorough evaluation of issues that are of particular concern for these types of digital health technologies. Doing so should allow MMAs to be accepted as part of standard clinical care alongside other more familiar medical and health technologies.
However, there are broader policy issues outside the scope of HTA regarding the use of MMAs that need to be considered. These policy issues could possibly impact the nature of individual clinical consultations and the trust that clinicians and patients have in the interventions being used. Some of the concerns, in particular about the jurisdictional responsibility for apps, as distinct from the responsibilities of health practitioners, will need to be considered in a wider context than can be captured through an HTA (Reference Duggal, Brindle and Bagenal32). For example, in Australia uncertainties around data ownership and IP may fall under the jurisdiction of the Australian government department responsible for proprietary knowledge and ideas applied to inventions, trademarks and inventions (IP Australia (33)) rather than with the Federal Department of Health. Additionally, even though real-world-data is integrated into the module – in the form of data collection and monitoring – it is most likely that the private organisations which monitor and create MMAs would be responsible for its collection, not governmental bodies. Thus, it is likely this domain of the module may be limited in its practical application that is, by its dependence on private organisations releasing the raw or analysed data on a regular basis. In the event that the private organisations did release the raw data to the appropriate government bodies, they would need to have the relevant resources (including human) to constantly evaluate the incoming data.
Similarly, clinical liability matters may be best dealt with by professional indemnity insurers and authorities that regulate the healthcare professions, such as the Australian Health Practitioner Regulation Agency (APHRA) (34). In particular, there needs to be clarity as to whether professional indemnity insurance can adequately deal with the use of apps in clinical consultations, or any other forms of malpractice that could occur through the professional use of the technology.
Limitations
The limitations of the analysis and synthesis conducted to develop the MMA evaluation module is that for some sections the SaMD: Clinical evaluation regulatory document was used as the gold standard for measuring specific indicators (11). Regulation and reimbursement have different aims, and thus the borrowing of concepts from a regulatory guidance documents may not be relevant for HTA processes developed in countries where there is a mature MMA regulation system (35;36). However, our review of regulatory processes in English-speaking countries found that only the U.S. is close to having a mature regulatory system for MMAs (9;37–40). Patients were not included in the interviews, which resulted in module not directly addressing their perspectives. As HTA in Australia has consumer representatives involved in the HTA decision-making process, it is likely that the patient perspective would still be represented; however, other countries using the module may not have that facility.
A limitation of using the interviews as a way of identifying possible pathways and impediments to MMA reimbursement is that the participants could have produced biased answers (25;41;42). The epistemology of pragmatism assumes that the participants will answer the questions truthfully (Reference Feilzer43–Reference Scott and Briggs45). The questions prepared and asked by the facilitator were open-ended and single barreled to avoid leading the participant(s) (25;41;42). A limited number of participants were recruited to the interviews, and this could have affected the results. However, the study design attempted to address this by assessing information power (see Supplementary material A) instead of seeking data saturation (Reference Malterud, Siersma and Guassora19).
The module was only tested by modifying the guidelines used by the Australian Federal Department of Health to determine what medical service should be funded by the national health insurance scheme. Further research needs to be conducted into how successfully it could be adapted to other HTA processes internationally.
Conclusion and Policy Implications
In conclusion, various steps need to be taken to facilitate the evaluation of MMAs. We have chosen to create a module that can be used to adapt existing HTA processes to address the unique technology-specific characteristics of MMAs. The module recommends making provisions for the analytical (reliability) validity, cybersecurity concerns, software updates, incorporation of post-market performance data, assessment of compatibility issues (e.g. platform and operating systems), as well as MMA-specific ethical and legal considerations.
Use of the MMA evaluation module in an HTA would enable policy makers to decide if an app should be reimbursed or not, particularly when used in the context of a clinical consultation. Thus, the module could be used to inform policy decisions.
Other implications are that broader policy changes are needed to ensure that MMAs are evaluated properly and that the technology can be completely integrated into the health system. These policies need to improve stakeholder trust in MMAs, including through gaining clarity on professional liability for health practitioners who use or recommend MMAs during clinical consultations as well as who owns the health data that the apps produce and/or the IP (e.g. for the app or the code, or app content). Other considerations are around how policies should be adjusted to address the rapid lifecycle of MMAs as well as cybersecurity concerns and the privacy and confidentiality of patient health data.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000288
Acknowledgments
The authors would like to acknowledge Ms Nicola Bennett for transcribing the interviews and Ms Jacqueline Parsons for reviewing the Framework for health technology assessment (HTA) of mobile medical applications (MMA) for reimbursement purposes.
Funding
Magdalena Moshi is a recipient of an Australian Government Research Training Program Scholarship and the University of Adelaide, School of Public Health, Higher Degree Research Student Support Fund.
Conflict of Interest
None.
