Mobile medical applications (MMA) are software applications (apps) with a therapeutic or diagnostic purpose (1). MMAs are a part of larger group of software known as software as a medical device (SaMD); a class of medical software that can act as a medical device (1). These apps are increasingly being used globally by patients and health practitioners to treat and monitor chronic health conditions (such as diabetes) (2;3). Currently, software—including MMAs—in Australia is regulated as a general medical device, or an in vitro diagnostic (IVD) medical device, and when applicable as an active implantable medical device (AIMD) (2;4). MMAs are also subject to pre and post market regulatory oversight by the Australian federal regulatory agency, the Therapeutic Goods Administration (TGA) (2). All devices regulated in Australia are listed on the Australian Register of Therapeutic Goods (ARTG) (5).
MMAs pose similar regulatory challenges to other forms of contemporary digital health software. These technologies are characterized by different methods for determining whether their intended purpose is achieved (software validation), as well as a fast and iterative design and development process, compared with more traditional forms of medical software and/or hardware (6–Reference Shuren, Patel and Gottlieb10). However, unlike traditional software forms, MMAs are easily accessible and installable software, and can be deployed on readily available nonspecialized hardware (off-the-shelf) over various platforms. These platforms may also be interconnected with other datasets, systems, and devices, by means of the Internet and other networks (6–9).
Current software regulation used by the TGA in Australia focuses on assessing the risks presented by traditional forms of software and/or hardware, predominantly direct physical harms such as infection (2). However, the risks posed by apps are mainly indirect, through the information they provide, and are thus more challenging to regulate (9). In this, MMAs resemble in vitro diagnostic medical devices (tests) as they do not directly improve health outcomes, but the output they provide has the ability to alter the management of the patient (Reference Horvath, Lord and StJohn11). Other jurisdictional regulatory bodies such as the United States (US) Food and Drug Administration (FDA) (3) and that of the European Commission (12) have experienced similar challenges with MMAs. In response, in 2013 the FDA implemented an MMA-specific approach to regulatory evaluation (3).
The regulatory challenges presented by MMAs and other contemporary software led the International Medical Device Regulator's Forum (IMDRF), to which Australia and the United States are signatories, to create a working group aimed at harmonizing the regulation of standalone medical device software (13). In 2017, the IMDRF published a final guidance document on Software as a Medical Device (SaMD): Clinical Evaluation (13;14). The guidance document provides the level and type of clinical evidence needed to appraise software for regulatory purposes (13).
This study explores the regulation of MMAs in various international jurisdictions, and tests whether the approach used in Australia by the TGA to regulate MMAs (and/or accompanying hardware) is consistent with international standards and suitable to assess the challenges that MMA technology presents.
Methods
Two methods were used to critically review international approaches to the regulation of MMAs and to benchmark the Australian approach against its international counterparts: a policy analysis and a review of case studies.
Policy Analysis
The aim of the policy analysis was to identify policies for the evaluation of MMAs in the regulatory setting developed in the nine IMDRF member jurisdictions. These policies were then compared and contrasted relative to the SaMD: Clinical Evidence (13) guidance document (Supplementary Material) (13;15).
To identify documents for the policy analysis, documents available on the relevant jurisdictional regulatory agencies’ websites were reviewed (Supplementary Material). Other relevant policy documents were identified through the snowballing of sources.
Inclusion Criteria
The included policy documents had to be in English as well as published in or after January 2013 (IMDRF published its first SaMD guidance document in 2013) and before April 2018. The policy documents were limited to guidance documents that directly or indirectly addressed the regulation of MMAs and/or software within the IMDRF member countries. Legislative documents or Acts were excluded as they had not been enforced by the relevant jurisdictional regulatory agency (e.g., TGA).
Case Studies
The case studies were used to assess to what extent these regulatory policies had been enacted in Australia and the United States. The United States was selected as companies’ regulatory submissions to the FDA were accessible. The case studies evaluated MMA applications which had been submitted, reviewed, and approved by the relevant jurisdictional regulators. Data were extracted from these documents using a form identifying elements of the IMDRF's SaMD: Clinical Evaluation (13) guidance document (Supplementary Material). The data extraction form is available upon request.
Acquisition of Submissions to Regulatory Bodies
The ARTG was searched in August 2017 to identify MMAs and/or the accompanying hardware that were representative of each type of regulation pathway (i.e., general medical device, IVD and AIMD). Subsequently, Freedom of Information Act 1982 (FOI) requests to the Australian Department of Health in August 2017 and January 2018 to obtain the regulatory evaluation/ individual document but with sponsor and manufacturer information redacted.
The corresponding FDA evaluation of these MMAs were obtained through searching their 510K database and downloading the approved submissions (16). Unlike the TGA, the FDA reviewed the MMAs and/or any accompanying hardware together.
Evaluation of Submissions to Regulatory Bodies
The TGA and FDA evaluation documents of the MMAs and/or their accompanying hardware (where applicable) that were used as case studies are listed in Table 1. Relevant information was extracted into to the data extraction form and by the first (M.M.) and second (J.P.) authors to determine what the regulatory bodies did and did not assess. Any disagreements between the evaluators was resolved by discussion, and if consensus was not achieved, a third author (T.M.) was consulted.
Included MMAs and Accompanying Hardware (Where Applicable)
AIMD, active implantable medical device; ARTG (Australian Register of Therapeutic Goods),- the databases used to catalogue all registered medical devices in Australia; IVD, in vitro diagnostic medical device; MMA, mobile medical applications; 510K Database, the database used to catalogue all registered medical device in the United States.
Results
Policy Analysis
Nine of the ten included regulatory bodies had policy guidance information on medical software regulation available; however, only five were in English (2;3;17–23). The jurisdictions of Australia, Canada, the European Economic Area (EEA), Singapore, and the United States were included in the analysis (2;3;17–19).
Independent review (e.g., internal non-conflicted experts, third parties, external experts)
Policy documents from Australia, the EEA, and Singapore all recommend that the clinical evaluation of a medical device should be performed by a qualified individual(s). This is consistent with IMDRF SaMD clinical evaluation recommendations. The regulatory bodies also state the applicants should justify the choice of clinical evaluator(s) as well as provide evidence of the expert(s) experience and/or proficiency (24–26). The United States and Canadian guidance documents did not include any clause to indicate that the submission to the jurisdictional regulatory body should involve a clinical evaluator (3;18;27;28).
Continuous Learning Using Real World Performance Data
Also in line with IMDRF guidance, the five jurisdictional regulatory agencies require postmarket surveillance of medical devices, encouraging the manufacturers to monitor the performance and safety (e.g., adverse events) of medical devices (29–33). However, contrary to the recommendation in SaMD: Clinical Evaluation (13), these regulatory agencies do not review the device surveillance approach taken by the manufacturers, assess how burdensome the approach is, nor determine how the collected information could be integrated into the functionality of the regulated software in premarket assessments.
Description and Current Use of the Technology
All five regulatory agencies appraised the intended purpose of MMAs (1–3;17;24–26;28;34). However, only the EEA considered the input (e.g., physiological status, laboratory results, images, etc.), output (e.g., treat, diagnose, inform, etc.), and/or algorithm (e.g., model based logic, equations, rules, etc.), used by the app (13;17;35). The TGA requires a statement of both the intended purpose and the manner of supply of the apps (2).
Effectiveness
All of the guidance documents recommend assessing the clinical effectiveness of SaMDs, in terms of clinical association and product performance. It was difficult to determine whether particular evidence was specifically required for the evaluation of MMAs, as this was not stated. Regarding clinical association, all jurisdictions require a literature review, information on clinical equivalency of the product, any research that has been conducted on the product, and any performance data that the manufacturer possesses on the device (e.g., postmarket data) (1–3;17;24–26;28;34). Regulatory bodies in Australia, Singapore, and the United States verify whether the app/software meets the requirements of its specifications and whether it has been validated through testing as fulfilling its intended purpose (24;26;28;35).
Safety
Like Australia, the other jurisdictional members of the IMDRF use a physical risk-based (Table 2) approach for medical device software, which aims to minimize the effect of physical harm from the device by balancing this against the benefits of its intended purpose. In all these jurisdictions, the higher the risk the medical devices poses, the more regulatory controls that are applied to the technology, and the more systematic the clinical evaluation (Figure 1) (1–3;17;18;24–26;28;34;36–39). The system (risk-classification) recommended by the IMDRF to classify the dangers posed by SaMDs, assesses the harm posed by comparing the impact the software output will have on clinical decision making against the severity of the condition being treated by the software (9;14). However, none of the included jurisdictions used the risk-classifications suggested in the IMDRF's SaMD: Clinical Evaluation (13) guidance document.
The relationship between the application of Australian regulatory-controls and risk-classification.
Risk Classification of Medical Devices in Included Jurisdictions
AIMD, active implantable medical device; IVD, in vitro diagnostic medical device; Non-IVD, a medical device that is neither an AIMD nor IVD.
Similarly, none of the regulatory bodies included any aspects of information security (cybersecurity) for software as recommended by the IMDRF (13).
Technical Characteristics
Australia, Singapore, the United States, and the EEA all clarify in their guidance documents that their jurisdictional influence only applies to MMA software and/or any relevant attachable hardware (1–3;17;19;27;34). These jurisdictions clearly conveyed that their jurisdictional authority does not extend to the mobile platforms (e.g., smartphone, etc.) or the devices’ operating system (e.g., Android, etc.) that interacts with the app and/or hardware (1–3;17;19;27;34).
Summary of Findings
Regulatory agencies in Australia, the EEA, and Singapore all to some extent addressed four of the five recommendations within the SaMD: Clinical Evaluation (13) guidance documents. The United States and Canada did not assess if an independent reviewer (clinical evaluator) is necessary to appraise submissions. Canada was the only jurisdiction that did not assess technical characteristics of SaMD. Each of the five jurisdictions did not assess safety—, as defined by the SaMD: Clinical Evaluation (13) guidance documents, both in terms of how risk is classified for software and in terms of cybersecurity.
Case-Studies
We could not identify any standalone software versions of MMAs that were regulated by both Australian and American regulatory agencies. For Australia, we selected the SkinVision skin cancer detection app (Class I), and for the United States we selected DANA (unclassified), an MMA that assesses a person's medical and psychological state (40;41). These MMAs were selected as they were two apps that acted as a medical device using solely a mobile platform and did not require an attachment and/or any additional hardware. For software that related to the IVD and implantable MMA regulatory pathways, we were able to identify apps that were regulated in both jurisdictions. We selected the iHealth Align Glucose Meter for the IVD pathway and the Reveal LINQ Insertable Cardiac Monitoring (ICM) System for the implantable MMA pathway. In Australia, the TGA classified the MMA and the attachable hardware of iHealth Align Glucose Meter as IVD Class 1 and IVD Class 3, respectively (42;43). The FDA classified the app and hardware together as Class II (44). As with the glucometer, the TGA classified the MMA and ICM hardware as Class III and AIMD, respectively (45;46), while the FDA classified the devices together as Class II (47). See Table 1 for more information.
Independent Review
The use of an independent reviewer in the case studies was only used by the TGA for high risk software and hardware submissions. The low risk TGA submissions of Class I (SkinVision) and IVD Class 1 (iHealth app) did not require independent review of the device. These applications to the TGA were generally accepted within 24 hours of submission. The higher risk devices such as Class III (ICM app), IVD Class 3 (iHealth Align Glucose Meter), or AIMD (ICM System), were all reviewed by TGA reviewers as well as professionals with clinical expertise whose curriculum vitae (CV) was provided with the submission (40;42;43;45;46). Regarding submissions to the FDA, it was unclear if the reviewers had direct involvement with the development and/or testing of the product, or if the FDA ever reviews their CVs.
Continuous Learning Using Real World Performance Data
Only the TGA considered how real world experience data could be integrated into the lifecycle of the medical device software and hardware within premarket evaluations. In the Class III (ICM app) and AIMD (ICM System) submissions, the TGA only assessed whether the manufacturers would continue to monitor the software aftermarket authorization (45;46). The TGA did not fully comply with IMDRF recommendations and examine if the MMA product performance was inferior or superior to the metric stated in the submission, and/or if the data could be used to enable or disable functions in the software. Unlike Australia, the FDA did not integrate any real world data into the regulatory process.
Description and Current Use of the Technology
The TGA's and FDA's appraised submissions all reviewed the intended purpose of the software and hardware, the intended population, and output(s) of the technology (e.g., inform, treat, diagnose) (40–47). From the selected submissions, the TGA did not enquire about the input (e.g., digitized content such as laboratory results, symptoms, images). However, the TGA did review the software output (e.g., inform, treat, diagnose) for IVD Class 1 (iHealth app) and Class I (SkinVision) software, AIMD (ICM) and IVD Class 3 (iHealth Align Glucose Meter), but not the Class III (ICM app) software (41;44;47). The FDA did review the input for Class II (ICM and iHealth Align Glucose Meter) software and devices as well as the software's algorithm (e.g., equations, model based logic, rules, knowledge base, reference base) (41;44;47). However, the FDA only reviewed the output for IVD MMAs (iHealth Align Glucose Meter), not for the active implantable (ICM) or standalone (DANA) MMAs (41;44;47).
Effectiveness
Effectiveness in terms of clinical association or product performance (Table 3) was not assessed by the TGA in any of the software classified as Class I (SkinVision), or devices in IVD Class 1 (iHealth app) (40;42). Unlike the TGA, each MMA assessed by the FDA did evaluate valid clinical association and product performance in all submissions to some extent, including reviewing the clinical equivalency of all included MMAs. The FDA also examined if any scientific validity studies had been conducted on IVD (iHealth Align Glucose Meter) and active implantable (ICM) devices (41;44;47). The TGA evaluated the clinical association domain through the use of a literature review and experience data for its active implantable devices (ICM), whereas the FDA did not (41;44–47). Regarding clinical validation, the FDA reviewed the relevance of the data used to demonstrate MMA effectiveness for all of the selected device types (41;44;47). Neither the TGA nor FDA assessed analytical validity for low risk (SkinVision and/or iHealth app) or unclassified devices (DANA). Only Class II (ICM and iHealth Align Glucose Meter) and above medical devices in either jurisdiction had their performance verified and validated (41;43–47).
SaMD: Clinical Evaluation Categories and Sub-categories Addressed by Each Submission
AIMD, active implantable medical device; IVD, in vitro diagnostic medical device; Non-IVD, a medical device that is neither an AIMD nor IVD; SaMD, software as a medical device.
✔ Domain was addressed.
~ Domain was partially addressed.
✘ Domain was not addressed.
a Software.
b Hardware (attachment to platform).
c Software and hardware combined as a single device.
Safety
The case-studies demonstrated that information required by the TGA and FDA changed depending on the risk-classification and the intended purpose of the device. For IVD Class 3 (iHealth Align Glucose Meter) devices the TGA determined whether the sponsor and/or manufacturer had been transparent with their information as well as if the device could withstand being configured in unintended ways. Unlike the FDA, the TGA also inquired about how the information is displayed by the software and how the technology could affect workflow for Class III (ICM app) devices.
With respect to information security (cybersecurity), the TGA reviewed transmission data and if software could resist system interactions for Class III (ICM app) devices. The FDA did not appraise MMA cybersecurity in any form.
Technical Characteristics
The case studies demonstrated that there was limited consideration of information about the mobile platform or operating system. In Australia, for some classes of MMAs (not including hardware), the regulatory agency required information about the platform that the software was run on. However, for Class I (SkinVision) and IVD Class 1 (iHealth app) MMAs, the submissions only stated that the devices were medical software, inferring use of an off-the-shelf device. The FDA assessed whether MMAs were run on nonspecialized medical platforms (e.g., smartphones) for all the included devices. For the IVD device, the FDA inquired about the type of platform and operating system that the attachment and app were to use.
Summary of Challenges
The case studies have highlighted several challenges faced by Australian regulations of MMAs. When compared with IMDRF recommendations, the challenge revolves around how the Australian system should change its risk-classification system so that it is based on MMA content and information, instead of the physical risk it poses. Another challenge is integrating technology specific items (e.g., cybersecurity, analytical validity, etc.) into current regulation and ensuring that MMA regulatory submissions undergo independent clinical review– not just for devise that are Class II and above.
Discussion
In English speaking IMDRF jurisdictions, the current regulatory oversight of MMAs and/or accompanying hardware does not completely comply with the IMDRF SaMD: Clinical Evaluation (13) guidance on the regulation of software.
However, the Australian software regulation is consistent with the approaches used internationally. Most jurisdictions use a physical harm-based risk-classification system, which determines how extensive and thorough (Figure 1) the clinical evaluation a MMA should be (2;3;24;26;30;36;37). Figure 1 illustrates the current Australian physical based risk classification. It is concerning that none of the IMDRF member jurisdictions review software safety using the method that they have recommended, i.e., that the risk-classification should be based on the consequences to patients of the information supplied by the software (output). Much like medical tests, MMAs can produce health consequences for patients indirectly, if the patient takes a course of action (e.g., treatment) based on the information provided. The credibility of this information is critical.
The challenges presented by digital health software to current device regulation is resulting in changes in regulatory processes internationally. In April 2017, the European Commission adopted new medical device legislation (implemented in 2020) to ensure that regulatory processes can adapt to the significant progress in technology and science that has occurred in the past two decades, and which will likely continue in the future (12).
Furthermore, the United States has had to explore a different way of regulating SaMDs to address the unique challenges they present (3). The FDA is currently piloting a new method to regulate software (including MMAs) (Reference Shuren, Patel and Gottlieb10). It explores a precertification (based on SaMD: Clinical Evaluation) approach that assesses the SaMD developer for their software testing, designs, and other matters (7;10;48–50). The reason for exploring this new approach is because the existing method was considered inappropriate for the regulation of SaMDs, given the technology is easily adaptable with a fast life-cycle (10;50).
Unlike the European Commission and the FDA, the TGA has not altered its approach to MMA regulation, but it does acknowledge its complexity (51). Like the other regulatory agencies, the main reason that the TGA approach to regulation of MMAs is not compliant with SaMD: Clinical Evaluation (13;14), is because of the risk-classification approach used. Software has no direct physical interaction with the user (e.g., exchange or administer energy and/or supply energy for imaging, monitoring physiology processes), so the device is generally classified as Class I (Figure 1) (2;26). Thus, submissions for MMAs will only have to provide a minimal level of evidence.
An MMA that has direct interaction with an AIMD is automatically a Class III (this explains why the Reveal LINQ ICM app is Class III instead of Class I) (2;52). If the TGA measured the risk posed by software by reviewing the impact of the MMA content on the user, instead of its physical harm, as the IMDRF recommended for SaMD, the clinical evaluations and regulatory controls applied to medical apps may be more extensive. With regard to hardware that accompany MMAs, the current “physical risk” approach of the TGA may be appropriate as these devices generally have direct contact with patients and as such pose physical harms (2).
Technical Considerations
The TGA regulatory approach does not assess information security in MMAs or the applicable hardware. However, the TGA issued a Medical Devices Safety Update (53) in 2016 on medical device cybersecurity, in which it “advises medical device sponsors and asset owners to perform risk assessments by examining the specific clinical use of potentially affected products in the host environment” (53). Traditionally medical devices did not have the networking or connective capabilities and could only be “hacked” through being physically altered (Reference Out and Tettero54). However, with the networked and connective nature of MMAs and applicable hardware, these devices are now vulnerable to ransomware and other forms of malicious software (Reference Out and Tettero54–59). This is particularly concerning as software enables third parties to remotely control the device (e.g., in 2017 the FDA issued a Safety Communication about cybersecurity vulnerabilities in Abbott's implantable cardiac pacemaker), as well as alter the programming (54;56;57;60;61). The ability of malicious software to affect the safety and efficacy of the device could ultimately endanger the life of a patient (54;56;57;61).
Areas for Future Research
The development of a risk-classification process that can assess the downstream harms posed by SaMDs and accompanying hardware is needed. Methods for evaluating the cybersecurity of MMAs/SaMDs and accompanying hardware are also urgently required. Research would also be helpful on the impact of commercially accessible nonspecialized platforms on MMAs. Finally, further research could be conducted to investigate if our study findings are applicable to non-English language regulatory agencies.
In the long-term, research could explore the barriers to more robust regulation of MMAs to protect the Australian population from preventable harm. Better regulation of MMAs in Australia could also potentially create a pathway to reimbursement, as it is the first step to a device being eligible for government reimbursement schemes (62). If MMAs and/or the accompanying hardware are not properly regulated, it may prevent a device that could potentially provide benefits to the population's health from being publicly funded. This raises the question of how a health technology assessment would be conducted on an MMA for reimbursement purposes, given the indirect impact of MMA information on patient health outcomes (Reference Moshi, Tooher and Merlin63). Perhaps methods used in the evaluation of in vitro diagnostics could be adopted.
Limitations
There were various limitations to this research. Due to the policy analysis being limited to jurisdictional documents available in English, potentially important information was excluded. Furthermore, the case studies did not include FDA Class I or III software. However, according to regulatory policy, the clinical evaluations conducted for FDA Class I or III are almost identical to the ones that we reviewed. The study used the SaMD: Clinical Evaluation (13;14) as the benchmark (gold standard) to measure the regulation of SaMDs, and there may be differing views on the validity of this standard.
In conclusion, the Australian TGA's regulation of MMAs is consistent with approaches used by similar international regulatory agencies. These approaches all focus on evaluating the physical risks posed by traditional medical devices. However, unlike risks posed by traditional medical devices, the main harm posed by MMAs relate to the information provided and how this is subsequently used in clinical decision making. At present, none of the approaches used by international regulatory agencies adequately assess the harms and risks posed by potential misinformation in apps. To protect the Australian public as well as global app users from the threats posed by MMAs, which mimic the challenges of IVDs, proper regulation that addresses the unique challenges of this technology is required.
Policy Implications
To address the unique challenges presented by software as a medical device, the Australian TGA should adopt the risk-classification approach recommend by the IMDRF. Any hardware that accompanies the MMA should continue to be regulated in accordance with current TGA evaluation and risk classification. The TGA should also create a method for evaluating the information security of the apps, and other software and hardware with connectivity capabilities, due to cybersecurity threats. Other IMDRF jurisdictions should consider similar changes to their regulation of MMAs and medical device software more generally. With a clearer understanding of the information and connectivity risks and benefits associated with MMAs, there is a greater potential for the development of reimbursement pathway for these technologies.
If the FDA's software precertification program is successfully implemented and integrated into the U.S. regulatory environment, the TGA could also consider trialing a similar system in Australia.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000461
Author ORCIDs
Magdalena Ruth Moshi, 0000-0002-2868-6505.
Conflicts of interest
None.
