Introduction
Health apps
Digital health interventions including apps (also known as “applications”), wearable devices, and artificial intelligence systems have the potential to provide effective care that reduces resource demands on health services and is convenient for users (1).
Heath technology assessment (HTA) organizations, including the United Kingdom's (UK) National Institute for Health and Care Excellence (NICE), have started to conduct assessments for apps (Reference Dillon, Powell, Campbell and Salmon2). For the purposes of this study, we have employed the UK Medicines and Healthcare products Regulatory Agency (MHRA) definition for apps. Apps are software programs that are designed as medical devices for preventing, diagnosing, monitoring, or managing conditions (3). They can be used via portable Internet-enabled technologies such as mobile phones or accessed via the web on desktop computers.
Evaluating the effectiveness of apps can be challenging because their evidence base is often immature (1;Reference Dillon, Powell, Campbell and Salmon2). This makes it difficult to assess them using traditional HTA methods (1;Reference Dillon, Powell, Campbell and Salmon2). It can also be challenging to retrieve evidence about apps in bibliographic databases because brand names and specific terminology for the technologies (i.e. app, apps, or applications) are not always reported in titles or abstracts. Inconsistent, descriptions for apps such as “Internet-based” (Reference Barnes, Miller and Bostock4), “web-delivered” (Reference McGrath, Espie, Power, Murphy, Newell and Kelly5), or “online” interventions (Reference Bourne, DeVos, North, Chauran, Green and Brown6) are used instead. In addition, relevant subject headings are not always applied to reference records in bibliographic databases (Reference Barnes, Miller and Bostock4–Reference Bourne, DeVos, North, Chauran, Green and Brown6). These issues can lead to evidence about apps being missed in searches if only specific terminology for the interventions is used. Conversely, large volumes of irrelevant results are retrieved when wide ranges of descriptions for apps and their associated technologies are used in searches.
Search filters
Filters are pre-made, reusable collections of search terms that are applied to literature searches to retrieve research with a common feature in bibliographic databases (Reference Glanville, Bayliss, Booth, Dundar, Fernandes and Fleeman7;Reference Jenkins8). Filters are created to retrieve evidence reliably for challenging search topics such as apps. A range of filters are available for study designs (such as RCTs or systematic reviews), topics (such as health conditions, geography, and gender), and health interventions (9).
Filters are developed and validated using a gold standard (GS) set of known relevant references (also known as a “reference set”) (Reference Jenkins8). The GS set references are divided into a development and validation set. The development set is used to inform the search terms for filter and the validation set is used to validate the filter's retrieval ability (Reference Jenkins8). Validation is important because it allows conclusions to be made about the generalizability of a search filter's performance, providing users with an indication of how successfully they work (Reference Jenkins8).
Recall (also known as “sensitivity”), precision, and specificity are commonly used to measure filter performance (Reference Jenkins8). Recall is the proportion of relevant references retrieved from the GS set (Reference Jenkins8). Precision and specificity provide an indication of the retrieval efficiency of filters. Precision is the proportion of relevant results retrieved of the total number of references (relevant and irrelevant) (Reference Jenkins8). Specificity is the number of irrelevant references that are not retrieved as a proportion of all irrelevant references (Reference Jenkins8).
The ideal filter would have high recall, precision, and specificity (Reference Jenkins8). However, filters with high recall usually have low precision and specificity (Reference Jenkins8). They retrieve a high proportion of relevant results, but they also retrieve a large volume of irrelevant results (Reference Jenkins8). Filters with high precision and specificity often have low recall (Reference Jenkins8). They tend to be used when comprehensive evidence is not needed (Reference Jenkins8).
The NICE health apps search filters
Filters for apps were not available previously (9). A team of NICE information specialists undertook a project to develop filters for apps with high recall.
The target for high recall was defined as >90 percent in line with the threshold used in previous filter development studies (Reference Beynon, Leeflang, McDonald, Eisinga, Mitchell and Whiting10). Recall was of importance for the filters because it was known that the inconsistent descriptions used for apps can lead to relevant evidence being missed in searches. In addition, comprehensive evidence about topics is required to inform NICE guidance. However, it was acknowledged that including all possible descriptions for apps was likely to retrieve a large volume of irrelevant results and reduce the filters’ precision and specificity. Therefore, decisions were made during development to limit the filters’ retrieval of irrelevant results while aiming to achieve the >90 percent recall target.
MEDLINE and Embase (Ovid) were chosen for the filters because they are used routinely at NICE in line with best practice for systematic literature searching (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas and Chandler11).
Objective
To develop and validate search filters to retrieve evidence about health apps from MEDLINE and Embase (Ovid) with high recall.
Methods
Creating the GS set
Journal sources
A selection of eight journals from the “medical informatics” section of the National Library of Medicine (NLM) Journal Catalog (12) was used to identify references about apps for the GS set:
• BMC Medical Informatics and Decision Making
• IEEE Journal of Biomedical and Health Informatics
• International Journal of Medical Informatics
• Journal of Medical Internet Research
• Journal of Biomedical Informatics
• Journal of the American Medical Informatics Association
• Studies in Health Technology and Informatics
• Telemedicine Journal and E-Health
Journals with a medical informatics focus were chosen because it was thought that they had the potential to yield the most references about apps.
Hand searching process
Hand searching is a manual process for identifying relevant references for GS sets (Reference Jenkins8). All journal contents published between 2015 and 2018 were downloaded from MEDLINE into EPPI-Reviewer reference management software. There was a total of 8,827 references. The references were ordered by primary author surname to mix their publication dates and journal sources. The references were then split into five equal sets and hand searched independently by five members of the project team using details from the titles and abstracts.
In Embase, 9,186 references from the medical informatics journals were available. The unique Embase references that were not available in MEDLINE were downloaded into a separate EPPI-Reviewer review and hand searched by a project team member.
Inclusion criteria
The team used the MHRA criteria for apps to guide their selection of relevant references for the GS set (3). In addition, the team referred to references from published NICE advice products on apps as a guide to identifying relevant references that did not have specific terminology for apps (see Supplementary File 1 for the NICE advice references).
References on the following were selected for the GS:
• single apps
• reviews of multiple apps
• development of apps
• classification of apps
• evaluation of apps
• user opinions of apps
• safety and regulatory issues about apps
All references selected for inclusion were double screened to ensure their relevance. Any queries regarding the suitability of a reference for inclusion were resolved through discussion. A total of 689 references formed the MEDLINE GS set and 703 formed the Embase GS set.
Creating the development set and validation set
The 689 relevant MEDLINE GS set references were randomized and then divided to create a development and validation set using the free online RANDOM.org randomization tool (13) on 5 November 2019. A total of 344 references formed the development set. These 344 references were used to inform the development of both the MEDLINE and Embase filters because all were available in both databases.
The remaining 345 references from the MEDLINE GS set formed the MEDLINE validation set. The remaining 359 references from the Embase GS formed the Embase validation set. The Embase validation set included 343 MEDLINE validation sets references that were available in Embase and sixteen extra relevant references that were available in Embase but not in MEDLINE. The validation set references were used to validate the filters.
A sample size of at least 100 relevant references has been suggested to provide a reasonable confidence interval for filter validation (if the filter retrieves 90 percent of the validation set references) (Reference Sampson, Zhang, Morrison, Barrowman, Clifford and Platt14). Both the MEDLINE and Embase validation sets therefore contained enough references.
In MEDLINE and Embase, the primary author, title, and publication date for each development and validation set reference were used to form individual search lines to create two separate search strategies for each set. During the project, the draft filters and final filters were run against the development set or validation set search strategy using the AND Boolean operator to assess their retrieval of the references.
Identifying the filter search fields and search terms
Search fields
The 344 development set reference records were exported from Ovid MEDLINE and Embase into Excel spreadsheets using the “Excel sheet” export option. This enabled the content for every search field from the MEDLINE and Embase development set records to be displayed in separate columns in the Excel spreadsheets. The columns were scanned and the following fields were found to contain terms of relevance to apps:
• Subject heading
• Title
• Abstract
The “keyword” field also contained terms about apps. However, it was not selected for the filter because some records did not have any keyword terms.
Subject headings
Medline filter
MEDLINE uses the NLM Medical Subject Headings (MeSH) controlled vocabulary thesaurus. The DataBasic online free word counter (15) was used to identify relevant MeSH terms for the filter from the development set references. The contents of the Excel spreadsheet column for the MeSH terms of the 344 development set records were copied and then pasted into the DataBasic word counter tool. An Excel spreadsheet containing single words up to three-word phrases for the MeSH terms was then generated and downloaded from the DataBasic site.
The candidate MeSH terms for the filter were not selected objectively using frequency analysis. Instead, the authors used their knowledge of the topic and scope note descriptions to choose the candidate MeSH terms subjectively (see Supplementary File 2 for the candidate filter terms). For example, some MeSH terms that were deemed to be relevant for apps and their associated technologies such as “Computers, handheld/” and “Therapy, Computer-Assisted/” had a low frequency of occurrence in the development set. However, it was thought that relevant low-frequency terms such as these could be useful to retrieve app references from outside the development set and they were selected. Conversely, in an attempt to reduce the retrieval of irrelevant results, MeSH terms for technologies that were deemed to be of wider relevance such as “Telemedicine” and “User-Computer Interface” were not selected despite them having a higher frequency of occurrence.
Embase filter
Embase uses the Elsevier Emtree controlled vocabulary thesaurus. As with the MeSH terms, the DataBasic word counter was used to identify the candidate Emtree terms from the development set references for the Embase filter using the methods described above for the MEDLINE filter (see Supplementary File 2 for the candidate filter terms).
Title and abstract fields
The DataBasic word counter was also used to identify title and abstract free-text terms for the filter from the 344 development set references.
As with the MeSH and Emtree terms, the candidate terms for the filter were not selected objectively using frequency analysis. For example, some relevant title and abstract terms such as “smartwatch” and “eMental” had a low frequency of occurrence in the development set references. However, it was thought that relevant low-frequency terms such as these could be useful to retrieve evidence about apps from outside the development set and they were selected.
Draft filter development
Medline draft filter creation
The MEDLINE filter was drafted first using the MEDLINE database. Where relevant, candidate MeSH terms were “exploded” to retrieve narrower MeSH terms. For instance, the candidate MeSH term “cell phone” was exploded to also retrieve the narrower “text messaging” candidate MeSH term. The MeSH terms were combined with the OR Boolean operator and they retrieved 80.2 percent of the development set references.
To increase recall, the candidate title and abstract terms were then added to the MeSH terms using the OR Boolean operator. A combination of single terms and phrases were used. Truncation was applied to retrieve singular and plural versions of the terms where relevant. Single terms were used when they were deemed to be able to retrieve results of relevance to apps (e.g. apps, smartphone, ehealth). For the phrases, proximity operators were used to reduce the number of individual candidate phrases needed. For example, the candidate phrases “mobile technology,” “mobile health intervention,” “mobile device,” and “mobile-based application” were searched as (mobile* adj2 (based or application* or intervention* or device* or technolog*)).ti,ab to retrieve them all. Iterations were made by increasing and decreasing the proximity operators for the phrase search to optimize the draft filter's retrieval of the development set references.
Some candidate title and abstract terms of potential relevance for apps were not included in the draft filter. For instance, it was assumed that nonspecific terms such as “tablet” and “computer” would retrieve large volumes of results that are irrelevant to apps even when used in phrase combinations. In addition, brand names for the devices and online systems for apps were not selected because only a small number of them were identified from the development set (iPhone, iOS, Android). Adding all the possible brand names that are associated with apps to the draft filter would have been difficult due to their large variety.
MEDLINE draft filter testing
Development set
When the first MEDLINE draft filter was complete, its recall in retrieving the 344 development set references was recorded. Target for recall was >90 percent. The following calculation was used:
• Recall: No. of relevant known references retrieved by filter/Total no. of relevant references × 100 to express as a percentage (Reference Jenkins8)
External MEDLINE development set: NICE advice app references
As journals with a medical informatics focus had been used to source the development set references, a further analysis was conducted to test the draft filter's ability to retrieve app references from other journals. The evidence sections of published NICE advice products about apps were scanned to identify the references. The availability of the references was checked in MEDLINE and Embase by entering the primary author name, title, and publication date for each reference into the databases. Thirty references had MEDLINE records and thirty-one had Embase records (see Supplementary File 1 for the NICE advice references). The references were published in a variety of specific and multi-disciplinary health journals.
Search strategies containing the primary author, title, and publication date on single lines for each available reference were created in MEDLINE and Embase. The first MEDLINE draft filter was run against the strategy for MEDLINE references using the AND Boolean operator to assess its recall of them.
MEDLINE draft filter modifications
Modifications were made to the first MEDLINE draft filter to increase its efficiency in retrieving the development and external development set references. The free-text terms for the filter had been searched in both the title and abstract fields. The effect of searching some single terms in the title field and the effect of using proximity operators to create phrase searches for the same terms in the abstract field was investigated within MEDLINE (see Supplementary File 3 for the MEDLINE draft filter modifications). The intention was to maintain the baseline of the draft filter's recall while reducing the number of search hits retrieved to improve its efficiency.
The most efficient version of the filter became the final MEDLINE filter and its recall against the development and external development sets calculated.
Embase draft filter creation
The Embase filter was created when the final MEDLINE draft filter was developed. The candidate Emtree terms were combined with the title and abstract terms from the final MEDLINE draft filter using the OR Boolean operator to create the Embase draft filter (see Supplementary File 4 for the Embase draft filter).
Embase draft filter testing
Development set
The Embase draft filter's recall for the 344 development set references was calculated in Embase.
External Embase development set: NICE advice app references
The Embase draft filter's recall was also calculated against thirty-one NICE advice references in Embase (see Supplementary File 1 for the NICE advice references).
Filter validation
The MEDLINE and Embase filters were validated by calculating their recall against the references in their validation sets in MEDLINE and Embase.
Overall filter performance measures
A 2 × 2 table was used to calculate the filters’ recall, specificity, and precision against the full set of references identified by hand searching. There was a total of 8,827 references for the MEDLINE filter (689 GS set references, 8,138 irrelevant references) and 9,186 references for the Embase filter (703 GS set references, 8,483 irrelevant references).
Results
First MEDLINE draft filter
The first MEDLINE draft filter and its recall results can be found in Supplementary File 3. The filter retrieved 99.1 percent recall (341/344) of the development set references and met the >90 percent recall target. The draft filter also retrieved all thirty of the NICE advice references from the external MEDLINE development set (100% recall).
Final filters
Medline
Development set references
The number of hits that the first draft filter retrieved from MEDLINE was high at 412,508. Following modifications, the final MEDLINE filter retrieved 195,023 hits (see Supplementary File 3 for the draft MEDLINE filter modifications). Despite the 53 percent reduction in search hits compared to the first MEDLINE draft filter, the final MEDLINE filter continued to retrieve 99.1 percent of the development set references and met the >90 percent recall target as shown in Table 1.
Recall of the NICE health apps search filters against the development and validation sets
External MEDLINE development set: NICE advice app references
The final MEDLINE filter retrieved all thirty of the NICE advice references (100%) as shown in Table 1.
Embase
Development set references
The Embase filter achieved 98.5 percent recall of the development set references and met the >90 percent target as shown in Table 1 (the Embase filter can be found in Supplementary File 4).
External Embase development set: NICE advice app references
The Embase filter retrieved twenty-nine out of the thirty-one (93.5%) NICE advice references as shown in Table 1.
The two missing NICE advice references (Reference Lowres, Neubeck, Salkeld, Krass, McLachlan and Redfern16;Reference Marks, Kenwright, McDonough and Whittaker17) did not contain the title and abstract field free-text terms from the MEDLINE and Embase filters. The references were retrieved in MEDLINE by the final MEDLINE filter because they had MeSH terms for “Cell phone/” (Reference Lowres, Neubeck, Salkeld, Krass, McLachlan and Redfern16) and “Therapy, Computer-Assisted/” (Reference Marks, Kenwright, McDonough and Whittaker17) in their database records. In Embase, the database records for the references did not have Emtree terms for apps or their associated technologies.
Filter validation
Medline
The final MEDLINE filter was validated by calculating its recall against the 345 validation set references. The filter achieved 98.6 percent recall (95 percent confidence interval [CI], 96.7–99.5) as shown in Table 1. The filter can be found in Supplementary File 5.
Embase
The Embase filter was validated by calculating its recall against the 359 validation set references. The filter achieved 98.6 percent recall (95 percent CI, 96.8–99.6) as shown in Table 1. The filter can be found in Supplementary File 5.
Overall filter performance measures
The 2 × 2 calculation table for the filters’ performance is shown in Table 2. The MEDLINE filter achieved 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter achieved 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.
Overall performance measures for the NICE health apps search filters
Discussion
The NICE health apps search filters for MEDLINE and Embase are the first published filters for retrieving evidence about the interventions. They retrieve evidence about apps from the databases with high recall. The filters can be applied to literature searches to retrieve evidence on a range of app-related topics including the effectiveness of the interventions for specific health conditions, evaluation methods, or safety and regulatory issues.
The precision and specificity of the filters is low despite attempts made to limit their retrieval of irrelevant results during development. However, this was anticipated because they contain nonspecific terminology to describe apps. It is expected that the specificity of the filters would be increased in “real life” searches. This is because the references used to calculate the specificity of the filters were from medical informatics journals and they are not representative of the range of content available in MEDLINE and Embase.
App interventions are a relatively new method of providing healthcare and the technologies that support them are continually developing. It is acknowledged that future developments may result in changes to terminology that could affect the relevance of the terms used in the filters. The filters will be updated as required to ensure that they remain fit for purpose.
Medical informatics journals were used to source the GS set references. It is possible that terminology used to describe apps in these specialist journals may differ to other journals. However, the retrieval ability of the filters was tested against an external development set of NICE advice references about apps from a variety of healthcare journals and they performed well. Although the number of references used to test the filters was small, this suggests that the filters can retrieve app evidence across journals.
Literature searches on additional databases can be required for systematic searches. A variety of controlled vocabulary thesauri are used across different databases and the creation of filters without subject headings has been identified as a research need (Reference Lefebvre, Glanville, Beale, Boachie, Duffy and Fraser18). A post-hoc test of the recall of the title and abstract field terms from the filters against the validation sets was conducted to assess if they could be used in other databases. In the test, >90 percent recall was achieved in MEDLINE and Embase. If the representation of titles and abstracts in other databases do not vary substantially from their representation in Medline and Embase, this indicates that the title and abstract field terms from the filters could potentially achieve >90 percent recall in any database. We intend to conduct more research on the retrieval performance of the filters in other databases.
Limitations
Defining apps is challenging when specific terminology (“app,” “apps,” “applications”) is not used in the titles or abstracts of references. Due to the inconsistent descriptions for apps, it is possible that some relevant references may not have been identified for the GS set during hand searching.
The irrelevant references identified during the hand searching process were not double-screened. It is possible that some relevant references may have been therefore omitted accidentally. In addition, titles and abstracts were used to screen references which means it is possible that relevant references may not have been selected because they only contain app descriptions in their full-text articles.
Subjective decisions were made to select candidate terms for the filters and frequency analysis was not conducted to inform their inclusion and combinations in the filters. Furthermore, the candidate terms were not reviewed by an expert in the field. It is also acknowledged that exploding some of the candidate subject headings may have resulted in the inclusion of noncandidate terms in the filter.
The validation set references were generated from the same set of medical informatics journals as the development set references. It is possible that the recall of the filters may have been lower if they had been tested against apps references from independent sources.
Conclusions
The novel NICE health apps search filters can be used to retrieve evidence about the interventions from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about apps by information professionals, researchers, and clinicians.
Implications for practice
Prior to the development of the NICE health apps search filters it could be challenging to retrieve evidence about the interventions from bibliographic databases. As a result, relevant evidence about health apps could have been missed in searches.
The NICE health apps search filters improve searching practice for the interventions. The filters retrieve evidence about health apps from MEDLINE and Embase (Ovid) with high recall. They can be applied to literature searches to retrieve evidence on a range of app-related topics including the effectiveness of the interventions for specific health conditions, evaluation methods, or safety and regulatory issues.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S026646232000080X.
Acknowledgments
None.
Financial support
This research received no specific funding from any agency, commercial or not-for-profit sectors.
