Student health and emotional welfare is important for academic success. School psychologists and counsellors can play an important role in promoting the physical and mental health of students at risk by facilitating coping skills and adaptive behaviour change. Advances in technology are creating unprecedented opportunities for school psychologists and counsellors to promote health-related behaviours among students — in particular, apps linked to mobile devices. The need for school counsellors and psychologists to be aware that students are seeking support via apps has been recognised, along with the need to judge the quality of apps (Furlonger & Budisa, Reference Furlonger and Budisa2015). Users, however, receive little guidance and support in selecting health apps, and the health apps themselves currently do not require approval from the Therapeutic Goods Administration (Hale, Capra, & Bauer, Reference Hale, Capra and Bauer2015).
There is certainly some evidence that the better apps include behaviour change techniques that are systematic and designed to be an active component of an intervention intended to change behaviour (Higgins, Reference Higgins2016). One explanation for this finding is that behavioural theories provide a systematic way of explaining and predicting behaviour (Hale et al., Reference Hale, Capra and Bauer2015). There is, however, currently no consensus on what assessment measures are most appropriate when assessing the behavioural content of apps.
Accordingly, this study includes details of several taxonomies that have been used to evaluate the behavioural content of health-related apps; in particular, the app selection process, the rating procedures, and app characteristics. For the purposes of this study, a taxonomy is defined as classifying the evaluation criteria. Such taxonomies may assist school psychologists and counsellors to better evaluate such apps. This study is therefore aimed at two audiences: school psychologists and counsellors interested in evaluating health behaviour apps, and those who are interested in researching and developing their own apps. Such research and development is important as a high proportion (64%) of Australians own a smart phone and, on average, have 33 apps installed (Australian Communications and Media Authority [ACMA], 2014). The development of smartphones, and software apps has introduced a new dimension to health care in the 21st century (Whitehead & Seaton, Reference Whitehead and Seaton2016).
Apps specifically designed to facilitate the self-management of chronic and other health conditions have the potential to enable individuals to improve their ability to self-manage such conditions (Whitehead & Seaton, Reference Whitehead and Seaton2016). Empowering individuals to self-manage health conditions has the potential to reduce management demands commonly placed on medical professionals, family members, or other carers (Jordan, Briggs, Brand, & Osborne, Reference Jordan, Briggs, Brand and Osbourne2008). Smartphones present a convenient, compact, portable tool to assist in self-management programs, and given their rising prevalence, they provide a personal self-management assistant that is accessible by the majority of the population (Azevedo, de Sousa, Monteiro, & Lima, Reference Azevedo, de Souza, Monteiro and Lima2015; Patrick, Griswold, Raab, & Intille, Reference Patrick, Griswold, Raab and Intille2008). For example, apps can be used to prompt and track new behaviours, monitor progress between counselling visits, generate reminders, and facilitate social connectivity for the purposes of support.
Previous research has demonstrated that many self-management apps are available for individuals wanting to change, manage, and/or monitor behaviour associated with various health conditions (Azevedo et al., Reference Azevedo, de Souza, Monteiro and Lima2015; Huckvale, Car, Morrison, & Car, Reference Huckvale, Car, Morrison and Car2012; Sheehy, Cohen, & Owen, Reference Sheehy, Cohen and Owen2014) and general health behaviours, independent of any specific chronic health conditions, such as exercise and healthy eating (Sutton & Redman, Reference Sutton and Redman2016; West, Hall, Arredondo et al., Reference West, Hall, Arredondo, Berrett, Guerra and Farrell2013; West, Hall, Hanson et al., Reference West, Hall, Hanson, Barnes, Giraud-Carrier and Barrett2012).
While some apps have been found to provide users with safe, sound, self-management assistance based on medical guidelines (Reynoldson et al., Reference Reynoldson, Stones, Allsop, Gardner, Bennett, Closs, Jones and Knapp2014), few researchers have investigated whether such apps incorporate effective behaviour change strategies. While some self-management interventions, such as those for weight loss, may be considered complete and successful once an individual reaches a desired, healthy weight, many chronic health conditions present as a challenge over a lifetime. Such conditions require a lifelong change in behaviour for effective self-management. Behaviour change strategies play an important role in self-management interventions, as they enable individuals to bring about the desired behaviour change and increase the likelihood that interventions will be adhered to over time (Institute of Medicine (US) Committee on Health and Behavior Committee on Health and Behavior: Research, Practice and Policy, 2001).
Recent years have seen the development of several assessment taxonomies or measures for assessing the behaviour change content of self-management interventions targeting health- related behaviours (Abraham & Michie, Reference Abraham and Michie2008; Michie et al., Reference Michie, Ashford, Sniehotta, Dombrowski, Bishop and French2011; Michie et al., Reference Michie, Richardson, Johnston, Abraham, Francis, Hardeman, Eccles, Cane and Wood2013). Of the limited research currently available for the behaviour change content of health self-management apps, the content analysis study design has been a common approach (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Crane, Garnett, Brown, West, & Michie, Reference Crane, Garnett, Brown, West and Michie2015; Middelweerd, Mollee, van der Wal, Brug, & te Velde, Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014). Content analysis is a method for summarising any form of content by analysing themes and counting various aspects of the content, and it sits at the intersection of qualitative and quantitative research. It is a research technique used to make replicable and valid inferences by interpreting and coding data.
However, while the methodologies of such analyses share many similarities (e.g., using an assessment measure to identify behaviour change strategies present in apps), these taxonomies vary significantly (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014).
The current study, therefore, reviews the methodologies of previous content analysis studies of behaviour change strategies in health self-management apps with a view to identifying taxonomies that school psychologists and counsellors could use for the purposes of evaluating apps. Evaluations of taxonomies were based on the following criteria: (1) generalisability of the measure, that is, whether the measure can be applied to health self-management apps regardless of the targeted behaviour; (2) acceptable levels of interrater reliability in the studies to ensure the measure is reliable; (3) comprehensiveness of the measure, that is, the amount of behavioural content included in the measure; and (4) the user-friendliness of the measure, that is, how easily it can be understood and activated by a user.
Method
Literature Search and Selection
Search strategies
An electronic search of the literature was conducted on three databases: PsycINFO, Ovid MEDLINE, and ProQuest Psychology Journals. Boolean logic was employed for the search strategy, and the search terms used were “management” OR “self-management” OR “self-care” OR “self-regulation”, combined with the key words “app*” AND “behavio” AND “change” AND “interventio*”. The search was limited to include only peer-reviewed articles written in English. Following the initial search and removal of duplicates and articles with irrelevant titles, 203 articles were retained. Next, the titles and abstracts of the remaining articles were reviewed in line with the inclusion and exclusion criteria listed below. Twenty-seven articles met inclusion criteria and were included for a full text review. Following a full text review, only two articles were found to meet inclusion criteria for the final review. The reference lists of the 27 relevant studies were manually searched to identify any relevant studies not identified via the initial search. In addition, a university library search engine was used to identify any additional studies of relevance not found via the reference list search. Following this, 25 additional studies of relevance were identified for abstract and full text review, 13 of which were found to meet the inclusion criteria. Combined with the two studies located via the electronic database search, a total of 15 studies were included for the systematic review.
Inclusion and exclusion criteria
Studies were included for review if they were: (1) were published between 1996 and 2016; (2) analysed smartphone apps designed to assist in the self-management of chronic health conditions/health behaviours; (3) utilised a tool or framework or guideline or taxonomy to assess apps; (4) focused on behaviour change content of apps; and (5) utilised a content analysis study design. The 1996–2016 time period was selected. as the first smartphone designed by IBM appeared in 1993, with Blackberry releasing a smart phone in 2002. While both mobile phones were significantly less sophisticated than the smart phone that appeared in 2008, both were able to store data and prompt action. Thus, the decision was made to examine a 20-year period.
Extraction of Data for Review
Included studies were coded for the following data: study details (i.e., authors, year, journal), targeted behaviour/health condition, measures used to assess apps, structure and features of a measure, assessment procedure, apps analysed (number, format, selection process, content, source, and free or paid), app features (prices of paid apps; design of apps;interactivity and user-friendliness; technical features; user feedback, ratings and popularity), study findings, study strengths and limitations, critique of study (any noteworthy aspects regarding study design and reporting of study), and appraisal of measures and/or tools (positive/negative).
Inter-Observer Reliability
An independent observer not included in the research team assessed the studies included for full text review using the inclusion and exclusion criteria, with any discrepancies discussed prior to confirmation of the final sample of included studies. Inter-observer reliability was assessed using Cohen's kappa, and found to be κ = 0.86.
Results
Apps for the self-management of conditions and behaviours such as Type II diabetes, weight management, physical activity, and alcohol reduction were assessed in the reviewed studies. Many studies included a large number of apps, with most studies including paid and free apps for both Apple and Android formats.
Most studies employed more than one rater to assess the apps, with two raters being used most frequently. The raters were usually trained in the use of the assessment measure for assessing apps. They then tested all of the included apps prior to rating them. Testing is important, as some apps only use simple technologies such as short message service (SMS) text messaging for self-monitoring and automated feedback, while others are more sophisticated with real-time, graphic feedback and social functionality (Azar, Riley, & Mani, Reference Azar, Riley and Mani2015).
An initial test of interrater reliability was used in some studies to clarify items on the measure or taxonomy used after assessing a small number of apps, and the rest of the apps were assessed after this had taken place. Methods for resolving discrepancies included discussion, and the involvement of additional raters.
Research Methodology in Health App Content Analyses
The research methodology in health app content analyses was reviewed from three perspectives: apps selection process, rating procedures, and app characteristics. A summary of all the findings appear in Table 1.
Summary of Findings of Research Methodology in Health App Content Analyses
Note: Apps selection: targeted health condition, operating platform, language. Rating procedure: number of raters, raters’ background, rater training, rating tests, timeframe for apps testing. App characteristics: price, popularity, technical features.
Apps selection process
All 15 studies included apps based on uniquely designed inclusion and exclusion criteria. The inclusion and exclusion criteria across the studies were used to select apps in terms of their relevance to: (1) the targeted health condition, (2) the operating platform, and (3) the language.
After examining the 15 app studies, four studies included apps that promoted physical activity (Conroy, Yang, & Maher, Reference Conroy, Yang and Maher2014; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Yang, Maher, & Conroy, Reference Yang, Maher and Conroy2015), three assessed weight management (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Bardus, van Beurden, Smith, & Abraham, Reference Bardus, van Beurden, Smith and Abraham2016; Pagoto, Schneider, Jojic, DeBiasse, & Mann, Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013), and one assessed apps for both physical activity and healthy eating (Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014). Two studies assessed apps on smoking cessation (Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015); and finally, a range of single studies included apps for the following conditions: alcohol reduction (Crane et al., Reference Crane, Garnett, Brown, West and Michie2015), type 2 diabetes management (Hale, Capra, & Bauer, Reference Hale, Capra and Bauer2015), paediatric obesity prevention and treatment (Schoffman, Turner-McGrievy, Jones, & Wilcox, Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013), stress management (Payne, Wilkinson, West, & Bernhardt, Reference Payne, Wilkinson, West and Bernhardt2016), and medication adherence promotion (Morrissey, Corbett, Walsh, & Molloy, Reference Morrissey, Corbett, Walsh and Molloy2016).
Across the reviewed studies, the apps search was conducted via two operating platforms: iTunes and Google Play. Eight studies selected apps from both iTunes and Google Play (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Hale et al., Reference Hale, Capra and Bauer2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Yang et al., Reference Yang, Maher and Conroy2015); the remaining seven studies selected apps identified from iTunes (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015). Only English apps were selected for all reviewed studies.
Rating procedures
Of the 15 reviewed studies, one study involved only one rater (Hale et al., Reference Hale, Capra and Bauer2015), ten involved two raters (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015), one included three raters (Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013), two were assessed by four raters (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014), and one used nine raters (Yang et al., Reference Yang, Maher and Conroy2015). In all studies, raters conducted the apps assessments independently.
Three studies recruited raters with a behavioural health background, such as behaviour change experts and health psychologists (Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Hale et al., Reference Hale, Capra and Bauer2015). Of the remaining studies, seven studies recruited raters with no background in behavioural health (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Yang et al., Reference Yang, Maher and Conroy2015), and five studies did not specify their background (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015).
In only seven studies were raters trained (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Conroy et al., Reference Conroy, Yang and Maher2014; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Yang et al., Reference Yang, Maher and Conroy2015). In the other seven, there was no mention of training raters (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015), with only one providing an explanation why no raters were trained (Hale et al., Reference Hale, Capra and Bauer2015).
Rating tests were implemented to evaluate raters’ understanding of the content and scoring system and required those involved to rate a small sample of apps prior to the formal apps assessment and to check for interrater reliability and/or rater drift effects. These forms of rating training tests were only used in five studies (Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Yang et al., Reference Yang, Maher and Conroy2015).
When rating discrepancies were identified, they were discussed between raters to clarify any misinterpretations and to achieve consensus in 8 of the 15 studies (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015; Yang et al., Reference Yang, Maher and Conroy2015). In contrast, a third party was brought into the discussion to resolve discrepancies in four studies (Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015; Yang et al., Reference Yang, Maher and Conroy2015).
Of significance, four studies had raters who assessed apps over several days, thus strengthening their conclusions about their application and value (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014). Two studies had raters assess apps on one occasion (Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015), while eight did not specify the timeframe for assessment (Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Hale et al., Reference Hale, Capra and Bauer2015; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Yang et al., Reference Yang, Maher and Conroy2015). Interestingly, one study assessed apps based only on their app store descriptions, and did not download apps for detailed inspection (Conroy et al., Reference Conroy, Yang and Maher2014).
App characteristics (price, popularity, technical features)
Of the fifteen studies, twelve included free and paid apps (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Hale et al., Reference Hale, Capra and Bauer2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015; Yang et al., Reference Yang, Maher and Conroy2015), two included only free apps (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016), and one included only paid apps (Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016). Nine studies included free/paid apps as a variable for analysis against other variables (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013, Yang et al., Reference Yang, Maher and Conroy2015).
Several reviewed studies determined the ‘popularity’ of apps, based on the frequency of downloads, user ratings, and number of user ratings. Four studies included frequency of downloads and/or user ratings as part of the inclusion criteria for selection of apps (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013). In three studies, researchers recorded the frequency of downloads and/or user rating as variables for analysis (Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013); however, eight studies did not include ‘popularity’ of apps as a variable for analysis (Conroy et al., Reference Conroy, Yang and Maher2014; Hale et al., Reference Hale, Capra and Bauer2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015; Yang et al., Reference Yang, Maher and Conroy2015).
Technical features in apps influence how information is delivered to the mobile users. Examples of such technical features are manual or semi-automated tracking, phone and email alerts, motion sensors, and so on. Although the majority of the studies reviewed did not include technical features as a variable for analysis (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Conroy et al., Reference Conroy, Yang and Maher2014; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Hale et al., Reference Hale, Capra and Bauer2015; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016; Yang et al., Reference Yang, Maher and Conroy2015), six studies explored the inclusion of technical features and their relationships with other variables (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013; Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015).
Behaviour Change Rating Measures/Taxonomies
Behaviour change rating measures identified in the studies were reviewed on four components: generalisability across health conditions, interrater reliability/agreement, behavioural change strategies, and the design features of the measure. An overview of each measure appears in the following section. Detailed information on the measures’ scoring system and interrater reliability/agreement is presented in Table 2. A list of 14 key behaviour change strategies for self-management is presented in Table 3, providing a comparison of behaviour change content across the rating measures.
Summary of Findings for Behaviour Change Rating Measures
Key Behaviour Change Strategies in Self-Management across Generalizable Behaviour Change Rating Measures
Note: √ = included strategies.
Behaviour Change Techniques (BCT) Taxonomy
Nine studies included for review used the Behaviour Change Techniques (BCT) taxonomy as the rating measure (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016; Conroy et al., Reference Conroy, Yang and Maher2014; Crane et al., Reference Crane, Garnett, Brown, West and Michie2015; Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014; Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015; Yang et al., Reference Yang, Maher and Conroy2015). Behaviour change techniques are defined as the observable, replicable components of behaviour change interventions. They can be used on their own or in combination with other BCTs, and include goal setting, self-monitoring of behaviour, and action planning. Identifying BCTs involves a deductive process of classifying qualitative information using an established coding framework and instructions.
The original BCT taxonomy (26-BCTs) was developed by Abraham and Michie (Reference Abraham and Michie2008) through a compilation of techniques from behaviour change theories and meta-analysis of behavioural interventions. Over time, different versions of the BCT taxonomy were developed through research. The CALO-RE, for example, was a BCT version designed specifically for physical activity and healthy eating behaviours, which incorporated 40 techniques (Michie et al., Reference Michie, Ashford, Sniehotta, Dombrowski, Bishop and French2011). One of the most recent versions of the BCT incorporated 93 techniques, named the BCT Taxonomy v1 (BCTTv1). The 93 items were compiled through an evaluation of 85 behaviour change intervention descriptions, and is considered a methodological tool without a link to a theory (Michie et al., Reference Michie, Richardson, Johnston, Abraham, Francis, Hardeman, Eccles, Cane and Wood2013).
Among the nine studies, two used the 26-BCTs version (Direito et al., Reference Direito, Dale, Shields, Dobson, Whittaker and Maddison2014; Middelweerd et al., Reference Middelweerd, Mollee, van der Wal, Brug and Te Velde2014), two used the BCTTv1 (Morrissey et al., Reference Morrissey, Corbett, Walsh and Molloy2016; Yang et al., Reference Yang, Maher and Conroy2015), one used the CALO-RE (Conroy et al., Reference Conroy, Yang and Maher2014), one used a 33-BCTs version modified for weight management that was derived from the 26-BCTs and CALO-RE (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016), two used a modified 5-BCTs version for smoking cessation (Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015), and one used both a modified 41-BCTs version for alcohol reduction and a 22-BCTs version extracted from BCTTv1 (Crane et al., Reference Crane, Garnett, Brown, West and Michie2015).
Significantly, out of all the versions of the BCT taxonomy, the 26-BCTs and BCTTv1 were the only versions generalisable to all health conditions. Compared to other rating measures in this study, the 26-BCTs and BCTTv1 were the most comprehensive in behaviour change content, incorporating 13 key behaviour change strategies (see Table 3).
Behaviour Theory Content Survey (BTS)
Three studies utilised the Behaviour Theory Content Survey (BTS) as the rating measure (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013; Cowan et al., Reference Cowan, Van Wagenen, Brown, Hedin, Seino-Stephan, Hall and West2013; Hale et al., Reference Hale, Capra and Bauer2015). It assesses the inclusion and interactivity of 20 intervention strategies. The BTS was originally developed by Doshi, Patrick, Sallis, and Calfas (Reference Doshi, Patrick, Sallis and Calfas2003) to identify behaviour change strategies on internet websites promoting physical activity. The BTS consists of 20 intervention strategies derived from four traditional health promotion theories: health belief model (Rosenstock, Reference Rosenstock1974), theory of planned behaviour (Ajzen, Reference Ajzen1991), transtheoretical model (Prochaska, Reference Prochaska, DiClemente, Norcross and Goldfried2005), and social cognitive theory (Bandura, Reference Bandura, Bryant and Oliver2002). Each intervention strategy is scored out of 5 and is dichotomously rated for the inclusion of five dimensions of user interaction: (1) provides general information or guidelines, (2) assesses current practices or use of strategies, (3) provides feedback on assessment, (4) offers general assistance on behaviour change, and (5) offers individually tailored assistance in response to assessment and feedback. The levels are hierarchical, as level 5 (individual advice) is thought to be more effective than level 1 (providing general information). The BTS is the total of scores for all 20 intervention strategies; the maximum BTS score being 100, representing 20 strategies, each of which are scored out of 5 to indicate the level of interactivity.
The intervention strategies in the BTS are also categorised into five domains for interpretation: knowledge, behavioural strategies, cognitive strategies, emotion-focused strategies, and therapeutic interventions. Each intervention strategy was also explored on five different levels of user interaction, which included: provides general information and guidelines, assesses current practices or use of strategies, provides feedback on assessment, offers general assistance on behaviour change, and offers individually tailored assistance in response to assessment and feedback. The BTS incorporated a total of 12 key behaviour change strategies (see Table 3).
Persuasive Technology Content Survey (PTS)
The PTS was used in one study to evaluate weight management apps (Azar et al., Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013). The PTS has adapted principles from the Fogg Behaviour Model (FBM), which describes three components influencing individuals to engage in behaviour change: motivation, ability, and triggers. Unlike traditional behaviour change theories that generally do not account for the responsive and adaptive nature of technology, users of persuasive technology theory argue that technology can be designed and utilised to influence attitudes and behaviours (Fogg, Reference Fogg2003, Reference Fogg2009). The six items in the PTS were constructed with principles from FMB: motivators (social praise, financial incentives, non-financial incentives), ability (features that decreased barriers such as tailored information), and triggers used to promote adherence to apps (positive feedback for completing a task, prompts/cues to use apps). All items on the PTS were generalisable across all health conditions and incorporated four behaviour change strategies (see Table 3).
Precede-Proceed Model (PPM)
The PPM was used in one study to evaluate stress management apps (Payne et al., Reference Payne, Wilkinson, West and Bernhardt2016). This was developed as a health promotion framework that provided a series of steps that lead to behaviour change (Green & Kreuter, Reference Green and Kreuter1991). The framework grouped intervention strategies together into three factors: predisposing (related to knowledge/awareness, information, beliefs/values/attitudes, confidence/motivation), enabling (teach a skill, provide service/sell something, trace/record behaviour), and reinforcing (rewards, and feedback from others). The items of PPM were generalisable across different health conditions and contained four key behaviour change strategies (see Table 3).
Diabetes Prevention Program (DPP) measure
The DPP was identified in one study to assess weight management apps (Pagoto et al., Reference Pagoto, Schneider, Jojic, DeBiasse and Mann2013). The DPP was developed as an evidence-based lifestyle intervention effective for weight loss and risk reduction for diabetes (Whittemore, Reference Whittemore2011). The DPP consists of 20 key intervention strategies found to be effective in past trials (DPP Research Group, 2002a, 2002b). Among the 20 intervention strategies, 13 were tailored to weight management and physical activity, and therefore the measure is only relevant for health conditions such as diabetes, obesity, and other cardiovascular illnesses. The DPP was excluded from Table 3 since it is not generalisable across health conditions.
Expert Committee for Paediatric Obesity Prevention (ECPOP) assessment
One study used the ECPOP to assess apps (Schoffman et al., Reference Schoffman, Turner-McGrievy, Jones and Wilcox2013). Created by health experts from 15 national healthcare organisations, the ECPOP serves as a set of evidence-based health guidelines addressing paediatric obesity prevention, assessment, and treatment (Barlow, Reference Barlow2007). The guideline consists of eight intervention strategies and seven behavioural targets. Although the intervention strategies are generalisable across various health conditions, the behavioural targets are only suitable for paediatric obesity (e.g. ‘decrease TV time to less than 2 hours per day’). As an overall measure, the ECPOP is not considered generalisable across health conditions, and does not, therefore, appear in Table 3.
Rating Measures to Assess App Quality and Design
The Mobile App Rating Scale (MARS) was used in one study to evaluate the quality of apps (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016). MARS was developed as a simple and reliable tool for quality assessment of apps on four domains: engagement, functionality, aesthetics, and information quality (Stoyanov et al., Reference Stoyanov, Hides, Kavanagh, Zelenko, Tjondrongoro and Mani2015). A total of 19 constructs were grouped into the four domains, and each construct was scored on a 5-point scale (1 as adequate and 5 as excellent). The mean of each domain was calculated as the grand mean of the constructs within the domain, and the total MARS score was computed as the mean across the domains. The study reported substantial interrater reliability (kalpha = 0.80) between two raters (Bardus et al., Reference Bardus, van Beurden, Smith and Abraham2016).
Two reviewed studies developed their own rating measures to evaluate app engagement and ease-of-use (Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015). One measure contained a set of 11 criteria that were related to engagement with users, and the second measure contained 9 criteria that were related to ease-of-use for users. For both measures, the items were scored as either present or absent, and the score was recorded as a proportion or percentage of the total score. For example, an app that scored 7 out of 11 on the engagement measure would have a score of .64 or 64%. The scores for each app from the two raters were then averaged to achieve the final score. The two studies reported interclass correlation coefficients of .77 and .91 for the engagement measure, and .75 and .54 for the ease-of-use measure, respectively (Ubhi et al., Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016; Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015).
Discussion
A main aim of this study was to evaluate the methodologies of previous content analysis studies of behaviour change strategies in health self-management apps with a view to identifying taxonomies that school psychologists and counsellors could use for the purposes of evaluating and developing apps. It was evident not all taxonomies were able to be used to evaluate all types of apps and that the use of raters to evaluate apps did not always reach a criterion for best practice. It was also clear that the amount of behavioural content varied across the measures. Finally, user friendliness as defined by quality and design was measured in different ways even though the MARS rating scale was considered to be a simple, objective, and reliable tool for classifying and assessing the quality of mobile health apps (Stoyanov et al., Reference Stoyanov, Hides, Kavanagh, Zelenko, Tjondrongoro and Mani2015). These findings place the school psychologist and counsellor in a better position to evaluate apps by raising their awareness of how apps can be evaluated and by highlighting that behavioural content is a key component of the best apps if behaviour change is the goal. For those interested in researching and developing their own apps, the content analysis approach has provided guidance on how to interpret and code textual material in order to make replicable and valid inferences by systematically evaluating the content of apps.
Each of these findings are now discussed in detail. It was evident that five taxonomies were generalisable across the various health conditions: 26-BCTs, BCTTv1, BTS, PTS, and PPM. Such generalisable rating measures potentially provide a basis of control (or standardisation) for evaluating the same behaviour change content in apps under any condition, and therefore provide a fair comparison of content across studies, leading to the conclusion that behavioural health app researchers should consider the use of generalisable rating measures, instead of tailored rating measures such as the DPP measure.
Rating measures were compared on their comprehensiveness in behaviour change strategies crucial to self-management. It was shown that, among the generalisable measures, the 26-BCTs and BCTTv1 incorporated the highest number of behaviour change strategies, followed by BTS, with the least comprehensive being PTS and PPM.
A rating measure's overall design — in particular, the complexity of its scoring system and clarity of items — are factors that can influence raters’ ability to use the measure accurately. Both 26-BCTs and BCTTv1 use a simple binary scoring system and provide descriptive explanations (coding manuals) to help raters differentiate the meaning of each BCT. BTS's scoring system is more complex due to two levels of measurement (intervention strategies and user interactivity). The lack of explanations for items on the BTS could render the measure more challenging for raters to understand. Although the PTS and PPM both had simple scoring systems, the measures lacked detailed explanations on items. Consequently, future behaviour change rating measures may be improved by using a less complex scoring system and in-depth descriptions of each item in the measure.
An examination of the findings suggested a need for standardisation in the research methodology, particularly in the design and implementation of the apps selection process and rating procedures. It was shown that almost half of the studies did not select Google Play apps. This may be viewed as a considerable oversight by past health app content analyses, given that the number of Google Play apps is now comparable to or greater than that of Apple iTunes apps (Sims, Reference Sims2015). Considering Google Play's increased penetration in the apps market, future studies would benefit from including apps from both operating platforms to provide accurate and comprehensive samples for review. Indeed, only the top-ranked apps were included in four studies for review. The rationale proposed was that selecting the most popular apps would increase the sample's relevance to its audience. Results from the study by Azar et al. (Reference Azar, Lesser, Laing, Stephens, Aurora, Burke and Palaniappan2013), however, indicated that apps of lower ranking incorporated a similar level of behavioural change content as apps of higher rankings. Thus, selection of apps based on ranking may potentially exclude apps worthy of review. Future health apps content analyses should reconsider whether popularity is a good basis for apps selection criteria.
Content analysis as a research methodology was also prone to rater bias, as most studies recruited only two raters. Indeed, obtaining similar data through multiple observers (or raters) would increase confidence that the definition of the target behaviour was unambiguous and that the rating measure was not too difficult to use (Cooper, Heron, & Heward, Reference Cooper, Heron and Heward2007). Additionally, raters’ background and rater training should be considered at the outset of the study, since both factors can influence how content is identified. Of concern was the fact that seven studies did not report any form of training prior to app assessment. Providing adequate training can bridge the knowledge gap for raters without behavioural health background. Ubhi et al. (Reference Ubhi, Kotz, Michie, van Schayck, Sheard, Selladurai and West2016) reported that the addition of a structured rater training (involving 8 hours over 4 days) significantly increased the interrater reliability, compared to a past study conducted without any rater training (Ubhi et al., Reference Ubhi, Michie, Kotz, van Schayck, Selladurai and West2015). Finally, future content analyses are likely to benefit from conducting rating tests prior to apps assessment as they help to verify the raters’ understanding of the measure. It was shown that only half of the reviewed studies implemented rating tests prior to app assessment. Lastly, future content analyses should consider extending the timeframe for testing apps. For example, some technical features such as pop-up reminders are accessed only when apps are tested more than once. Disappointingly, only four reviewed studies reported that raters were allowed to test apps on more than one occasion (i.e., over several days).
The findings from the present study were in line with published studies that found behaviourally based apps were more effective than non-theory based ones (Nunday et al., Reference Nundy, Mishra, Hogan, Lee, Solomon and Peek2014). Indeed, current international diabetes guidelines endorse interventions that are based on behavioural theory. Behaviourally focused interventions that include interactive strategies have the greatest impact on metabolic and diabetes self-care outcomes. These findings have, in turn, influenced best practice guidelines for chronic disease management of lifestyle-related problems that include the recommendation that interventions be based on behaviour change theory (Azar et al., Reference Azar, Riley and Mani2015).
Limitations
The interrater reliability/agreement varied considerably across the different BCT versions. For example, in the three studies that used modified BCT versions there was almost perfect interrater reliability/agreement, while in the four studies that used the generalisable BCT versions (26-BCTs and BCTTv1), researchers recorded only moderate interreliability/agreement. This inconsistency in interrater reliability across the BCT versions could be due to the level of ease or difficulty in understanding the items. It was evident that the definitions of BCTs in the modified versions were easier to understand compared to the generalisable versions. Such a finding has suggested a need for adequate training using the generalisable versions of the BCT. Indeed, Wood et al. (Reference Wood, Richardson, Johnston, Abraham, Francis, Hardeman and Michie2015) have created and tested the use of BCTTv1 training modules using face-to-face and international teleconference workshops, to ensure that raters were able to use the measure competently and reliably.
One limitation that existed in all the behaviour change rating measures was their inability to measure how behaviour change content assisted users to self-manage, such as the required frequency, intensity, and duration of behaviour change strategies. The rating measures can indicate the amount of behaviour change content incorporated within apps, but they cannot measure how behaviour change strategies are applied, real time, by the user. It is not possible, therefore, to make conclusions on the rating measures’ ability to predict the actual effectiveness of an app as a behaviour intervention tool without further trial studies. Ultimately, digital tools still require effort to operate. This is an obvious issue to be explored in future research.
Conclusions
This study reviewed 15 health app content analyses that assessed behaviour change content. An interpretation of the findings confirmed an overall lack of consistency in the research methodology across the studies, particularly in the app selection process and rating procedures.
Future researchers, when using content analysis approach, should select apps from both iTunes and Google Play and be cautious on selecting apps based on popularity. It was also clear that rating procedures for apps should include multiple raters, provide adequate rater training, implement rating tests, and extend the timeframe for apps testing. Finally, future content analyses may benefit from standardisation in research methodology and rating measures, to provide a way to validly and reliably compare apps.
It was also evident that apps should contain effective behaviour change strategies, as such strategies play an important role in self-management interventions and in facilitating behaviour change. They should also ensure that an individual adheres to the intervention over extended periods of time. Self-management appears to be especially important for long-term chronic conditions such as asthma and diabetes.
Of interest to app designers, counsellors, and psychologists is the fact that only five behaviour change rating measures were identified as generalisable across all health conditions, with the BCTTv1 and 26-BCTs seen as the most comprehensive in behaviour change strategies, as well as most user friendly in design.
However, to test our conclusion that BCTTv1 and 26-BCTs are the most effective taxonomies for evaluating health related apps, we now plan to use them to assess in detail the functioning of asthma and diabetes management apps. Such a focus will allow us to determine not only whether they are a good fit for evaluating such apps but also whether asthma and diabetes management apps require further development to improve behaviour change content and incorporate more evidence-based health information.
Disclosure Statement
There are no conflicts of interest, financial or otherwise connected with this study.
