INTRODUCTION
The rates of neurodegenerative disorders are rising as the population ages, and costs of assessment are a barrier to timely diagnosis and treatment (Hurd, Martorell, Delavande, Mullen, & Langa, Reference Hurd, Martorell, Delavande, Mullen and Langa2013). Persons with Parkinson’s disease (PD) experience common barriers to assessment found with other neurodegenerative disorders (Bradford, Kunik, Schulz, Williams, & Singh, Reference Bradford, Kunik, Schulz, Williams and Singh2009), but motor impairment is an additional impediment to clinic and lab visits (e.g., slow gait, risk of falls, challenges to driving or using public transit). These limitations result in delayed diagnosis for some individuals long after symptoms emerge (Breen, Evans, Farrell, Brayne, & Barker, Reference Breen, Evans, Farrell, Brayne and Barker2013) as well as suboptimal treatment (Hogan et al., Reference Hogan, Bailey, Black, Carswell, Chertkow, Clarke and Thorpe2008). Nonetheless, clinics and labs are required for current methods of assessment of PD symptoms, including nonmotor symptoms such as cognitive impairment.
Neuropsychological testing to characterize cognitive deficits requires hours of time from persons with PD, who often experience daytime fatigue (Friedman et al., Reference Friedman, Brown, Comella, Garber, Krupp, Lou and Taylor2007), and current methods of testing capture cognitive performance only at a single time point. This is particularly problematic for persons with PD, which is a disorder marked by fluctuations in symptoms over time including changes in cognition, mood, motivation, and arousal level (Cronin-Golomb, Reynolds, Salazar, & Saint-Hilaire, Reference Cronin-Golomb, Reynolds, Salazar, Saint-Hilaire, Alosco and Stern2019).
Digital technology has the potential to address limitations (e.g., access, time, and cost) inherent to neuropsychological assessment in clinical populations such as PD (Parsey & Schmitter-Edgecombe, Reference Parsey and Schmitter-Edgecombe2013). Emerging digital assessments use computerized versions of the lengthy analog tests that only provide a snapshot of cognition in a single session (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012; Rentz et al., Reference Rentz, Dekhtyar, Sherman, Burnham, Blacker, Aghjayan and Sperling2016); these tests can now be completed at home for tracking cognitive function. An emerging literature indicates that smartphone assessments of cognition in clinical samples show strong convergent validity with their analog traditional neuropsychological assessments (Dagum, Reference Dagum2018; Moore et al., Reference Moore, Campbell, Delgadillo, Paolillo, Sundermann, Holden and Swendsen2020; Pal et al., Reference Pal, Mendelson, Clavier, Baggott, Coyle and Galloway2016; Schuster, Mermelstein, & Hedeker, Reference Schuster, Mermelstein and Hedeker2016; Sliwinski et al., Reference Sliwinski, Mogle, Hyun, Munoz, Smyth and Lipton2018; Timmers et al., Reference Timmers, Maeghs, Vestjens, Bonnemayer, Hamers and Blokland2014).
Another benefit of remote digital assessment is the ability to easily assess cognitive variability with a higher volume of trials than one-time testing allows. Intraindividual variability is a sensitive marker of function and risk for decline in neurodegenerative disorders (Christ, Combrinck, & Thomas, Reference Christ, Combrinck and Thomas2018; Haynes, Bauermeister, & Bunce, Reference Haynes, Bauermeister and Bunce2017). Specifically in PD, we know that greater intraindividual variability in processing speed, beyond mean changes, is associated with later stages of the disease (de Frias, Dixon, Fisher, & Camicioli, Reference de Frias, Dixon, Fisher and Camicioli2007). Repeated testing of cognition and state-based variables via a mobile assessment device, such as a smartphone, allows us to identify changes in disease symptoms, which can subsequently predict more subtle changes in disease state or progression. Furthermore, compared to the invasive and costly nature of biomarkers, within-person variability as measured via intensive longitudinal data could produce a sensitive metric of disease stage and diagnosis without undue burden to individuals. The ubiquity of smartphones allows for evidence-based assessment in the hands of those who may most benefit from it. The potential of this technology is becoming increasingly apparent given the ever-shrinking digital divide between younger and older adults as smartphone ownership among seniors continues to rise (Anderson & Perrin, Reference Anderson and Perrin2017).
Few studies to date have used ecological momentary assessment (EMA) in PD, which refers to frequent, typically brief assessments delivered in real-world environments (Csikszentmihalyi & Larson, Reference Csikszentmihalyi and Larson1987). Our lab demonstrated the feasibility of conducting a smartphone EMA study of mood and subjective sleep quality in PD (Wu & Cronin-Golomb, Reference Wu and Cronin-Golomb2020). Further, a single-case EMA study established proof-of-concept using smartphone technology to track intraindividual fluctuations in PD motor symptoms as they related to mood and then modeled these fluctuations using network analysis; they found that anxiety was prospectively associated with more tremor, and cheerfulness with less tremor (van der Velden, Mulders, Drukker, Kuijf, & Leentjens, Reference van der Velden, Mulders, Drukker, Kuijf and Leentjens2018). The mPower Study used a brief repeated-measures smartphone design to assess PD motor symptoms including motor and reaction time over six months (Bot et al., Reference Bot, Suver, Neto, Kellen, Klein, Bare and Trister2016; Lipsmeier et al., Reference Lipsmeier, Taylor, Kilchenmann, Wolf, Scotland, Schjodt-Eriksen and Lindemann2018). All elements of the assessment related to at least one component of the Unified Parkinson’s Disease Rating Scale (UPDRS); e.g., finger tapping (motor speed) was significantly correlated with UPDRS ability to independently dress oneself without difficulty (Lipsmeier et al., Reference Lipsmeier, Taylor, Kilchenmann, Wolf, Scotland, Schjodt-Eriksen and Lindemann2018). Another study used smartphone test data to develop an independent and valid disease severity score using a supervised and weighted machine learning algorithm. The score was significantly associated with the UPDRS and other standard PD motor tests such as Timed Up and Go (Zhan et al., Reference Zhan, Mohan, Tarolli, Schneider, Adams, Sharma and Saria2018). A limitation of this latter study was that all assessments occurred within the clinic setting and no measures of neuropsychological function beyond reaction time were included. To date, no studies have examined patterns of neuropsychological testing using smartphone technology in PD, although research has found cognitive correlates in PD (Lo et al., Reference Lo, Arora, Baig, Lawton, El Mouden, Barber and Hu2019).
The first purpose of the present study was to use a smartphone EMA design to assess executive function and working memory, which are among the earliest cognitive dysfunctions to arise in PD (reviewed in Cronin-Golomb et al., Reference Cronin-Golomb, Reynolds, Salazar, Saint-Hilaire, Alosco and Stern2019; Dirnberger & Jahanshahi, Reference Dirnberger and Jahanshahi2013; Miller, Neargarder, Risi, & Cronin-Golomb, Reference Miller, Neargarder, Risi and Cronin-Golomb2013) and are related to a wide range of other symptoms, including anxiety (Reynolds, Hanna, Neargarder, & Cronin-Golomb, Reference Reynolds, Hanna, Neargarder and Cronin-Golomb2017), depression and apathy (reviewed in Dirnberger & Jahanshahi, Reference Dirnberger and Jahanshahi2013), and impairments in spatial judgment (Salazar, Moon, Neargarder, & Cronin-Golomb, Reference Salazar, Moon, Neargarder and Cronin-Golomb2019) and gait (Morris et al., Reference Morris, Martini, Smulders, Kelly, Zabetian, Poston and Horak2019). While the cognitive measures we included are similar to tests used in other app-based measures of cognition (Moore, Swendsen, & Depp, Reference Moore, Swendsen and Depp2017), performance on the app-based tests have generally not been measured alongside performance on the traditional neuropsychological tests (Spatial Span [working memory] and Trail Making Test [executive function]) upon which they are based. A recent smartphone-based study of performance on a Stroop-based task by persons aged 50 years and above with HIV, a disorder that is like PD in that it affects the basal ganglia, showed good feasibility and convergent validity with respect to traditional neuropsychological measures of executive function and working memory (Moore et al., Reference Moore, Campbell, Delgadillo, Paolillo, Sundermann, Holden and Swendsen2020). We predicted a significant association in PD between scores on our traditional neuropsychological tests and overall mean score (across time points) on the respective smartphone cognitive assessments.
An additional objective was to examine how PD symptoms and individual differences relate to design feasibility and to intraindividual variability in cognitive performance. We predicted that the majority of participants, who were nondemented, would exhibit a completion rate similar to that seen in other studies of mobile cognitive assessment, roughly 70%–80% completion of surveys (Moore et al., Reference Moore, Swendsen and Depp2017). We hypothesized that those with lower baseline scores on a global cognitive screening measure, the Montreal Cognitive Assessment (MoCA), would have lower rates of response due to difficulty navigating the smartphone app interface without the assistance of a researcher or a clinician. We expected that those with greater symptom burden (higher UPDRS score) would have lower rates of response due to motor or other impairments (e.g., pain, fatigue).
Because the literature has demonstrated associations between specific contextual variables and within-person fluctuations in cognition (Weizenbaum, Torous, & Fuford, Reference Weizenbaum, Torous and Fuford2020), we developed exploratory hypotheses regarding contextually driven within-person fluctuations. Specifically, we examined whether lower mood and higher anxiety would be associated with worse performance on smartphone tasks, as prior research has shown negative mood (Brose, Schmiedek, Lövdén, & Lindenberger, Reference Brose, Schmiedek, Lövdén and Lindenberger2012; Jefferies, Smilek, Eich, & Enns, Reference Jefferies, Smilek, Eich and Enns2008) and stress (Sliwinski, Smyth, Hofer, & Stawski, Reference Sliwinski, Smyth, Hofer and Stawski2006) to be associated with poorer momentary performance on attention and working memory tasks, especially in older adults. Similarly, we assessed whether lower motivation related to worse smartphone performance due to poor task engagement, as shown in another EMA study of working memory (Brose et al., Reference Brose, Schmiedek, Lövdén and Lindenberger2012). As older adults typically perform best on working memory in the morning (West, Murphy, Armilio, Craik, & Stuss, Reference West, Murphy, Armilio, Craik and Stuss2002), we assessed whether time of day would be a significant predictor of performance. Subjective alertness can also be predictive of cognitive performance on attention accuracy tasks (Manly, Lewis, Robertson, Watson, & Datta, Reference Manly, Lewis, Robertson, Watson and Datta2002); hence, we hypothesized that alertness would be positively associated with performance. Although social activity has been associated with improved cognitive performance in certain domains, including memory and processing speed (Bielak, Mogle, & Sliwinski, Reference Bielak, Mogle and Sliwinski2017), the visual and auditory distraction in these settings may be less conducive to optimal performance, and variable noise in the surrounding environment is associated with worse working memory (Lange, Reference Lange2005). We examined whether being at home (versus in other environments) would produce a more controlled and familiar sensory environment and hypothesized that it may lead to improved cognitive performance.
We also considered whether smartphone cognitive performance would be related to self-reported on-off periods of medication, as off-periods are associated with an increase in motor symptoms including tremor as well as nonmotor symptoms such as pain and anxiety (Cheon, Park, Kim, & Kim, Reference Cheon, Park, Kim and Kim2009). In those without medication-related fluctuations, it was predicted that off-periods would be associated with poorer cognitive performance at that time point. The final of the secondary contextual hypotheses was whether factors such as recent exercise may be associated with improved cognitive performance, as previous work suggests widespread benefit of exercise to cognition in people with PD (Murray et al., Reference Murray, Sacheli, Eng and Stoessl2014; Reynolds et al., Reference Reynolds, Hanna, Neargarder and Cronin-Golomb2017).
METHODS
Participants
We initially recruited 30 community-dwelling participants who met the clinical criteria for mild to moderate idiopathic PD, following the United Kingdom Parkinson’s Disease Society Brain Bank diagnostic criteria (Hughes, Daniel, Kilford, & Lees, Reference Hughes, Daniel, Kilford and Lees1992). They were recruited from the Boston University Center for Neurorehabilitation and the Parkinson’s Disease and Movement Disorders Center at Boston Medical Center. Study protocols were approved by the Boston University Institutional Review Board, with consent obtained according to the Declaration of Helsinki. All participants were native-English speaking adults who owned a smartphone with iOS or Android operating systems. Exclusion criteria included a score of less than 20 on the Montreal Cognitive Assessment (MoCA) or a motor impairment (e.g., significant tremor or dyskinesia) that prohibited them from regularly using a smartphone. Out of the 30 enrolled, 27 completed the remote smartphone assessments in addition to the in-lab initial assessment. One dropped out due to other time commitments, and the other two experienced technical failures with their phone and/or the study app. All but one of the 27 participants were taking medication to treat PD. We calculated levodopa equivalent daily dosages (LEDD) based on convention (Tomlinson et al., Reference Tomlinson, Stowe, Patel, Rick, Gray and Clarke2010). See Table 1 for descriptive statistics of the final sample.
Participant characteristics (n = 27)
MoCA, Montreal Cognitive Assessment; UPDRS, Unified Parkinson’s Disease Rating Scale (disease severity); LEDD, levodopa equivalent daily dose.
Study Procedure
After enrollment by phone, participants were emailed a link to online questionnaires to complete before the in-lab assessment. In-lab (Study Day 0), after the MoCA confirmed eligibility, the UPDRS was administered by a trained examiner as well as two brief traditional neuropsychological measures: the WMS-III Spatial Span Test and the Trail Making Test A & B. Participants then downloaded the study app onto their own smartphone. They were guided through a slideshow presentation on how to use the app and practiced completing surveys and games with the researcher present. Immediately following the visit, participants were emailed the slideshow of instructions and the structure of smartphone assessment administration was described to them. On Study Days 1–10, smartphone assessment notifications appeared consistently at 9am, 12pm, 3pm, 6pm, and 9pm. The smartphone assessments consisted of a brief survey and two games (described below) and typically required five minutes to complete. Participants had the opportunity to respond to the assessments within two hours of the initial notification before the assessment window would close until the next time prompt (Table 2). Participants were encouraged to contact researchers if questions arose and were also emailed on Day 2 and Day 6 to check-in and troubleshoot any problems encountered using the study app. Participants were compensated at the rate of $15/hour for time spent in-lab and $1 for every smartphone assessment, with the opportunity to earn a $15 bonus if they completed at least 80% (40/50) of all assessments.
Study design
Study Measures
Online pre-study survey
The following questionnaires were completed online before the lab visit. Mood and Anxiety Symptoms Questionnaire (MASQ-short) assesses symptoms of depression and anxiety (Wardenaar et al., Reference Wardenaar, van Veen, Giltay, de Beurs, Penninx and Zitman2010). Positive and Negative Affect Scale assesses trait positive and negative affect (Watson, Clark, & Tellegen, Reference Watson, Clark and Tellegen1988). Motivation and Pleasure Scale assesses motivation and pleasure-seeking behavior (Llerena et al., Reference Llerena, Park, McCarthy, Couture, Bennett and Blanchard2013). Behavior Rating Inventory of Executive Functioning assesses self-perceived difficulties in executive functioning (Roth, Gioia, & Isquith, Reference Roth, Gioia and Isquith2005). Epworth Sleepiness Scale (ESS) assesses daytime sleepiness (Johns, Reference Johns1991).
In-lab baseline assessment
The following tests were administered in-lab: UPDRS, a standard assessment of PD symptoms including in-person interview and motor examination and Hoehn and Yahr (H&Y) stage rating (Fahn, Elton, & UPDRS program members, Reference Fahn, Elton, Fahn, Marsden, Goldstein and Calne1987); the neuropsychological measures, Spatial Span (Wechsler Memory Scale-III, Wechsler, Reference Wechsler1997 ) for visual working memory as the analog of the smartphone Backwards Spatial Span, and Trail Making Test A & B (Tombaugh, Reference Tombaugh2004), for processing speed and complex attention as the analog of to the smartphone Trails-B test.
mindLAMP smartphone assessments
The app-based assessment included questions of context, mood state, and cognition using the Mind Learn-Assess-Manage-Prevent app developed at the Department of Digital Psychiatry of Beth Israel Deaconess Medical Center, Boston (Torous et al., Reference Torous, Wisniewski, Bird, Carpenter, David, Eduardo and Keshavan2019). See screenshots of the app interface in Figure 1a–d. Code for the smartphone assessments is publicly available at: https://github.com/BIDMCDigitalPsychiatry
mindLAMP App Assessments. (a) Location & Social Context Survey: Indication of type of current location and social context. (b) Mood & Alertness Survey: five questions including rating happiness, sadness, anxiety, alertness, and level of motivation to try one’s hardest on subsequent cognitive games. (c) Backwards Spatial Span Test: four increasingly challenging trials of a task requiring one to tap the sequences of boxes that appeared on the screen during that trial but in reverse order. (d) Trails-B Test: four increasingly lengthy trials of a task requiring one to tap numbers and letters on the screen in a sequential and alternating manner.
Location & Social Context Survey assessed type of current location and social context (Figure 1a). Location options included home, work/school, someone else’s home, public place indoors, public place outdoors, and transportation. Social context survey options included alone, strangers, classmates/coworkers, acquaintances, close friends, family members, and romantic partner. Mood & Alertness Survey included five questions relating to mood (happy/sad), anxiety, alertness, and level of motivation to try one’s hardest on subsequent cognitive games (Figure 1b).
As noted above, executive function and working memory are among the earliest cognitive domains affected by PD and also are domains affected by contextual variables and for this reason we included the following tests:
Backwards Spatial Span Test consisted of four increasingly challenging trials requiring one to tap the sequence of boxes that appeared on the screen during that trial but in reverse order (Figure 1c). Trails-B Test consisted of four increasingly lengthy trials requiring one to tap numbers and letters on the screen in a sequential and alternating manner (Figure 1d).
Data Analysis
EMA response rate was calculated as the number of completed smartphone assessments divided by the total possible number of assessments for each person (50), and then within-person means were averaged to create a sample response rate mean. To determine whether there was reliable and systematic between-person variability that was greater than within-person variability in participants’ cognitive tests scores (i.e., Backwards Spatial Span and Trails-B), a multilevel modeling approach was taken using the software MPlus Version 8.1.7 that uses between- and within-person variance to calculate between-person reliability. The formula for reliability is where Var(BP) is the total variance in scores that is between persons, Var(WP) is the total variance in scores that is within persons, and n is the number of assessments (Sliwinski et al., Reference Sliwinski, Mogle, Hyun, Munoz, Smyth and Lipton2018).
$$BP\,{\rm{Reliablity}} = {{Var{\rm(BP)}} \over {\left( {Var{\rm(BP)} + {{Var{\rm(WP)}} \over n}} \right)}}$$
Scores were aggregated across time points to find the sample’s average within-person means and standard deviations across the full study period and by time point and day. In addition to whole-sample analyses, within-person means were calculated along with within-person standard deviation to identify whether a person’s level of variability in score was associated with the overall performance on the task. Pearson correlations were used to assess potential convergence between within-person smartphone cognitive test means and self-reported executive dysfunction (BRIEF-A). Because of the repeated nature of smartphone tests compared to the one-time BRIEF-A questionnaire, a multilevel model maximum likelihood regression was used with the BRIEF-A as a Level 2 fixed effect predictor of smartphone test performance, which was entered as a Level 1 random effect outcome variable. To assess convergent validity with traditional neuropsychological measures, in-lab test scores were entered as Level 2 fixed effect predictors of Backwards Spatial Span and Trails-B performance (the Level 1 outcome variable) each in their own separate model.
We also conducted multilevel model regressions to assess the extent to which smartphone test performance on the Backwards Spatial Span or Trails-B task related to context and state variables collected at the same time point. In this model, each contextual variable was entered as an independent Level-1 predictor of the Level-1 outcome: Backwards Spatial Span or Trails-B. Survey time point (1–50) and day of study were independently analyzed as Level-1 predictors to determine whether smartphone cognitive test performance improved over the course of the study period as a result of practice and familiarity with the task. A two-level regression model was used to determine whether Level-2 fixed effects (between-person measures) moderated the within-person relations between context and smartphone score. These fixed effects included demographics, PD-specific measures (UPDRS Total Score (Parts I-IV), UPDRS Motor Subscale (Part III), UPDRS tremor item total (items 20 and 21), H&Y stage), baseline cognitive score (MoCA), and online questionnaires of trait affect, anxiety, sleepiness tendency, and motivation.
RESULTS
Association with Sample Characteristics
LEDD was not significantly correlated with any cognitive measures (in-lab or smartphone) (r’s < .20, p’s > .33). Age was negatively associated with the smartphone Backwards Spatial Span Task (r = −.59, p < .01), and UPDRS total score was negatively associated with the smartphone Trails-B task (r = −.53, p < .01). No other participant characteristics were associated with smartphone or in-lab cognitive measures.
Feasibility
Adherence was 73% (SD = 10%); on average, participants responded to roughly 37 out of the 50 possible assessments over the course of the 10-day study period. Age, disease severity (UPDRS total), and baseline MoCA score were not significantly correlated with response rate to EMA prompts. Responses were captured in a variety of social contexts. Participants reported being alone 44% of the time when completing the surveys and games, followed by 32% with family members. Overall, responses were evenly distributed across the course of the day, with most responses in the afternoon (40%) and evening (37%), which is to be expected as two prompts occurred in the afternoon and two in the evening. Responses were most likely to occur within the physical context of being home (71%), with the remaining 29% relatively evenly distributed across other locations including public places indoors, outdoors, work, someone else’s home, and during transit (Figure 2).
Summary of Smartphone Survey Responses.
Smartphone Cognitive Test Psychometrics
Backward spatial span
Within-person mean accuracy was 85.9% (SD = 5.2) (Table 3). Within-person means were significantly negatively correlated with standard deviation (r = −.70, p < .001), meaning that high accuracy was associated with less intraindividual variability. Between-person reliability was high (.89). Day of the study (1–10) was significantly related to higher accuracy (B = .33, SE = .13, p = .01), indicating a practice effect; however, no significant residual variance indicated no individual differences in this effect. Specifically, when scores were aggregated by day, the greatest amount of variability occurred on Day 1 (mean score = 84.0 [SD = 2.1]), after which SD rapidly narrowed to the study average of 85.9 (SD = 5.2).
Smartphone cognitive test psychometrics
Trails-B accuracy
When scores were collapsed across time points within individuals, Trails-B accuracy was 96.2% (SD = 3.3), indicating a ceiling effect with little variability around score from moment to moment within individuals (Table 3). Within-person means were significantly negatively correlated with standard deviation (r = −.93, p < .001), meaning that high accuracy was associated with less intraindividual variability around the mean. Between-person reliability was .87. There was no association between day of study (B = .13, SE = .08, p = .12) and accuracy, nor was there significant residual variance between persons. This indicates the absence of a practice effect, but likely reflects the high and unvaried score accuracy across participants and across the study period.
Smartphone Cognitive Test Convergent Validity
The smartphone Backwards Spatial Span was significantly predicted by performance on the MoCA, WMS-III Spatial Span Test (including total score backward and forward and subtest scores), and the Trail Making Test (time on A & B) (Table 4). By contrast, smartphone Trails-B task accuracy was not predicted by any of the traditional neuropsychological measures, including baseline cognitive screen (MoCA), WMS-III Spatial Span Test, or Trail Making Test A & B Time (Table 4), potentially owing to the limited variability in scores on the smartphone measure. Neither smartphone test score was predicted by the BRIEF-A questionnaire score nor was BRIEF-A (subjective self-report) correlated with any of the traditional in-lab test scores, all from objective task-based measures.
Convergent validity between smartphone and traditional analog cognitive tests
Individual differences as independent predictors of smartphone test scores
Using multilevel regression models of between-person (Level 2) predictors of repeated (Level 1) measurements, we found that higher scores on the UPDRS tremor items (B = −.77, SE = .27, p = .005), UPDRS Part III motor subscale (B = −.25, SE = .07, p = .001), and UPDRS Part I-IV total score (B = −.13, SE = .05, p = .009) predicted lower overall smartphone Trails-B accuracy scores, but not Backwards Spatial Span scores. Older age predicted lower overall Backwards Spatial Span scores, but not Trails-B scores. Neither the UPDRS Part I mood-mentation-behavior subscale nor the UPDRS Part II ADL subscale significantly predicted smartphone test scores, although there was a trend for an association between the ADL subscale score and Trails-B accuracy (B = −.280, SE = .149, p = .060) (Table 5).
Multilevel model results of PD-specific individual differences and smartphone cognitive test scores
EMA contextual variables and smartphone test performance
For Backwards Spatial Span, being at home (vs. other location) predicted a higher score at the same time point (B = 7.19, SE = 3.01, p = .02); there was no significant between-person residual variance in this association. Having reported exercising in the last 3 h also predicted higher Backwards Spatial Span score at the same time point (B = 2.16, SE = .78, p = .01); there was significant between person residual variance associated with this effect (B = 1.26, SE ≤ .001, p ≤ .001), meaning that individual differences existed in the strength of the exercise and Span score association (Table 6). When entered as a Level 2 predictor, a person’s mean exercise frequency score (the percent of times a person endorsed having exercised in the past 3 h) predicted the strength of the relation between exercise and spatial span score. Specifically, when those who generally exercised less frequently endorsed having exercised in the last 3 h, Backwards Spatial Span score tended to be higher (B = −2.42, SE = .10, p < .001). The question remained as to whether specific individual differences may predict which participants exhibited variability in their accuracy scores in relation to context variables. Based on exploratory correlations between individual differences and contextual variables, we entered individual differences as Level 2 predictors in multilevel models. No individual differences examined predicted the exercise and spatial span association (Table 7).
Contextual predictors of smartphone score as individual multilevel models
Multilevel model results of between-person predictors of context-smartphone performance association
For Trails-B, there were no significant associations between contextual EMA variables and accuracy. There was a significant level of between-person variability, however, given the residual variance values in the relation between time of day and Trails-B score (B = 3.75, SE = 1.65, p = .02) and being at home (versus another location) and Trails-B score (B = 57.76, SE = 23.49, p = .01). Despite the lack of sample-level relations between Trails-B score and time-of-day, this association did vary significantly from person to person in the study, as illustrated by the examples in Figure 3: Participant 40011 tended to be more accurate as the day progressed, whereas Participant 40008’s accuracy did not differ across the morning, afternoon, and evening.
Examples of Between-Person Variability in the Association Between Time of Day and Smartphone Trails-B Accuracy.
Individual difference variables that were significantly correlated with Trails-B accuracy were entered as Level 2 or between-person predictors in individual multilevel models of context and performance (Table 7). Those who reported lower baseline daytime sleepiness tended to perform better on Trails-B later in the day. No individual difference variables predicted the strength of a participant’s association between Trails-B accuracy and being at home.
DISCUSSION
This study examined the utility and feasibility of repeated smartphone assessment of cognition and context in persons with PD. There is a paucity of data on smartphone cognitive assessment in PD, and while a visual working memory task has been incorporated into other apps used to measure symptoms in PD (Bot et al., Reference Bot, Suver, Neto, Kellen, Klein, Bare and Trister2016), to our knowledge this is the first study to report on remote repeated measurement of working memory and executive functioning, cognitive domains that are affected early in PD.
On both smartphone cognitive measures, higher within-person mean accuracy was associated with lower within-person variability. This indicates that those who were performing more poorly were likely to be less consistent. An implication is that traditional one-time testing may not always be sufficient to capture the extent of variability (the good days and bad days) that people with cognitive impairment experience in day-to-day life. On Backwards Spatial Span, participants’ scores increased over time, primarily on the first day of the study. We found that average performance was very high on the smartphone Trails-B task, indicating a ceiling effect. Increasing the difficulty or scoring of this measure would improve psychometrics. Despite this limitation, the measure was found to be reliable in differentiating between participants when taking into account within-person fluctuation of repeated measurements.
Convergent Validity
Performance on the smartphone Backwards Spatial Span task was related to a number of the traditional in-lab neuropsychological measures. Most notably, in-lab WMS-III Spatial Span (including total, forward, and backward scores) strongly predicted an individual’s performance on the smartphone task. This provides evidence that this smartphone task likely measures a similar construct as the traditional in-lab neuropsychological measure and is the first to validate it against traditional measures. The present study supports use of smartphone cognitive assessment as a valid adjunct to traditional neuropsychological measures, especially because smartphone tasks can be completed quickly, administered remotely, and repeated across time points with high reliability.
Performance on Trails-B was not related to performance on any of the traditional neuropsychological tests administered in-lab, likely due to ceiling effects. Further, the smartphone Trails-B was a measure of switching accuracy versus speed, whereas completion time, or speed, is the primary metric of performance in the traditional Trail Making Test B. Our Trails-B task accuracy reflects inhibition and planning, components of executive function, not processing speed. Our version measured total time per Trails-B game, but the number of trials within a game varied based on accuracy; hence, response time was not comparable within or between people. Updated time measurement and scores based on speed or screen tap latency could make this task a more sensitive measure of processing speed and executive function. For example, it may be that the latency or response time in between alternating numbers and letters in the task is what differentiates people more than the accuracy of switches, and this kind of sensitive timing ability is what differentiates digital assessment from current analog versions of this task.
When considering the use of smartphone cognitive assessment in clinical populations with motor impairments, such as PD, it is essential to understand relations between disease symptoms and performance on the tasks themselves. In our study, the Trails-B task required quick and spatially precise taps on small on-screen circles. We found that higher UPDRS motor and tremor subscale scores were associated with lower scores on the task itself. If the objective of the assessment is to measure changes in processing speed and motor accuracy as a result of disease, this could be valuable to capture in a smartphone cognitive measure. When the goal is to understand complex attention, switching, and executive function; however, it is clear that a participant’s motor impairments may obscure cognitive ability. By contrast, the Backwards Spatial Span task was not associated with PD symptoms as measured by the UPDRS, which signals its utility in measuring cognitive performance via smartphone in those with motor speed impairments.
EMA Contextual Variables and Smartphone Test Performance
An additional aim of this study was to explore the extent to which cognitive performance varied across different contexts. We found that being home rather than in another environment, and endorsing recent exercise, predicted higher Backwards Spatial Span scores. On this measure of working memory, it seems logical that being in a familiar and perhaps less distracting environment such as one’s home would be conducive to stronger performance.
In interpreting the direct effect of exercise on visual working memory performance, it is important to note that this is an observational finding only. One possibility is that the physiological effects of exercise could contribute to improved cognitive performance in PD. Indeed, a recent systematic review showed that a variety of exercise programs can lead to improvements in global cognitive function, processing speed, attention, and mental flexibility in people with PD (da Silva et al., Reference da Silva, da Iop, de Oliveira, Boll, de Alvarenga, Gutierres Filho and da Silva2018). A number of studies point toward increased levels of blood-derived neurotrophic factor as the mechanism of action in the relation between exercise and higher cognitive performance in PD (Hirsch, van Wegen, Newman, & Heyn, Reference Hirsch, van Wegen, Newman and Heyn2018). It is also possible that on days when people are feeling better, broadly, they may be more likely to perform better on a working memory task and be more motivated to exercise. We found significant residual variance in this association indicating heterogeneous individual differences in how much recent exercise associated with Backwards Spatial Span scores. Specifically, those who least frequently endorsed exercise were the ones most likely to have a higher score when they did report recent exercise. This finding may further support the correlational interpretation that people with PD who exercise more frequently may be exercising unconditionally on “good” days and “bad” days or despite other factors of variability in their daily lives. By contrast, those who endorse exercising less frequently may have a more conditional relationship with exercise, such that choosing to exercise may be associated with other factors (e.g., less physical discomfort, a less busy day, etc.) that could also be conducive to better working memory.
Whereas there were no direct effects between contextual predictors and Trails-B, there were significant individual differences in these effects; specifically, those with less symptom burden (UPDRS total score) and less daytime sleepiness (ESS score) showed a stronger effect between Trails-B performance and later time of day. While there is reason to be cautious about these findings given the ceiling effect and motor confounds seen with Trails-B, it is worth considering that those with greater disease burden may show less sensitivity to varying contexts; alternatively, they may be more homebound and so have less exposure to varying physical and social contexts. One potential future question is how variability in context and one’s response to different contexts relate to stage of a disorder. For example, could a decrease in the variety of contexts or one’s differential response to them as measured by repeated mobile testing help to predict advancing disease state? Alternatively, mobile testing may be most useful for those in early stages of a disease. It may also help identify contexts in which performance is lower and targeted interventions could be of benefit for people across the spectrum of disease severity.
Results are in line with other studies regarding engagement. Participants completed 73% of the EMA prompts on average. This response rate is similar to other studies of mobile cognitive assessment in a variety of clinical samples, which found average study response to be 79% (Moore et al., Reference Moore, Swendsen and Depp2017) and is higher than the 61% response rate in a study of smartphone assessment of PD motor symptoms (Lipsmeier et al., Reference Lipsmeier, Taylor, Kilchenmann, Wolf, Scotland, Schjodt-Eriksen and Lindemann2018).
Study Limitations
This study examined a sample of individuals with mild to moderate PD severity without dementia and cannot generalize to individuals with more severe motor and cognitive symptoms. It is also important to note that participants in this sample were self-selected and had either previously participated in research or had expressed interest in participating. This type of sample often leads to narrow demographics in terms of socioeconomic status, education, and race, which poses barriers to community-wide generalization. A limitation in our methods was that one of our smartphone cognitive measures elicited very high accuracy and was not able to capture processing speed; nevertheless, the data produced were useful in regard to assessment of the relation between accuracy and performance variability.
Future Directions
In our study design, smartphone assessments were prompted at fixed times during the day. As such, participants could anticipate the exact time of an assessment and may have altered their routine to accommodate test-taking, which could have restricted the range of locations and contexts in which tests were completed. Further investigation is needed to compare performance under conditions of random vs. fixed-time prompts within daytime intervals. A second avenue of investigation would be to intentionally time test administration to coincide with participant-specific activities and routines. Future studies should also aim to measure cognitive domains beyond visual working memory and the inhibition and planning components of executive function, in order to compare results with those presented here; investigators wishing to use our tests may do so through this link https://github.com/BIDMCDigitalPsychiatry. We also suggest expanding the scope of studies of smartphone-based assessments of cognition to include measures of activities of daily living (ADLs) beyond the UPDRS ADL subscale used in our study where a trend was seen between this score and smartphone Trails-B accuracy. This line of future investigation is supported by an extensive literature relating performance on traditional neuropsychological tests of executive function to ADLs in older adults with and without cognitive impairment (e.g., Bell-McGinty, Podell, Franzen, Baird, & Williams, Reference Bell-McGinty, Podell, Franzen, Baird and Williams2002; see Overdorp, Kessels, Claassen, & Oosterman, Reference Overdorp, Kessels, Claassen and Oosterman2016 for a review), including in PD (Higginson, Lanni, Sigvardt, & Disbrow, Reference Higginson, Lanni, Sigvardt and Disbrow2013; Kudlicka, Hindle, Spencer, & Clare, Reference Kudlicka, Hindle, Spencer and Clare2018).
Conclusion
This study demonstrated that repeated smartphone assessment of cognitive performance in persons with PD is a feasible and useful method of neuropsychological assessment. Our smartphone measures of executive functioning and visual working memory both displayed strong between-person reliability. There was convergent validity for the smartphone test of visual working memory, with its results predicted by traditional neuropsychological tests administered in-lab. Performance on this task was also unrelated to PD motor impairment and tremor score, which demonstrates feasibility even in those with notable motor-based symptoms of PD. Further, we found that certain contexts and conditions (being at home and recent exercise) differentially predicted performance on this task at that same time point. Together these findings provide support for further investigation of smartphone cognitive assessment in PD as a means of understanding idiographic patterns of symptoms, with the potential to better predict disease progression and provide targeted and individualized interventions.
ACKNOWLEDGMENTS
Anup Dupaguntla, BA, provided valuable support on data entry and data management. Aditya Vaidyam, MS, and Phillip Henson, MS, assisted greatly with technical support for the mindLAMP and data formatting and compilation. Tairmae Kangarloo, BS, provided assistance with initial data visualization and analysis planning. We are grateful to Terry Ellis, PhD, and Cristina Colon-Semenza, PhD, for referring individuals with PD to this study from the Boston University Center for Neurorehabilitation and to Olivier Barthelemy, PhD, for referring participants from other lab studies. We acknowledge with gratitude the efforts of these colleagues and especially the efforts of all of the study participants.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
FUNDING
This work was funded by a Boston University Digital Health Initiative Research Incubation Award (#2018–02–006) with the support of the Institute for Health System Innovation & Policy and the Hariri Institute for Computing and Computational Science & Engineering at Boston University.
