Introduction
Ecological momentary assessment (EMA) is a research paradigm that involves collecting real-time data on individuals’ behaviors, experiences, and internal states in their natural environment, often using a smartphone or other personal mobile device. This approach has shown promise in psychological research, with benefits over traditional onetime assessments that include reducing recall bias and enhancing ecological validity. EMA allows researchers to study the dynamics of behaviors and experiences as they unfold in real time and can provide insight into influences of contextual factors, individual differences, and response variability. For example, EMA can periodically query participants regarding their mood states to elucidate whether participants feel better or worse when alone versus with others, active versus sedentary, or at home versus in community settings. EMA studies can also facilitate time-series analysis for tracking changes and trajectories of symptoms and experiences and may be valuable for monitoring the effectiveness of interventions.
There is growing interest in employing EMA paradigms in studies of traumatic brain injury (TBI), but there are reasonable concerns regarding feasibility, especially with high-intensity paradigms using multiple bouts of data collection per day. EMA studies typically require access and comfort with mobile devices as well as abilities such as attention, initiation, and follow-through that may be compromised following TBI. Furthermore, for those who experience fatigue or have limited cognitive resources, the frequency and duration of assessments may be burdensome enough to affect adherence.
Despite these concerns, a recent scoping review concluded that, overall, collecting patient-reported outcomes via smartphone EMA is feasible and acceptable in the chronic acquired brain injury population, and studies consistently showed an advantage of using EMA over single static measures (Juengst et al., Reference Juengst, Terhorst, Nabasny, Wallace, Weaver, Osborne, Burns, Wright, Wen, Kew and Morris2021). For example, a study of 52 individuals with moderate-severe TBI found that the overall response rate for EMA assessments for a 2-week period was 81.4% (Juengst et al., Reference Juengst, Wright, Sander, Preminger, Nabasny and Terhorst2022). In one of the few studies to examine personal factors related to EMA response rate in individuals with moderate-to-severe TBI, we examined 2-week adherence to EMA in a small sample (n = 23; a subset of participants contributing to the current report) and found that response rate was quite variable, between 6% and 100% (mean 65%) (Rabinowitz et al., Reference Rabinowitz, Hart and Wilson2021). A higher response rate was significantly correlated with the integrity of episodic memory and years of education. However, the persistence of adherence over a more extended period of time has yet to be explored in this population. Persistence is important in EMA studies because the longer the interval, the more likely that all or most of the naturally occurring contexts of interest will be captured. In clinical populations such as TBI, extended sampling may also be needed to help offset response variability.
In the present study, we extend our prior work by examining trajectories of EMA adherence over a 7-week period, using mixed-effects modeling to account for individual variation in level of adherence and change in response rate over time. Based on our and others’ previous findings, we also examined the impact of participants’ memory function, education, age, and initial severity of the TBI.
Methods
This study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki and its subsequent amendments. All procedures involving human participants were approved by the Institutional Review Board of the Albert Einstein Medical Center, ensuring compliance with established ethical standards for research with human subjects. Informed consent was obtained from all participants included in the study, and measures were taken to protect their confidentiality and privacy throughout the research process.
Participants and procedures
Participants were enrolled in a 2-arm randomized controlled trial for anxiety and/or depression following moderate-severe TBI. Inclusion criteria are published elsewhere (Hart et al., Reference Hart, Rabinowitz, Vaccaro, Chervoneva and Wilson2019). In brief, participants were ≥6 months post prospectively documented moderate-severe TBI and had at least mild depression or anxiety but no suicidal ideation. One of the treatment arms involved EMA for the purpose of facilitating self-monitoring and exploration of the connections between emotional states and activities/other contextual factors. The data for the current project were supplied by 39 of 40 participants randomized to that condition (one was excluded due to a tremor that precluded the use of a smartphone). For EMA training, participants received a live demonstration followed by hands-on guided practice. EMA response rate was monitored virtually, and if a participant failed to respond for 2 consecutive days, a member of the research team made contact to troubleshoot. EMA findings were displayed graphically to participants to provide feedback during treatment.
Measures
Prior to randomization, participants underwent baseline neuropsychological testing that included the Rey Auditory Verbal Learning Test (RAVLT; Schmidt, Reference Schmidt1996) with the sum of five trials used as the measure of episodic memory function. As a measure of initial TBI severity, post-traumatic amnesia (PTA) duration was estimated using a structured interview that correlates well with prospective assessment (Hart et al., Reference Hart, Dijkers, Whyte, Braden, Trott and Fraser2010). EMA data were collected via the LifeData System, a flexible mobile platform that allows researchers to custom-design protocols and deliver them using a smartphone app called RealLife Exp. Participants were notified five times per day, pseudorandomized within a 14 h window corresponding with their typical waking hours, to answer a set of multiple-choice questions about their activity(ies) “over the last hour or so,” the social and physical context of the activity and associated levels of enjoyment and accomplishment, and the 20-item Positive and Negative Affect Schedule (PANAS; Watson et al., Reference Watson, Clark and Tellegen1988). For each prompt, participants received up to five reminders spaced 5 mins apart; late responses were accepted until 90 mins prior to the next scheduled prompt. The required amount of time to complete the EMA session was approximately 2–3 mins. A complete description of the EMA protocol has been published previously (Rabinowitz et al., Reference Rabinowitz, Hart and Wilson2021).
Data analysis
The average weekly EMA response rate was calculated as the percentage of prompts to which the participant responded each week (for the purpose of this analysis, a week was operationalized as the amount of time between therapy sessions, which were targeted to occur 7 days apart; however, the actual time between sessions varied according to the patient/therapist schedules). We used linear mixed-effect modeling to predict the average weekly EMA response rate. Age, years of education, PTA duration, and memory performance on the RAVLT were included as covariates.
Results
The 39 participants were 46 years old on average (range = 23–90; SD = 16) with a mean of 13.5 years of education (SD = 2). Participants were an average of 7 years post-injury (range 0.7–21.2, SD = 6.4), with a mean of 29.3 days of post-traumatic amnesia (SD = 26.1). Thus, the average participant was living with very severe, chronic TBI. The sample was 30% female; 35% self-identified as Black, 60% as white, and 5% as Hispanic. Mechanisms of injury included falls (23%), motor vehicle incidents (35%), and assaults (20%). On the RAVLT, participants had a mean score of 45.37, SD = 11.54.
The average response rate over the 7-week EMA period for the full sample was 65% but ranged widely, from 5% to 100%. Four participants stopped responding to EMA prompts during the 7-week period, coinciding with their discontinuation of treatment participation. A spaghetti plot of individual response trajectories is depicted in F igure 1. Three linear mixed models were fitted to the data predicting the average weekly EMA response rate. Random intercepts and slopes account for between-person variability in response rates and the responsiveness to predictor variables (in this case, time indicated by week number) across individuals. First, a random intercept model was employed, which revealed significant effects for week number (β = −0.0084, p = 0.00803), with an overall model fit indicated by Akaike information criterion (AIC) = −264.6 and Bayesian information criterion (BIC) = −251.18. Subsequently, a model with both random intercept and random slope was tested, indicating significant effects for both intercept (β = 0.707, p < 0.001) and week number (β = −0.0129, p = 0.0527), with AIC = −363.27 and BIC = −343.13.
Spaghetti plot of random slopes.
Random intercepts and slopes are depicted in F igure 2. Finally, a full model incorporating additional covariates (week, education, memory, age, and PTA duration) demonstrated significant effects for week number (β = −0.0142, p = 0.0354), education (β = 0.0549, p = 0.0108), and a marginally significant effect for memory (β = 0.008, p = 0.0511) with an overall model fit reflected by AIC = −372.15 and BIC = −338.58. Comparisons between models indicated a significant improvement in fit when transitioning from the random intercept to random intercept and slope models (χ2 = 102.66, p < 0.001), as well as from the random intercept and slope to the full model (χ2 = 16.88, p = 0.00204). These findings suggest that including random slopes and additional covariates (week and education) significantly enhanced the prediction of changes in the EMA response rate over time.
Random effects.
Discussion
The present study provides a granular examination of the response rate to smartphone-delivered EMA in a sample of individuals with chronic moderate-to-severe TBI. We found that EMA adherence in this population averaged 65%. Recent meta-analyses suggest that response rates are lower in clinical samples as compared to healthy control cohorts and lower for studies that do not provide financial incentives (Wen et al., Reference Wen, Schneider, Stone and Spruijt-Metz2017; Williams et al., Reference Williams, Lewthwaite, Fraysse, Gajewska, Ignatavicius and Ferrar2021). In the present study, no financial incentives were provided for the completion of the EMA protocol, although participants were encouraged by their therapists to respond, and barriers to responding were discussed in treatment sessions. Promoting adherence was not a specific target of the protocol. In addition, the data from four participants who withdrew from treatment were retained in the current investigation. There is currently no universally agreed-upon minimum acceptable EMA response rate; depending on the scientific question and specific parameters of the study, an average of 65% may be suitable. One interpretation of the current findings, however, is that individual variation may be a more important focus compared to the average response across a clinical sample.
The relationship between adherence and protocol duration/intensity is not entirely straightforward. A recent meta-analysis by Wrzus and colleagues found that studies with more assessments per day tend to include fewer assessment days, but the total number of assessments was not related to adherence or dropout rates (Wzrus et al., Reference Wrzus and Neubauer2023). While 4–5 assessments may seem burdensome to some investigators, studies with a higher sampling frequency (6+ times per day) are associated with higher adherence rates in clinical samples (Wen et al., Reference Wen, Schneider, Stone and Spruijt-Metz2017). The overall response rate in the present study was similar to that reported in clinical studies sampling participants 4–5 x/day (Wrzus & Neubauer, Reference Wrzus and Neubauer2023).
Overall, there was a small but statistically significant linear decay in the EMA response rate over 7 weeks of participation. However, individual trajectories were variable, as evidenced by the significant effect of random slope in linear mixed-effects models. Individual variation in responding is associated, in part, with episodic memory function and educational attainment – greater educational attainment and more intact episodic memory had a positive effect on response rate (the effect was significant for educational attainment and marginally significant for episodic memory). There were no significant effects of age or PTA duration.
These findings could offer important insights into the feasibility of EMA protocols among individuals with TBI or other conditions that affect cognition, thus informing the design of future studies. On average, the response rate decreased over time by 0.84% points, suggesting that as the study progressed from 1 week to the next, participants, overall, were less likely to respond to the prompts. It is not surprising that the response rate somewhat decreased over time, as participants likely grew inured to the notifications as the study progressed. However, it was encouraging that this decline was relatively small. It is also encouraging that age was not a significant factor, despite the wide age range of the sample. This finding is consistent with a recent meta-analysis reporting high adherence to EMA protocols in samples of elderly individuals (Yao et al., Reference Yao, Yang, Wang, Pan and Xu2023), and other recent work demonstrating that most older adults find smartphone technology acceptable and incorporate smartphone use into their daily life (Wilson et al., Reference Wilson, Byrne, Rodgers and Maden2022). Regarding education and episodic memory performance, there could be several reasons for the association of these variables with adherence rates. Perhaps educational attainment and less complete recovery of memory are associated with a third variable such as cognitive reserve, which has been proposed as a factor in TBI recovery (Schneider et al., Reference Schneider, Sur, Raymont, Duckworth, Kowalski, Efron and Stevens2014). Further research could investigate other possibilities, such as a more direct link between impaired memory and a tendency to forget to respond later if one is distracted when the prompt arrives.
These results underscore that individual characteristics may interfere with responding for some individuals with moderate-to-severe TBI. Research participants with cognitive difficulties may require intensive up-front training, as well as ongoing monitoring and support to maximize adherence (Rabinowitz & Juengst, Reference Rabinowitz and Juengst2022). Developing memory strategies such as checking for missed notifications at or after meals, or enlisting the support of a family member, may also be helpful.
There are important limitations of the present study that bear noting. As mentioned previously, since EMA was not a primary outcome in this study, financial incentives for EMA completion – a standard practice for promoting adherence – were not provided. Conversely, participants’ EMA data were used to provide personalized feedback in therapy sessions, which may have been a motivator for completing assessments.
In summary, this study supports the feasibility of relatively lengthy, high-intensity EMA in chronic moderate-to-severe TBI but also highlights the need for attention to individual factors that may affect response rate. Comprehensive training and ongoing support mechanisms may be important for maximizing adherence, especially among those with memory impairments.
Acknowledgments
None.
Funding statement
This research was supported by a grant from the National Institute of Disability Independent Living and Rehabilitation Research #90DPTB0004 (PI: Rabinowitz).
Competing interests
The authors declare no competing interests.
