Cognitive training has long been investigated as a non-pharmacological alternative to the prevention or delay of cognitive impairment in older adults and yet the findings regarding its efficacy have been mixed (Rebok et al., Reference Rebok, Carlson and Langaum2007). One possible explanation for the heterogeneity of treatment response may be the nature of the populations studied. With diverse cognitive processes targeted for improvement, there are also diverse outcome measures. There is, of course, heterogeneity in the range and level of retention of cognitive abilities of older adults and this too impacts their ability to benefit from training. For example, patients with lower pre-training episodic memory scores appear not to benefit as greatly from training programs that focus on this skill (McKitrick et al., Reference McKitrick, Friedman, Brooks, Pearman, Kraemer and Yesavage1999).
The focus of this study was on identifying characteristics of older adults who benefit from mnemonic training. In mnemonic training, people are taught strategies to improve the learning and retention of new information (Brooks et al., Reference Brooks, Friedman and Yesavage1999). Our previous work has focused on providing mnemonics for practical problems such as recall of lists and name and faces (Yesavage et al., Reference Yesavage2008). Mnemonics employed for these problems include the method-of-loci and the name–face mnemonic. Briefly, the method-of-loci involves associating a list of items to a fixed list of places (or “loci”). The name–face mnemonic is a three-part technique for learning faces and names in which a person forms mental associations between facial features and a name. Thus, both techniques involve forming associations with novel, to-be-remembered information.
In previous work we assessed the characteristics of older adults who demonstrated short-term benefits (i.e., assessed immediately after participation in the training program). Persons who demonstrated a short-term response were younger, and had higher pre-training scores on measures of mental rotation, processing speed, attention, and verbal intelligence (McKitrick et al., Reference McKitrick, Friedman, Brooks, Pearman, Kraemer and Yesavage1999). Ultimately an intervention is not meaningful unless the benefits are sustained. Two approaches which utilize different time frames have been used for examining long-term treatment response. The first examines change from pre-training to a final follow-up point to establish what can be conceptualized as long-term effects of treatment (Anschutz et al., Reference Anschutz, Camp, Markley and Kramer1987; Ball et al., Reference Ball2002). The second approach examines change from end-treatment (i.e., immediately following completion of cognitive training) to a final follow-up visit and is a measure of maintenance of treatment response (Anschutz et al., Reference Anschutz, Camp, Markley and Kramer1987; Scogin and Bienas, Reference Scogin and Bienias1988). With long-term treatment response, the outcome of interest is one of significant improvement from baseline to follow-up, while with maintenance of treatment response the outcome of interest is retention of skill from end-treatment to follow-up. Improved understanding of factors that determine long-term treatment response and those that determine retention of initial treatment gain would allow providers to better match older adults with appropriate treatments based on their abilities resulting in a more cost-efficient treatment.
In order to identify the characteristics of treatment responders we applied a signal detection method (Kraemer, Reference Kraemer1992) to one-year follow-up data from a cognitive training program (Yesavage et al., Reference Yesavage2008). We performed separate analyses to reflect the long-term effects of treatment (i.e., difference between initial scores and scores at 52 weeks) and a separate analysis to represent the maintenance of treatment (i.e., difference between the final scores after training and the scores at 52 weeks) to best identify characteristics of long-term treatment responders and those who are able to maintain initial treatment response.
Methods
The current study utilized data collected in a prior Randomized Controlled Trial (RCT) (Yesavage et al., Reference Yesavage2008). Briefly, that study examined the effectiveness of donepezil as an augmentation to cognitive training. Participants were randomly assigned to a donepezil + cognitive training group or a placebo + cognitive training group. Pharmacological treatment in that study lasted 12 months. After the first 12 weeks of pharmacological treatment, all participants underwent two weeks of cognitive training. The results indicated that cognitive training was equally effective in both donepezil and placebo groups, thus donepezil did not “add” any benefit to cognitive training. Given that there was no additional effect with donepezil and that both treatment groups received the same cognitive training, we decided to collapse the two treatment groups into one sample for the current study.
Participants
Of the original 168 community-dwelling study participants, only those with complete data at baseline and one-year follow-up were included in the current analyses. This resulted in 120 participants (age 55 to 83 years, mean (M) = 64.7, standard deviation (SD) = 7.5) with equal numbers of men and women, and an average of 16.5 years of education (SD = 2.33). The current sample had a range of cognitive function from normal to Mild Cognitive Impairment (MCI) (MMSE M = 28.7, range = 25–30), with 30.8% (n = 37) of the sample classified as MCI. Following the criteria outlined by Petersen et al. (Reference Petersen2005), we considered scores below an education-adjusted cutoff on the Logical Memory II subscale (Delayed Paragraph Recall) of the revised Wechsler Memory Scale (Wechsler, Reference Wechsler1987) to be indicative of MCI (Yesavage et al., Reference Yesavage2008). No participants were taking anti-cholinergic medications at the time of randomization.
Procedures and measures
Participants in the original study (Yesavage et al., Reference Yesavage2008) received either donepezil or placebo for 52 weeks. After the initial 12 weeks of pharmacological treatment, participants began a two-week cognitive training program for two hours daily, Monday through Friday, in a classroom setting. Training included both non-mnemonic (e.g., visualization techniques) as well as mnemonic training (e.g., method-of-loci and name–face recall). In the method-of-loci, target words are associated with locations in the participant's home. Participants are then encouraged to visualize the target words at each location and to use verbal elaboration to bring the item to mind. For the name–face recall technique, names are associated with concepts and images that link the name to a feature on a picture of the person's face. Thus, both techniques use visual associations for remembering information.
Primary outcome measures were recall of name–face pairs and delayed recall of a list of words. Multiple versions of each task existed, thus both tasks were counterbalanced across the different testing time points. In the word-list recall task, participants were asked to remember a list of 16 words and to recall the list after a 5-min delay and then again after a 30-min delay. In the name–face recall task, 12 face-slides were presented consecutively with the associated name read aloud and simultaneously presented visually underneath the image of the face. For testing, the slides were presented again, although ordered differently and without the associated names. These primary outcome measures were chosen because of the well-established findings that training effects are usually specific to the training intervention (Schaie et al., Reference Schaie, Willis, Hertzog and Schulenberg1987).
Selection of predictors
Selection of predictors for Receiver operating characteristics (ROC) analyses was based upon our prior research and that of our colleagues. Drawing from the McKitrick et al. (Reference McKitrick, Friedman, Brooks, Pearman, Kraemer and Yesavage1999) study as well as that of Yesavage et al. (Reference Yesavage2008), the following measures were selected as possible predictors of long-term treatment response and maintenance: Logical Memory I and II (LM I; LM II), a measure of episodic memory; Paired Associates I and II (PA I; PA II), a measure of associative memory, and Digit Span (DS), a measure of working memory; which are all subtests from the Wechsler Memory Scales – Revised (WMS-R; Wechsler, Reference Wechsler1987). We also included Revised Visual Retention Test (BVRT; Benton, Reference Benton1955), a measure of visual memory; Trail Making Test A and B (TMTA and TMTB; Reitan, Reference Reitan1958), measures of executive function; Symbol-Digit Modalities Test (SDMT; Smith, Reference Smith1982), a measure of attention; and a Clock-Drawing Task, a measure of visuospatial ability. In those assessments in which two or more subscales exist (i.e., Trails A and B), subscales were entered as separate predictors in the analyses. Baseline scores on these tests, as well as age, gender, medication status (donepezil or placebo in the original Yesavage et al., Reference Yesavage2008 study), ethnicity, education, and MCI status were used as predictors of long-term response and maintenance of response in the present ROC.
Statistical analysis
In signal detection analysis, the “signal” is a binary outcome and the “detection” is a set of predictor variables. We began by setting criteria for significant benefit based on our primary outcome measures, improvement (or gain) in delayed recall on word-list tasks and in a name–face recall task. With signal detection analysis we derived an ROC curve and used this function to predict the participants who meet the criterion for treatment benefit. An advantage of the ROC over, for example, a multiple regression is that it assesses prediction of a clinically meaningful change rather than assessing improvements as a continuous measure. Furthermore, signal detection analysis is not vulnerable to multi-collinearity to which multiple regression is subjected, thus it allows for possible interactions among all the predictors.
For this study two separate ROC curves were derived that evaluated two measures: word-list and name–face associations. Outcomes were presented in terms of the improvements from baseline to the end of 52 weeks (long-term effects) and the durability of training effects from the point at which the training ended to the long-term maintenance point at 52 weeks (maintenance). This resulted in four analyses of recall: (1) long-term effects of word-list, (2) long-term effects of name–face tasks, (3) maintenance of word-list, and (4) maintenance of name–face tasks. The predictors included demographic and cognitive variables assessed at baseline.
Using a success criterion or “gold standard” established a priori, the ROC searches possible predictors of the criterion and the predictor's optimal cut-off score. The success criterion can be thought of as what would constitute clinically meaningful change. In the current study, meaningful change served as the “signal” or binary outcome and was defined as the mean gain score at 52 weeks, determined separately, for each of the recall tasks. For maintenance of treatment response, the mean gain score was calculated from end-treatment (14 weeks) to 52 weeks. For the analysis of long-term effects of cognitive training, the mean gain score was calculated from baseline to 52 weeks (see Figure 1).
Study timeline.
In an ROC, after a success criterion and its cut-off score are identified, the predicting variable is used to divide the sample into two sub-samples; this process is repeated in the two sub-samples until the stopping rule of p > 0.05 is reached. Thus, signal detection is an iterative process that continually re-runs on each sub-sample established until non-significant results are found. The ROC-derived decision trees for the cognitive training are displayed in Figures 2 through 5.
Long-term effect for delayed recall of word-list.
Long-term effect for recall of name–face task.
Maintenance of delayed recall word-list.
Maintenance of name–face task.
Results
Outcome analyses: long term effects of cognitive training (Baseline to 52-week follow-up)
Delayed Word-List Recall
For the word-list recall task, the mean gain score from baseline to 52-week follow-up was 4.4 words; thus, the “success criterion” was set at ≥4 (i.e., participants who recalled at least 4 more words at 52 weeks than they did baseline were “successful”), and 55.8% of the participants (n = 67) succeeded. The significant predictors of treatment response were baseline performance on the delayed word-list task, and baseline scores on measures of attention (SDMT), episodic memory (LM I), and delayed associative memory (PA II). No other predictors reached the 0.05 significance level.
As seen in Figure 2, participants who remembered fewer than 11 words at baseline on the word-list task had a 66.7% chance of reaching the success criterion (≥4 words at the 52-week follow-up visit) whereas participants remembering 11 or more words at baseline, had a 12.5% chance. For participants scoring <11 at baseline, the second predictor of success at 52 weeks was attention. Specifically, those who scored ≥48 at baseline on the SDMT had an 80.3% chance of reaching the success criterion at follow-up; those scoring <48 had a 42.9% chance. Continuing with those who scored ≥48 on the SDMT, the next significant predictor was delayed performance on a measure of associative memory. Those who scored ≥6 on the PA II had an 88.2% chance of succeeding whereas those who scored <6 had a 40% chance of success. Alternatively, for those participants who scored <48 on SDMT, the next significant predictor of treatment response was episodic memory (LM I). Participants who scored ≥22 on the LM I had a 66.7% chance of success at follow-up compared with the 18.8% chance of success in those who scored <22 on LM I. In sum, those older adults with the highest rates of long-term treatment response for delayed word-recall initially remembered fewer words at study entry but had better baseline attention and associative memory performance.
Name–Face Recall
The mean gain score from baseline to the 52-week follow-up visit was 1.4 names, thus ≥1 was the success criterion and 58.3% of the sample (n = 70) succeeded. The significant predictors of treatment response were age and delayed episodic memory. No other variables reached the 0.05 significance level.
As seen in Figure 3, those who were less than 65 years of age had a 70.7% chance of success compared with 37.8% chance held by those who were 65 years of age or older at long-term follow-up. For those participants who were at least 65 years of age or older, delayed recall on a measure of episodic memory was the next significant predictor of treatment response. Older participants who scored 20 or greater on LM II at baseline had a 63.2% chance of successful long-term treatment response compared with a 19.2% chance for those scoring less than 20. Thus, for those participants who were 65 years of age or older, better baseline delayed episodic memory predicted long-term treatment response for recall of a name–face pair.
Outcome analyses: maintenance of treatment response (end of treatment to 52-week follow-up visit)
Delayed Word-List Recall
The mean gain score from end-treatment to the 52-week follow-up visit was 1.4 words. Thus, the “gold standard” for successful maintenance of treatment response was set at 1 (i.e., participants remembered one more word on the delayed word-list recall task at 52-weeks than they did at end-treatment). The primary significant predictor was participants’ initial training gains (i.e., the change from baseline to end-treatment (14 weeks) or end-treatment gain) on the delayed recall of the word-list task. No other variables reached the 0.05 significance level.
Figure 4 highlights the fact that at the 52-week follow-up visit, those who had end-treatment gain scores of fewer than 5 words had a 73.3% chance of successful maintenance whereas those with end-treatment gains of 5 words or more had a 37.8% chance of successful maintenance. Following the former group (<5 words at end-treatment), the next significant predictor was, again, end-treatment gains. In this next round of predictors, those with end-treatment gain scores of less than 1 (i.e., those who had little to no change from baseline to end-treatment on delayed recall of the word-list) had a success rate of almost 88%. Those with end-treatment gain scores of at least one word had a 58.3% chance of success. Thus, individuals who had the highest rate of successful maintenance of treatment response (87.2%) were those who initially did not respond to treatment (end-treatment gain scores of <1).
Name–Face Recall
The mean gain score from end-treatment to the 52-week follow-up visit was 0.58, thus the success criterion was set at one name–face pair for the name–face task. The significant predictors of maintenance of treatment response were end-treatment gains (baseline to end-treatment) on the name–face task and associative memory (PA II). No other variables reached the 0.05 significance level.
As seen in Figure 5, those participants with end-treatment gains of less than 1 had a 67.9% chance of successful maintenance of treatment response compared with those with end-treatment gains of 1 or more (34.4% chance of success). For the former group, the next significant predictor was associative memory. Those who scored 7 or more on PA II had an 84.8% chance of success compared with those scoring less than 7 who had a 43.5% chance of success. In sum, initial treatment non-responders with higher baseline associative memory (PA II) scores had the highest rates of successful maintenance of treatment response (~85%).
Discussion
This study demonstrated that different neurocognitive variables predicted long-term treatment response and maintenance of initial treatment response in older adults. Thus, it would seem that the goal of the intervention (i.e., long-term effects vs. maintenance) might determine which older adults are most likely to derive benefit or meaningful change.
Delayed recall at baseline (word-list task), attention (SDMT), associative memory (PA II), and episodic memory (LM I) were significant predictors of the long-term effects of cognitive training on delayed recall of a word-list. The group that had the highest rates of success (i.e., long-term gain of ≥4 words) remembered fewer words at baseline and had better working and associative memory. The group that had the lowest rate of success remembered more words at baseline. Initially this may seem counterintuitive; however, we believe that this is an evidence of a possible ceiling effect with this measure. The measure consists of 16 words, so those who remembered 11 or more words (n = 24) at baseline had less room for improvement than those who remembered fewer words. It is also possible that this group had less motivation for improvement as they were able to remember over half of the words on the list prior to participating in any cognitive training.
It is interesting that attention was one of the predictors for long-term effects of cognitive training for the word-list. Attention was assessed using the SDMT, which is similar to the Digit Symbol task. This measure has also been related to memory abilities perhaps because learning the associations between the symbols and digits obviates the need to refer to the key and enables participants to complete more items in the allotted time. Thus, participants performing better on the SDMT may have been faster at forming associations such as those used in the method-of-loci, the mnemonic taught in the study for the word-list task. This single predictor divided the group with 80% of those with higher baseline attention having gain scores (baseline to 52-week follow-up visit) of 4 or more words. Those who had lower baseline attention had less than a 43% chance of success.
Associative memory (PA II) was also predictive of successful treatment response for long-term effects of cognitive training on the word-list task. As previously stated, the method-of-loci utilizes associative strategies to link images and to-be-remembered words. The PA II task is a measure of cued recall. The examiner offers the first word of a previously presented pair of unrelated words, and the participant is to provide the second or “associated” word (Mitrushina et al., Reference Mitrushina, Boone, Razani and D'Elia2005). It follows that those with better baseline associative memory may have been able to better utilize the method-of-loci on the word-list task. Episodic memory (LM I) also predicted response such that those with better episodic memory had much higher rates of success (66.7% vs. 18.8%). Episodic memory involves recollection of specific events, situations, and experiences. In the method-of-loci, participants were instructed to remember target words by associating the words with locations in the participant's home. It may be that those with better episodic memory were better able to recall the locations in their homes and then apply the method-of-loci to lists of to-be-remembered information. In sum, persons with maximum room for improvement in terms of their ability to recall a list of to-be-remembered information and also have better attention and associative memory skills are most likely to benefit from the cognitive training based upon the method-of-loci.
Whereas the long-term effects of cognitive training on word-list recall had multiple predictors of treatment response, prediction of response on name–face recall required only two predictors: age and delayed verbal memory (LM II). This ability is highly relevant to the name–face mnemonic, which requires both the ability to form mental associations and to recall verbal information. Our results also suggest that age was the best predictor of response with those 65 years of age or younger having the best response (70% vs. 38%). Delayed verbal memory was also a good predictor of response (19% vs. 63%). These results indicate that if one were older, a relatively poor performance on LM II reduced your successful long-term treatment response to 19%. It would seem that cognitive training featuring the name–face technique to improve recall of name–face pairs is most appropriate for persons 65 years of age or younger or for those 65 years of age and over who also have good verbal memory skills.
In the maintenance of treatment effect analyses, the primary predictor of success was initial treatment response (end-treatment gain). Specifically, for the delayed word-list recall, the initial predictor was an end-treatment gain score of less than 5 words. People who remembered fewer words at end-treatment were more successful at maintaining those gains than persons who had greater end-treatment gains (≥5 words). Surprisingly, those who did not respond to the initial cognitive training were the group that was most likely to maintain their performance (87%). A similar pattern emerged with the recall of name–face task in the analyses of maintenance of treatment response. Predictors included end-treatment gain scores of less than 1 (no initial treatment response) and better associative memory (PA II). Thus, those who did not respond to the initial cognitive training but had better associative memory performance were most likely to have successful maintenance. This implies that those who initially benefited from the cognitive training did not maintain those benefits at 52 weeks. Prior research has demonstrated that the staying power of cognitive training programs is significantly improved when participants have periodic “booster sessions” during which the use of the mnemonics is reinforced (Ball et al., Reference Ball2002).
The flip side to the identification of treatment responders is the identification of those who do not respond to treatment, the treatment non-responders. Considering the long-term effects of cognitive training on word-list recall, those who had the lowest rate of success were those who had the highest baseline ability. Given the strong possibility of a ceiling effect with this group, we examined the group with the second lowest rate of success (18.8%). These participants had remembered fewer words at baseline and had worse attention and episodic memory. To increase the benefit that persons with this neurocognitive profile can obtain from cognitive training using the method-of-loci, it will be necessary to include pre-training or other augmentation strategies that target attention and episodic memory. It may be the case that if this group was able to improve their attention and episodic memory, they may be better able to use the method-of-loci technique for delayed recall tasks. In the analyses of long-term effects of cognitive training on the name–face task, treatment non-responders were older and had poorer delayed verbal recall. Within the older participants, those with better verbal recall had markedly higher rates of success (63.2% vs. 19.2%). So within the older group (aged >65), those with lower baseline verbal recall had the lowest rates of success at 52 weeks. To increase the benefits that this group derives from cognitive training, it will be necessary to provide pre-training targeting verbal recall prior to teaching the name–face technique.
For the maintenance of treatment response, the non-responders for both of the recall tasks were those who had originally responded to the cognitive training. These initial responders were unable to maintain their gains, which resulted in them being “re-classified” as non-responders. While this may be further evidence for the necessity of booster sessions, it is also possible that the inability to maintain the initial treatment response represents a regression to the mean.
It appears that older adults may draw upon a variety of cognitive abilities when undergoing cognitive training. Persons experiencing MCI might be able to benefit from cognitive training that uses mnemonics that draw upon a diverse of neurocognitive abilities. However, because of the relatively smaller number of sample participants with MCI (n = 37), we were unable to complete a separate ROC for this group to better delineate these specific cognitive abilities. There is a growing literature examining cognitive training for persons with MCI. A review of these studies found improved performance on objective measures of memory, with some reporting moderate to large effect sizes (Belleville, Reference Belleville2008). The results of the current study support the utility of cognitive training in older adults with and without cognitive impairment, shed light on the heterogeneity of predictors of long-term treatment gain, and emphasize the value of tailoring training to the specific requirements of the individual.
Several limitations of the current study merit further discussion. As previously stated, we believe that several measures may have had limited predictive value because of ceiling effects. Our sample was both at the younger end of old age (M age = 64.7 years) and well educated (M years of education = 16.5), thus it is possible that a different pattern of results would emerge in an older or less educated sample. We also believe that a larger sample size may have resulted in identification of further predictors. The primary outcome measures in the study were selected to best capture training effects. As these outcomes were based on performance on specific memory tasks, it is unknown to what degree the training generalized to “everyday” memory ability. Future studies may wish to include a clinical measure of the strategies that participants employ when approaching the specific memory tasks required for everyday life.
The current work provides valuable information regarding the different abilities that both healthy and cognitively impaired older adults draw upon when approaching tasks of delayed recall. Providers should consider what older adults’ strengths are (i.e., verbal memory, associative memory, working memory) as well as their ages when choosing cognitive training modalities. It also appears to be important for the cognitive training program to include some sort of booster session, such as regularly scheduled single-day or web-based refresher courses, to maintain the initial treatment response.
Conflict of interest
None.
Description of author's roles
J. Fairchild was responsible for the statistical design of the study, for carrying out the statistical analysis, and wrote the paper. L. Friedman collected the original data and assisted with writing the paper. A. Rosen assisted with writing the paper. J. Yesavage designed the study, collected the original data, and assisted with writing the paper.
Acknowledgments
This research was supported by the National Institute of Mental Health (Grant R01 MH35182); the Sierra-Pacific Mental Illness Research, Education, and Clinical Center; and the Medical Research Service of the Department of Veterans Affairs.
