Statement of Research Significance
Research Question(s) or Topic(s): The current study investigated whether language characteristics of spoken responses while recalling a short story differ between individuals with normal cognition and those with cognitive impairment. Main Findings: Subtle differences in how someone recalls a short story, such as lower use of prepositions and poorer recall of specific words from the story, can help identify individuals with cognitive impairment. These features also predict future cognitive decline over 7 years. Study Contributions: The identification and scoring of linguistic features from spoken responses to cognitive tests captures novel information about language and memory function. This approach enables the extraction of information about multiple cognitive processes from a single test, whereas traditional summary scores conflate this information. Moreover, these features provide added value for predicting cognition after 7 years and may capture very early cognitive changes, allowing for earlier interventions to prevent cognitive decline.
Introduction
Deterioration of language performance is frequently observed in patients with cognitive impairment and dementia, and can include semantic impairment, decreased fluency, and low content density (Fraser et al., Reference Fraser, Meltzer and Rudzicz2016; Kempler & Goral, Reference Kempler and Goral2008; Taler & Phillips, Reference Taler and Phillips2008). However, the traditional scoring of many neuropsychological tests, particularly those not classically considered as language tests, may not capture these changes. Moreover, evidence suggests that disturbances in language can manifest even at the stage of mild cognitive impairment (MCI), preceding more overt signs of dementia (Boschi et al., Reference Boschi, Catricalà, Consonni, Chesi, Moro and Cappa2017; Verma & Howard, Reference Verma and Howard2012). By leveraging automated linguistic analysis, these potential early markers can be explored with great efficiency, thereby offering a promising approach for early diagnosis and intervention in dementia.
Recent advances in computational analysis have opened new avenues for detecting speech and language features from voice recordings of neuropsychological assessments. Unlike natural speech, cognitively demanding tasks that require engagement of multiple cognitive functions, such as episodic memory, attention, and executive function, may be more susceptible to impairment in response to early neuropathological changes than the processes typically involved during everyday conversations. For example, a recent study using acoustic features from a delayed paragraph recall task with a high memory load showed better classification accuracy for cognitive status than acoustic features collected from tasks with low to moderate memory loads, such as natural speech and repetition tasks (Bae et al., Reference Bae, Seo, Ko, Ham, Kim and Lee2023). Similarly, the Framingham Heart Study was able to accurately classify participants by cognitive status using natural language processing methods and extraction of acoustic features from testing sessions (Amini et al., Reference Amini, Hao, Zhang, Song, Gupta, Karjadi, Kolachalama, Au and Paschalidis2023; Tavabi et al., Reference Tavabi, Stück, Signorini, Karjadi, Al Hanai, Sandoval, Lemke, Glass, Hardy, Lavallee, Wasserman, Ang, Nowak, Kainkaryam, Foschini and Au2022). Other research has focused on specific cognitive tasks designed to elicit spontaneous speech, such as picture description tasks. Several studies have shown that linguistic markers, such as semantic content, syntactic complexity, speech efficiency, and frequency of noun and verb use on the Boston Diagnostic Aphasia Examination Cookie Theft (CT) task differentiate between individuals with normal cognition (NC) and those with dementia due to Alzheimer’s disease (AD) (Ahmed et al., Reference Ahmed, de Jager, Haigh and Garrard2013; Fraser et al., Reference Fraser, Meltzer and Rudzicz2016; Kavé & Goral, Reference Kavé and Goral2016; Lian & Wang, Reference Lian and Wang2025). Moreover, a longitudinal study found that the presence of altered grammatical structures, repetitiveness, and misspellings among responses from cognitively healthy adults was associated with incident AD, suggesting that language features may be preclinical markers of decline (Eyigoz et al., Reference Eyigoz, Mathur, Santamaria, Cecchi and Naylor2020).
The logical memory (LM) subtest of the Wechsler Memory Scale is a widely used paragraph recall test designed to measure episodic memory, which is commonly affected in normal aging and more substantially in cognitive impairment due to AD and related dementias (Tromp et al., Reference Tromp, Dufour, Lithfous, Pebayle and Després2015). LM performance has one of the strongest predictive values among neuropsychological assessments for future conversion to dementia (Silva et al., Reference Silva, Guerreiro, Santana, Rodrigues, Cardoso, Maroco and de Mendonça2013). Although this test is primarily described as an episodic memory task, cognitive tests are not domain-specific, often drawing on multiple cognitive processes. The verbal response format of LM draws on language function, allowing for analysis of linguistic features of a participant’s response. Traditional scoring of LM tallies the total number of correctly recalled story units and does not account for related responses that do not meet the strict criteria of allowable responses or utterances that are unrelated to the story. These unscored responses, such as repeating words or phrases, using semantically similar words, or commenting on their performance (e.g., “I can’t remember”) are rich in cognitive-related information. Unlike the previous analyses of picture description tasks, which place greater demands on semantic knowledge (Mueller et al., Reference Mueller, Hermann, Mecollari and Turkstra2018), analyses of LM responses may reveal different linguistic markers, reflecting both episodic memory and language deficits.
In early AD and MCI, initial encoding (captured by the immediate recall task condition of LM) can remain relatively preserved. In contrast, early neuropathological changes in the hippocampus disrupt memory consolidation, leading to rapid information decay and profound decline in delayed recall task conditions (Cerami et al., Reference Cerami, Dubois, Boccardi, Monsch, Demonet and Cappa2017; Welsh et al., Reference Welsh, Butters, Hughes, Mohs and Heyman1992). Therefore, the immediate and delayed recall task conditions of LM may also reveal differences in linguistic characteristics related to learning and storage/retrieval processes, respectively. Additionally, specific subtypes of MCI have distinct neuropsychological profiles (i.e., amnestic versus dysexecutive profiles) (Bondi et al., Reference Bondi, Edmonds, Jak, Clark, Delano-Wood, McDonald, Nation, Libon, Au, Galasko and Salmon2014; Libon et al., Reference Libon, Bondi, Price, Lamar, Eppig, Wambach, Nieves, Delano-Wood, Giovannetti, Lippa, Kabasakalian, Cosentino, Swenson and Penney2011; Pa et al., Reference Pa, Boxer, Chao, Gazzaley, Freeman, Kramer, Miller, Weiner, Neuhaus and Johnson2009) and thus are likely to be associated with differences in LM recall performance and language use. Taken together, capturing full verbal responses to LM allows us to extract more comprehensive information from a single test, which may be more sensitive for detecting cognitive impairment, differentiating MCI subtypes, and predicting cognitive decline.
This study aimed to identify interpretable linguistic features embedded in paragraph recall responses that differentiate between cognitively healthy individuals and those with cognitive impairment and determine whether they predict cognitive decline over time. Building on prior research on picture description responses, we hypothesized that responses from individuals with cognitive impairment have fewer semantic content words and verbs compared to responses from cognitively healthy individuals and that the loss of content words is exacerbated in delayed recall in comparison to immediate recall. Longitudinally, we hypothesized that these baseline linguistic markers from paragraph recall predict incident cognitive decline. In exploratory analyses, we investigated linguistic profiles associated with MCI subtypes with the hypothesis that different linguistic features are associated with the amnestic versus dysexecutive MCI due to their distinct neuropsychological profiles.
Methods
Study participants and data collection
The Long Life Family Study (LLFS) is a longitudinal family-based cohort study in the United States and Denmark aimed at determining the genetic, behavioral, and environmental factors related to healthy aging and longevity among 4,953 individuals from 539 families selected for evidence of familial longevity (Wojczynski et al., Reference Wojczynski, Jiuan Lin, Sebastiani, Perls, Lee, Kulminski, Newman, Zmuda, Christensen and Province2022). To date, participants have completed up to three in-home visits (Visit 1: 2006–2009, Visit 2: 2014–2017, Visit 3: 2020–2024) which included a neuropsychological assessment comprised of the Mini Mental State Examination (MMSE), WMS-R Logical Memory, Number Span, semantic (animal) fluency, and the Digit Symbol Substitution Test across all visits and added the Hopkins Verbal Learning Test – Revised (HVLT-R), Clock Drawing Test, Trail Making Tests and letter fluency at Visit 2 and thereafter. The neuropsychological assessment was voice-recorded at Visit 2 for a subset of LLFS participants (N = 710) from the Boston field site. The recordings of the LM immediate recall and delayed recall responses were manually transcribed and cross-checked by a second transcriber to ensure uniform quality.
A study partner completed the Dementia Questionnaire (Kawas et al., Reference Kawas, Segal, Stewart, Corrada and Thal1994; Silverman et al., Reference Silverman, Breitner, Mohs and Davis1986) to identify cognitive problems in the participant’s daily life. Participants’ cognition was monitored longitudinally (annually for participants age 70 years and older and every three years for younger participants) using a modified version of the Telephone Interview for Cognitive Status (TICS-m), a brief 12-item test of global cognitive status with a maximum score of 51 points (Brandt et al., Reference Brandt, Spencer and Folstein1988; Welsh et al., Reference Welsh, Breitner and Magruder-Habib1993). Cases flagged for possible cognitive impairment based on test scores or informant reports of cognitive concerns, were reviewed in clinical consensus conferences with neurologists and neuropsychologists and assigned a Clinical Dementia Rating Scale (CDR) score (Hughes et al., Reference Hughes, Berg, Danziger, Coben and Martin1982).
For this analysis, participants were classified with NC if they were not flagged for consensus review or had a CDR score of 0 at Visit 2 during consensus review. Participants were classified with cognitive impairment (CI) if they had a CDR score of 0.5 or greater (i.e., MCI or dementia) at Visit 2. Participants with CDR = 0.5 and lower than expected performance in episodic memory (i.e., HVLT-R and/or LM) were classified as having an amnestic profile (aMCI). Participants with CDR = 0.5 and lower than expected performance in executive function (i.e., letter fluency, Trail Making Test B, and/or the Digit Symbol Substitution Test) were classified as having a dysexecutive profile (dysMCI). All participants provided written informed consent. Study procedures were completed in accordance with the Helskinki Declaration and approved by the Institutional Review Boards at Boston University and Washington University in St. Louis.
Statistical analyses
Linguistic feature extraction and feature selection
Linguistic features were extracted with the 2015 version of Linguistic Inquiry and Word Count (LIWC), a text analysis tool developed to quantitatively assess the psychological, emotional, and cognitive characteristics of written or spoken language (Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010). LIWC compares each word in a sample to internal dictionaries and calculates the percentage of words in 93 linguistic dimensions including emotional, cognitive, and structural components. The program was developed to measure aspects of language use that reflect thoughts and emotions, however, these linguistic features have also been used for detection and classification of dementia in conversational speech as well as in story recall and picture description tasks (Asgari et al., Reference Asgari, Kaye and Dodge2017; Boland et al., Reference Boland, Jensen, Davidson and Taler2024; Haulcy & Glass, Reference Haulcy and Glass2021; Weiner & Schultz, Reference Weiner and Schultz2018).
After generating the 93 LIWC variables, we removed 41 variables with either low frequency (e.g., words related to religion or death) or irrelevance to the LM task (e.g., punctuation) since this was a spoken rather than written response. An additional 6 variables in the immediate recall condition and 10 in delayed recall were excluded due to redundancy (Pearson correlation r > 0.6 or r < −0.6). Five categories of pronouns were combined into three variables: first person, second person, and third person pronouns, resulting in a final set of 44 immediate and 40 delayed recall variables. Since most variables were skewed, we standardized them with median absolute deviation (MAD), which also created beta estimate weights on the same scale. For variables with zero-inflation (22 in immediate; 20 in delayed recall) we used two types of binary transformation (less than median versus median or greater; 0 versus 1+ for variables with a median value of 0).
A previous study using 68 word categories from LIWC to predict cognitive status had decreased model performance compared to models using selected features, indicating that some word categories are not related to cognitive function (Asgari et al., Reference Asgari, Kaye and Dodge2017). Therefore, we performed additional feature selection to remove variables not associated with cognitive status. We used individual regression models for each linguistic feature adjusted by age, sex, and education to predict cognitive status (NC versus CI), evaluating statistical significance with Bonferroni-corrected p-values to maintain a family-wise error rate of α = .05. We estimated the regression coefficients using Generalized Estimating Equations (GEE) with exchangeable correlation to account for possible within-family correlation.
Linguistic polyfeature scores
Since the effect size of each linguistic feature associated with cognitive impairment is small, we developed a novel method of quantifying the features into a linguistic polyfeature score (PFS). This method was derived from the concept of polygenic risk scores which quantify an individual’s genetic susceptibility to a specific trait or disease by aggregating the effects of numerous genetic variants across the genome. Although each variant contributes only a small amount of risk, their combined influence can help estimate the likelihood of developing complex conditions such as heart disease, diabetes, or AD (Bae et al., Reference Bae, Song, Ali, Sasaki, Tesi, Lords, Leshchyk, Abe, Hirose, Arai, Barzilai, Weiss, Hulsman, van der Lee, van Schoor, Huisman, Pijnenburg, van der Flier, Reinders and Sebastiani2025; Wray et al., Reference Wray, Goddard and Visscher2008). For the linguistic PFS we combined the individual LIWC metrics from the immediate recall condition that were significantly associated with cognitive impairment into a single weighted sum score called an immediate recall PFS (PFS-IR). Each feature (median absolute deviation standardized or dichotomized) was weighted based on the beta estimate and adjusted for the standard error in the GEE model. Similarly, the individual LIWC features from delayed recall that were associated with cognitive impairment were combined into a delayed recall PFS (PFS-DR).
Cross-sectional analyses
To estimate the association between the linguistic PFS-IR and PFS-DR and cognitive impairment (i.e., NC versus CI), we used logistic regression models for each linguistic PFS adjusted by age, sex, and education to predict cognitive status and estimated the regression coefficients using GEE with exchangeable correlation to account for possible within-family correlation.
To assess the discriminative power of the linguistic PFSs independently and in comparison to traditional cognitive test scores, we built the following four logistic regression models using GEE with exchangeable correlation for within family relatedness and cross-validation to distinguish between CI and NC: (1) PFS-IR, (2) PFS-DR, (3) PFS-IR and PFS-DR, and (4) traditional LM delayed recall score. Each model was adjusted for age, sex, and education. Model performance was evaluated using a 10-fold cross-validation framework to ensure robust out-of-sample estimates. Since cognitive status was highly imbalanced (14% CI and 86% NC), stratified cross-validation was used to ensure that the prevalence of CI was preserved across folds and precision-recall curves, rather than receiver operating characteristic curves, of each classifier were plotted. For each classifier, we compared evaluation metrics including positive predictive value, negative predictive value, Matthews Correlation Coefficient (MCC), and F1. MCC serves as a holistic complement to traditional metrics for evaluating binary classification quality incorporating critical confusion matrix measures to offer a balanced assessment of model performance and is especially useful for imbalanced classes (Chicco & Jurman, Reference Chicco and Jurman2023; Richter-Laskowska et al., Reference Richter-Laskowska, Sobotnicka and Bednorz2025). The F1 score is the harmonic mean of precision and recall and represents the balance between these two metrics (Rainio et al., Reference Rainio, Teuho and Klén2024). To compare the precision-recall performance of the classifiers, we estimated the 95% confidence interval using bootstrapping with 2,000 resamples. We used R version 4.3.2 for all cross-sectional analyses and the ‘geepack’ package for regression models with GEE.
Longitudinal analysis
We assessed whether linguistic PFS scores predict change in TICS-m scores over follow-up. For model selection, we followed a two-stage approach. First, we used the HPGENSELECT procedure in SAS, to perform iterative backward selection with a significance level of 0.1 with the following covariates: PFS-IR, PFS-DR, age, sex, education, time (months following Visit 2), and interactions between each PFS and time. Based on the result from the iterative backward selection, we implemented a repeated measures model with GEE to identify the association between PFS and TICS-m over time. The final crude model incorporated backwards variable selection (post-hoc), ensuring that removal of non-significant variables did not have a confounding effect on either PFS (i.e., 10% change of effect rule).
In secondary analyses, we constructed two adjusted models. In the first adjusted model, we added APOE genotypes to control for the genetic risk conferred by APOE ϵ2 and ϵ4 alleles. APOE alleles were based on the single nucleotide polymorphisms rs7412 and rs429358, which were genotyped using real-time PCR and classified as: ϵ2 (rs7412 = T; rs429358 = T), ϵ3 (rs7412 = C; rs429358 = T), and ϵ4 (rs7412 = C; rs429358 = C). APOE genotype was categorized into reference, protective, and risk genotypes, respectively (0 = ϵ3/ϵ3, 1 = ϵ2/ϵ2 or ϵ2/ϵ3, 2 = ϵ3/ϵ4 or ϵ4/ϵ4). Participants with the ϵ2/ϵ4 genotype were excluded. In the second adjusted crude model, we added the traditional LM delayed score to determine if the PFS provides additional information beyond manual scoring. Finally, the analysis was repeated using a sample restricted to participants without MCI and dementia at Visit 2 to ensure that results were not driven by participants with prevalent cognitive impairment who are expected to show more decline after Visit 2 and to understand the predictive power of the linguistic PFS in a cognitively healthy sample. All models were repeated measures models with GEE. Longitudinal analyses were conducted in SAS version 9.4.
Exploratory analyses
In exploratory analyses, we used logistic regression models adjusted by age, sex, and education to determine whether each LIWC feature was associated with aMCI in reference to NC, evaluating statistical significance with Bonferroni-corrected p-values to maintain a family-wise error rate of α = .05. We estimated the regression coefficients using GEE to account for possible within-family correlation. Similar logistic regression models with GEE were used to determine the LIWC features associated with dysMCI in reference to NC. Participants with both amnestic and executive impairments were included in both analyses. We used R version 4.3.2 for secondary analyses and the ‘geepack’ package for regression models with GEE.
Results
Participant characteristics are summarized in Table 1. Of the 710 participants included in the analysis (mean age = 72.3 ± 11.0 years, 53.5% female), 598 participants had NC and 112 had CI. On average, those in CI group were older, were less likely to have a bachelor’s or higher degree and had lower MMSE and LM scores than the NC group.
Participant characteristics
Table 1. Long description
The table presents participant characteristics for 598 individuals with normal cognition to 112 with cognitive impairment and additional columns for subgroups of 24 participants with amnestic mild cognitive impairment and 46 participants with dysexecutive mild cognitive impairment. Key variables include age, sex, educational level, Mini-Mental State Examination scores, and logical memory scores. Participants with cognitive impairment are older, less likely to have a college degree, and have lower Mini-Mental State Examination and logical memory scores than those with normal cognition at a significance level of p<.001.
Note: The p values represent comparisons between the group with normal cognition and the group with cognitive impairment, NC = normal cognition, CI = cognitive impairment, MCI = mild cognitive impairment, MMSE = Mini-Mental State Examination, SD = standard deviation.
Our initial regression models identified eight features in immediate recall responses and seven features in delayed recall responses that were associated with cognitive impairment as shown in Table 2 (see Supplementary Tables 1 and 2 for the results of all LIWC variables). During immediate recall, the CI group had a higher proportion of negations, words related to cognitive processes (see Table 2 for examples), and words indicating a time orientation in the present than the NC group (all Bonferroni-corrected p < .05; see Table 2). In contrast, some features were less common among participants with CI such as prepositions and words related to semantic categories of biological processes, work, space, and leisure. During delayed recall, the CI group, again, had a higher proportion of negations and a lower proportion of words related to biological processes and work in their responses in comparison to the NC group. Unique to the delayed recall condition, the CI group used more first-person pronouns but fewer words in the past tense and words related to numbers and drives (i.e., motivations) in their responses compared to the NC group. Figure 1 presents a Venn diagram summarizing the unique and overlapping features from immediate and delayed recall responses.
Venn diagram of linguistic features that are associated with cognitive impairment. Note: ‘+’ indicates a positive association and ‘−’ indicates a negative association with the outcome of cognitive impairment.
Linguistic features associated with cognitive impairment
Table 2. Long description
The table presents results of the twelve Linguistic Inquiry and Word Count (LIWC) features from immediate recall and delayed recall tasks that are significantly associated with cognitive impairment. Three linguistic features were associated with cognitive impairment in both immediate and delayed recall conditions: Negations, Biological Processes, and Work. Five features were associated with cognitive impairment only during the immediate recall task: Cognitive Processes, Present Focus, Space, Prepositions, and Leisure. Finally, four features were associated with cognitive impairment exclusively during the delayed recall task: First Person pronouns, Past Focus, Number, and Drives.
Note: Statistical significance was evaluated using Bonferroni-corrected p-values to adjust for multiple hypothesis testing. A family-wise error rate of α = 0.05 was maintained. LIWC = Linguistic Inquiry and Word Count.
PFS-Immediate Recall (PFS-IR) and PFS-Delayed Recall (PFS-DR) scores were created for each participant using the weighted sum of the features associated with cognitive impairment from immediate recall and delayed recall, respectively. The distributions of the PFS-IR scores (median: −4.5, range: −50.8 to 121.3) and PFS-DR scores (median: −0.9, range: −38.6 to 176.2) are shown in Supplementary Figure 1. Linguistic PFS scores had a moderate negative correlation with traditional scores of LM for both immediate and delayed recall (Spearman correlation coefficient, ρ = −0.531, p < .001; ρ = −0.527, p < .001, respectively, Supplementary Figure 2), indicating that the PFS, in part, captures different information than traditional scoring. Supplementary Figure 3 shows an example of two participants who both scored 7 on delayed recall using traditional scoring, but their PFS scores differed by more than 50 points.
Both PFSs were associated with cognitive status (PFS-IR OR = 1.05, 95% CI: 1.03–1.07, p < .001; PFS-DR OR = 1.07, 95% CI: 1.05–1.09, p < .001, Supplementary Table 3), with a higher linguistic PFS associated with cognitive impairment. As shown in Figure 2, classifiers with either the immediate or delayed PFS with demographic information (PFS-IR: AUC-PR = 0.75, f1 = 0.68, MCC = 0.62; PFS-DR: AUC-PR = 0.77, f1 = 0.70, MCC = 0.65) were more robust than a classifier with only demographic information (AUC-PR = 0.68, f1 = 0.64, MCC = 0.58). The model with both PFS-IR and PFS-DR (AUC-PR = 0.78, f1 = 0.71, MCC = 0.66) yielded a similar performance to the more parsimonious classifier with only PFS-DR and demographics. Although the classifier with the traditional LM score and demographics achieved a slightly higher performance (AUC-PR = 0.81, f1 = 0.73, MCC = 0.68) compared to the PFS-DR model, a paired bootstrap analysis demonstrated that this difference was not statistically significant (95% CI [−0.118, 0.035], p = 0.292).
Precision-recall curves for classification of cognitive status. Note: Demo = demographics, PFS-IR = immediate recall linguistic polyfeature score, PFS-DR = delayed recall linguistic polyfeature score, Lmdly = traditional logical memory delayed score.
A subset of 514 participants completed at least two TICS-m after their digitally recorded in-person visit with an average follow-up time of 7 years (mean = 88 months, standard deviation [SD] = 32 months, range = 6 – 133 months). Table 3 shows the results of the longitudinal analysis in which PFS-DR had a significant negative association with TICS-m scores (β = −0.07, 95% CI [−0.10, −0.05], p < .001). Since the most extreme linguistic PFS scores are likely to be the most informative for identifying preclinical cognitive impairment, a 30-point higher PFS-DR score (i.e., approximately 2 SD in this sample) translates to a 2-point difference in TICS-m score over 7 years. Notably, the interaction term between PFS-DR and follow-up time (PFS-DR × time, p = 0.51) was not statistically significant. Therefore, while higher baseline PFS-DR is associated with lower cognitive screener scores over follow-up, it does not predict a steeper trajectory or faster rate of cognitive decline. Next, we added APOE genotypes to the base model; however, the results were largely unchanged (Supplementary Table 4). When the LM traditional score was introduced to the base model, PFS-DR remained a significant predictor of TICS-m scores over follow-up (β = −0.04, 95% CI [−0.07, −0.01], p = 0.012, Table 4). Finally, the base model was applied to a restricted sample, omitting participants with cognitive impairment at baseline (n = 65) who are more likely to experience cognitive decline, leaving a sample of 449 participants with NC. In this cognitively healthy sample, PFS-DR was a marginally significant predictor of TICS-m scores over follow-up (β = −0.02, 95% CI [−0.03, 0.00] p = 0.060, Supplementary Table 5).
Prediction of TICS-m scores over 7 years of follow-up
Table 3. Long description
Higher delayed recall linguistic polyfeature scores (PFS-DR) were significantly associated with lower TICS-m scores over 7 years of follow-up after adjusting for age, sex, education, and time. Older age, male sex, lower educational attainment, and increased time elapsed in months after logical memory testing were also significantly associated with lower TICS-m scores.
Note: TICS-m = Modified Telephone Interview for Cognitive Status, PFS-IR = immediate recall linguistic polyfeature score, PFS-DR = delayed recall linguistic polyfeature score.
Prediction of TICS-m scores including traditional logical memory scores over 7 years of follow-up
Table 4. Long description
Higher delayed recall linguistic polyfeature scores (PFS-DR) were significantly associated with lower TICS-m scores over 7 years of follow-up when adjusting for age, sex, education, time, and traditional logical memory delayed recall scores. Older age, male sex, lower educational attainment, increased time elapsed in months after logical memory testing, and lower logical memory delayed scores were also significantly associated with lower TICS-m scores.
Note: TICS-m = Modified Telephone Interview for Cognitive Status, PFS-IR = immediate recall linguistic polyfeature score, PFS-DR = delayed recall linguistic polyfeature score.
Exploratory analyses
Of the participants with cognitive impairment, 24 had an amnestic profile and 46 had a dysexecutive profile. As shown in Table 5, participants with aMCI had a higher percentage of function words and a lower percentage of leisure-related words in their immediate recall responses compared to participants with NC. Among delayed recall features, participants with aMCI had a higher percentage of negations, present-focused words, first-person pronouns, authentic language, and words related to cognitive processes in their delayed recall responses. In contrast, they had a lower percentage of clout-related language (i.e., a summary measure of confidence in responses) and work-related words compared to the NC group.
Linguistic features associated with amnestic mild cognitive impairment
Table 5. Long description
The table presents the nine Linguistic Inquiry and Word Count (LIWC) features significantly associated with amnestic mild cognitive impairment in reference to normal cognition across immediate recall and delayed recall tasks. Two linguistic features were associated with amnestic mild cognitive impairment in immediate recall: greater use of function words and less use of leisure words. Seven linguistic features were associated with amnestic mild cognitive impairment in delayed recall: greater use of negations, words with a present tense focus, authentic language, words related to cognitive processes, and first person pronouns, and a lower clout rating, and less use of words related to work.
Note: Statistical significance was evaluated using Bonferroni-corrected p-values to adjust for multiple hypothesis testing. A family-wise error rate of α = 0.05 was maintained. LIWC = Linguistic Inquiry and Word Count.
Participants with a dysexecutive profile of MCI had a lower percentage of words related to biological processes and leisure in their immediate recall responses compared to participants with NC, as shown in Table 6. In delayed recall, participants with dysMCI had a higher percentage of present-focused words and negations, but a lower percentage of clout-related language and past-focused words, compared to the NC group.
Linguistic features associated with dysexecutive mild cognitive impairment
Table 6. Long description
The table presents the six Linguistic Inquiry and Word Count (LIWC) features significantly associated with dysexecutive mild cognitive impairment in reference to normal cognition across immediate recall and delayed recall. Two linguistic features, use of words related to biological processes and leisure, were associated with a lower odds of dysexecutive cognitive impairment in reference to normal cognition. Four linguistic features were associated with a lower odds of dysexecutive mild cognitive impairment in reference in to normal cognition in delayed recall: lower use of words with a present tense focus and negations, a higher clout rating, and greater use of words with a past tense focus.
Note: Statistical significance was evaluated using Bonferroni-corrected p-values to adjust for multiple hypothesis testing. A family-wise error rate of α = 0.05 was maintained. LIWC = Linguistic Inquiry and Word Count.
Discussion
We found that linguistic features extracted from a paragraph recall task were able to detect cognitive impairment and predict change in cognition over time. We used a readily available text analysis tool to extract linguistic features from a commonly used test of episodic memory. The most notable features associated with cognitive impairment were prepositions, negations, pronouns, and words related to time orientation and the contents of the story. These findings add to the growing body of evidence that linguistic markers can be extracted from tests not necessarily designed to examine language function, and may be meaningful for detecting cognitive impairment (Amini et al., Reference Amini, Hao, Zhang, Song, Gupta, Karjadi, Kolachalama, Au and Paschalidis2023; Petti et al., Reference Petti, Baker and Korhonen2020; Tavabi et al., Reference Tavabi, Stück, Signorini, Karjadi, Al Hanai, Sandoval, Lemke, Glass, Hardy, Lavallee, Wasserman, Ang, Nowak, Kainkaryam, Foschini and Au2022; de la Fuente Garcia et al., Reference de la Fuente Garcia, Ritchie and Luz2020). The current analysis specifically focused on semantic and linguistic features from participants’ verbal responses on a paragraph recall task. Our findings align with prior work showing that both semantic content (e.g., story-related terms) and lexical features (e.g., preposition use) in spoken and written responses are clinically informative for assessing cognitive status and detecting preclinical impairment (Ahmed et al., Reference Ahmed, de Jager, Haigh and Garrard2013; Eyigoz et al., Reference Eyigoz, Mathur, Santamaria, Cecchi and Naylor2020; Petti et al., Reference Petti, Baker and Korhonen2020; Taler & Phillips, Reference Taler and Phillips2008; Weissenbacher et al., Reference Weissenbacher, Johnson, Wojtulewicz, Dueck, Locke, Caselli, Gonzalez, Knight, Nenkova and Rambow2016).
Summary scores of linguistic features detect cognitive impairment and predict a lower cognitive performance over time
Subtle linguistic features, rather than coarse linguistic measures such as total word count, were associated with cognitive status but had small effect sizes. Thus, the creation of linguistic PFSs (i.e., the combined summary scores of these features) was key to the reliable differentiation of participants with CI from those with NC. The PFS is unique from traditional scoring of the LM task because it captures linguistic features beyond the semantic content (i.e., story details), such as sentence complexity and markers of self-monitoring (e.g., “I don’t remember her name”). Since our analysis indicated that the PFS has comparable classification performance to traditional LM scoring, it suggests that these methods could be used to automate scoring in clinical or research settings where manual scoring is not feasible. There is also the opportunity to improve the PFS by adding relevant features not currently available in LIWC (e.g., word categories for all LM story details, common confabulations) and potentially surpass the predictive ability of manual LM scoring. An additional opportunity would be to adapt these methods for other languages (e.g., LIWC has dictionaries in more than 15 different languages), thereby extending their applicability to diverse linguistic populations and potential to enhance cross-cultural research. Taken together, our approach could enable the implementation of sensitive and specific neuropsychological assessments into cognitive screening processes in multiple settings, particularly if integrated with an automated pipeline for digital voice capture, auto-transcription, and scoring in a seamless process.
In our longitudinal analysis, we found that the linguistic PFS predicted future decline on a cognitive screener over 7 years of follow-up. Furthermore, our results showed that the linguistic PFS predicted lower scores on a cognitive screener over follow-up even when including the traditional LM score, suggesting that it provides additional information about future cognitive performance not captured by traditional, manual scoring of the LM task.
Moving forward, additional work is needed to refine the feature selection process. Ideally, the predictive model for feature selection would use a training sample rather than the full sample as done in this analysis, which may have led to overly optimistic estimates of predictive performance. We were unable to use a training sample because it would have further reduced the size of our CI group (n = 112) and potentially led to unstable feature selection and loss of subtle but meaningful linguistic features for the PFS. Additionally, the selected features and their predictive ability for prevalent and incident cognitive impairment need to be validated across diverse cohorts and clinical samples prior to implementation in real-world settings.
Individual linguistic features are associated with cognitive impairment
As mentioned above, coarse linguistic measures, such as total word count, were not associated with cognitive status, as also seen in other studies (Chapin et al., Reference Chapin, Clarke, Garrard and Hinzen2022; Kavé & Goral, Reference Kavé and Goral2016). One previous study did find a difference in word count (König et al., Reference König, Köhler, Tröger, Düzel, Glanz, Butryn, Mallick, Priller, Altenstein, Spottke, Kimmich, Falkenburger, Osterrath, Wiltfang, Bartels, Kilimann, Laske, Munk, Roeske and Teipel2024), however, they used much longer discourse than the LM responses in our study, which generally contained less than 60 s of content. In our study, a lower proportion of content words, such as those related to work, leisure, and numbers, was also associated with cognitive impairment, which is expected as these categories are closely related to the content of the stimulus story. This suggests that participants with cognitive impairment conveyed less content information than participants with normal cognition, which aligns with a deficit in episodic memory. Specific to the immediate recall condition, a lower use of prepositions was associated with cognitive impairment. This is consistent with findings from Lindsay and colleagues of “semantically empty language” and reduced use of prepositions during a picture description task among individuals with dementia due to AD (Lindsay et al., Reference Lindsay, Tröger and König2021), and suggests poorer encoding of the story details among those with CI in our study. Additionally, a greater use of ‘negations’ and words related to ‘cognitive processes’ in participants’ immediate recall responses was associated with an increased likelihood of cognitive impairment in our study. Responses such as “don’t know” or “can’t remember” contribute to these linguistic categories and may reflect an awareness of cognitive deficits, uncertainty in their responses, or a hesitancy to respond. In the delayed recall condition, there was an enrichment of the use of first-person pronouns and a lower use of the past tense among participants with cognitive impairment, which may reflect a shift toward commenting about their performance instead of adhering to the task of providing story details as the story is set in the past tense. This result is partially congruent with a previous study that found a higher proportion of pronouns during immediate, rather than delayed recall, of multiple stories among participants with MCI (Boland et al., Reference Boland, Jensen, Davidson and Taler2024). These differences in linguistic features for each task condition suggest that they are related to different cognitive processes. During immediate recall, difficulties with learning are revealed through simplified grammar and expressions of uncertainty. In delayed recall, it appears that remembering and retrieving specific details of the story, such as the number of children and the amount of money, are particularly sensitive to cognitive impairment.
These findings show that information-rich responses from participants hold nuances that are not fully captured by the traditional LM scoring system. Clinicians practicing the “Boston Process Approach” have long appreciated that qualitative features of a person’s process while completing a task, not just their final score, are informative for understanding cognitive function (Ashendorf et al., Reference Ashendorf, Swenson and Libon2013; Milberg et al., Reference Milberg, Hebben, Kaplan, Grant and Adams2009). Some features, such as repetitions of the contents of the story or self-commentary (e.g., “I don’t remember”) are feasible for a clinician to detect, whereas more subtle features (e.g., preposition use) are less so. Digital recording of responses and the NLP techniques used in the current study allow for the systematic quantification of these subtle, process-based features, and also make them accessible to the general clinician.
Subtypes of cognitive impairment are associated with distinct linguistic profiles
Some linguistic features were similarly altered in those with amnestic MCI or dysexecutive MCI in comparison to cognitively healthy participants. However, amnestic MCI was uniquely associated with a higher proportion of first-person words, cognitive process words, and a higher ‘authentic’ score (i.e., a summary measure of personal content; Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010) which, taken together, suggest a shift toward talking about oneself rather than recalling the story. These distinct patterns of word use and linguistic features may aid in the early differentiation of MCI subtypes; however, these results must be interpreted with caution, as the amnestic MCI and dysexecutive MCI sample sizes were small.
Limitations
A limitation of the current study is that the clinical consensus conferences included review of traditional LM scores in the determination of cognitive status. As a result, the content of the LM responses, and therefore the LIWC variables, is not completely independent of the outcome variable of cognitive status. For example, cognitive impairment was identified in part through poor LM traditional scores, and therefore it is not surprising that LIWC features such as “biological processes” and “work”, which overlap with traditional scoring elements of cook, cafeteria, and employed, are also related to cognitive status. This circularity issue is likely to have inflated the predictive accuracy of our models and be more pronounced for the aMCI group in the MCI subtype analysis. Secondly, this study was limited to linguistic features that are coded by the LIWC program. The program was designed to assess psychological constructs, such as positive and negative affect, whereas LM was developed to assess episodic memory. Therefore, LIWC may not capture all relevant linguistic features associated with learning and recall. Additionally, our study may have been underpowered to detect all LIWC features associated with cognitive impairment due to the small sample size. For example, word count trended toward significance (OR = 0.67, 95% CI: 0.51–0.89, p = .225) but did not reach significance after Bonferroni correction. Furthermore, the LIWC program does not incorporate the syntactic structure of responses and thus may not accurately capture word meanings that vary depending on context within a sentence. Lastly, this study focused on a text-based, rather than audio-based, analysis of voice recordings. Future studies should integrate additional features from digital voice recordings, such as acoustic and temporal patterns of responses, which have the potential to increase the sensitivity and specificity of determining cognitive status and detecting preclinical cognitive impairment.
In summary, this study identified novel, subtle features of verbal responses on a brief assessment of episodic memory that were informative for detecting cognitive impairment and predicting cognitive performance over time. These findings suggest that analyzing linguistic features during recall tests can capture information about learning, memory, language, and self-monitoring processes. Furthermore, after further refinement of the approach and validation across diverse cohorts and clinical samples, these methods could enable automated scoring of paragraph recall tests when gold-standard manual scoring is not feasible, such as in primary care settings or self-administered modalities.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1355617726102215.
Acknowledgments
We thank the participants of the Long Life Family Study for the generous donation of their time and data and their contribution to knowledge about healthy aging.
Funding statement
This work was supported by the National Institute on Aging (grants U01 AG023746, U01 AG023712, U01 AG023749, U01 AG023755, U01 AG023744, U19 AG063893, and K01 AG057798).
Competing interests
Dr. Au serves as a Scientific advisor to Signant Health and Novo Nordisk. Dr. Libon receives royalties from Oxford University Press and Linus Health and serves as a consultant to Linus Health. All other authors declare no conflicts of interest.
Open access
