Hostname: page-component-74656f9884-c52kr Total loading time: 0 Render date: 2026-11-08T23:49:18.557Z Has data issue: false hasContentIssue false
  • English
  • Français

Measuring disorganized speech in schizophrenia: automated analysis explains variance in cognitive deficits beyond clinician-rated scales

Published online by Cambridge University Press:  25 April 2018

K. S. Minor ,
J. A. Willits ,
M. P. Marggraf ,
M. N. Jones  and
P. H. Lysaker
K. S. Minor*
Affiliation:
Department of Psychology, Indiana University- Purdue University Indianapolis, Indianapolis, IN, USA
J. A. Willits
Affiliation:
Department of Psychology, University of California-Riverside, Riverside, CA, USA
M. P. Marggraf
Affiliation:
Department of Psychology, Indiana University- Purdue University Indianapolis, Indianapolis, IN, USA
M. N. Jones
Affiliation:
Department of Psychology, Indiana University, Bloomington, IN, USA
P. H. Lysaker
Affiliation:
Roudebush VA Medical Center, Indianapolis, IN, USA Department of Psychiatry, Indiana University School of Medicine, Indianapolis, IN, USA
*
Author for correspondence: K. S. Minor, E-mail: ksminor@iupui.edu
Rights & Permissions [Opens in a new window]

Abstract

Background

Conveying information cohesively is an essential element of communication that is disrupted in schizophrenia. These disruptions are typically expressed through disorganized symptoms, which have been linked to neurocognitive, social cognitive, and metacognitive deficits. Automated analysis can objectively assess disorganization within sentences, between sentences, and across paragraphs by comparing explicit communication to a large text corpus.

Method

Little work in schizophrenia has tested: (1) links between disorganized symptoms measured via automated analysis and neurocognition, social cognition, or metacognition; and (2) if automated analysis explains incremental variance in cognitive processes beyond clinician-rated scales. Disorganization was measured in schizophrenia (n = 81) with Coh-Metrix 3.0, an automated program that calculates basic and complex language indices. Trained staff also assessed neurocognition, social cognition, metacognition, and clinician-rated disorganization.

Results

Findings showed that all three cognitive processes were significantly associated with at least one automated index of disorganization. When automated analysis was compared with a clinician-rated scale, it accounted for significant variance in neurocognition and metacognition beyond the clinician-rated measure. When combined, these two methods explained 28–31% of the variance in neurocognition, social cognition, and metacognition.

Conclusions

This study illustrated how automated analysis can highlight the specific role of disorganization in neurocognition, social cognition, and metacognition. Generally, those with poor cognition also displayed more disorganization in their speech—making it difficult for listeners to process essential information needed to tie the speaker's ideas together. Our findings showcase how implementing a mixed-methods approach in schizophrenia can explain substantial variance in cognitive processes.

Keywords

Automated analysiscognitiondisorganized symptomsschizophreniaspeech

Information

Type
Original Articles
Information
Psychological Medicine , Volume 49 , Issue 3 , February 2019 , pp. 440 - 448
Copyright
Copyright © Cambridge University Press 2018 

Introduction

Communicating information cohesively is an essential skill when we relay instructions to others, explain the rationale behind our actions, or tell a loved one about our day. Disruptions in cohesive communication have been reported since early conceptualizations of schizophrenia – Bleuler (Reference Bleuler and Zinkin1911) described how many with the disorder exhibit breakdowns in connections between conscious thoughts – and are often captured by measuring disorganized symptoms. This cluster of symptoms has shown ties to deficits in neurocognition, social cognition, and metacognition (Ventura et al. Reference Ventura2010, Reference Ventura, Wood and Hellemann2013; Minor & Lysaker, Reference Minor and Lysaker2014; Minor et al. Reference Minor2015c). Neurocognitive deficits involve poor performance on non-social tasks in domains ranging from the attentional capacity to executive functioning (Green, Reference Green1996; Green et al. Reference Green, Kern and Heaton2004). Social cognitive impairments reflect problems in an array of processes (e.g. theory of mind, emotion recognition) needed to draw social inferences about others (Green et al. Reference Green2008). Metacognitive deficits encompass difficulties integrating previous experiences to form complex ideas about oneself and others (Lysaker & Klion, Reference Lysaker and Klion2017; Lysaker et al. Reference Lysaker2018). These three cognitive areas represent distinct, but related, processes (Pinkham et al. Reference Pinkham2003; Allen et al. Reference Allen2007; Fanning et al. Reference Fanning, Bell and Fiszdon2012; Lysaker et al. Reference Lysaker2013).

In previous studies, one issue with linking disorganized symptoms to cognitive deficits has centered on the methodology implemented to measure disorganization. Typically, clinician-rated scales have been used. Although many scales are time-efficient and have been validated in schizophrenia samples (Andreasen & Grove, Reference Andreasen and Grove1986; Bell et al. Reference Bell1994), disadvantages of this approach include dependence on ordinal rating systems and a failure to account for how specific facets of communication produced by examinees compare to ‘typical’ communication (Cohen & Elvevag, Reference Cohen and Elvevag2014; Elvevag et al. Reference Elvevag2016). Research using clinician-rated scales has shown small to moderate associations between disorganized symptoms and cognitive deficits (Ventura et al. Reference Ventura2010, Reference Ventura, Wood and Hellemann2013; Hamm et al. Reference Hamm2012; Minor & Lysaker, Reference Minor and Lysaker2014). However, since both constructs stem, in part, from breakdowns in connecting conscious thoughts, there may be additional variance that has not been identified.

One strategy to account for additional variance in cognitive deficits is to implement behaviorally-based process instruments in combination with or in place of clinician-rated scales. These instruments quantify disorganization in speech samples using either trained raters (Docherty, Reference Docherty2012; Docherty et al. Reference Docherty2013; Minor et al. Reference Minor2016) or automated analysis (Elvevag et al. Reference Elvevag2007; Bedi et al. Reference Bedi2015; Minor et al. Reference Minor2015a). In both cases, specific instances of disorganization are identified and a summary score is calculated to reflect the total proportion of disorganization within each speech sample. Although the trained rater strategy benefits from a systematic approach to identify explicit behaviors, a key drawback is that the training required to produce reliable ratings and the number of hours needed to rate speech samples is time-intensive.

Automated analysis has emerged in recent years as a methodology with the potential to leverage strengths of clinician-rated and trained rater approaches (see Table 1). Using speech or writing samples, automated analysis can rapidly estimate how frequently people use certain categories of words (Buck et al. Reference Buck, Minor and Lysaker2015; Minor et al. Reference Minor2015a; Fineberg et al. Reference Fineberg2016; Abplanalp et al. Reference Abplanalp2017; Minor et al. Reference Minor2018) and compare explicit communication from individuals to a large corpus of text to objectively rate disorganization (Elvevag et al. Reference Elvevag2007; Manschreck et al. Reference Manschreck2012; Merrill et al. Reference Merrill2017; Gupta et al. Reference Gupta2018; Marggraf et al. Reference Marggraf2018). In schizophrenia samples, automated analysis has been used to show how disorganization occurs across multiple levels of language, ranging from basic (e.g. words, phrases) to more complex (e.g. across sentences, paragraphs, conversations) processes (Maher et al. Reference Maher2005; Elvevag et al. Reference Elvevag2010; Moe et al. Reference Moe2016; Willits et al. Reference Willitsin press). Many of these studies identify disorganization at deep levels of language processing and are unlikely to be apparent using other methodologies. A recent example from Bedi and colleagues (Reference Bedi2015) showed how automated analysis of language structure could predict which people at clinical high risk for psychosis would convert to a psychotic disorder over a 30-month period. However, an important gap in the literature centers on comparing automated analysis with other assessments of disorganized speech to determine if automated metrics provide added value.

Table 1.

Comparing three methodologies of assessing speech disorganization in schizophrenia

Implementing automated analysis alongside a clinician-rated scale of disorganization would address this gap while holding the potential to account for additional variance in cognitive processes. To date, few studies have tested relationships between disorganized symptoms measured via automated analysis and cognitive deficits, with existing studies focusing primarily on word use categories rather than disorganization (Buck et al. Reference Buck, Minor and Lysaker2015; Minor et al. Reference Minor2015a). Based on previous work, both basic and complex language processes appear to be good candidates for predicting neurocognition, social cognition, and metacognition in people with schizophrenia (Docherty et al. Reference Docherty1996; Ditman & Kuperberg, Reference Ditman and Kuperberg2010; Buck & Penn, Reference Buck and Penn2015; Merrill et al. Reference Merrill2017). Testing these relationships would indicate if automated analysis offers incremental variance compared with a traditional measure of disorganized symptoms. In addition, administering both a clinician-rated measure and automated analysis carries the benefit of testing how much variance in cognitive processes is explained when integrating these two methodologies.

Objectives and hypotheses

The aims of this study were three-fold. First, we determined if disorganization measured using automated analysis was related to neurocognition, social cognition, and metacognition in schizophrenia. Our expectation was that deficits in both basic and complex language markers would be significantly linked with deficits in all three cognitive processes. Second, we tested if measuring disorganization via automated analysis added incremental variance to predictions of cognitive processes by comparing it with a clinician-rated measure. We hypothesized that automated analysis would account for significant variance when predicting neurocognition, social cognition, and metacognition and that basic and complex markers would serve as individual predictors. To compare methodologies, we also examined if the clinician-rated scale accounted for variance beyond automated analysis. Finally, we determined how much variance in neurocognition, social cognition, and metacognition was explained when integrating clinician-rated and automated measures of disorganization.

Methods

Participants

Eighty-one outpatients from a Midwestern VA Medical Center participated in this study. All participants had: (1) a DSM-IV diagnosis of Schizophrenia (n = 56) or Schizoaffective Disorder (n = 25) confirmed via the Structured Clinical Interview for DSM-IV-TR Disorders-Patient Edition (SCID-I/P; First et al. Reference First2002); (2) no hospitalizations or changes in medication within 30 days of testing; (3) no documented intellectual disability based on medical record review; and (4) no active substance dependence. Participants were part of a randomized controlled trial on cognitive remediation. To avoid potential confounds with a treatment designed to improve cognitive functioning, the current study focused solely on baseline testing. University and Veterans Affairs Institutional Review Boards approved all procedures and participants gave written informed consent prior to study onset.

Measures

Disorganization

Automated analysis was conducted on speech generated in response to the Indiana Psychiatric Illness Interview (IPII; Lysaker et al. Reference Lysaker2002), a semi-structured interview that assesses perceptions of one's life and illness. The open-ended nature of the IPII was a key reason for its selection; its format differs from many structural interviews and speech tasks in that subjects control how long they speak with little input or affective prompting from examiners. IPII interviews were typically 30–60 min in length, allowing subjects to generate substantial samples for analysis (Total words: Mean = 2786, s.d. = 2117). Each interview was recorded and transcribed with examiner speech removed.

Automated analysis of IPII interviews was conducted using Coh-Metrix 3.0 (McNamara et al. Reference McNamara2014). Coh-Metrix is a discourse processing software that analyzes language by measuring coherence and cohesion, which are closely tied to disorganization (Andreasen, Reference Andreasen1979; Elvevag et al. Reference Elvevag2007). It has been used in several studies (Crossley et al. Reference Crossley, Salsbury and McNamara2009; McCarthy & Jarvis, Reference McCarthy and Jarvis2010; see McNamara et al. Reference McNamara2014) and has shown the ability to differentiate high and low levels of organization in healthy adults (McNamara et al. Reference McNamara, Sun and Miyake2006) and people with schizophrenia (Willits et al. Reference Willitsin press). Coh-Metrix contains 108 indices across 11 categories and provides a comprehensive view of language characteristics ranging from basic to complex processes. Basic indices typically examine speech at word or phrase levels, whereas complex indices measure organization within sentences, between sentences, and across paragraphs.

For this study, we selected two basic (syllables per word, type-token ratio) and five complex indices (narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion; see Table 2 for descriptions of all indices) a priori. Basic indices were chosen based on their expected overlap with cognitive processes. Specifically, syllables per word corresponds with verbal intelligence which is inversely linked to disorganization (O'Leary et al. Reference O'Leary2000; Ventura et al. Reference Ventura2010); type-token ratio has been used as a disorganization marker (Manschreck et al. Reference Manschreck, Maher and Ader1981, Reference Manschreck1991; Allen, Reference Allen1983) and is a key component of systems measuring language (e.g. CAST; Maher et al. Reference Maher2005). Complex indices were selected due to their ability to capture discourse characteristics across full speech samples. The specific complex indices chosen here were narrowed down based, in part, on a principal component analysis study conducted by co-creators of Coh-Metrix 3.0 that showed these five indices accounted for the majority of variance in 54 separate categories (Graesser et al. Reference Graesser, McNamara and Kulikowich2011). To date, Coh-Metrix has only been used to analyze speech in one schizophrenia study (Willits et al. Reference Willitsin press) and written text in one study of people at clinical high risk for psychosis (Gupta et al. Reference Gupta2018).

Table 2.

Means, standard deviations, and descriptions of automated analysis indices (n = 81)

Notes. Multidimensional indices reflect z-scores from Coh-Metrix 3.0. Additional information on all indices available in McNamara et al. (Reference McNamara2014).

The disorganized symptoms subscale of the Positive and Negative Syndrome Scale (PANSS; Kay et al. Reference Kay, Fizszbein and Opler1987; see Bell et al. Reference Bell1994) was used as the clinician-rated measure in this study. PANSS items range from 1 (absent) to 7 (extreme) and have been used extensively to rate symptoms in people with schizophrenia (Kay et al. Reference Kay, Fizszbein and Opler1987; Bell et al. Reference Bell1992; Minor et al. Reference Minor2015c). Seven items comprise the disorganized subscale: difficulty in abstract thinking, stereotyped thinking, conceptual disorganization, lack of insight, poor attention, tension, and mannerisms/ posturing. Our sample demonstrated moderate disorganization with considerable range between participants (M = 16.63, s.d. = 3.75, range: 11–27). The PANSS has shown strong psychometric properties (Kay et al. Reference Kay, Fizszbein and Opler1987; Bell et al. Reference Bell1992; Lysaker et al. Reference Lysaker2013); it was rated by clinicians who had undergone several weeks of PANSS training. Training consisted of completing sample ratings on videotapes of clinical interviews. Before administering the PANSS, clinician's ratings had to demonstrate good inter-rater reliability (α = 0.80) with a consensus rating group with several years of experience using the PANSS.

Neurocognition

The Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS; Nuechterlein et al. Reference Nuechterlein2008), which was designed specifically to assess neurocognitive functioning in schizophrenia (De Herdt et al. Reference De Herdt2013; McCleery et al. Reference McCleery2015; Minor et al. Reference Minor2015b), was used as the neurocognition measure. In this study, a composite neurocognition score was created for each participant using six of the seven MATRICS domains (processing speed, attention, working memory, verbal learning, visual learning, reasoning, and problem-solving). The MATRICS social cognition measure, the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT; Mayer et al. Reference Mayer, Salovey and Caruso2002), was excluded from this score and included with social cognition. Equal weight was given to the remaining six domains.

Social cognition

A composite score was created for social cognition that gave equal weight to three separate domains: (1) emotional processing (measured using the MSCEIT); (2) emotion recognition [Bell-Lysaker Emotion Recognition Task (BLERT); Bell et al. Reference Bell, Bryson and Lysaker1997]; and (3) theory of mind [Hinting task (Corcoran et al., Reference Corcoran, Mercer and Frith1995; Greig et al. Reference Greig, Bryson and Bell2004) and Social Attributions Test-Multiple Choice (SAT-MC; Bell et al. Reference Bell2010)]. The MSCEIT (Nuechterlein et al., Reference Nuechterlein2008; Eack et al., Reference Eack2010), BLERT (Bell et al. Reference Bell, Bryson and Lysaker1997), Hinting task (Bell et al. Reference Bell2010), and SAT-MC (Pinkham et al. Reference Pinkham2014) have all demonstrated strong psychometric properties in schizophrenia; the BLERT and Hinting task were selected by the Social Cognition Psychometric Evaluation (SCOPE) study as being among the best social cognition measures in schizophrenia (Pinkham et al. Reference Pinkham2014).

Metacognition

The Metacognition Assessment Scale-Adapted (MAS-A; Lysaker et al. Reference Lysaker2005, adapted from Semerari et al. Reference Semerari2003) was used to assess metacognition. The MAS-A assesses a person's ability to integrate implicit and explicit information about themselves and others to form an integrated sense of how to respond to psychosocial challenges. The MAS-A is scored from 0 to 28, with higher scores signifying greater metacognitive capacity (Lysaker et al. Reference Lysaker2013). MAS-A scores were calculated using raters who had exhibited good inter-rater reliability (α ⩾ 0.80). In schizophrenia cohorts, the MAS-A has demonstrated good reliability and validity in several studies (see Lysaker & Dimaggio, Reference Lysaker and Dimaggio2014).

Analysis

Data analysis was conducted in three parts. First, composite scores were calculated for neurocognition and social cognition. For neurocognition, standardized t-scores were converted to z-scores and summed using a weighted average of 1.0 for all MATRICS domains except social cognition. For social cognition, measures were converted to z-scores and summed, with the MSCEIT (emotion processing) and BLERT (emotion recognition) given weighted averages of 1.0 and the Hinting task and SAT-MC (theory of mind) given weighted averages of 0.5. This strategy gave equal weight to each social cognition domain. Second, correlations between cognitive variables (neurocognition, social cognition, metacognition) and Coh-Metrix indices were conducted to test relationships between cognitive processes and disorganization. Correlations between disorganization measures were also analyzed. Finally, two sets of stepwise regressions were run with neurocognition (regression one), social cognition (two), and metacognition (three) as outcome variables. In the first set, PANSS disorganized symptoms were entered as the step one predictor, and Coh-Metrix indices as step two predictors. The order was reversed in the second set. For all regressions, only those Coh-Metrix indices that exhibited a significant relationship with cognitive variables in correlational analyses were entered.

Results

Participants were primarily middle aged (M = 49.74, s.d. = 10.71), male (n = 77, 95%), African-American (n = 46, 57%), currently unmarried (n = 47, 58%), unemployed (n = 68, 84%), reported only immediate family or providers in their social network (n = 53, 65%), completed high school/GED (n = 74, 91%), and were earning below $20 000 annually (n = 69, 85%). On average, they were first hospitalized in their late twenties (M = 29.43, s.d. = 12.35) and were currently prescribed antipsychotic medications (Chlorpromazine equivalent M = 363.64, s.d. = 416.10). Post-hoc analyses of associations between disorganization methods showed small convergence between the clinician-rated scale with referential cohesion, r(79) = −0.24, p = 0.029, syllables per word, r(79) = −0.12, p = 0.298, type-token ratio, r(79) = −0.21, p = 0.067, syntactic simplicity, r(79) = 0.20, p = 0.079, word concreteness, r(79) = −0.11, p = 0.312, and deep cohesion, r(79) = −0.11, p = 0.324. Medium convergence was observed with narrativity, r(79) = −0.31, p = 0.005.

Links between cognitive processes and disorganization using automated analysis

As shown in Table 3, several small to medium significant relationships were observed between cognitive processes and disorganization measured via automated analysis. Neurocognition was associated with one basic index (syllables per word) and inversely associated with one complex index (syntactic simplicity). Social cognition was inversely associated with one complex index (syntactic simplicity) but was not linked with basic indices. Metacognition was inversely associated with one basic index (type-token ratio) and positively related to one complex index (referential cohesion). Three complex indices (narrativity, word concreteness, and deep cohesion) were not significantly associated with any cognitive process. These findings generally supported hypotheses that basic and complex indices of disorganization would be related to neurocognition, social cognition, and metacognition.

Table 3.

Relationships between disorganization using automated analysis with neurocognition, social cognition, and metacognition (n = 81)

***p < 0.001; **p < 0.01; *p < 0.05; +p < 0.10.

Variance in cognitive processes explained by automated analysis when accounting for a clinician-rated measure of disorganized symptoms

To determine if automated analysis explained added variance in cognitive processes, stepwise regression models were conducted with a clinician-rated scale of disorganized symptoms entered as the step one predictor. In the first model, one basic (syllables per word) and one complex (syntactic simplicity) automated index were entered simultaneously as step two predictors of neurocognition. The overall model was significant, F (3,77) = 9.73, p < 0.001, with clinician-rated and automated measures of disorganization accounting for 28% of the variance in neurocognition (adjusted R 2 = 0.25). Automated indices explained 10% of the variance beyond the clinician-rated scale. All three variables entered (clinician-rated disorganization, syllables per word, syntactic simplicity) were significant predictors of neurocognition. For social cognition, one complex index (syntactic simplicity) was entered into the second step of the regression. The overall model was significant, F (2,78) = 15.05, p < 0.001, accounting for 28% of the variance in social cognition (adjusted R 2 = 0.26). Clinician-rated disorganization was the lone significant predictor. The automated index (syntactic simplicity) was on the cusp of significance and contributed 3% of variance beyond the clinician-rated scale. In the third model, metacognition was the outcome variable; one basic (type-token ratio) and one complex (referential cohesion) index were entered as predictors. The overall model was significant, F (3,77) = 11.40, p < 0.001, accounting for 31% of the variance in metacognition (adjusted R 2 = 0.28). Automated indices explained 26% of the variance after accounting for the clinician-rated scale. Both clinician-rated disorganization and the basic automated index (type-token ratio) were significant individual predictors. These findings generally supported our hypothesis that disorganization measured via automated analysis could explain added variance in cognitive processes (Table 4).

Table 4.

Stepwise regressions using clinician-rated (step one) and automated measures (step two) of disorganized symptoms as predictors of cognitive variables in schizophrenia (n = 81)

***p < 0.001; **p < 0.01; *p < 0.05; +p < 0.10.

Variance in cognitive processes explained when reversing stepwise regressions

Stepwise regression models were also reversed with automated analysis indices entered in step one and the clinician-rated scale entered in step two (Table 5). For neurocognition, the clinician-rated scale accounted for 12% of the variance after controlling for automated indices. The clinician-rated scale accounted for the most variance, 20%, in social cognition and the least variance, 9%, in metacognition whenever automated indices were controlled for. These findings illustrate the unique variance in cognitive processes accounted for by a commonly-used clinician-rated scale when controlling for automated indices.

Table 5.

Reverse stepwise regressions entering automated (step one) before clinician-rated measures (step two) to predict cognitive variables in schizophrenia (n = 81)

***p < 0.001; **p < 0.01; *p < 0.05; +p < 0.10.

Discussion

Our primary goals in this study were to use automated analysis to delineate links between disorganization and cognitive processes and determine if automated analysis added incremental variance in predicting cognitive processes compared with a commonly used clinician-rated measure. Three key findings emerged. First, complex automated indices were significantly associated with all three cognitive processes and basic indices were related to neurocognition and metacognition, but not social cognition. Second, automated analysis explained significant variance in neurocognition and metacognition even when accounting for the clinician-rated scale. Third, automated analysis and the clinician-rated instrument combined to account for between a quarter to a third of the total variance in all three cognitive processes.

When assessing disorganized symptoms broadly, previous research has shown these symptoms are related to cognitive processes (Ventura et al. Reference Ventura2010, Reference Ventura, Wood and Hellemann2013; Hamm et al. Reference Hamm2012; Minor & Lysaker, Reference Minor and Lysaker2014); the current study is novel in its implementation of automated analysis to illustrate which specific facets are linked with neurocognition, social cognition, and metacognition. Regarding basic language indices, findings indicated that automated analysis reveals relationships between neurocognition and metacognition with disorganization at fundamental levels of discourse. The most intriguing finding was that people with poor metacognitive capacity produced a greater frequency of different words (i.e. higher type-token ratio). Although counterintuitive at first glance, this may reflect the inverse relationship between type-token ratio and a number of words spoken (McCarthy & Jarvis, Reference McCarthy and Jarvis2010), as people with lower metacognitive capacity tended to generate less speech in clinical interviews (Buck et al. Reference Buck, Minor and Lysaker2015). A post-hoc analysis testing this explanation showed that number of words spoken was significantly associated with metacognition—accounting for 14% of the variance—and inversely related to type-token ratio—accounting for 55% of the variance. A second possibility is that those with poor metacognitive capacity frequently shifted between topics—leading to more unique words—without providing the connective threads necessary to frame the context for their audience. This is in line with the observed association between metacognition and referential cohesion found here and supports findings of greater disorganization in those with schizophrenia who display poor metacognition (Hamm et al. Reference Hamm2012; Minor & Lysaker, Reference Minor and Lysaker2014).

Neurocognition, social cognition, and metacognition were each associated with at least one complex automated index, suggesting that these different cognitive abilities are connected to disorganization at sophisticated levels – within sentences, between sentences, and across paragraphs – of discourse. Associations were in the small to medium effect size range. In all instances, those who performed poorly in a cognitive area also produced speech that lacked clarity, making it challenging for listeners to devise meaning. Specifically, automated analysis captured how those with poor neurocognition and social cognition used opaque, unfamiliar sentence structures. This style forces listeners to work to understand the speaker's intentions, creating obstacles when processing information. The automated analysis also showed how people with poor metacognitive capacity produced speech containing few overlapping words and ideas; thus, leaving out the important context for listeners by omitting explicit threads to connect concepts. Without these connective threads, the audience is typically unable to integrate essential information to tie the speaker's ideas together. Although it was surprising some complex indices (narrativity, word concreteness, deep cohesion) were not related to cognitive processes, our general findings supported previous observations of links between speech disorganization and cognitive impairment (Ventura et al. Reference Ventura2010, Reference Ventura, Wood and Hellemann2013; Minor & Lysaker, Reference Minor and Lysaker2014).

The size of observed associations using automated analysis was generally in line with previous studies implementing clinician-rated or trained rater approaches. Clinician-rated scales of disorganization have demonstrated small (neurocognition; Ventura et al. Reference Ventura, Wood and Hellemann2013) or small to medium effect size associations (social cognition; Ventura et al. Reference Ventura, Wood and Hellemann2013; metacognition; Hamm et al. Reference Hamm2012; Minor & Lysaker, Reference Minor and Lysaker2014) with cognitive processes. The trained rater approach has yielded small to medium associations with neurocognitive and social cognitive variables (Docherty, Reference Docherty2012; Docherty et al. Reference Docherty2013), typically outperforming clinician-rated scales as well as automated analysis based on the links observed in this study. In line with the relatively small associations observed here, the trained rater approach has also demonstrated only minor or modest convergence with clinician-rated scales (Docherty et al. Reference Docherty, Miller and Lewis1997; Docherty, Reference Docherty2005, Reference Docherty2012). Comparing automated and trained rater methods is an important future avenue of research.

Going beyond associations, one of the most compelling aspects of this study is that it determined how much unique variance automated analysis explained in cognitive processes beyond a traditional measure of disorganized symptoms. Estimates of neurocognition, social cognition, and metacognition varied widely. On the low end, automated analysis minimally explained social cognition (3%) and no automated index was a significant predictor. In contrast, automated analysis had a moderate role in neurocognition, explaining 10% of the variance beyond the clinician-rated measure. Both syllables per word (basic index) and syntactic simplicity (complex index) individually predicted neurocognition. This highlights how people with schizophrenia displaying poor neurocognition produce simple words that should be easy for others to comprehend, yet their sentences contain unfamiliar structures that are difficult for their audience to follow. Of the cognitive processes tested here, automated analysis explained the most unique variance in metacognition (26%). Type-token ratio (basic index) was a significant individual predictor, indicating that those with schizophrenia who used a greater frequency of different words tended to display poor metacognitive capacity even after accounting for a clinician-rated scale.

When controlling for automated analysis, the clinician-rated scale accounted for significant variance in neurocognition (12%), social cognition (20%), and metacognition (9%). Compared with automated analysis, the clinician-rated scale accounted for more unique variance in social cognition (20% v. 3%), marginally more variance in neurocognition (12% v. 10%), and considerably less variance in metacognition (9% v. 26%). One important difference between these two methods is that a composite score of seven clinician-rated disorganized symptoms was tested compared with only one or two automated indices (which were selected based on associations with the cognitive process). Taken together, our findings showcase the unique variance accounted for by two different methodologies and illustrate the value of implementing each method for explaining variance in different cognitive processes.

Whereas automated analysis and the clinician-rated scale accounted for a wide range of variance in cognitive processes alone, the amount explained whenever they were combined was more consistent – between 28 and 31%. Both automated analysis and clinician-rated scales assess disruptions in conscious thoughts, with the former using computational software to characterize disorganized symptoms at basic and sophisticated levels of discourse (McNamara et al. Reference McNamara2014) and the latter relying on clinician's impressions of a person's level of disorganized speech and behavior (Kay et al. Reference Kay, Fizszbein and Opler1987). A key contribution of this study is the observation that integrating these approaches is useful for illustrating the role of disorganization in neurocognition, social cognition, and metacognition. Given that the study sample exhibited mild to moderate disorganization, a novel future direction would be to select subjects based on the presence of formal thought disorder and determine if a mixed-methods approach accounts for additional variance in cognitive processes. This strategy would be compatible with calls from funding agencies (Insel, Reference Insel2014) and researchers in the field (see Cohen et al. Reference Cohen2017).

Study strengths include using novel automated indices to calculate objective assessments of disorganization and comparing automated analysis to traditional measurements. One limitation is that data were only collected at one time point. This is particularly relevant for speech samples, given that Coh-Metrix has only been used in one previous schizophrenia study (Willits et al. Reference Willitsin press) and reproducibility is a critical issue when testing new methodologies. Our group is currently collecting longitudinal data to measure the test-retest reliability of Coh-Metrix indices on several types of speech data, including IPII interviews. A second limitation is that the clinician-rated scale contained items that may not be associated with language dysfunction (e.g. lack of insight). Although previous studies have shown that the seven items used here cluster together (Bell et al. Reference Bell1994), future studies may benefit from implementing scales where all items are designed to assess behavioral disorganization. A third limitation is that no control group was recruited to compare how automated analysis predicted cognitive functioning in healthy adults. In a previous study, Coh-Metrix indices did differentiate schizophrenia and control groups (Willits et al. Reference Willitsin press); however, this study did not test relationships between disorganization and cognitive processes. A fourth limitation concerns generalizability; our sample tended to be better educated, had fewer females, and were hospitalized later in life than many schizophrenia cohorts. Although these characteristics are common in VA samples (Harvey et al. Reference Harvey2000; Thorp et al. Reference Thorp2012; Firmin et al. Reference Firmin2016), future work should replicate our findings using community subjects.

In sum, this is among the first studies showing how automated analysis can be implemented to predict cognitive processes in schizophrenia. Significant associations were observed between automated indices and neurocognition, social cognition, and metacognition. When accounting for a traditional clinician-rated measure, automated analysis varied in its ability to predict cognitive processes: whereas little variance was accounted for in social cognition, substantial variance was explained in metacognition and multiple indices were significant predictors of neurocognition. Finally, integrating automated analysis with a clinician-rated measure explained between a quarter to a third of the variance in all three cognitive processes. This finding highlights the benefits of implementing a mixed-methods approach. Future studies should replicate these results using longitudinal data and determine how disorganization predicts cognitive processes in community and formal thought disorder samples.

Acknowledgements

We would like to acknowledge and thank the staff and clients at the Roudebush VA Medical Center in Indianapolis, IN. In addition, the staff at Indiana University and the developers of Coh-Metrix 3.0 were essential partners in this research study.

Financial support

Research supported by grants from The Veterans Rehabilitation Research and Development Service (VA RR&D) D6629R and Indiana University Collaborative Research Grant (Doc # 24787213): Building Statistical Language Processing Algorithms for the Automated Coding of Semi-Structured Interview Data in Clinical Schizophrenia.

Declaration of interest

None.

Ethical standards

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

References

Abplanalp, SJ, et al. (2017) Using lexical analysis to identify emotional distress in psychometric schizotypy. Psychiatry Research 255, 412–417.Google Scholar
Allen, HA (1983) Do positive symptom and negative symptom subtypes of schizophrenia show qualitative differences in language production? Psychological Medicine 13, 787–797.Google Scholar
Allen, DN, et al. (2007) Factor analytic support for social cognition as a separable cognitive domain in schizophrenia. Schizophrenia Research 93, 325–333.Google Scholar
Andreasen, NC (1979) Thought, language and communication disorders. II. Diagnostic significance. Archives of General Psychiatry 36, 1325–1330.Google Scholar
Andreasen, N and Grove, WM (1986) Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophrenia Bulletin 12, 356–358.Google Scholar
Bedi, G, et al. (2015) Automated analysis of free speech predicts psychosis onset in high-risk youths. NPJ Schizophrenia. doi: 10.1038/npjschz.2015.30.Google Scholar
Bell, MD, Bryson, GJ and Lysaker, PH (1997) Positive and negative affect recognition in schizophrenia: a comparison with substance abuse and normal control subjects. Psychiatry Research 73, 73–82.Google Scholar
Bell, M, et al. (1992) The positive and negative syndrome scale and the brief psychiatric rating scale: reliability, comparability, and predictive validity. Journal of Nervous and Mental Diseases 180, 723–728.Google Scholar
Bell, MD, et al. (1994) Five component model of schizophrenia: assessing the factorial invariance of the PANSS. Psychiatry Research 52, 295–303.Google Scholar
Bell, MD, et al. (2010) Social attribution test--multiple choice (SAT-MC) in schizophrenia: comparison with community sample and relationship to neurocognitive, social cognitive and symptom measures. Schizophrenia Research 122, 164–171.Google Scholar
Bleuler, E (1911) Dementia praecox or the Group of Schizophrenia. (Zinkin, J., Trans.) New York: International Universities Press.Google Scholar
Buck, B and Penn, DL (2015) Lexical characteristics of emotional narratives in schizophrenia: relationships with symptoms, functioning, and social cognition. Journal of Nervous and Mental Diseases 203, 702–708.Google Scholar
Buck, B, Minor, KS and Lysaker, PH (2015) Differential lexical correlates of social cognition and metacognition in schizophrenia: a study of spontaneously-generated life narratives. Comprehensive Psychiatry 58, 138–145.Google Scholar
Cohen, AS and Elvevag, B (2014) Automated computerized analysis of speech in psychiatric disorders. Current Opinion in Psychiatry 27, 203–209.Google Scholar
Cohen, AS, et al. (2017) Can RDoC help find order in thought disorder? Schizophrenia Bulletin 43, 503–508.Google Scholar
Corcoran, R, Mercer, G and Frith, C (1995) Schizophrenia, symptomatology and social inference: investigating theory of mind in people with schizophrenia. Schizophrenia Research 24, 397–405.Google Scholar
Crossley, SA, Salsbury, T and McNamara, DS (2009) Measuring L2 lexical growth using hypernymic relationships. Language Learning 59, 307–334.Google Scholar
De Herdt, A, et al. (2013) Neurocognition in clinical high risk young adults who did or did not convert to a first schizophrenic psychosis: a meta-analysis. Schizophrenia Research 149, 48–55.Google Scholar
Ditman, T and Kuperberg, G (2010) Building coherence: a framework for exploring the breakdown of links across clause boundaries in schizophrenia. Journal of Neurolinguistics 23, 254–269.Google Scholar
Docherty, NM (2005) Cognitive impairments and disordered speech in schizophrenia: thought disorder, disorganization, and communication failure perspectives. Journal of Abnormal Psychology 114, 269–278.Google Scholar
Docherty, NM (2012) On identifying the processes underlying schizophrenic speech disorder. Schizophrenia Bulletin 38, 1327–1335.Google Scholar
Docherty, NM, Miller, TN and Lewis, MA (1997) Communication disturbances in the natural speech of schizophrenic patients and non-schizophrenic parents of patients. Acta Psychiatrica Scandinavica 95, 500–507.Google Scholar
Docherty, NM, et al. (1996) Working memory, attention, and communication disturbances in schizophrenia. Journal of Abnormal Psychology 105, 212–219.Google Scholar
Docherty, NM, et al. (2013) Effects of social cognitive impairment on speech disorder in schizophrenia. Schizophrenia Bulletin 39, 608–616.Google Scholar
Eack, SM, et al. (2010) Assessing social-cognitive deficits in schizophrenia with the Mayer-Salovey Emotional Intelligence Test. Schizophrenia Bulletin 36, 370–380.Google Scholar
Elvevag, B, et al. (2007) Quantifying incoherence in speech: an automated methodology and novel application to schizophrenia. Schizophrenia Research 93, 304–316.Google Scholar
Elvevag, B, et al. (2010) An automated method to analyze language use in patients with schizophrenia and their first degree relatives. Journal of Neurolinguistics 23, 270–284.Google Scholar
Elvevag, B, et al. (2016) An examination of the language construct in NIMH's research domain criteria: time for reconceptualization! American Journal of Medical Genetics Part B: Neuropsychiatric Genetics 171, 904–919.Google Scholar
Fanning, JR, Bell, MD and Fiszdon, JM (2012) Is it possible to have impaired neurocognition but good social cognition in schizophrenia? Schizophrenia Research 135, 68–71.Google Scholar
Fineberg, SK, et al. (2016) Self-reference in psychosis and depression: a language marker of illness. Psychological Medicine 46, 2605–2615.Google Scholar
Firmin, R, et al. (2016) Veteran identity as a protective factor: a grounded theory comparison of perceptions of self, illness, and treatment among veterans and non-veterans with schizophrenia. American Journal of Psychiatric Rehabilitation 19, 294–314.Google Scholar
First, MB, et al. (2002) Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition (SCID-I/P). New York: Biometrics Research, New York State Psychiatric Institute.Google Scholar
Graesser, AC, McNamara, DS and Kulikowich, J (2011) Coh-Metrix: providing multilevel analyses of text characteristics. Educational Researcher 40, 223–234.Google Scholar
Green, MF (1996) What are the functional consequences of neurocognitive deficits in schizophrenia? American Journal of Psychiatry 153, 321–330.Google Scholar
Green, MF, Kern, RS and Heaton, RK (2004) Longitudinal studies of cognition and functional outcome in schizophrenia: implications for MATRICS. Schizophrenia Research 72, 41–51.Google Scholar
Green, MF, et al. (2008) Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophrenia Bulletin 34, 1211–1220.Google Scholar
Greig, TC, Bryson, GJ and Bell, MD (2004) Theory of mind performance in schizophrenia: diagnostic, symptoms, and neuropsychological correlates. Journal of Nervous and Mental Diseases 192, 12–18.Google Scholar
Gupta, T, et al. (2018) Automated analysis of written narratives reveals abnormalities in referential cohesion in youth at ultra high risk for psychosis. Schizophrenia Research 192, 82–88.Google Scholar
Hamm, JA, et al. (2012) Metacognition and social cognition in schizophrenia: stability and relationship to concurrent and prospective symptom assessments. Journal of Clinical Psychology 68, 1303–1312.Google Scholar
Harvey, PD, et al. (2000) Clinical, cognitive and functional characteristics of long-stay patients with schizophrenia: a comparison of VA and state hospital patients. Schizophrenia Research 43, 3–9.Google Scholar
Insel, TR (2014) The NIMH Research Domain Criteria (RDoC) project: precision medicine for psychiatry. American Journal of Psychiatry 171, 395–397.Google Scholar
Kay, SR, Fizszbein, A and Opler, LA (1987) The positive and negative syndrome scale for schizophrenia. Schizophrenia Bulletin 13, 261–276.Google Scholar
Lysaker, PH and Dimaggio, G (2014) Metacognitive capacities for reflection in schizophrenia: implications for developing treatments. Schizophrenia Bulletin 40, 487–491.Google Scholar
Lysaker, PH and Klion, RE (2017) Recovery, Meaning-Making, and Severe Mental Illness: A Comprehensive Guide to Metacognitive Reflection and Insight Therapy. New York: Routledge.Google Scholar
Lysaker, PH, et al. (2002) Insight and personal narratives of illness in schizophrenia. Psychiatry 65, 197–206.Google Scholar
Lysaker, PH, et al. (2005) Metacognition amidst narratives of self and illness in schizophrenia: associations with neurocognition, symptoms, insight, and quality of life. Acta Psychiatrica Scandinavica 112, 64–71.Google Scholar
Lysaker, PH, et al. (2013) Social cognition and metacognition in schizophrenia: evidence of their independence and linkage with outcomes. Acta Psychiatrica Scandinavica 127, 239–247.Google Scholar
Lysaker, PH, et al. (2018) Promoting recovery from severe mental illness: implications from research on metacognition and metacognitive reflection insight therapy. World Journal of Psychiatry 8, 1–11.Google Scholar
Maher, BA, et al. (2005) Quantitative assessment of the frequency of normal associations in the utterances of schizophrenia patients and healthy controls. Schizophrenia Research 78, 219–224.Google Scholar
Manschreck, TC, Maher, BA and Ader, DN (1981) Formal thought disorder, the type-token ratio and disturbed voluntary motor movement in schizophrenia. British Journal of Psychiatry 139, 7–15.Google Scholar
Manschreck, TC, et al. (1991) Reduced primacy and related features in schizophrenia. Schizophrenia Research 5, 35–41.Google Scholar
Manschreck, TC, et al. (2012) Frequency of normative word associations in the speech of individuals at familial high-risk for schizophrenia. Schizophrenia Research 140, 99–103.Google Scholar
Marggraf, MP, et al. (2018) Semantic coherence in psychometric schizotypy: an investigation using latent semantic analysis. Psychiatry Research 259, 63–67.Google Scholar
Mayer, JD, Salovey, P and Caruso, DR (2002) Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT): User's Manual. Toronto, ON: Multi-Health Systems, Inc.Google Scholar
McCarthy, PM and Jarvis, S (2010) MTLD, voc-D, and HD-D: a validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods 42, 381–392.Google Scholar
McCleery, A, et al. (2015) Latent structure of cognition in schizophrenia: a confirmatory factor analysis of the MATRICS consensus cognitive battery (MCCB). Psychological Medicine 45, 2657–2666.Google Scholar
McNamara, DS, et al. (2006) Validating Coh-Metrix. In Sun, R and Miyake, N (eds). Proceedings of the 28th Annual Conference of Cognitive Science Society. Austin, TX: Cognitive Science Society, pp. 573–578.Google Scholar
McNamara, DS, et al. (2014) Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge, UK: Cambridge University Press.Google Scholar
Merrill, AM, et al. (2017) Evidence that communication impairment in schizophrenia is associated with generalized poor task performance. Psychiatry Research 249, 172–179.Google Scholar
Minor, KS and Lysaker, PH (2014) Necessary, but not sufficient: links between neurocognition, social cognition, and metacognition in schizophrenia are moderated by disorganized symptoms. Schizophrenia Research 159, 198–204.Google Scholar
Minor, KS, et al. (2015 a) Lexical analysis in schizophrenia: how emotion and social word use informs our understanding of clinical presentation. Journal of Psychiatric Research 64, 74–78.Google Scholar
Minor, KS, et al. (2015 b) The impact of premorbid adjustment, neurocognition, and depression on social and role functioning in patients in an early psychosis program. Australian and New Zealand Journal of Psychiatry 49, 444–452.Google Scholar
Minor, KS, et al. (2015 c) Conceptual disorganization weakens links in cognitive pathways: disentangling neurocognition, social cognition, and metacognition in schizophrenia. Schizophrenia Research 169, 153–158.Google Scholar
Minor, KS, et al. (2016) Affective systems induce formal thought disorder in early-stage psychosis. Journal of Abnormal Psychology 125, 537–542.Google Scholar
Minor, KS, et al. (2018) Words matter: implementing the electronically activated recorder in schizotypy. Personality Disorders: Theory, Research, and Treatment 9, 133–143.Google Scholar
Moe, AM, et al. (2016) Idea density in the life-stories of people with schizophrenia: associations with narrative qualities and psychiatric symptoms. Schizophrenia Research 172, 201–205.Google Scholar
Nuechterlein, KH, et al. (2008) The MATRICS consensus cognitive battery, part 1: test selection, reliability, and validity. American Journal of Psychiatry 165, 203–213.Google Scholar
O'Leary, DS, et al. (2000) Cognitive correlates of the negative, disorganized, and psychotic symptom dimensions of schizophrenia. Journal of Neuropsychiatry & Clinical Neurosciences 12, 4–15.Google Scholar
Pinkham, AE, et al. (2003) Implications for the neural basis of social cognition for the study of schizophrenia. American Journal of Psychiatry 160, 815–824.Google Scholar
Pinkham, AE, et al. (2014) The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophrenia Bulletin 40, 813–823.Google Scholar
Semerari, A, et al. (2003) How to evaluate metacognitive functioning in psychotherapy? The metacognitive assessment scale and its applications. Clinical Psychology & Psychotherapy 10, 238–261.Google Scholar
Thorp, SR, et al. (2012) Older patients with schizophrenia: does military veteran status matter? American Journal of Geriatric Psychiatry 20, 248–256.Google Scholar
Ventura, J, Wood, RC and Hellemann, GS (2013) Symptom domains in neurocognitive functioning can help differentiate social cognitive processes in schizophrenia: a meta-analysis. Schizophrenia Bulletin 39, 102–111.Google Scholar
Ventura, J, et al. (2010) Disorganization and reality distortion in schizophrenia: a meta-analysis of the relationship between positive symptoms and neurocognitive deficits. Schizophrenia Research 121, 1–14.Google Scholar
Willits, JA, et al. (in press) Evidence of disturbances of deep levels of semantic cohesion within personal narratives in schizophrenia. Schizophrenia Research. doi: 10.1016/j.schres.2017.11.014.Google Scholar
Figure 0

Table 1. Comparing three methodologies of assessing speech disorganization in schizophrenia

Figure 1

Table 2. Means, standard deviations, and descriptions of automated analysis indices (n = 81)

Figure 2

Table 3. Relationships between disorganization using automated analysis with neurocognition, social cognition, and metacognition (n = 81)

Figure 3

Table 4. Stepwise regressions using clinician-rated (step one) and automated measures (step two) of disorganized symptoms as predictors of cognitive variables in schizophrenia (n = 81)

Figure 4

Table 5. Reverse stepwise regressions entering automated (step one) before clinician-rated measures (step two) to predict cognitive variables in schizophrenia (n = 81)