Systemizing empathy: Teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia

OFER GOLAN; SIMON BARON-COHEN

doi:10.1017/S0954579406060305

Systemizing empathy: Teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia

Published online by Cambridge University Press: 28 March 2006

OFER GOLAN and

SIMON BARON-COHEN

Show author details

OFER GOLAN: Affiliation:
Autism Research Centre, University of Cambridge
SIMON BARON-COHEN: Affiliation:
Autism Research Centre, University of Cambridge

Article contents

Abstract
Mind Reading: A Systematic Guide to Emotions
Experiment 1
Experiment 2
General Discussion
References

Rights & Permissions

Abstract

This study evaluated Mind Reading, an interactive systematic guide to emotions, for its effectiveness in teaching adults with Asperger syndrome (AS) and high-functioning autism (HFA) to recognize complex emotions in faces and voices. Experiment 1 tested a group of adults diagnosed with AS/HFA (n = 19) who used the software at home for 10–15 weeks. Participants were tested on recognition of faces and voices at three different levels of generalization. A matched control group of adults with AS/HFA (n = 22) were assessed without any intervention. In addition, a third group of general population controls (n = 24) was tested. Experiment 2 repeated the design of Experiment 1 with a group of adults with AS/HFA who used the software at home and met in a group with a tutor on a weekly basis. They were matched to a control group of adults with AS/HFA attending social skills training and to a general population control group (n = 13 for all three groups). In both experiments the intervention group improved significantly more than the control group on close, but not distant, generalization tasks. Verbal IQ had significant effects in Experiment 2. Using Mind Reading for a relatively short period of time allows users to learn to recognize a variety of complex emotions and mental states. However, additional methods are required to enhance generalization.

The first author was supported by the National Alliance for Autism Research, the Corob Charitable Trust, the Cambridge Overseas Trust and B'nai B'rith Leo Baeck scholarships. The second author was supported by the Shirley Foundation, the Medical Research Council, and the Three Guineas Trust. We are grateful to the following for helping with this study: our participants, Red Green and Blue Ltd, Jessica Kingsley Ltd, Pamela Yates for arranging and running the social skills groups, Andrea Macleod and Rob Whiskens (Autism West Midlands), Jill Howard (Autism London), Lynne Moxon (European Services for People with Autism), and Harriet Fisher and James Graham (The Interact Centre). Many thanks to Jacqueline Hill, Chris Ashwin, Sally Wheelwright, Matthew Belmonte, Yael Golan, Sarah Johnson, Emma Chapman, and Ilaria Minio Paluello. A version of this work was presented at the International Meeting For Autism Research (IMFAR), May 2004.

Information

Type: REGULAR ARTICLE
Information: Development and Psychopathology , Volume 18 , Issue 2 , June 2006 , pp. 591 - 617

DOI: https://doi.org/10.1017/S0954579406060305 [Opens in a new window]
Copyright: 2006 Cambridge University Press

The ability to recognize and understand emotions and mental states develops in the first year of life, with infants able to discriminate different emotions from facial expressions and vocalizations (Fernald, 1992; Haviland & Lelwica, 1987). Development of emotion recognition skill continues through childhood and adulthood as part of “theory of mind” (Astington, Harris, & Olson, 1988), or what is also referred to as “mindreading” (Wellman, 1992) or “empathizing” (Baron-Cohen, 2002, 2003).

It is well established that emotion recognition and mental state recognition are core difficulties in people with autism spectrum conditions (ASCs; Baron-Cohen, 1995; Hobson, 1994). These underlie the social difficulties that are diagnostic. Such difficulties have been found through cognitive, behavioral, and neuroimaging studies, and across different sensory modalities, both visual and auditory (Frith & Hill, 2004).

Because the human face is central in both the expression and communication of emotion, the majority of studies have focused on the face and tested recognition of six emotions (happiness, sadness, fear, anger, surprise, and disgust). These “basic emotions” are expressed and recognized universally (Ekman, 1993; Ekman & Friesen, 1971). Some studies reveal emotion recognition deficits among individuals with ASC, compared to typical or clinical control groups, using both static (Celani, Battacchi, & Arcidiacono, 1999; Deruelle, Rondan, Gepner, & Tardif, 2004; Macdonald et al., 1989) and dynamic stimuli (Hobson, 1986a, 1986b; Yirmiya, Sigman, Kasari, & Mundy, 1992). Other studies have found children and adults with high-functioning autism (HFA) or Asperger syndrome (AS) have no difficulties in recognizing these basic emotions from pictures (Adolphs, 2001; Grossman, Klin, Carter, & Volkmar, 2000) or films (Loveland et al., 1997), and that the deficit only becomes apparent when testing recognition of more “complex” emotions (such as embarrassment, insincerity, intimacy, etc.) in both adults and children with ASC (Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001; Baron-Cohen, Wheelwright, & Jolliffe, 1997; Baron-Cohen, Wheelwright, Spong, Scahill, & Lawson, 2001; Golan, Baron-Cohen, & Hill, in press). These findings suggest recognition of basic emotions is relatively preserved among high-functioning individuals with ASC, and that they show greater difficulties recognizing more complex emotional and mental states.

Neuroimaging studies of emotion recognition from faces reveal that people with ASC show less activation in brain regions central to face processing, such as the fusiform gyrus (Critchley et al., 2000; Pierce, Muller, Ambrose, Allen, & Courchesne, 2001; Schultz et al., 2003). Behavioral studies show that children and adults with ASCs process faces differently compared to controls: participants with ASC tend to process faces in a feature-based approach, whereas controls process faces configurally (Hobson, Ouston, & Lee, 1988; Schultz et al., 2003; Teunisse & De Gelder, 1994; Young, 1998). There is also evidence of reduced activation in brain areas that play a major role in processing of emotion, such as the amygdala, when individuals with ASC process social–emotional information (Ashwin, Baron-Cohen, Wheelwright, O'Riordan, & Bullmore, 2005; Baron-Cohen et al., 1999; Critchley et al., 2000).

Emotion recognition from voices has been studied less frequently. Here, too, there are contradictory findings in relation to recognition of basic emotions (Boucher, Lewis, & Collis, 2000; Loveland, Tunali Kotoski, Chen, & Brelsford, 1995; Loveland et al., 1997). Regarding recognition of complex emotions from voices, several studies report a deficit in performance in high-functioning adults with ASC compared to controls (Golan et al., in press; Golan, Baron-Cohen, Hill, & Rutherford, 2005; Kleinman, Marciano, & Ault, 2001; Rutherford, Baron-Cohen, & Wheelwright, 2002). To date, there are no reported neuroimaging studies assessing emotion recognition in voices in ASC, perhaps because the sound of the scanner itself (the echo-planar system) makes auditory brain scanning difficult. One study used positron emission tomography to measure brain activity of participants with ASC and matched controls while listening to theory of mind stories. Both groups showed similar patterns of brain activity, but activation in the medial frontal area of the brain was less intensive and extensive in the AS group, compared to controls (Nieminen-von Wendt et al., 2003).

Studies assessing the ability of individuals with ASC to identify emotions and mental states from context have also shown deficits relative to the general population or to other clinical control groups (Baron-Cohen, Leslie, & Frith, 1986; Fein, Lucci, Braverman, & Waterhouse, 1992). For example, adolescents and adults with ASC have difficulties answering questions on the Strange Stories Test (Happe, 1994; Jolliffe & Baron-Cohen, 1999). This test assesses the ability to provide context-appropriate mental state explanations for nonliteral statements made by story characters (e.g., ironic or sarcastic statements). When using this task in a neuroimaging study, reduced activation of the left medial prefrontal cortex was found in people with ASC compared to matched controls (Happe et al., 1996).

The integration of cross-modal emotional information from faces, voices, and context allows understanding and prediction of others' emotions and mental states (Baron-Cohen, 1995), and the more information available, the easier should be the recognition of these emotions or mental states. Studies assessing complex emotion and mental state recognition from ecologically rich social situations, containing multimodal sources of information, show a deficit in individuals with ASC, compared to controls (Golan, Baron-Cohen, Hill, & Golan, in press; Heavey, Phillips, Baron-Cohen, & Rutter, 2000; Klin, Jones, Schultz, Volkmar, & Cohen, 2002). These difficulties may be related to a failure to pick up the right emotional cues, and/or to a failure integrating them, explained by weak central coherence in the cognitive level (Frith, 1989), and underconnectivity between brain regions in the neurobiological level (Belmonte, Allen, et al., 2004; Belmonte, Cook, et al., 2004; Critchley et al., 2000). Although emotion recognition deficits in ASC are life-long, some high-functioning individuals develop compensatory strategies that allow them to recognize basic emotions. However, when recognition of more complex emotions and mental states is required, many find them hard to interpret.

In contrast to these difficulties, individuals with ASC show good and sometimes even superior skills in “systemizing” (Baron-Cohen, 2003). Systemizing is the drive to analyze or build systems, to understand and predict the behavior of nonagentive events in terms of underlying rules and regularities. Individuals with ASC are hyperattentive to detail and prefer predictable, rule-based environments, features that are intrinsic to systemizing. In addition, individuals with ASC are superior to controls on various tasks that involve searching for detail, analyzing and manipulating systems (Baron-Cohen, Richler, Bisarya, Gurunathan, & Wheelwright, 2003; Baron-Cohen, Wheelwright, Spong, et al., 2001; Jolliffe & Baron-Cohen, 1997; Lawson, Baron-Cohen, & Wheelwright, 2004; O'Riordan, Plaisted, Driver, & Baron-Cohen, 2001; Shah & Frith, 1993).

If high-functioning individuals with ASC possess good systemizing skills, it is possible they could use them to compensate for some of their empathizing difficulties. This might be hard to implement, because the socioemotional world is a context-related open system (Lawson, 2003), often unpredictable and difficult to conceptualize with strict rules. However, if provided with a system of emotions, it is plausible that systemizing skills could be harnessed to help individuals with ASC learn to recognize emotions.

Past attempts to teach emotion recognition to adults and children with ASC have either focused on the basic emotions (Hadwin, Baron-Cohen, Howlin, & Hill, 1996; Howlin, Baron-Cohen, & Hadwin, 1999), or have been part of social skills training courses, usually run in groups (Barry et al., 2003; Howlin & Yates, 1999; Rydin, Drake, & Bratt, 1999). These training programs typically do not focus specifically on systematically teaching emotion recognition, but instead address other issues, such as conversation, reducing socially inappropriate behavior, personal hygiene, and so forth. In such groups it is difficult to target the individual's specific pace of learning. Finally, such groups are socially demanding, and might therefore deter more socially anxious participants.

Other attempts to teach individuals with ASC social skills have used computer-based training (Bernard-Opitz, Sriram, & Nakhoda-Sapuan, 2001; Bolte et al., 2002; Hetzroni & Tannous, 2004; Rajendran & Mitchell, 2000; Silver & Oakes, 2001; Swettenham, 1996). The use of computer software for individuals with ASC has several advantages: first, individuals with ASC favor the computerized environment because it is predictable, consistent, and free from social demands, which they may find stressful. Second, users can work at their own pace and level of understanding. Third, lessons can be repeated over and over again, until mastery is achieved. Fourth, interest and motivation can be maintained through different and individually selected computerized rewards (Bishop, 2003; Moore, McGrath, & Thorpe, 2000; Parsons & Mitchell, 2002). Previous studies have found that the use of computers can help individuals with autism pass false belief tasks (Swettenham, 1996), recognize basic emotions from cartoons and still photographs (Bolte et al., 2002; Silver & Oakes, 2001), and solve problems in illustrated social situations (Bernard-Opitz et al., 2001). However, participants find it hard to generalize their knowledge from learnt material to related tasks.

The computer-based interventions above used drawings or photographs for training, rather than more lifelike stimuli. This might have made generalization harder than if more ecologically valid stimuli were used. In addition, the programs teaching emotion-recognition focused on basic emotions, and only on facial expressions. No reported program to date has systematically trained complex emotion recognition in both visual and auditory channels, with lifelike faces and voices. The study reported here evaluates such a program in two formats: individually based, and via a group intervention. The question tested in both was “Can the good systemizing skills that individuals with ASC possess be used to teach them to improve their recognition of complex emotions?” To test this question, emotions and mental states had to be presented as a system for learners to analyze and study.

Mind Reading: A Systematic Guide to Emotions

Mind Reading (Baron-Cohen, Golan, Wheelwright, & Hill, 2004) is an interactive guide to emotions and mental states. It is based on a taxonomic system of 412 emotions and mental states, grouped into 24 emotion groups, and six developmental levels (from age 4 to adulthood). The emotions and mental states are organized systematically, according to the emotion groups and developmental levels. Each emotion group is introduced and demonstrated by a short video clip giving some clues for later analysis of the emotions in this group. Each emotion is defined and demonstrated in six silent films of faces, six voice recordings, and six written examples of situations that evoke this emotion. The resulting library of emotional “assets” (video clips, audio clips, or brief stories) comprise 412 × 18 = 7,416 units of emotion information to learn to recognize or understand. This is therefore a rich and systematically organized set of educational material. The face videos and voice recordings comprise actors of both genders, various ages, and ethnicities, to facilitate generalization. Faces and voices are presented separately for each emotion (i.e., silent face films and faceless voice recordings) to encourage analysis of the emotion in each modality. All face video clips and voice recordings were validated by a panel of 10 independent judges, and were included in Mind Reading if at least 8 judges agreed the emotional label given described the face/voice (p < .05, binomial test).

This emotion database is accessed using three applications: (a) The emotion library allows users to browse freely through the different emotions and emotion groups, play the faces, voices and scenarios giving examples of the emotions, read stories, add their own notes, and compare different emotional expressions in the face and the voice using a scrapbook. (b) The learning center uses lessons, quizzes, and several reward collections to teach about emotions in a more structured and directive way. In addition to teaching about the 24 emotion groups, it also includes lessons and quizzes about the 20 and 100 most commonly used emotions, as well as a “build your own lesson/quiz” option. The various reward collections were chosen for their potential appeal to users with ASC and were arranged systematically (e.g., pictures and information about space elements, clips of birds arranged by families, different types of trains to collect, etc.). A reward is given when a quiz question is answered correctly on the first attempt. (c) The game zone comprises five educational games, allowing users to enjoy a game while studying about emotions. The software was created for the use of children and adults of various levels of functioning. Vocal and animated helpers give instructions on every screen. Figure 1 shows screen shots from the software (www.jkp.com/mindreading).

Screenshots from Mind Reading: The Interactive Guide to Emotions, by S. Baron-Cohen, O. Golan, S. Wheelwright, & J. J. Hill, 2004, London: Jessica Kingsley Limited. Copyright 2003 by the University of Cambridge. Reprinted with permission. [A color version of this figure can be viewed online at www.journals.cambridge.org]

The study reported here includes two experiments. Experiment 1 tested for any improvement in adults with AS/HFA in emotion recognition skills following independent use of the software, and the extent to which these users can generalize their acquired knowledge. Experiment 2 tested the same in a group of participants who, in addition to using the software individually, attended a weekly group session led by a tutor. Using group discussion, role play, worksheets, and analysis of emotions in newspapers and television programs, this group aimed to consolidate participants' computer-acquired knowledge to improve generalization. Both interventions took place over a period of 10–15 weeks, to assure a meaningful period for training, recognizing that a longer duration might lead to individuals dropping out. This allowed us to test whether the educational software (used in both experiments) has an effect independent of any human tutor or social group effects, or whether addition of a human tutor and social group enhances any effects. In both experiments, participants were tested before and after the intervention. A no-computer intervention control group of adults with AS/HFA was matched to the intervention group in both experiments. This AS/HFA control group was also tested before and after a similar period of time. In Experiment 1 the AS/HFA control group had no intervention, thereby simply controlling for the passage of time. In Experiment 2 the AS/HFA control group undertook a 10–15 week social skills course, but without using Mind Reading, thereby controlling for the specific use of this educational software. Finally, a typical control group from the general population was matched to the intervention groups in both experiments. This group was only tested once, to obtain baseline measures.

For both experiments we predicted the following:

the performance of participants with AS/HFA would be lower than typical controls on all emotion recognition tasks at Time 1;
AS/HFA groups using the software would perform better on all emotion recognition tasks at Time 2, compared to Time 1, across generalization levels; and
participants using the software would improve more than AS/HFA controls on all tasks.

In addition, the association of software usage time with improvement on the different tasks was examined for the intervention groups.

Experiment 1

This experiment compared the effect of using Mind Reading at home alone to that of taking the assessment twice with no intervention, at three levels of generalization. The need for a no-intervention AS/HFA group was to assess whether any improvement was related to the intervention or merely to taking the tasks twice.

Method

Design

A controlled trial design with multiple repeated measures was used for this experiment. Two groups, the home intervention and the AS/HFA control, were assessed twice; at Time 1 and, 10–15 weeks after, at Time 2. In each assessment, generalization was assessed using stimuli in two modalities (visual and auditory) and three generalization levels:

Close generalization: this was tested using faces and voices that were included in the computer intervention but were presented using different software, and with more challenging multiple choice answers.
Feature-based distant generalization: using faces and voices that were not included in the training, we tested the ability to transfer emotion recognition skills separately in faces and voices.
Holistic distant generalization: using scenes from feature films, this level comprised holistic socioemotional stimuli, including faces, voices, body language, and context. Participants were then asked to identify the emotion of one of the characters in the scene, thus testing the ability to transfer acquired knowledge to complex holistic stimuli. This level was only assessed at Time 2.

In addition, the two groups were compared to the typical control group at Time 1.

Participants

Three groups took part in this experiment; one AS/HFA intervention group, one AS/HFA control group, and one typical control group. Participants in the clinical groups had all been diagnosed with AS/HFA in specialist centers using established criteria (American Psychiatric Association, 1994; World Health Organization, 1994). Participants filled in the Autism Spectrum Quotient (AQ; Baron-Cohen, Wheelwright, Skinner, Martin, & Clubley, 2001), to assess their self-reported level of autistic traits. Eighty percent of the participants scored above a cutoff point of 32, which matches the percentage originally reported by Baron-Cohen et al. (2001). Eighty-eight percent of the participants scored above 26, which has recently been suggested in two separate studies as a more sensitive cutoff point for the AQ (Kurita, Koyama, & Osada, 2005; Woodbury-Smith, Robinson, Wheelwright, & Baron-Cohen, 2005). Participants were recruited from several sources, including a volunteer database, a local clinic for adults with AS/HFA, and an advert in the National Autistic Society magazine Communication. Participants were accepted to the study only if they had not participated in any related intervention during the last 3 months and had no plans for engaging in another intervention while the study was ongoing. Participants were randomly allocated into two groups:

Software home users. Nineteen participants (14 males, 5 females) were asked to use the software (provided free of charge) at home by themselves for 2 hr/week over a period of 10 weeks, a total of 20 hr. Participants were included in the study if they completed a minimum of 10 hr of work with the software. If they did not complete this minimum, participants were given an extension of up to 4 weeks to do more work with the software. Out of 24 participants originally recruited to this group, 3 withdrew during the 10-week period and 2 others were excluded at the end, as they failed to reach the 10-hr minimum. No specific pattern was found for these participants: they varied in their age range (21–43), education (3 had carried on studying beyond compulsory education, 2 had not), and employment status (2 were unemployed, 3 were employed). Their IQ, AQ, and Time 1 assessment task scores ranged within 1 SD of their group means. All of them related dropping out/not completing their work to being too busy and not getting to do the required amount of work. This finding will be discussed further.

AS/HFA control group. Twenty-two participants (17 males, 5 females) attended the assessment meetings with a 10–15-week period between them, during which they did not take part in any intervention related to emotion recognition.

Typical control group. Twenty-eight participants were recruited for this group from a local employment agency. Participants reported no psychiatric history and no occurrence of ASC in their families. After screening for autistic spectrum conditions using the AQ (Baron-Cohen, Wheelwright, Skinner, et al., 2001), four participants were excluded for scoring above the cutoff of 26. The remaining 24 participants (19 males, 5 females) attended one assessment meeting.

All participants were given the Wechsler Abbreviated Scale of Intelligence (WASI), comprising the vocabulary, similarities, block design, and matrix reasoning tests. The WASI produces verbal, performance, and full-scale IQ scores, with correlations of .88, .84, and .92 with the full Wechsler scales (Wechsler, 1999). All participants scored above 70 on both verbal and performance scales.

All participants completed the AQ (Baron-Cohen, Wheelwright, Skinner, et al., 2001). One-way analysis of variance (ANOVA) of AQ scores was significant, F (2, 62) = 81.01, p < .001. Tukey post hoc comparisons revealed that the two clinical groups scored significantly higher on the AQ compared to the typical control group. Participants were asked to report any psychiatric comorbid diagnoses. Five participants in each AS/HFA group had another psychiatric diagnosis, such as depression or attention deficit/hyperactivity disorder. No difference was found between the AS/HFA groups in the proportion of participants with psychiatric comorbidity, χ² (1) = .07, ns (see Table 1).

Means (standard deviations) and ranges of age, IQ, and AQ scores for the intervention and control groups of Experiment 1

The three groups were matched on age, verbal and performance IQ, handedness, and gender. They spanned an equivalent range of employment and educational levels. As shown in Table 1, no significant differences were found between the groups for age, F (2, 62) = 2.14, ns, verbal IQ, F (2, 62) = 2.31, ns, performance IQ, F (2, 62) = 0.53, ns, gender, χ² (2) = 0.18, ns, handedness, χ² (2) = 1.91, ns, education, χ² (2) = 0.47, ns, and employment, χ² (2) = 0.20, ns. In addition, no difference was found between the two AS/HFA groups in the time between the two assessment meetings, t (27.7) = 1.57, ns.

Instruments.

The Cambridge Mindreading (CAM) Face–Voice Battery. This battery (Golan et al., in press) tests complex emotion and mental state recognition from short silent clips of faces and from voice recordings. Fifty faces and 50 voices, taken from Mind Reading, test recognition of 20 different complex emotions and mental states (e.g., intimate, insincere). The face task comprises silent clips of adult actors, both male and female, of different ethnicities, expressing the emotions in the face. An example showing one frame from one of the clips is shown in Figure 2. The voice task comprises recordings of short sentences expressing various emotional intonations. In both tasks four adjectives were presented after each stimulus is played and participants were asked which adjective best describes how the person feels. Both tasks were run on a computer, using the experimental software DMDX (Forster & Forster, 2003). A handout of definitions of all the adjectives used in the battery was available for the participants at the beginning and through the assessment. There was no time limit for answering. The battery provides an overall facial and an overall vocal emotion recognition score (max = 50 for each of them), as well as individual scores for each of the 20 emotions assessed (pass/fail, i.e., recognized above chance or not) and an overall number of the emotions correctly recognized (max = 20). Individuals with AS/HFA have been found to score significantly lower than controls on all three scores of the battery (Golan et al., in press). Test–retest correlations, calculated for the AS/HFA control group in the current study were found to be r = .94 for the face task and r = .81 for the voice task, p < .001 for both. This battery was used for the assessment of close generalization in both faces and voices.

Examples of visual tasks from the three generalization levels: (top) close generalization adapted from the CAM Face task from Golan et al. (in press), (middle) feature-based distant generalization adapted from Reading the Mind in the Eyes Test from Baron-Cohen et al. (2001), and (bottom) holistic distant generalization adapted from Reading the Mind in Films task (Golan & Baron-Cohen 2006). Screenshot from Lost for Words, by D. Longden, 1999, London: Yorkshire Television. Copyright 1999 ITN. Reprinted with permission. [A color version of this figure can be viewed online at www.journals.cambridge.org]

Reading the Mind in the Eyes task (revised, adult version). The task (Baron-Cohen, Wheelwright, Hill, et al., 2001) has 36 items, in which participants are presented with a photograph of the eyes region of the face and must choose one of four adjectives or phrases to describe the mental state of the person pictured (see Figure 2 for an example). A definition handout is provided at the beginning of the task. In the present study, the pictures and adjectives were presented on the computer screen (using DMDX software), to avoid possible difficulties due to communication with a human examiner (Ozonoff, 1995a). Items were presented in a random order. There was no time limit for answering. Test–retest correlation, calculated for the AS/HFA control group in the current study was found to be r = .86 (p < .001). This task was used as a facial feature based generalization task.

Reading the Mind in the Voice task (revised). We used a revised version (Golan et al., 2005) of the original task (Rutherford et al., 2002). In the original task, 40 segments of speech, taken from BBC drama series, were played on audio tape to the participants, who were asked to choose one out of two possible answers, describing the speaker's mental state in each item. The task was revised as follows: each of the test items was allocated two more foils, taken from the same level: one level above or one level below the correct answer (based on the emotion taxonomy in Mind Reading). Foils were selected to match the content of the verbalizations but not the intonation, thus making the task harder to answer. This avoided ceiling effects to which the original version of the test was prone. Seven items were removed because the authors found the correct answer inappropriate to the verbalization. Eight more items were excluded after validation by a sample of 15 typically developing adults. The final task included 25 items with four possible answers for each of them. The test items were “cleaned” as far as possible and played on a computer (using DMDX software) in random order, preceded by an instruction slide and two practice items. Participants were given a definition handout before the beginning of the task. There was no time limit for answering. Test–retest correlation, calculated for the AS/HFA control group in the current study was r = .80 (p < .001). This task was used as a vocal feature based generalization task.

Reading the Mind in Film task. This task (Golan, Baron-Cohen, Hill, et al., in press) comprises 22 short social scenes taken from feature films. Each scene includes visual, vocal, and some contextual information. Scenes are presented on the computer screen (using DMDX software). Participants are presented with four adjectives and are asked to choose the one that best describes the way a target character feels at the end of the scene. Foils were selected to match some aspects of the scene (e.g., content of speech) but not all of them (e.g., facial expression, intonation, etc.), thus making the task harder to answer. A handout of definitions of all the adjectives used in the task was available for the participants at the beginning and through the assessment. There was no time limit for answering. Task score is the number of correctly recognized emotions (max = 22). Participants with AS/HFA were found to perform significantly worse on this task, comparing to matched controls. Figure 2 presents a screenshot from one of the task items. This task was used as a holistic distant generalization task and was only taken at Time 2.

Procedure

Participants were individually tested at the Autism Research Center in Cambridge. The first author and three trained assistants helped the participants through the assessments. The assistants were blind as to which group the participants belonged. Participants in the intervention group were asked to help in the evaluation of a new piece of software. It was explained they would need to commit to using Mind Reading for 2 hr/week over a period of 10 weeks, and to be assessed before and after this training period. Participants of both control groups were asked to take part in an emotion recognition study, helping to validate new tasks. For this reason, participants in the AS/HFA control group were asked to come for two assessments, separated by a 10- to 15-week period. Participants' written consent was obtained. All participants were told they were free to withdraw from the study at any time.

In the first assessment background information was collected, followed by the emotion recognition tasks. Participants were seated in front of IBM compatible computers with 15-in. monitors and were given headphones for the voice tasks. The emotion recognition tasks were presented to them in random order (with the exception of the Film task). The CAM battery included two breaks in each task due to its length. Participants were also allowed a break to freshen up between tasks. In between the tasks, two subtests of the WASI were administered. After the assessment was completed, Mind Reading was introduced to the participants of the intervention group in detail, including a presentation of the emotion taxonomy, the different areas, and a demonstration of a systematic analysis of an emotion, comparing different faces and voices to identify the unique facial/intonation features of this emotion. Participants were encouraged to analyze the facial and vocal stimuli systematically. They were asked to use the emotions library and learning center as they please, but not to use the game zone for more than a third of the usage time. They were also told Mind Reading logs their work, and they were asked to bring the log file to the second assessment meeting, for usage time verification. The whole assessment meeting took about 3 hr. During the time between the two assessments participants of the intervention group were approached by telephone at least once, to check they were still committed to the study and to working with the software.

In the second assessment participants took the same emotion recognition tasks, including the Film task and the other two WASI subtests. Task administration order was changed. The log files of participants in the intervention group were checked to verify they had used the software for the required amount of time. Average usage time was 17.5 hr (SD = 6.7, range = 10–36). These participants were then asked for their feedback about the program, their experience with it, and comments about its usefulness. Participants of the AS/HFA control group were asked about any possible effects of the first assessment meeting on their interest in emotions during the period between the assessments. All participants were then debriefed about the aims and design of the study and were rewarded with a complimentary copy of Mind Reading (or were allowed to keep the copy they used, with no charge). This assessment meeting took about 3 hr.

Results

The performance of the three groups on the emotion recognition tasks at Time 1 was explored first. Five one-way analyses of variance were conducted, testing group differences on the emotion recognition tasks used at Time 1. Using Holm's sequential rejective Bonferroni procedure (Holm, 1979; see also Zhang, Quan, Ng, & Stepanavage, 1997), significant differences were found between the groups on the CAM face task, F (2, 62) = 13.82, p < .001, voice task, F (2, 60) = 11.53, p < .001, the number of emotional concepts recognized, F (2, 60) = 12.77, p < .001, the Reading the Mind in the Eyes task, F (2, 62) = 6.10, p < .01, and the Reading the Mind in the Voice-R task, F (2, 62) = 4.92, p < .02. Preplanned comparisons with Bonferroni corrections revealed no significant differences between the two clinical groups on any of the task scores, and significantly higher scores of the typical control group on all tasks, comparing to the two AS/HFA groups. These findings support hypothesis 1. Table 2 shows the means and standard deviations of the groups' emotion recognition scores at Time 1.

Experiment 1 means (standard deviations) of the three groups on all tasks at Time 1 and Time 2

Next, five multivariate analyses of covariance (MANCOVA) with repeated measures were conducted, to examine the differences between the intervention and AS/HFA control group on the various tasks at Time 1 and Time 2. Age, verbal, and performance IQ were used as covariates. Using Holm's sequential rejective Bonferroni procedure, significant Time × Group interactions were found on CAM faces, F_Wilks (1, 35) = 11.82, p < .002, CAM voices, F_Wilks (1, 34) = 7.51, p < .01, CAM number of concepts recognized, F_Wilks (1, 33) = 8.38, p < .01, but not for Reading the Mind in the Eyes, F_Wilks (1, 36) = 1.46, ns, or Reading the Mind in the Voice-R, F_Wilks (1, 36) = 0.47, ns. No main effects were found significant, but the effect of the covariate verbal IQ was significant in relation to CAM voice scores, F (1, 34) = 5.11, p < .05.

Simple main effects analyses for the three CAM scores with Bonferroni corrections revealed the intervention group improved significantly from Time 1 to Time 2 on all three scores: for faces, t (17) = 5.37, p < .001; for voices, t (16) = 5.24, p < .001; for concepts recognized: t (15) = 3.96, p < .005. The AS/HFA control group scores did not change significantly from Time 1 to Time 2 on the CAM voices, t (21) = 1.43, ns, and the number of concepts recognized, t (21) = 1.25, ns, but did so on the CAM faces task, t (21) = 3.51, p < .005. However, when a t test was conducted on Time 2–Time 1 score differences, the improvement of the intervention group was significantly greater than that of the AS/HFA control group, t (38) = 3.38, p < .005 . Table 2 and Figure 3 show the mean scores of all tasks at Time 1 and Time 2.

Mean scores (with standard error bars) of the three groups on two levels of generalization for Experiment 1. [A color version of this figure can be viewed online at www.journals.cambridge.org]

Then, we computed score differences for the 20 CAM concepts (Time 2 score minus Time 1 score). A multivariate ANOVA (MANOVA) was conducted for the 20 difference scores, with group as the independent variable. The MANOVA did not yield a significant group difference beyond emotional concept, F_Wilks (20, 19) = 1.36, ns, but yielded significant individual between group effects for the concepts: grave, F (1, 38) = 5.81, p < .05, uneasy, F (1, 38) = 5.2, p < .05, lured, F (1, 38) = 6.98, p < .05, intimate, F (1, 38) = 5.70, p < .05, insincere, F (1, 38) = 6.79, p < .05, and nostalgic, F (1, 38) = 18.48, p < .001. These findings suggest that the intervention group's recognition of these emotional concepts improved more than it did in the AS/HFA control group. However, care should be taken when interpreting these findings, due to the lack of an overall effect for the MANOVA, and the multiple tests conducted under it.

Next, an analysis of the Film task scores was conducted, to test for holistic distant generalization. This task was only used at Time 2, so differences could only be measured between groups. Therefore, a one way ANOVA was conducted on the three groups' Film task scores and was significant, F (2, 62) = 7.68, p < .01. Preplanned contrasts with Bonferroni correction revealed no difference between the intervention group (M = 11.8, SD = 3.8) and the AS/HFA control group (M = 12.8, SD = 3.4) for this task, t (62) = 1.07, ns. However, the typical control group (M = 15.5, SD = 2.4) scored significantly higher than both groups, t (62) = 3.81, p < .001. These results suggest the intervention group did not perform better than the AS/HFA control group on this level of generalization.

Last, a correlation analysis was conducted between the time participants had used the software and the improvement scores of each task. Nonparametric analysis was used, due to the small group size. Significant Spearman rho correlations were found for software usage time with improvement on the CAM voice task (r_Spearman = .59, p < .01), and with Reading the Mind in Film task scores (r_Spearman = .45, p < .05). No other correlations were significant.

Discussion

In this experiment, the use of Mind Reading at home was compared to no intervention. Results showed that following 10–20 hr of using the software over a period of 10–15 weeks, users significantly improved in their ability to recognize complex emotions and mental states from both faces and voices, compared to their performance before the intervention, and relative to the control group. This finding is interesting, considering the short usage time and the large number of emotions included in the software, and because participants were not asked to study these particular emotions.

Time 1 results confirmed the marked difficulties individuals with ASC have in recognizing emotions from faces, voices, and eyes, and from social situations including the above with context and body language. These findings replicate results of previous studies (Baron-Cohen et al., 1997; Baron-Cohen, Wheelwright, Hill, et al., 2001; Golan, Baron-Cohen, & Hill, in press; Golan, Baron-Cohen, Hill, et al., in press; Heavey et al., 2000; Kleinman et al., 2001; Rutherford et al., 2002). Because no differences were found between the clinical groups at Time 1, any difference at Time 2 can be attributed to the intervention.

The intervention group improved significantly on close generalization measures, including faces, voices, and the emotions individuals with AS/HFA had particular difficulties with (Golan et al., in press). Improvement in the ability of individuals with ASC to recognize mental states such as intimate, insincere, or grave might have a positive effect on their confidence, willingness, and functioning in interpersonal situations. This, together with participants' reports of greater attention to faces and emotions, and improved eye contact, suggests that the analysis of emotions using Mind Reading allows people with ASC to improve emotion recognition skills from both faces and voices.

Improvement following the intervention was limited to close generalization tasks, that is, to faces and voices taken from Mind Reading. No difference was found between the intervention and AS/HFA control group on either feature based or holistic tasks of distant generalization. Similar findings of poor generalization have been found in studies teaching theory of mind, emotion recognition, and social skills to individuals with ASC. These findings will be discussed in more detail later. However, software usage time was positively correlated with film task scores, suggesting that the more participants used the software, the higher they scored on the distant generalization task. It is possible that a longer period of usage would have led to improved generalization among software users.

Interestingly, the control group significantly improved on the CAM face task between Time 1 and Time 2, despite having no formal intervention. This could be the result of taking the same task for a second time after a relatively brief interval. In addition, when interviewed at the end of the second assessment, many participants in this group reported greater interest in emotions following the assessment at Time 1. Therefore, it is possible that the assessment itself served as a limited short-term intervention, arousing participants' awareness of the importance of faces and emotions. This new awareness was not sufficient to cause an improvement on voices, but did allow for improvement on faces, which might suggest this domain is more easily changed through intervention. However, the improvement in this group's CAM face task scores was significantly smaller than that of the software home users' group.

Unlike our previous findings (Golan et al., in press), verbal IQ was found to have a significant effect on CAM voice task scores. The inclusion of verbal content in the voice task segments of speech (rather than just intonation) might account for this effect and for the lack of it in the face task. However, no such effect was found for Reading the Mind in the Voice (Revised), which also includes verbal content in its items. Experiment 2 was in part intended to help determine how central the role is that verbal IQ plays in the ability to recognize emotions.

Experiment 2

This experiment compared the effect of using Mind Reading with a weekly support of a tutor in group meetings to that of taking part in a 10-week social skills training course without using the software. Once again, improvement of emotion recognition skills in the three levels of generalization was the target outcome measure. This experiment allowed us to examine any extra value of using the software to that of attending group training for social skills that had less if any systematic method for teaching emotion recognition.