Introduction
Cognition enhancement in schizophrenia is a major public health goal and it presents a vexing challenge for drug development and psychosocial treatment. To increase the focus on cognition-enhancing drugs for schizophrenia, the National Institute of Mental Health (NIMH) sponsored the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Initiative (Green et al. Reference Green, Nuechterlein, Gold, Barch, Cohen, Essock, Fenton, Frese, Goldberg, Heaton, Keefe, Kern, Kraemer, Stover, Weinberger, Zalcman and Marder2004; Marder & Fenton, Reference Marder and Fenton2004; Buchanan et al. Reference Buchanan, Davis, Goff, Green, Keefe, Leon, Nuechterlein, Laughren, Levin, Stover, Fenton and Marder2005). A key product of this Initiative was a standard cognitive battery that can be used in clinical trials of schizophrenia to measure cognitive improvement: the MATRICS Consensus Cognitive Battery (MCCB) (Nuechterlein & Green, Reference Nuechterlein and Green2006; Kern et al. Reference Kern, Nuechterlein, Green, Baade, Fenton, Gold, Keefe, Mesholam-Gately, Mintz, Seidman, Stover and Marder2008; Nuechterlein et al. Reference Nuechterlein, Green, Kern, Baade, Barch, Cohen, Essock, Fenton, Frese, Gold, Goldberg, Heaton, Keefe, Kraemer, Mesholam-Gately, Seidman, Stover, Weinberger, Young, Zalcman and Marder2008).
An initial decision regarding the composition of the MCCB was to include social cognition as one of the domains. Managing Emotions (Branch 4) of the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT-ME) was selected as the representative measure of social cognition (Mayer et al. Reference Mayer, Salovey and Caruso2002). The MSCEIT is a standardized and fairly comprehensive measure of emotion processing, including the abilities of perceiving emotions, facilitating thought, and understanding emotions, in addition to managing emotions. The Managing Emotions branch includes two subtests, Emotion Management and Emotional Relations and its selection was based on a review of published findings for candidate social cognition tests, as well as the results from a psychometric study with schizophrenia patients (Nuechterlein et al. Reference Nuechterlein, Green, Kern, Baade, Barch, Cohen, Essock, Fenton, Frese, Gold, Goldberg, Heaton, Keefe, Kraemer, Mesholam-Gately, Seidman, Stover, Weinberger, Young, Zalcman and Marder2008). The MSCEIT-ME is designed to asses ‘the ability to be open to feelings, and to modulate them in oneself and others so as to promote personal understanding and growth’ (Mayer et al. Reference Mayer, Salovey and Caruso2002). Although management of emotions is similar to emotion regulation, this measure differs from other measures of emotion regulation that have been used in schizophrenia in that the MSCEIT-ME involves knowledge about how to effectively regulate one's own emotions. As such, it is not a direct physiological measure of regulation in the way that electroencephalography or functional magnetic resonance imaging paradigms are.
At the time the decision was made to include social cognition as a domain in the MCCB (Green et al. Reference Green, Nuechterlein, Gold, Barch, Cohen, Essock, Fenton, Frese, Goldberg, Heaton, Keefe, Kern, Kraemer, Stover, Weinberger, Zalcman and Marder2004), there were relatively few studies of social cognition in schizophrenia. Since then, social cognition has emerged as a major focus for schizophrenia research, as well as for psychopathology research more generally. Social cognition refers to those cognitive functions that are employed in socially-relevant situations (Harvey & Penn, Reference Harvey and Penn2010) and includes emotion perception, emotion regulation, social perception, mentalizing, and attributional style, as well as complex social abilities like empathy (Pinkham et al. Reference Pinkham, Penn, Green and Harvey2016). It is critically important for social and vocational outcomes in schizophrenia and it has somewhat higher correlations than non-social cognitive deficits with functional outcomes (Fett et al. Reference Fett, Viechtbauer, Dominguez, Penn, van Os and Krabbendam2011). Social cognition has become a treatment target in schizophrenia, as evidenced by energetic development of training interventions (Wolwer et al. Reference Wolwer, Frommann, Haufmann, Piaszek, Streit and Gaebel2005; Roberts & Penn, Reference Roberts and Penn2009; Horan et al. Reference Horan, Kern, Tripp, Hellemann, Wynn, Bell, Marder and Green2011), and promising pharmacological treatments (Pedersen et al. Reference Pedersen, Gibson, Rau, Salimi, Smedley, Casey, Leserman, Jarskog and Penn2011; Davis et al. Reference Davis, Lee, Horan, Clarke, McGee, Green and Marder2013; Fischer-Shofty et al. Reference Fischer-Shofty, Brune, Ebert, Shefet, Levkovitz and Shamay-Tsoory2013).
Clinical trials, especially large trials that are intended for approval by the U.S. Food and Drug Administration and regulatory agencies in other countries, are often conducted internationally. Similarly, advancements in psychosocial treatment are quickly internationally adapted and their treatment outcomes studied. The MCCB was initially published only in English. To be useful for large multi-site trials, the MCCB needed to be translated into other languages. Extending a cognitive battery to a new language involves two distinct steps: (1) translation and cultural adaptation of tests, and (2) norming the tests on community samples so that scores from different countries have the same performance metric and can be combined. The MCCB has now been translated into over 20 languages (Green et al. Reference Green, Harris and Nuechterlein2014).
For many non-social cognition tests (e.g. speed of processing, memory, and vigilance) the procedures for translation, adaptation, and norming are fairly well understood (Hambleton et al. Reference Hambleton, Merenda and Spielberger2005). The process is also quite clear for social cognition tests that involve basic processes such as identifying emotions in facial images. However, the situation for social cognition assessments that involve inference and knowledge of cultural norms is much more complex. This complexity is reflected in tests of social cognition such as the MSCEIT-ME, where participants are asked to evaluate the usefulness of different strategies of emotion management across a wide range of situations that are presented in vignettes. For example, a vignette might describe a person experiencing negative emotions at their workplace due to being passed over for promotion in favor of a less qualified colleague, and ask about the appropriateness and effectiveness of different emotion regulation strategies in this situation. The participant is asked to rate each strategy on a 5-point rating scale on how effective it would be, with the scale anchored at 1 as ‘Very Ineffective’ and at 5 as ‘Very Effective’. The most appropriate and effective solutions to social cognition problems like the one above will presumably depend on social context, and there may not be a ‘correct’ answer.
The MSCEIT-ME assessment uses a ‘consensus-based’ scoring method in which the value or score assigned to a response to a social problem is based on the proportion of a population that selects that response. For example, if 75% of the population selects a particular response as the preferred choice, then that response is subsequently assigned an equivalent proportion score for that item (e.g. 0.75 in a range of 0–1.00). The original norms of the MSCEIT-ME that generate the scores were established using a large (N = 2112) sample of English-speaking participants drawn from seven countries. Most participants were from the USA (1240), with others from South Africa (231), India (194), the Philippines (170), the UK (115), Scotland (122), and Canada (37).
Indeed, in early use of the MCCB outside of the USA, questions about cross-cultural validity of the MCCB focused mainly on the MSCEIT-ME as opposed to the other non-social cognitive tests. This concern is similar to what was seen when clinicians were surveyed about the cross-cultural adaptability of intermediate measures of functioning (Velligan et al. Reference Velligan, Rubin, Fredrick, Mintz, Nuechterlein, Schooler, Jaeger, Peters, Buller, Marder and Dube2012). These intermediate measures, such as the MSCEIT, are based on participants’ evaluation and response to situations that are presented verbally or as text.
Because of the importance of social cognition deficits in schizophrenia and other psychiatric disorders, and their increasing role as treatment targets, development of a valid international scoring system for the MSCEIT-ME test was viewed as a crucial step in multi-national adoption of the MCCB for clinical trials.
In this paper, we describe a data-driven approach to address questions about the degree to which scoring on the MSCEIT-ME is influenced by differences in cultural norms that influence the judgment of the appropriateness of social behavior. Our goal was to develop a single scoring system that can be used internationally for the MSCEIT-ME. This work was done using six different published translations of the MSCEIT-ME (Hindi, Japanese, Simplified Chinese, Russian, Spanish for Spain, Spanish for Central and South America) that were generated using the same established procedures for translation and cultural adaptation as the other tests in the MCCB. We obtained normative samples from six corresponding regions (India, Japan, China, Russia, Spain, and Central and South America), each of which was stratified by age, sex, and education according to each region's population. This broad sample of different languages and different cultural contexts provided a solid basis for evaluating the consequences of language translation and differing cultural contexts.
Methods
These analyses are based on an international sample of 1597 participants from six selected regions ranging from 204 to 350 participants per region. In each of the regions recruitment was stratified to ensure a representative sample of the age, gender and educational distribution. In all regions exclusion criteria were a diagnosis of a major psychiatric disorder, a history or current alcohol or drug abuse, a psychotic disorder in first degree relatives, and inability to understand the language of the test sufficiently to comprehend testing instructions. All participants were compensated small amounts (less than US$40) for their time and transportation costs. Local IRBs approved all protocols. There were some differences in recruitment strategies, and in the number and selection of the recruitment sites. All regions included urban and rural areas in their recruitment, with some regions specifically selecting recruitment sites in rural and urban communities to ensure that they were represented in the final sample. Most sites recruited using local advertisements as well as other strategies. The specific recruitment strategies for each region and sample sizes are provided in Table 1.
Sample and sampling characteristics.
To create an international scoring system we needed to select items from the original MSCEIT-ME that would create comparable scores regardless of language and country of origin. Consistent with this goal, we evaluated how discrepant the consensus ratings were across the different regions and excluded items that showed large differences in consensus ratings.
To quantify the degree of discrepancy between consensus ratings in different regions we decided to use a measure of discrepancy known as L1 (Manhattan norm), which is defined as the sum of the absolute differences between the percentages of choices for each of the response options. This distance metric has two key advantages: First, it directly mirrors the scoring algorithm of the MSCEIT-ME. A respondent's score on an item in the MSCEIT-ME is the percentage rating of the consensus scoring of the response option. For example, if the selected response option was also chosen by 17% of the participants that created the norms for consensus scoring, it is scored as 0.17. Hence, the L1 norm of the difference between two sets of consensus ratings is proportional to the average difference in a respondent's score if their performance on that item were to be scored based on the consensus rating of one region v. the other consensus rating of another region. Second, the L1 norm is less influenced than other discrepancy metrics by the skew of the frequency distribution of the consensus ratings within each country. That is an important feature because consensus ratings of behaviors generally show a large degree of skew in that one choice is often identified as the best by the majority of participants, and other choices are considered inappropriate and are infrequently endorsed. Third, as long as the consensus ratings are unimodal and have the same overall pattern the L1 norm also measures the variability of participants’ responses and how much any individual response on average deviates from the population mode. We computed the L1 value for each possible pairwise comparison between regions and then averaged those values. Items with the highest average discrepancies were considered for exclusion.
To capture another important aspect of the differences in response pattern between regions that is not measured by the L1 norm, we also identified and excluded items that were considered to be ‘reversed.’ We determined for each item the median response option based on the five-point rating scale from ‘very ineffective’ to ‘very effective’, i.e. 50% of the respondents considered the presented strategy that or more effective, and 50% considered it effective to that degree or less. Items were considered reversed in which, based on the median response option, one region rated the action as effective and another region rated the action as ineffective. Reversal/non-reversal can also be viewed as general measure of agreement between the shapes of the response patterns, as they are typically unimodal and their general shape can be described by the location of the mode.
Although some situations presented in the vignettes might not be appropriate across all regions, we expected that most of the items would be minimally affected by cultural and language differences based on how the original vignettes were selected. The situations presented in the MSCEIT-ME were designed to cover a wide range of social situations while still being sufficiently generic so that most people could relate to them. In addition, the original norms were established on an English speaking sample that was culturally heterogeneous – it included English speakers mainly from North American, but also from other countries (e.g. South Africa, India), who might not consider English their primary language and who were likely to have a range of ethnic identities. Thus, heavily culturally-loaded items would tend to be dropped before inclusion due to undesirable psychometric properties in the original MSCEIT-ME development process.
For scoring the items included in the international scoring system of the MSCEIT-ME, we decided to use the original consensus norms instead of establishing a new set of norms based on the seven regional norming samples for two reasons. First, the regional samples are large enough (n > 200) to determine if the consensus ratings of the different language versions differ from the original ratings, but they are small compared to the sample for the original consensus norms. Second, using the original norms and modifying the items included provides a more straightforward path to future expansion of the international MSCEIT-ME scoring method to other countries. If the international consensus ratings were to be based on an aggregate of the regional norming samples available now, adding new regions would necessitate calculation of new consensus ratings that would change the overall scoring system. Hence, it seemed desirable to anchor the international MSCEIT-ME on norms with the largest norming sample available and choose items that were as culturally neutral as possible. Finally, to confirm that the international scoring system provides a valid representation of participants’ responses, we scored the MSCEIT-ME using region-specific norms to determine whether the international version and the region-specific versions of the MSCEIT-ME resulted in similar scores for participants.
Results
When examining pairs of regions for discrepancy, no one region or pair of regions stood out as particularly distinctive. The average discrepancies ranged from 51.6 (India v. Japan) to 27.1 (India v. Latin America), with a mean of 43.22 and a standard deviation of 9.88 (Table 2). These discrepancies represent the sum absolute differences for all response options for each items, i.e. the discrepancy score of 43.22 represents an average difference of 8.64% per response option (43.22/5) between the consensus ratings for that item for that pair of regions. No pair of regions showed a discrepancy that was more than one standard deviation larger than the mean. The absence of regions that were extremely dissimilar from one another based on the original norms supports our decision to use the original norms as the basis for the international scoring system. The consensus ratings of all regions that these analyses are based on are available in online Supplement 1.
Average distances across all items.
A small number of items (3 of 29) showed a reversal. Since a reversal indicates a major conceptual difference in the evaluation of effectiveness between regions, these items were dropped from scoring for the international version without further discussion. Three additional items were dropped in a second phase, meaning a total of 6 out of 29 items were excluded in the international version of MSCEIT-ME scoring. The final decision for dropping each item was made using a consensus-based decision process based on the statistical properties of the item (the discrepancy score), and case-by-case considerations. One of the goals of this process was to generate an international scoring system that maintained the overall structure of the MSCEIT as much as possible, including all of the original vignettes. Thus, when two items from the same vignette showed some discrepancies between regions, the poorer item was considered for omission. There was only one vignette in which two items were dropped. In that case, one item was reversed and the other had an average discrepancy score which was in the top 10% of discrepancies. The final set of items omitted from scoring for the international MSCEIT-ME consists of D13, D22, D43, D44, H12, and H32.
When evaluating the performance of the international MSCEIT-ME scoring method after the removal of these six items compared with region specific norms, we found very strong associations between the performances on these two scoring methods (most correlations were r > 0.8). The only region that did not reach this threshold was Russia (r = 0.725), but we consider this degree of convergence to be still acceptable (Table 3). This pattern of strong associations indicates that the items selected for the international scoring system are sufficiently homogeneous across cultures to be used for valid comparison of scores.
Associations between international MSCEIT-ME scoring method and region specific norms
We did not identify common elements in the content of the six excluded items. The content of the excluded items was not specific to scenarios based on gender (two feature female protagonists, four male protagonists), valence of the emotion (two feature managing positive emotions, four feature managing negative emotions), or setting (two are set in a work environment, one in a family setting, and four deal with personal emotions). Details are provided in online Supplement 1.
We compared the performance of the international norms excluding the six items with the original norms and found that for the vast majority of cases the international norms reduced overall discrepancies (compare Tables 2 and 4). The only exception was for some dyads including Russia.
Average distances on the items included in the international norms.
As a further analysis we evaluated whether the relationship between other non-social cognitive measures that are part of the MCCB and social cognition measured with the MSCEIT-ME changed based on which scoring algorithm is used. While in some cases the changes are statistically significant, the effect sizes of these changes are very small. In no case does the effect size reach the threshold Cohen defines as small (q = 0.10). Details are available in online Supplement 2.
Discussion
Overall, these results show that participants evaluated strategies for the management of emotions in social situations very similarly across diverse geographical regions. Only a small proportion of items showed evidence of either major conceptual differences or differences in the degree of consensus for a response option. The international scoring version, which was created by excluding items that contributed the most variability between countries, produced scores that correlated highly with scores based on region-specific data, and provides a reliable measure of social cognition that is comparable across different populations.
The items that showed strong response variability across cultures did not show a consistent or predictable content. Several items that appeared to be culturally-loaded (e.g. normative behavior at the workplace, gender roles) did not result in strongly discrepant response patterns. This suggests greater international similarity than differences in people's ability to assess strategies for successful emotion management.
As mentioned earlier, one potential reason for the relatively culture-neutral nature of the MSCEIT-ME might be that the original English language norms of the MSCEIT-ME are based on English speaking participants from seven countries (US, Canada, UK, Scotland, India, Philippines, and South Africa) representing a broad range of different cultural backgrounds, including participants who would not consider English their first language. Thus, the items on the MSCEIT-ME were already to some extent pre-selected for absence of strong cultural effects. Additionally one of the limitations of this study is that we do not have detailed information about each participant's primary language, or more fine-grained cultural and ethnical self-identification. Similarly, the vignettes of the MSCEIT-ME explicitly ask the participant to evaluate the behavior of somebody else. Evaluating behavior from somebody else's perspective, even if those individuals are engaged in scenarios that are not commonly observed in one's own socio-cultural environment, requires the same social cognitive processes across cultures. If the vignettes are implicitly read as ‘Evaluate the behavior of this person from their perspective, even if the situation is not something you would be likely experience yourself’, then respondents can formulate their response using perspective-taking (seeing things from someone else's point of view). This ability is reinforced by the global reach of media and the internet, and at least partial familiarity with other cultures. Thus, the relative robustness of the original MSCEIT-ME, with only a limited number of items showing strong cultural variability might be partly attributable to respondents’ perspective-taking.
While we consider the MSCEIT-ME with the international scoring system to be a relatively culture-independent measurement of emotion management, certain aspects of social cognition are culture dependent. Thus, while there are some social situations and emotion management strategies that are relatively invariant across cultures, the details of appropriate social behavior in some social situations depend on the specific cultural context, and the ability to evaluate behavior for its adherence (or violation) of social norms. Thus, one could develop a social cognition measure to detect the degree to which an individual is skilled in social situations that require nuanced navigation of social norms within a specific culture.
The purpose of the international MSCEIT-ME scoring system, and the international versions of the MCCB of which it is a part, is to provide researchers with reliable and validated measurement instruments that can be used across the widest possible range of geographical locations, and yield comparable results. The international scoring system for the MSCEIT-ME is now available and included with the MCCB test kits. The results of this adaptation process suggest that versions of social cognition tests can be developed that are valid and reasonably culture independent.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/S0033291717001052
Acknowledgements
The data presented in this paper were supported, in part, by an extension to contract HHSN 278 2004 41003C from NIMH. Further, this project was made possible by the diligent work of our collaborators conducting the research on the performance of the MCCB in each included region (in alphabetical order):
China: Robert K. Heaton1, Hua Jin 1, Chuan Shi2,3,4 and Xin Yu2,3,4
1Department of Psychiatry, University of California at San Diego, CA, USA; 2Peking University Institute of Mental Health (Sixth Hospital), Beijing, China; 3National Clinical Research Center for Mental Disorders (Peking University Sixth Hospital) Beijing, China; 4Key Laboratory of Health (Peking University) Beijing China.
India: Mark Marsico1.
1Center for Observational & Real-World Research, Merck & Co., Inc., Kenilworth, USA.
Japan: Kunimasa Arima1, Yasuhiro Kaneda2, Crystal Lin3, Ichiro Sora4 and Kiyohisa Takahasi5
1National Komorokogen Hospital; 2Iwaki Clinic; 3Sakata Clinic; 4Department of Psychiatry Kobe University Graduate School of Medicine; 5Japan Foundation for Neuroscience and Mental Health.
Latin America: Delfina de Achával1,2, Salvador Rogelio Apiquian3, Rodrigo Nel Cordoba Rojas4,5, Elsa Y. Costanzo1,2 and M. Guinjoan1,2.
1Adult Psychiatry Service, FLENI Foundation, Argentina; 2Department of Psychiatry & Mental Health, University of Buenos Aires School of Medicine, Argentina; 3Arete Proyectos, Mexico City, Mexico; 4Grupo Cisne, Bogotá, Colombia; 5University of Rosario, Bogotá, Colombia.
Russia: Andrey Alexeev1, Taissia Lepilkina1, Margarita Morozova1, Yana Morozova1 and Victoria Irzhevsky1
1Moscow Research Institute of Psychiatry, Moscow, Russia.
Spain: R. Rodriguez-Jimenez1,2, A. Bagney1,2, J.L. Santos3, A. Lobo2,4 and T. Palomo2,5.
1Department of Psychiatry, Instituto de Investigación Hospital 12 de Octubre (imas12), Madrid, Spain; 2Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Spain; 3Department of Psychiatry, Hospital Virgen de La Luz, Cuenca, Spain; 4Department of Psychiatry, Universidad de Zaragoza, Instituto de Investigación Sanitaria Aragón (IIS Aragón), Spain; 5Department of Psychiatry, Universidad Complutense de Madrid, Spain.
Disclosures
The MSCEIT is published by Multi-Health Systems Inc. Development of the MATRICS Consensus Cognitive Battery was supported by the NIMH (N01MH2206).
Michael Green and Keith Nuechterlein are unpaid officers of MATRICS Assessment, Inc.; Robert Kern was formerly a paid officer of MATRICS Assessment, Inc. Gill Sitarenios is the Chief Scientist and a paid employee at Multi-Health Systems, Inc.
