Introduction
Although stigma has been defined as including components of knowledge, attitudes and behaviour (Thornicroft, Reference Thornicroft2006; Thornicroft et al. Reference Thornicroft, Rose, Kassam and Sartorius2007), it is often only attitudes that are measured as outcomes in intervention studies and population surveys (Thornicroft, Reference Thornicroft2006; Van Brakel, Reference Van Brakel2006; NICE, 2007). Moreover, ‘attitude’ scales often comprise knowledge- and behaviour-related items rather than disaggregating these as separate constructs. Behaviours, however, are central to discrimination and could be argued to be the most meaningful outcome from the perspective of mental health service users/consumers (Rose et al. Reference Rose, Thornicroft, Pinfold and Kassam2007; Rüsch et al. Reference Rüsch, Corrigan, Wassel, Michaels, Olschewski, Wilkniss and Batia2009). Theoretically, it is recognised as a significant, yet sometimes overlooked, component of stigma. A review by Link & Phelan (Reference Link and Phelan2009) discussed expanding previous definitions of stigma to incorporate discriminatory behaviour. Several studies have demonstrated the significance and high prevalence of discrimination experienced by service users/consumers across a range of contexts and cultures (Hamre, Dahl & Malt, Reference Hamre, Dahl and Malt1994; Angermeyer & Matschinger, Reference Angermeyer and Matschinger1997; Link et al. Reference Link, Phelan, Bresnahan, Stueve and Pescosolido1999; Rose & Lucas, Reference Rose, Lucas, Knapp, McDaid, Mossialos and Thornicroft2007; Thornicroft et al. 2009) further emphasising the importance of assessing and documenting behavioural trends.
Given the growth in interest in population level interventions, such as national campaigns aimed at reducing mental health-related stigma, (Sartorius, Reference Sartorius1997; Hocking, Reference Hocking2003; Vaughan & Hansen, Reference Vaughan and Hansen2004; Crisp et al. Reference Crisp, Gelder, Goddard and Meltzer2005; Henderson & Thornicroft, Reference Henderson and Thornicroft2009; Mental Health Commission of Canada, 2009; Myers et al. Reference Myers, Woodhouse, Whitehead, McCollam, McBryde and Pinfold2009) reliable and valid outcome measures that can assess their effectiveness are essential. Specific measures of behavioural discrimination have been developed for assessing the effectiveness of some public health campaigns revealing those interventions that are effective (Evans, Price & Blahut, Reference Evans, Price and Blahut2005) and ineffective (Guy et al. Reference Guy, Goller, Leslie, Thorpe, Grierson, Batrouney, Kennedy, Lewis, Fairley, Ginige, Zablotska and Hellard2009). In these cases, however, the range of behaviours that were targeted for change (e.g. smoking and testing for sexually transmitted infections) were narrower.
Some measures have been developed to assess stigma-related behaviour within a mental health context. For example, Corrigan measured willingness to sign a pledge against discrimination (Corrigan et al. Reference Corrigan, Edwards, Green, Diwan and Penn2001). This type of assessment, however, would not be feasible for the evaluation of an entire population. Additional research has assessed hypothetical behaviour using vignettes. Imagined behaviour, however, may not be congruent with actual behaviour as emotional response and social context may not be easily interpretable (Thornicroft et al. Reference Thornicroft, Rose, Kassam and Sartorius2007).
The lack of measurements for behavioural outcomes is a significant concern for the evaluation of anti-stigma interventions and limits our confidence in the effectiveness of such programmes. No psychometrically tested instrument to assess behavioural discrimination at the population level has been developed. The aim of this project is to adapt and improve upon a vignette-based stigma measure, The Star Social Distance Scale, to assess previous, current and intended behaviour. The original scale was developed by Shirley Star and was based on previous research into attitudes of social distance towards people with mental illness in American society and has been adapted for and used in many research studies (Star, Reference Star1952; Phillips, Reference Phillips1967). Development of a new instrument fills a gap in the existing literature and tools by advancing measures used to evaluate anti-stigma interventions and encouraging the inclusion of behavioural outcomes. The objective of this paper is to present the psychometric properties of the Reported and Intended Behaviour Scale (RIBS), a newly developed measure of mental health stigma-related behaviour, which can be used with the general public and is feasible to use with large populations.
Methods
Instrument development
Items from the Star Social Distance scale that were considered to be applicable to the general population were selected and adapted for design of the RIBS. Items that gave a premise of having had specialised experiences, such as hiring and firing or being a parent, were excluded. Strong consensus validity and comprehensibility were found, as rated by service users/consumers and international experts in the field of stigma research (the panel included seven individuals who were not authors of the paper).
Instrument design
The RIBS inquires about reported and intended behaviour among four different contexts: (1) living with, (2) working with, (3) living nearby and (4) continuing a relationship with someone with a mental health problem. To provide a frame of reference for respondents, we specify in the instructions that ‘someone with a mental health problem’ refers to ‘people seen by healthcare staff’. It is possible, however, that the term ‘mental health problems’ might be revised according to the cultural context. The choice of the term mental health problems reflects a British context and a compromise between mental illness/mental distress.
The first four items of the RIBS are designed to assess the prevalence of behaviour in each of the four contexts whereas items 5–8 ask about intended behaviour within the same contexts. Because individuals may or may not have had the opportunity to engage in the behaviours listed in items 1–4, these data are used only to assess prevalence and are not included in the final score. It was felt that both reported and intended behaviour would be important to include in this instrument as it is important to understand how reported behaviour may be associated with future (intended) behaviour. For example, someone may have lived with someone with a mental illness in the past, but perceived it negatively and therefore would refrain from engaging in that behaviour in the future. Assessing both reported and intended behaviour will allow us to distinguish this group of people from those who would report certain behaviours, feel positive about them and would be more likely to engage in those behaviours in the future. Of note, when examining only respondents who reported engaging in the behaviour currently or previously, 48, 76, 67 and 82% of individuals (living with, working with, living nearby and continuing a relationship, respectively) agreed that they would be willing to do so in the future.
Item wording was revised based on expert and layperson consultation. The terms ‘would’ and ‘could’ were both tested for comprehensibility and validity in the format ‘In the future, I would/could [engage in specific behaviour]’. Layperson interviews and expert consultation both suggested that the term ‘would’ was clearer and suggested a greater degree of intention than ‘could’. Response options also reflect consultation with experts and laypeople. Work with the pre-pilot sample suggested that the majority of respondents preferred having multiple response options. It was suggested that ‘don't know’ be added as an additional response option and this option was endorsed in Studies 1 and 3 by 6 and 7%, respectively (Table 2).
Pre-pilot
Following instrument refinement, based on expert review, we performed a pre-pilot test on a purposively selected sample of 30 laypersons in South London. This test was used specifically to clarify wording, comprehensibility and response format (Willis, Reference Willis2005).
Empirical testing of RIBS
Study 1
Following pre-pilot testing of the instrument, the RIBS was piloted using face-to-face interviews (n = 92) during September 2008 in Cambridge, England prior to a brief, locally based anti-stigma campaign carried out as part of the Time to Change campaign and preceding the national campaign that began in 2009 (Henderson & Thornicroft, Reference Henderson and Thornicroft2009). Although the instrument is designed to be a general measure of behavioural discrimination among the general public against people with mental health problems which can be applied to evaluate a range of anti-stigma interventions, the evaluation of the Time to Change campaign provided a unique opportunity for us to test this measure. Participants were recruited via a market research panel through Consumer Insight which was contracted to assist with data collection and campaign tracking. As the campaign was designed to target people in specific age and socio-economic status (SES) groups, respondents were restricted to precisely these groups, i.e. adults aged 25–45 and in SES groups B, C1 and C2 (defined as middle, lower-middl and skilled working class, respectively). All participants were residents of Cambridge and participants were chosen to be representative of the population living in Cambridge. Quotas were set to include equal distributions of males and females and target SES and age groups. Information gathered from these tests was used to further refine the instrument.
Study 2
A random selection of 60 participants from Study 1 were asked whether they would complete the survey at a later time. Thirty-seven people agreed to complete the RIBS twice, 1 week apart to assess test–retest reliability.
Finalising the mainstage instrument
Study 3
Final refinements were made to the RIBS following the pilot testing in Cambridge (Studies 1 and 2) and psychometric properties of the instrument were determined in 403 adults across England, Scotland and Wales. Quotas were set again to include equal distributions of age, sex and SES, and the sample was designed to be representative of the population in England. Participants in this sample completed the RIBS online because initial pilot testing revealed ceiling effects in a few items, suggesting a social desirability bias. Participants were recruited via an online fieldwork provider, Research Now. Research Now has access to 89 000 panel members meeting eligibility criteria. Members of the panel who met eligibility criteria were randomly selected and invited to take part in the survey via email. Due to concerns that the face-to-face format of the interviews may result in social desirability bias, especially given the sensitive content of the scale, participants in this sample completed the RIBS online and anonymously. A meta-analysis in 1999 concluded that online assessment enhances participants' perceptions of anonymity compared to interviews, which is linked to decreased social desirability distortion and increased self-disclosure (Richman et al. Reference Richman, Kiesler, Weisband and Drasgow1999).
Data analysis
Scoring
RIBS items 5–8 were scored on an ordinal scale (1–5). Items in which the respondent strongly agreed with engaging in the stated behaviour had a value of 5, whereas individuals who strongly disagreed that they could engage in the stated behaviour received 1 point. The total score for each participant was calculated by adding together the response values for items 5–8. ‘Don't know’ was coded as neutral (i.e. 3) for the purposes of determining a total score. As items 1–4 only calculate the prevalence of behaviours and respondents may or may not have engaged in those behaviours, they were not given a score value.
Statistical analysis
Each item's psychometric performance was assessed by response frequencies, internal consistency reliability using Cronbach's alpha (Cronbach, Reference Cronbach1984) and (in Study 2), retest reliability. Overall, test–retest reliability of the RIBS was also evaluated. For test–retest, a weighted kappa was performed for each item (assuming responses are ordinal). Lin's statistic (Lin, Reference Lin1989) was used to calculate the overall test–retest statistic for the entire RIBS scale using the ‘concord’ command in Stata. Analyses were carried out using Stata version 10 and SAS version 9.1.
This study was classified as exempt by the King's College London, Psychiatry, Nursing and Midwifery Research Ethics Subcommittee. All participants were given information on the study and could refuse or accept to take part in the study.
Results
Sample characteristics
The total sample of all studies included 495 distinct individuals (Study 1, n = 92; Study 2, n = 37; Study 3, n = 403). Of note, participants in Study 2 represented a subset of participants from Study 1. Participant characteristics are shown in Table 1 (i.e. biographic characteristics, socio-demographic characteristics, geographical characteristics and social contact). For all studies, the majority of respondents were white, married and working full time. Approximately half of respondents in each study said they know or have known someone with a mental illness.
Participant characteristics of study samples
*This sample is a subset of sample 1 retested at a later point.
Response frequencies for Study 3, n = 403
Feasibility and acceptability
The average time for online, self-completion of the RIBS was 1 min and 1 s (range 10.6 s–9 min 23 s). The average time for interviewer administration of the RIBS was 58 s (range 6 s–3 min 9 s). Overall, participants tended to use the full range of response options (see Table 2) although responses tended to be skewed towards agreement with the item, indicating the potential for a ceiling effect. No individual items indicated significant floor effects. There was, however, variation in the distribution of item responses and those that involved a higher degree of social contact (e.g. In the future, I would be willing to live with someone with a mental health problem) tended to have a more normally distributed response pattern; however, for items that implied less social contact (e.g. working with someone or living nearby to someone), responses tended to be more skewed towards agreement. For example, 33% of respondents agreed strongly or slightly to the statement: In the future, I would be willing to live with someone with a mental health problem, while working with someone or living nearby to someone were more commonly endorsed (61 and 62%, respectively). Interestingly, the item that inquired if one could continue a relationship with a friend who developed a mental health problem was the most highly endorsed (75%). Although this item also implies a high degree of social contact, responses may have been mediated by a higher number of people having that experience as this was the most frequently reported behaviour (44%).
Reliability
Overall test–retest reliability was 0.75 (Table 3). Item retest reliability based on a weighted kappa ranged from 0.62 to 1.0 suggesting moderate/substantial agreement between the two time points. The Cronbach's alpha that would be achieved if each item were eliminated from the scale was calculated and that value ranged from 0.72 to 0.81 in the final Study 3. The overall internal consistency, based on Cronbach's alpha among items 5–8 was 0.85. Given that the alpha for removing any of the items was lower than the overall alpha, the scale reliability would decrease if any of the items were removed. This supports keeping all items in the scale.
Item means, standard deviations and test–retest reliability
*Only items 5–8 were evaluated for mean and standard deviation as potential responses included a range of values for these items.
Discussion
The aim of this study was to develop and psychometrically evaluate a questionnaire to assess reported and intended stigmatising/discriminatory behaviours towards people with mental health problems. The RIBS was found to be a brief and feasible instrument for and could thus be easily added to an existing survey with minimal additional response burden. The RIBS demonstrated overall moderate/substantial test–retest reliability and substantial internal consistency.
The RIBS also demonstrates methodological strengths. A limitation of many studies assessing internal consistency is that the alpha score tends to decrease once it is applied to a new sample. For the RIBS, the overall alpha of 0.85 is well above the minimum threshold of 0.7; so, any decrease in subsequent samples is not likely to pose a problem. Additionally, many scales are tested on homogeneous samples that can lead to an inflated alpha. In these cases, the alphas would be expected to drop when they are applied at the population level, using a more heterogeneous sample. A second strength of this study is that the test sample was heterogeneous and population based and so the scale's scores are likely to be robust when applied at the population level.
The measurement of reported and intended behaviours together is important in that it allows for the investigation of both types of information and their relationship over time at the population level. Although we are interested in actual prevalence of certain behaviours, it is also important to contextualise behaviour through measurement of future intentions. The RIBS allows for comparison of individuals who report having a specific experience and have positive future intentions with those who had a similar experience, but do not intend to engage in that experience again. Moreover, changes in responses to these items could indicate overall population trends reflecting more recognised intergroup contact between individuals with and without mental health problems. Finally, the RIBS provides the ability to explore how trends in intended behaviour may affect the actual prevalence of such behaviours over time.
There are some limitations to the RIBS. It is possible that social desirability influenced participants. Although we noticed that the ceiling effect may have been lessened in Study 3 v. Study 2, we cannot be sure to what extent social desirability still plays a role in response patterns. Additionally, especially with exposure to interventions designed to improve the public's understanding of the prevalence of mental health, it is possible that people may broaden their definition of mental health problems and therefore respond with more positive reported or intended behaviours without shifting actual behaviour. Therefore, when using the RIBS, it might be useful to include a separate question concerning definitions and perceptions of mental health problems to anchor responses when tracking behaviour over time. Indeed, it is important to consider a wide range of measures when assessing stigma (Link and Phelan, Reference Link and Phelan2009; Evans-Lacko et al. Reference Evans-Lacko, Little, Meltzer, Rose, Rhydderch, Henderson and Thornicroft2010a, Reference Evans-Lacko, London, Little, Henderson and Thornicroftb) and including the RIBS as part of a multifaceted assessment. Therefore, this paper also emphasises the importance of using knowledge, attitude and behavioural measures together as recommended by the National Institute for Health and Clinical Excellence (Thornicroft, Reference Thornicroft2006; NICE, 2007) when evaluating anti-stigma/anti-discrimination interventions. Another limitation is that some of the behaviours reported by respondents may not always be under their control. Although we are not able to enquire about the specific circumstances of each experience in this brief measure, use of the intended behaviour questions alongside the reported behaviour questions allows for some interpretation or evaluation of the reported behaviour.
Despite these limitations, the RIBS improves on previous instruments, such as the Star Social Distance Scale, which has not been updated for more than 50 years. There is ample evidence of the negative impact of stigmatising behaviour in the community (Thornicroft, Reference Thornicroft2006), interpersonal (Rüsch et al. Reference Rüsch, Corrigan, Wassel, Michaels, Olschewski, Wilkniss and Batia2009; Thornicroft et al. Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009; Rossi et al. Reference Rossi, Ricci, Cicolini and Lasalvia2010), work (Thornicroft et al. Reference Thornicroft, Brohan, Rose, Sartorius and Leese2009; Brohan et al. Reference Brohan, Henderson, Little and Thornicroft2010; Rossi et al. Reference Rossi, Ricci, Cicolini and Lasalvia2010) and housing (Wahl, Reference Wahl1999) domains. Although there are other behaviours which may impact significantly on the lives of service users or other ways in which service users are ‘treated differently’, the RIBS focuses on indicators of reported and intended behaviour at the population level in domains that are significant in the personal lives of service users and assesses experiences that are common among the general public. Additional research might further investigate the association between RIBS responses and behavioural actions in experimental settings, for instance, via video, to further validate the RIBS. Future research which incorporates the RIBS as an evaluation tool alongside other measures could allow for a more comprehensive evaluation of the impact of anti-stigma campaigns and could also facilitate delineation of intervention mechanism by examining the interrelatedness of various knowledge, attitude and behavioural outcomes.
The assessment of behaviours may be essential to the evaluation of interventions intended to reduce stigma and/or discrimination related to mental illness, since behaviour is the outcome that often has the most impact on individuals. It is possible that mental health-related knowledge and attitudes may inform or mediate behaviours; but, it is necessary to include a measure that specifically examines behaviour in order to test these relationships. The RIBS was found to be a brief, feasible and psychometrically robust measure for assessing mental health-related reported and intended behavioural discrimination.
Acknowledgements
We are grateful to the comments and feedback throughout the development of the RIBS provided by our advisory group: Patrick Corrigan, Steven Hinshaw, Tony Jorm, Bruce Link, Jo Phelan, Vanessa Pinfold and Norman Sartorius; Time to Change evaluation team members: Ann Law, Elly Lewis-Holmes and Jillian London for their collaboration on the evaluation: Sue Baker, Maggie Gibbons, Paul Farmer, Paul Corry, Mark Davies, Katherine Crawshaw and Nichola Jones. We would also like to thank Consumer Insight for their assistance with data collection. Time to Change is England's most ambitious programme to end the discrimination faced by people with mental illness, and improve the nation's wellbeing. Mind and Rethink are leading the programmes, which is funded with £16m from the Big Lottery Fund and £4.5m from Comic Relief and evaluated by the Institute of Psychiatry at King's College, London. This study was supported by Comic Relief, The Big Lottery Fund (grant number WB/2/010250044), SHiFT (the UK Department of Health) and the Cambridgeshire and Peterborough NHS Foundation Trust.
Declaration of Interests
GT has received grants for research purposes for stigma-related research in the last 5 years from Lundbeck UK and from the National Institute for Health Research, and has acted as a consultant to the UK Office of the Chief Scientist. SEL, DR, CH and GT are part of the independent Time to Change Evaluation Team and are supported by a grant from Big Lottery, Comic Relief and Shift. The funding sources (Big Lottery, Comic Relief and Shift) had no role in the writing of this Comment. GT is funded through an NIHR Specialist Mental Health Biomedical Research Centre at the Institute of Psychiatry, King's College London and the South London and Maudsley NHS Foundation Trust. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
Appendix 1
