EPSILON Study

The present study is part of a larger research project, the EPSILON (European Psychiatric Services: Inputs Linked to Outcome Domains and Needs) Study, a comparative cross-national cross-sectional study of the characteristics, needs, quality of life, patterns of care, associated costs and satisfaction levels of people with schizophrenia in five European countries (Reference Becker, Knapp and KnudsenBecker et al, 1999) which aims to :

1. (a) produce standardised versions of five instruments in key areas of mental health service research in five European languages (Danish, Dutch, English, Italian and Spanish), each of which was converted from the original into the other four languages;

2. (b) obtain and compare data from five regions in different European countries about social and clinical variables, characteristics of mental health care and its costs; and

3. (c) test both instrument and cross-instrument hypotheses.

Five instruments were converted for use from their original language into the other study languages by: (a) accurate translation and back-translation into/from the other languages, (b) checks of cross-cultural applicability using focus groups, and (c) assessment of instrument reliability. Fuller details on the translation and cross-cultural adaptation process are given in this supplement by Knudsen et al (Reference Knudsen, Vázquez-Barquero and Welcher2000). The five instruments were the Camberwell Assessment of Need (CAN) (Reference Phelan, Slade and ThornicroftPhelan et al, 1995), Client Service Receipt Inventory (CSRI) (Reference Beecham, Knapp, Thornicroft, Brewin and WingBeecham & Knapp, 1992), Involvement Evaluation Questionnaire (IEQ) (Reference Schene, van Wijngaarden and KoeterSchene et al, 1998), Lancashire Quality of Life Profile (LQoLP) (Reference OliverOliver, 1991; Reference Oliver, Huxley and BridgesOliver et al, 1996) and Verona Service Satisfaction Scale (VSSS) (Reference Ruggeri and Dall'AgnolaRuggeri & Dall'Agnola, 1993).

The VSSS reliability results are presented in this paper, while the results for the other scales are given in other papers in this supplement (Reference Chisholm, Knapp and KnudsenChisholm et al, 2000; Reference Gaite, Vázquez-Barquero and Arriaga ArrizabalagaGaite et al, 2000; Reference McCrone, Leese and ThornicroftMcCrone et al, 2000; Reference van Wijngaarden, Schene and Koetervan Wijngaarden et al, 2000).

Study sites

Six partners in five European countries (The Netherlands, Denmark, England, Spain and Italy) joined forces. The teams are located in Amsterdam, Copenhagen, London (Centre for Economics in Mental Health, and Section of Community Psychiatry), Santander and Verona. The criteria used to identify study centres and the key summary characteristics of each study site are given in Becker et al (Reference Becker, Knapp and Knudsen1999).

Case identification

Cases included in the study were adults aged 18-65 inclusive with an ICD-10 diagnosis of any of F20 to F25. Administrative prevalence samples of people with these diagnoses were identified either from Psychiatric Case Registers (in Copenhagen and in Verona) or from the case-loads of local specialist mental health services (in-patients, out-patients and community). Patients needed to have been in contact with mental health services during the 3-month period preceding the start of the study. Thus, an administrative prevalence sample of people with schizophrenia in contact with mental health services was used in each site as the sampling frame. Cases identified were diagnosed using the item group checklist (IGC) of the Schedule for Clinical Assessment in Neuropsychiatry (SCAN) (Reference WingWorld Health Organization, 1992). Only patients with an ICD-10 F20 research diagnosis were included in the study.

The exclusion criteria were: current residence in prison, secure residential services or hostels for long-term patients; co-existing learning disability (mental retardation), primary dementia or other severe organic disorder; and extended in-patient treatment episodes longer than one year. The numbers of patients finally included in the study varied from 52 to 107 between the five sites, with a total of 404.

Verona Service Satisfaction Scale - European Version (VSSS-EU)

The VSSS-EU was developed from the Italian VSSS-54 version, which was first translated into Danish, Dutch, English and Spanish by professional translators. The resulting four translations were then backtranslated into Italian. The back-translations were checked by the authors of the VSSS and compared with the original Italian version. Small discrepancies were examined and alterations made to the Italian version in order to preserve the precise meaning of each question, while still producing an understandable and acceptable translation into the various languages. Specific items were changed to adapt them to the context of each country's mental health system. When those changes occurred, the local researchers made a list of modifications. The next step was the focus group process, which discussed the content and the language of the translated instruments. In the light of the comments and recommendations, the instrument was then revised both in its original Italian version and in each of the four translations. On the basis of the focus groups' discussions, minor changes were made to the wording of the items, and a further change was made by deciding to assess separately the skills and behaviour of psychiatrists, psychologists, nurses and social workers, who in the previous version had been assessed in pairs (psychiatrists/psychologists, nurses/social workers) (see Appendix).

Conceptually, the items in the VSSS-EU cover seven dimensions: Overall Satisfaction, Professionals' Skills and Behaviour, Information, Access, Efficacy, Types of Intervention and Relative's Involvement. Items in the first five dimensions cover all areas belonging to Ware's taxonomy of satisfaction (Reference Ware, Snyder and WrightWare et al, 1983). The last two dimensions, on the other hand, examine domains that have not been assessed systematically in previous studies and have been specifically developed for the VSSS.

Each conceptual dimension of the VSSS-EU consists of a certain number of items that cover various aspects of satisfaction with services (see Appendix for a detailed description of each item) :

1. (a) the Overall Satisfaction dimension consists of three items which cover general aspects of satisfaction with psychiatric services;

2. (b) the Professionals' Skills and Behaviour dimension consists of 24 items which cover various aspects of satisfaction with the professionals' behaviour, such as technical skills, interpersonal skills, cooperation between service providers, respect of patients' rights, etc.; psychiatrists, psychologists, nurses and social workers are assessed in separate items;

3. (c) the Information dimension consists of three items which cover aspects related to satisfaction with information on services, disorders and therapies;

4. (d) the Access dimension consists of two items which cover aspects related to satisfaction with service location, physical layout and costs;

5. (e) the Efficacy dimension consists of eight items which cover aspects related to satisfaction with overall efficacy of the service, and service efficacy on specific aspects such as symptoms, social skills and family relationships;

6. (f) the Types of Intervention dimension consists of 17 items which cover various aspects of satisfaction with care, such as drugs prescription, response to emergency, psychotherapy, rehabilitation, domiciliary care, admissions, housing, recreational activities, work, benefits, etc.

7. (g) the Relative's Involvement dimension consists of six items which cover various aspects of the patient's satisfaction with help given to his/her closest relative, such as listening, understanding, advice, information, help coping with the patient's problems, etc.

The VSSS-EU is designed for self-administration and can be completed without prior training. In cases of cognitive deficit, severe psychopathology or low level of literacy, a research worker may assist the patient and/or the relative by reading through the items with them. Special care must be taken to guarantee confidentiality and anonymity and, in the case of assisted administration, to stress the independence of the research worker from the clinical team. Administering the questionnaire takes 20-30 minutes.

In the VSSS-EU, subjects are asked to express their overall feeling about their experience of the mental health service they have been using in the past year.

For items 1-40, satisfaction ratings are on a 5-point Likert scale (1=terrible, 2=mostly dissatisfactory, 3=mixed, 4=mostly satisfactory, 5=excellent), presented with alternate directionality to reduce stereotypic response. Items 41-54 consist of three questions: first, the subject is asked if he/she has received the specific intervention (Question A: “ Did you receive the intervention x in the last year?”). If the answer is ‘yes’, he/she is asked his/her satisfaction on a 5-point Likert scale (Question B). If the answer is ‘no’, he/she is asked Question C: “Do you think you would have liked to receive intervention x?” (6=no, 7=don't know, 8=yes). These questions permit the estimation of the subject's degree of satisfaction both with the interventions provided and with the professional's decision not to provide an intervention (if that was the case). The latter may be considered a measure of underprovision of care, from the patient's point of view.

Data management

SPSS (Version 6.1 and above) was used for data entry. Data consistency and homogeneity were ensured by the coordinating centre (London) preparing the SPSS templates for use at all participating sites; this ensured consistent data structures across sites. Subsequent cross-instrument merging and analysis was possible as the same patient/carer identifiers were used throughout the study.

Reliability assessment procedures

Reliability testing in the EPSILON project has been conducted on several levels, depending on the nature of the instruments involved and the way they were administered (interviews v. questionnaires). Details on the general methodology of the reliability study are given elsewhere in this supplement (Reference Koeter and van WijngaardenSchene et al, 2000).

As far as the VSSS-EU is concerned, the reliability tests have been performed: (a) on the VSSS-EU total mean score; (b) on the mean scores of each dimension; and (c) item by item, both for the pooled sample and across the five sites. Three kinds of reliability tests have been used: Cronbach's α (Reference CronbachCronbach, 1951), to check the internal consistency of the whole questionnaire and the different dimensions; the intraclass correlation coefficient (ICC) (Reference Bartko and CarpenterBartko & Carpenter, 1976), to evaluate test-retest reliability of the VSSS-EU total mean score and dimension mean scores; and Cohen's weighted κ (Reference CohenCohen, 1968), to evaluate test-retest reliability of single VSSS-EU items. Additional statistics estimated were the standard errors of measurement, which were obtained from the analysis of variance used to estimate the intraclass correlations.

Statistical analysis was performed using SPSS for Windows, release 7.5 (Reference NorusisNorusis, 1997), the Amsterdam α -testing program, ALPHA.EXE based on Feldt et al (Reference Feldt, Woodruff and Salih1987) and EXCEL for tests of the homogeneity of ICCs.

Internal consistency

Table 1 shows the mean scores for each VSSS-EU dimension in the various EPSILON sites and the tests of homogeneity of variance (Levene test) and of means. The latter tests are not discussed in the present study, but will be investigated in a future paper. Assuming that the standard error of measurement is the same in all countries, the reliability coefficient depends on the variance in each sample. Conversely, where standard errors of measurement do differ, sample variables will be affected, since they are composed both of variance in true scores and variation due to measurement error. Therefore, when computing reliability coefficients and testing differences between samples, differences between sites in terms of both sample variance and standard errors of measurement should be considered.

Table 1

The Verona Service Satisfaction Scale - European Version (VSSS-EU) dimensions and total means in the pooled sample and by site

Dimension	Pooled n=399		Amsterdam n=58		Copenhagen n=51		London n=83		Santander n=100		Verona n=107		Test of equality of means (P-value)	Test of equality of s.d. (P-value)
	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.	mean	s.d.
Overall satisfaction	3.83	0.79	3.90	0.80	4.04	0.79	3.45	0.67	3.79	0.84	4.01	0.72	0.05	0.27
Professionals' skills and behaviour	3.88	0.57	3.97	0.51	4.13	0.56	3.46	0.40	3.94	0.53	4.00	0.60	<0.01	0.03
Information	3.39	0.93	3.66	0.75	3.69	0.86	3.26	0.65	2.93	1.09	3.64	0.88	<0.01	<0.01
Access	3.83	0.73	3.63	0.73	4.19	0.71	3.97	0.67	3.84	0.67	3.64	0.79	<0.01	0.35
Efficacy	3.56	0.74	3.69	0.69	3.80	0.76	3.22	0.55	3.41	0.77	3.81	0.74	<0.01	0.03
Types of intervention	3.64	0.42	3.65	0.47	3.72	0.42	3.68	0.24	3.42	0.41	3.75	0.46	<0.01	<0.01
Relative's involvement	3.39	0.96	3.57	0.92	3.32	1.21	2.91	0.67	3.39	0.96	3.75	0.91	<0.01	<0.01
Total score	3.70	0.50	3.79	0.46	3.89	0.48	3.45	0.34	3.59	0.51	3.86	0.54	<0.01	0.01

The variability of the scores differed across the sites, both in the VSSS-EU total mean score and in all VSSS-EU dimensions, with the exception of Overall Satisfaction and Access; therefore the lack of homogeneity across the sites for the total mean scores and most of the dimension scores may account to some extent for the degree of variability in the reliability coefficients across the sites.

Table 2 shows the α coefficients and the test for the equality of α across the sites. Alpha coefficients indicate the degree to which items exhibit a positive correlation (internal consistency above 0.7 is considered adequate; Reference Bech, Mault and DenckerBech et al, 1993).

Table 2

Internal consistency of the Verona Service Satisfaction Scale - European Version (VSSS-EU): α coefficients (95% CI) in the pooled sample and by site

		Pooled		Amsterdam		Copenhagen		London		Santander		Verona
	No. of items	n	α	n	α	n	α	n	α	n	α	n	α	Test of equality of α (P-value)
Overall satisfaction	3	384	0.80	57	0.80	45	0.82	83	0.77	97	0.83	102	0.73	0.67
			(0.77-0.83)		(0.71-0.87)		(0.73-0.88)		(0.69-0.83)		(0.78-0.88)		(0.65-0.80)
Professionals' skills and behaviour1	16	275	0.91	38	0.90	24	0.89	75	0.85	72	0.90	66	0.91	0.58
			(0.89-0.92)		(0.89-0.95)		(0.82-0.95)		(0.80-0.89)		(0.86-0.93)		(0.88-0.94)
Information	3	342	0.72	51	0.67	40	0.60	82	0.63	72	0.79	97	0.72	0.39
			(0.68-0.76)		(0.51-0.78)		(0.37-0.75)		(0.49-0.73)		(0.71-0.85)		(0.63-0.79)
Access	2	386	0.06	53	0.16	49	0.96	83	0.08	99	0.77	102	0.29	<0.01
			(-0.11 to 0.20)		(-0.33 to 0.47)		(0.93-0.97)		(-0.33 to 0.36)		(0.68-0.83)		(0.02-0.49)
Efficacy	8	254	0.87	37	0.83	28	0.82	58	0.77	69	0.89	62	0.89	0.11
			(0.84-0.87)		(0.75-0.89)		(0.73-0.90)		(0.68-0.84)		(0.85-0.92)		(0.85-0.92)
Types of intervention	17	115	0.73	19	0.81	8	0.61	52	0.62	4	-2	32	0.77	0.71
			(0.67-0.79)		(0.69-0.90)		(0.17-0.88)		(0.48-0.74)				(0.66-0.86)
Relative's involvement1	5	298	0.89	40	0.85	26	0.93	67	0.81	92	0.91	73	0.88	0.04
			(0.87-0.91)		(0.77-0.91)		(0.88-0.97)		(0.72-0.87)		(0.87-0.94)		(0.82-0.91)
Total score	54	74	0.96	16	0.95	4	—2	33	0.93	2	—2	19	0.96	1.00
			(0.94-0.97)		(0.91-0.98)				(0.90-0.96)				(0.93-0.98)

It should first be noted that the high degree of variability in the number of cases is usually not due to missing values but to the fact that some items (i.e. items on the performances of the different professionals assessed in the VSSS, items on the relatives, items on response of the service to emergencies during the night or at weekends) are not applicable to all patients. For this reason, α values in the VSSS-EU total mean scores have been computed for a small number of cases only. However, they were always over 0.90 and were similar across the sites, with the exception of Relative's Involvement. In the VSSS-EU dimensions, α coefficients ranged from 0.60 (Information dimension, in Copenhagen) to 0.93 (Relative's Involvement dimension, in Copenhagen) and did not differ significantly across the sites. The dimension Access, which consists of just two items (Costs of Service and Physical Layout) measuring different constructs, is a special case. In this dimension, α varied greatly, ranging from 0.08 (London) to 0.96 (Copenhagen), and differed significantly across the sites. Dimensions constituted by a higher number of items are expected to have higher α values. This was true for the dimension Professionals' Skills and Behaviour, but not for the dimension Types of Intervention, which is not expected to have high internal consistency, due to the wide range of different interventions explored by the questionnaire.

On the whole, α values of the pooled sample were good, with the above-mentioned exception of the Access dimension, and ranged from 0.72 (Information) to 0.91 (Professionals' Skills and Behaviour).

Stability

The test-retest reliability was studied by considering the ratings both in each dimension (ICC) and item by item (weighted κ).

Table 3 shows the test-retest reliability for the VSSS-EU total mean score and VSSS-EU dimension scores, both in the pooled sample and at each EPSILON site. ICCs and the standard errors of measurements are reported.

Table 3

Test-retest reliability of Verona Service Satisfaction Scale - European Version (VSSS-EU) dimensions and total means in the pooled sample and by site

Sub-scale	Pooled		Amsterdam		Copenhagen		London		Santander		Verona		Test of equality of ICCs
	n=289		n=49		n=43		n=81		n=50		n=66		(P-value)
	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m	ICC	(s.e.)m
Overall satisfaction	0.66	0.44	0.50	0.51	0.67	0.46	0.76	0.33	0.80	0.37	0.52	0.52	0.01
Professional's skills and behaviour	0.76	0.25	0.66	0.29	0.80	0.24	0.78	0.19	0.86	0.19	0.72	0.31	0.06
Information	0.75	0.14	0.49	0.52	0.75	0.42	0.66	0.37	0.84	0.44	0.76	0.41	<0.01
Access	0.56	0.47	0.56	0.48	0.51	0.49	0.73	0.33	0.43	0.51	0.51	0.55	0.05
Efficacy	0.75	0.34	0.70	0.36	0.76	0.35	0.77	0.26	0.90	0.23	0.59	0.45	<0.01
Types of intervention	0.69	0.22	0.56	0.32	0.64	0.25	0.82	0.09	0.90	0.13	0.66	0.26	<0.01
Relative's involvement	0.78	0.43	0.72	0.47	0.82	0.53	0.70	0.38	0.89	0.32	0.67	0.48	<0.01
Total score	0.82	0.20	0.73	0.24	0.85	0.19	0.82	0.15	0.93	0.13	0.76	0.25	<0.01

The VSSS-EU total mean score in the pooled sample shows a high degreee of reliability (0.82; 95% CI 0.78-0.85), although there are some differences between the sites, with all sites above 0.70 and three sites (Copenhagen, London and Santander) above 0.80. Each of the VSSS-EU dimension mean scores in the pooled sample had a degree of reliability ranging from 0.56 to 0.78. The reliability of the VSSS-EU dimension mean scores across the sites was over 0.50 in all cases, with the exception of the Access dimension in Santander (0.43) and the Information dimension in Amsterdam (0.49). Overall, there was some degree of variability in the ICC coefficients across the sites. This could be due either to differences in measurement errors between different sites, or to lack of homogeneity in the samples. The lower reliabilities tend to be associated with higher standard errors of measurement, and therefore lack of homogeneity in the samples is unlikely to be the explanation for differences in reliability; rather, the performance of the instrument itself differs.

Table 4 shows Cohen's weighted κ for both the pooled sample and the five EPSILON sites, calculated in each VSSS-EU item by taking the disagreements' weight to be equal to the square of the distances. According to Landis & Koch (Reference Landis and Koch1977), a κ coefficient of 0.2-0.4 indicates fair agreement, 0.4-0.6 indicates moderate agreement, 0.6-0.8 indicates a substantial agreement and 0.8-1.0 indicates almost perfect agreement, although alternative schemes are possible (see Reference Koeter and van WijngaardenSchene et al, 2000, this supplement).

Table 4

Test-retest reliability of individual items of the Verona Service Satisfaction Scale - European Version (VSSS-EU) for the pooled sample and in each study site. Number (percentage) of items in Cohen's weighted κ bands suggested by Landis & Koch (Reference Landis and Koch1977)

Weighted κ	Strength of agreement	Pooled n=289	Amsterdam n=49	Copenhagen n=43	London n=81	Santander n=50	Verona n=66
		n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
0.81-1.0	Almost perfect			2(3)	14 (22)	10 (16)
0.61-0.80	Substantial	31 (49)	8 (13)	23 (36)	36 (57)	33 (52)	8 (12.7)
0.41-0.60	Moderate	32 (51)	37 (59)	25 (40)	11 (17)	16 (25)	26 (41.2)
0.21-0.40	Fair		18 (29)	10 (16)	2 (3)	4 (6)	22 (34.9)
0-0.20	Slight			3 (5)			7 (11.1)

In the pooled sample, all items generated κ coefficients from 0.4 to 0.8, indicating a moderate-substantial agreement. In the various EPSILON sites the percentage of items with κ coefficients exceeding 0.4 (generally accepted as the minimum value) ranged from 54% (Verona) to 97% (London).

A paired sample t-test on the difference between test and retest for the VSSS-EU total mean scores and dimension sub-scores revealed no significant differences, both pooled across sites and at individual sites (with the exception of Access in London and Santander, where the retest values were respectively higher and lower than the time 1 values, P <0.001), thus showing no overall tendency for patients to respond more or less favourably after an interval of time ranging from 1 to 2 weeks.

Overall Satisfaction (3)

• 11. Amount of help received

• 20. Kind of services

• 21. Overall satisfaction

Professionals' Skills and Behaviour (16)

• 3a. Professionalism and competence of the psychiatrists

• 3b. Professionalism and competence of the psychologists

• 16a. Thoroughness of psychiatrists

• 16b. Thoroughness of psychologists

• 22a. Professional competence of the nursing staff

• 22b. Professional competence of social workers

• 35a. Thoroughness of nurses

• 35b. Thoroughness of social workers

• 6a. Psychiatrists' manner

• 6b. Psychologists' manner

• 5a. Ability of psychiatrists to listen and understand problems

• 5b. Ability of psychologists to listen and understand problems

• 25a. Nurses' manner

• 25b. Social workers' manner

• 28. Nurses' knowledge of patient's medical history

• 37a. Ability of nurses to listen and to understand problems

• 37b. Ability of social workers to listen and to understand problems

• 2. Behaviour and manners of reception or secretarial staff

• 33. Instruction on what to do between visits

• 18. Cooperation between service providers

• 17. Referring to general practitioner or other specialists

• 40. Opportunity of being followed up by the same professionals

• 10. Confidentiality and respect for patient's rights

• 7. Punctuality of the professionals when patient comes for an appointment

Information (3)

• 12. Explanation of specific procedures and approaches used

• 29. Information on diagnosis and prognosis

• 19. Publicity or information on mental health services which are offered

Access (2)

• 4. The appearance, comfort level and physical layout

• 8. Costs of the service

Efficacy (8)

• 9. Effectiveness of the service in attaining well-being and preventing relapses

• 1. Effectiveness of the service in helping patient deal with problems

• 24. Effectiveness of the service in helping patient improve knowledge and understanding of his/her problems

• 13. Effectiveness of the service in helping patient to relieve symptoms

• 26. Effectiveness of the service in improving the relationship between patient and relative

• 34. Effectiveness of the service in helping patient improve capacity to look after her/himself

• 31. Effectiveness of the service in helping patient establish good relationships outside family environment

• 38. Effectiveness of the service in helping patient improve abilities to work

Types of Intervention (17)

• 14. Response of the service to crisis or urgent needs during office hours

• 15. Response of the service to emergencies during the night, weekends and Bank Holidays

• 39. Help received for unexpected outcomes, discomfort or side-effects of medication

• 41. Medication prescription

• 42. Individual rehabilitation

• 43. Individual psychotherapy

• 44. Compulsory treatment in hospital

• 45. Family therapy

• 46. Living in sheltered accommodation

• 47. Participation in the recreational activities organised by mental health services

• 48. Group psychotherapy

• 49. Sheltered work

• 50. Informal admission to hospital

• 51. Practical help by the sevice at home

• 52. Help in obtaining welfare benefits or exemptions

• 53. Help to find open employment

• 54. Help from the service to join in recreational activities separate from the mental health services

Relative's Involvement (5)

• 30a. Ability of psychiatrists to listen and understand the concerns and the opinions relative may have about patient

• 30b. Ability of psychologists to listen and understand the concerns and the opinions relative may have about patient

• 23. Recommendations about how relative could help

• 32. Information to relative about diagnosis and prognosis

• 36. Effectiveness of the service in helping relative to deal better with patient's problems

• 27. Effectiveness of the service in helping relative improve his/her understanding of patient's problems