Introduction
Britain, as elsewhere in the developed world, has a steadily aging population, with major epidemiological implications. For dementia alone, the annual cost to the U.K. economy is predicted to escalate from £17 billion to over £50 billion in 30 years (Comas-Herrera et al., Reference Comas-Herrera2007). Substantial investment is needed in mental health services for older people – services which have suffered from ageism and therapeutic nihilism, and have been described as the Cinderella of healthcare provision (Warnes et al., Reference Warnes, Warren, Nolan, Warnes, Warren and Nolan2000). However, the socio-political force of an aging electorate is growing. Acknowledging that mental health problems are neglected in older people, the National Service Framework for Older People (NSF-OP; Department of Health, 2001) issued standards for services, but no extra resources were provided. A recent survey showed that only 17% of old age psychiatrists attributed any improvement in services to the NSF-OP (Tucker et al., Reference Tucker2007).
While national policy encourages transformation of mental health services for older people, there is a lack of empirical evidence to guide this process, partly due to scientific difficulties. The Cochrane Collaboration places the randomized controlled trial atop a hierarchy of study design, and the primacy of experimental method has been reinforced by the Medical Research Council (Campbell et al., Reference Campbell2000), the largest health research funding body in the U.K. This positivist orientation in healthcare is shown by a preponderance of research on treatments amenable to controlled study, while evidence at whole service level remains poorly developed. Donabedian (Reference Donabedian1988) formulated quality of care as a dynamic of structure, process and outcomes, and as this complexity cannot be represented by a priori outcome indicators alone, health services evaluation has evolved as a pragmatic, methodologically eclectic enterprise. However, a Department of Health review of evidence on organizational change (Iles and Sutherland, Reference Iles and Sutherland2001) found ad hoc evaluation activity with disparate dissemination. Local studies should be designed to contribute to the knowledge base and the advancement of evidence-based service development. This was the intention of the evaluation presented here.
Theory of evaluating organizational change
While the bureaucratic system of Max Weber describes the official functioning of an organization, the human relations school of Elton Mayo illuminated a complex social network outside the realm of management control (Roethlisberger and Dickson, Reference Roethlisberger and Dickson1939). Lewin (Reference Lewin1952) considered organizations as a “force field” of opposing pressures, with much energy required to disturb equilibrium. Contrasting with the conservative entities described by earlier theorists, Drucker (Reference Drucker1995) described an “age of uncertainty” of rapid cultural and technological change, with ongoing organizational development necessary for survival. The public sector, often criticized as inward-looking, has not escaped this pressure, as governments have attempted “to lift organizations from their old tramlines and on to wholly new paths” (Wheeler and Grice, Reference Wheeler and Grice2000). However, a common phenomenon is resistance to change (Plant, Reference Plant1987). Mintzberg (Reference Mintzberg1979) described the health service as a professional bureaucracy, whereby administrative authority is constrained by autonomous disciplines whose interests may be served by the status quo. According to Drucker (Reference Drucker1995, p. 81), “changing a team demands the most difficult learning imaginable: unlearning”. Senge (Reference Senge1992) described how organizations may be handicapped by “learning difficulties” at every level:
• excessive commitment of workers to their own remit rather than the collective product
• externalizing of blame
• reactive rather than proactive management
• seeing the current situation but not longer-term patterns
• lack of awareness of gradual threats to status quo
• reliance on experiential learning to the neglect of research and development.
Lewin (Reference Lewin1952) described a three-stage process of organizational change: unfreezing the existing state by promoting the new direction, moving by enacting the plan, then refreezing through reinforcement of desired behavior. Management manuals emphasize direction in organizational change, but progress is often more chaotic than an envisaged orderly sequence. Change being a process rather than an event, leaders should present a vision for getting from A to B, but like a yacht maneuvering against wind, current and tide, the precise route need not be fixed (Collins, Reference Collins1998). As health services operate in a multi-agency community environment, change is best observed through the lens of systems thinking (Iles and Sutherland, Reference Iles and Sutherland2001).
The fundamental question about a developmental intervention is: does it work? Scriven (Reference Scriven, Tyler, Gagne and Scriven1967) presented a dichotomy of summative and formative evaluation, the former measuring outcomes of program objectives, and the latter concerned with the experience of implementation. Evaluation research has shifted from naive experimentalism towards understanding the process of change (Kushner, Reference Kushner2000). Qualitative methods generate explanatory data beyond the capacity of an input-output design, but “hard” numerical evidence is preferred by policy-makers and the dominant professions, while qualitative findings are considered “soft” due to inherent subjectivity and perceived lack of methodological rigor (Morse, Reference Morse and Morse1994). Verbose reports are likely to frustrate busy managers seeking concise indicators of program effectiveness. Consequently, major program evaluation is rarely a purely qualitative pursuit.
Pursuing both measurement and meaning, evaluation researchers have led the trend in social science towards methodological pluralism. In the realist evaluation model of Pawson and Tilley (Reference Pawson and Tilley1997), the context of an intervention is not an impediment but fundamental to generating theory. Change is a product of structure and agency, and the task is to determine what works, for whom, and in what circumstances, depicted by the following configuration:
Intervention and setting
The mental health service for older people in Lambeth, an inner London borough, evolved as distinct strands. On the west side the service originated at Tooting Bec Hospital, opened in 1904 to divert the “senile mentally infirm” from standard mental institutions. Accommodating over 2,000 patients in spartan multi-storey blocks, the hospital displayed a cheeseparing functionalism, reflecting societal neglect of its stigmatized clientele. Following the administrative reforms of the NHS and Community Care Act 1990, Tooting Bec was replaced by a comprehensive, multidisciplinary community base. The service in the east of the borough was centered at the Maudsley Hospital, with a small team of nurses based in the community. Both sides of Lambeth were subsumed by the South London and the Maudsley NHS Trust, but continued to operate as separate entities, with no overall modus operandi.
Stimulated by the NSF-OP, senior management conducted a service review, concluding that substantial redevelopment was necessary. In consultation with staff, social services and service user representatives, a sustainable change program “Improving the Quality of Care for Older People in Lambeth” (IQCOL) was devised, focusing on accessibility for all sections of the community, seamless interface with other care providers, and a commitment to quality. Priorities for service improvement were distilled into six operational objectives:
1. Improve access to the service. Existing east and west Community Mental Health Trusts (CMHTs) were divided, creating four localized teams, each with a central referral point and exclusive links to general practices. Additional recruitment ensured a multidisciplinary complement in each team.
2. Create an assertive outreach capacity. This new function aimed to maintain vulnerable patients in the community, and facilitate early discharge from hospital. A support worker was recruited in each team to provide intensive input under professional supervision.
3. Guarantee a rapid response to urgent and emergency referrals. New standards stipulated that referrals classified as emergencies would be seen within four hours, and urgent cases on the same day.
4. Improve dementia assessment and treatment. This objective focused on assessing and treating people at an earlier stage of the illness.
5. Introduce systematic carer support. The Caring and Coping with Loss program (Moore et al., Reference Moore1998), based on cognitive behavioral principles, was chosen. Cohorts of carers participate in a series of eight sessions. CMHT practitioners were to be trained to provide this support to carers of patients newly diagnosed with dementia.
6. Tune the service to the needs of an ethnically diverse population. It was envisaged that policies would be formulated to ensure acceptability of the service for people of minority ethnic groups.
The steering committee (comprising the clinical director for older people, borough manager and a consultant psychiatrist) and evaluator held monthly meetings throughout the program. It was envisaged that the investment would be maintained after program completion, depending on evaluation evidence.
Evaluation design
The realist evaluation model (Pawson and Tilley, Reference Pawson and Tilley1997) gives methodological primacy to the individual interview, as it provides relatively unimpeded opportunity for participants to describe their experience of program implementation. In an idiographic cohort design, CMHT workers were interviewed six months into IQCOL, and at the end. Thematic analysis entailed the three concurrent activities of data reduction, data display and drawing conclusions described by Miles and Huberman (Reference Miles and Huberman1994), and was facilitated by N6 data analysis software (QSR International, 2002). Verisimilitude of findings was checked in feedback sessions. The interview topic guide was informed by the domains of the Work Environment Scale (WES; Moos, Reference Moos1994), which was used to measure the quality of work environment, as perceived by staff. The versatility of the WES is shown by its wide application across the occupational spectrum. A review of organizational climate instruments by Kimberly and Cook (Reference Kimberly and Cook2008, p. 15) stated: “This measure stands out as one with good face validity, good psychometric properties, and a history of use in mental health care settings.” The WES comprises ten subscales, each with scores ranging from of 0 to 9, as follows:
• Involvement
• Peer cohesion
• Supervisor support
• Autonomy
• Task orientation
• Work pressure
• Clarity
• Management control
• Innovation
• Physical comfort
Primary analysis, using SPSS statistical software (Norušis Reference Norušis2001), compared ratings from baseline to end of the program, with additional exploratory analysis of patterns in CMHTs across three time points. The WES and interviews formed an integrative conceptual framework for studying the progress and impact of IQCOL from staff perspectives.
A good outcome measure directly relates to the program, and is feasible within practice constraints. The evaluator collaborated with team administrators to develop a realistic data collection strategy. A comprehensive record of referrals included one year of baseline and two years of prospective data, providing outcome data for objectives relating to access and assessment. Assertive outreach was studied prospectively through activity journals and a focus group. Data were also collected on rapid response and carer support groups.
The program and evaluation were conducted as an iterative exercise (Figure 1), with a dual focus on process and outcomes. Findings from all sources were presented and discussed at staff feedback meetings, held at six months, one year and close of program. Finally, the borough service manager was interviewed for an overall review of IQCOL.
Evaluation schedule.
Results
Staff perspectives
In phase one, all available team members agreed to participate in qualitative interviews. The cohort was slightly reduced in phase two by staff departures and prolonged leave (see Table 1).
Qualitative cohort
The new teams were launched at the beginning of the program, and many participants saw advantages in the restructured service.
The two areas are quite different demographically, and I can see that specializing in having smaller teams means that we can focus much more on the needs of those particular areas (psychiatrist).
Nonetheless, a strong attachment to the original east and west structures persisted, with a close working relationship seen as necessary for support and efficiency.
I prefer the joint meetings because even though we are supposed to be practicing as two teams; you can't really separate because you have two patients fighting for one bed, and it's thrashing out who needs the admission the most (nurse).
The team leaders, who covered both teams on their respective sides of the borough, found their area more difficult to cover than when it operated from one base. While appreciating additional investment in the service, they expressed a cautious attitude towards IQCOL. While the evaluator's impression was that team leaders prioritized staff morale over the program agenda, SW team members felt isolated: -
Moving here we feel dumped . . . this IQCOL thing separating the team hasn't done any good (nurse).
Staff shortages raised doubts about the viability of the SW team. To ease the burden on practitioners, the west side reverted to a single referral duty system. In phase two, NW team members saw such interventions as retrograde, as a nurse argued:
It has to be one way or the other; I mean you either have separate teams or you have one team. I struggle to understand what is going on. There is never enough staff on that side and we're always having to deal with everything there as well as here. What is the point if you're still going to do everything together?
Views on IQCOL were related to age and experience. In a remarkably stable workforce, many of the nurses had worked in the local service for over 20 years. Their experience was valued by younger colleagues.
They are very knowledgeable and skilled; particularly the older ones – they know the community inside out (occupational therapist).
While an “if it ain't broke, don't fix it” attitude was expressed by older nurses, their younger counterparts appeared more receptive towards IQCOL, as did the recently trained psychologists and occupational therapists. One nurse attributed problems in the SW team to an outdated outlook:
There's always been two camps: there's the younger, newer clinicians who tend to think about their work in a different way; and some of the older staff members who have been around for a longer time, in a system that was different than the system we're trying to push now. The younger camp ended up staying north and the older camp going to the south.
Integration of the service was not a specific objective of IQCOL, but rather a tacit assumption of the redevelopment. In phase one, east and west sides remained distinct, with rivalry clearly apparent. Boundaries are important for group cohesion, but should remain permeable for communication and collaboration, rather than parapets to keep external groups at bay. In phase two, many participants spoke positively about increased contact for clinical purposes and at IQCOL meetings, encouraging identification with a pan-Lambeth service.
A high response rate was achieved for the WES, averaging 94% over three time points. From baseline to year 2, statistically significant differences arose in peer cohesion and physical comfort; with differences approaching statistical significance in work pressure, autonomy and involvement (Table 2). Multiple testing increases the chances of false positives, but a positive trend is discernable (note that for work pressure, lower ratings are favorable).
Change in mean scores from baseline to year 2
There was much concordance of WES and interview data in the differing trajectories of teams. However, WES ratings were generally higher than health worker norms reported by Moos (Reference Moos1994). Fear of detection and competitiveness between teams possibly influenced responses. While WES bias was inclined towards positive ratings, an opposite effect was possible in interviews. Beyond their peer group, team members rarely have opportunity to discuss their concerns, and unproblematic aspects of their work situation were perhaps overlooked.
Outcomes of program objectives
Access to the service
As existing record systems did not differentiate new and cross-referrals; the baseline dataset, produced from fragmentary clinical and administrative sources, presented an overestimate of referrals to the service. In consultation with the steering group, a 30% adjustment was made, informed by reliable referral data from Lewisham, a neighboring borough of similar demography. For prospective data, central referral points enabled comprehensive data collection by team administrators. Prospective data showed a modest increase over the IQCOL period, and a provisional 13% increase from baseline (Tables 3 and 4).
Total referrals
Distribution of referrals by sector
The increase in referrals was small, but stress was unevenly distributed in the system. Imbalance between east and west was inevitable because the new structure retained the existing longitudinal skew, but the disproportionately higher demand on the SW team was unanticipated, exacerbating staffing and logistic problems. Despite redistribution of workload midway through the program by transferring responsibility for some general practices and care homes, the west side continued to receive two-thirds of referrals.
Assertive outreach
This aspect of the program was reported in a previous paper (McCrae et al., Reference McCrae2008). Journal entries showed that support workers were mostly involved in social and practical interventions, with few cases identified as assertive outreach. In the focus group, attended by the four support workers and two nurses, the dominant issue was role interpretation. While the support workers disliked becoming an extra pair of hands, the nurses regarded social support as the aspect of care most appreciated by patients, as confirmed in a recent evaluation of support workers by Huxley and colleagues (Reference Huxley2009). Restrictive deployment of support workers was ostensibly about accountability, but nurses also seemed to display underlying insecurity about erosion of professional territory, as observed with role innovations elsewhere (Warne, Reference Warne2004). Two support workers left during the program and were not replaced due to financial cutbacks. Introducing assertive outreach was thus unsuccessful here, and it was perhaps unrealistic to rely on a single non-professional worker in each team to take this new model forward. The concept of assertive outreach was poorly understood, as practitioners failed to see how concentrated input might ultimately reduce demand on the service. A separate team might have been more effective, although a randomized trial by Killaspy and colleagues (Reference Killaspy2006) found little difference in outcomes between assertive outreach and standard CMHT care. As well as cost implications, specialized teams create boundary issues (Harrison and Traill, Reference Harrison and Traill2004). Further evidence is needed on application of the model in services for older people.
Rapid response
It transpired that practitioners did not identify urgent referrals, but incomplete data collected for the first six months suggested that referrals of pressing need were handled promptly. With no change to existing practice, measurement of this objective was not pursued by the evaluation.
Dementia assessment and treatment
Among dementia referrals, there was a slight increase in mean Mini-mental State Examination (MMSE; Folstein et al., Reference Folstein, Folstein and McHugh1975) scores from 18.1 in the first six months to 18.5 in the last six months of the program, although the average remained in the moderate range of cognitive decline. The mean difference of 0.45 was not statistically significant (95% confidence interval –2.08 to 2.98). With relatively minor changes to a generic CMHT model, IQCOL was unlikely to attract the increase in early referrals achieved by the nearby Croydon Memory Service (Banerjee et al., Reference Banerjee2007).
Systematic support for carers of people with dementia
Two series of the systematic support model were completed, each with eight carers. As access was restricted by lack of a sitting service for carers, closer collaboration with the voluntary sector was indicated.
Tuning the service to cultural diversity
The evaluator perceived initial defensiveness among staff (themselves ethnically mixed) towards the idea that they should develop their repertoire of cultural sensitivity. As mental health care is person-centered, Iliffe and Manthorpe (Reference Iliffe and Manthorpe2004) appealed against “pigeon-holing” people into blunt ethnic categories. Nonetheless, evidence shows underuse of mental health services by minority ethnic groups (National Institute for Mental Health, 2003), particularly where attitudes differ from the Eurocentric model of mental health. An initial priority was to improve ethnicity recording as a foundation for further work in this area. Some practitioners claimed that older people do not like being asked about ethnic status. However, recording improved during the program, and comparison with official census data indicated that the service was reaching all segments of the community.
Review by borough manager
Acknowledging the mixed fulfillment of objectives (Table 5), the borough manager asserted that IQCOL had promoted an ongoing process of quality improvement and a unified sense of purpose, while paving the way for further development.
Synthesis of program objectives
Discussion
Inferences should be drawn from this study with caution, due to limitations in design and generalizability. For a program aimed at improving quality of care, it might seem unusual that perspectives of patients were not sought. Comparison is fundamental to measuring change such as referral processes, but patients are likely to judge services on their personal encounter with a clinician, and they would have had limited awareness of the organizational interventions here. While the raison d'être must remain at the forefront of health services research, there should be clear scientific and ethical rationale for engaging patients in time-consuming research activities. GPs could have contributed useful views on the interface with primary care. In a study by Hull and colleagues (Reference Hull2002), wide variation in general practice referral rates to CMHTs was attributed to differences in the relationship between services. Such a phenomenon is likely to transcend developments in the mental health service, requiring a level of enquiry beyond the scope of this study. As Donabedian (Reference Donabedian1988) explained, quality of care is a multifaceted phenomenon, and there are many angles from which services may be judged. Here, the evaluation was concerned with the staff responsible for implementing the program: did IQCOL help or hinder practitioners in performing their roles?
The vehicle for change was the workforce, which generally adapted to the challenge, but there was variation in response. Staff on the east side embraced the restructured teams and, as in Lewin's model, refroze into new norms. The SW team did not thaw from its original position, while the initially keen NW team became disillusioned. Attitudes towards IQCOL varied by profession and age, with critics typically the more experienced nurses. Older employees tend to have found the work which best suits them, as aspirations and attainments become better matched (Campbell et al., 1976), but they may be less amenable to changes to established working practices and relationships. As Menzies (Reference Menzies1960) showed in her psychoanalytic study of nurses, a defensive posture reduces anxiety, affording a modicum of control in an unpredictable environment. Gournay (Reference Gournay2005) commented on how nurses trained in the asylum era are slow to embrace new models of practice. Here, inflexibility was displayed towards assertive outreach and support workers.
Interaction with peers appeared to be an important aspect of the job for nurses, and progression to smaller units diminishes the breadth of relationships that provide a supportive foundation for practice. Organizational theorists find that smaller teams tend to be more innovative and client-focused (Peters, Reference Peters1987), but Burns (Reference Burns2004) observed that undersized CMHTs often fail. West and colleagues (Reference West, Cooper and Robertson2001) identified three components of team quality: task effectiveness, well-being of members, and team viability. These inter-relate, as in the SW team, where a lack of critical mass led to stress, dissatisfaction and absenteeism. As a similarly small team on the east side coped well, perhaps the difficulties of the SW team were related to personality traits. If management had manipulated team membership to balance predilections for stability and creativity, the program could have proceeded more smoothly. However, disruption of valued relationships might have led to early retirements and a recruitment crisis at the worst time. With concessions to continuity, the workforce remained intact.
Consultation with staff led to various ideas being pursued by IQCOL, but the means of achieving desired outcomes were not clearly stated. This problem has arisen elsewhere, as in the evaluation of Health Action Zones (Judge and Bauld, Reference Judge and Bauld2001). The program produced concrete change in the restructuring of teams, but abstract goals were elusive. Without continual drive for change, a workforce is inclined to the comfort zone of time-honored practices. Many practitioners believed that they were already providing satisfactory rapid response and cultural sensitivity. According to Seddon (Reference Seddon2008), a continual development process is necessary in public services, with practitioner involvement crucial to devising meaningful measures of achievement. The program made some important achievements in refining and expanding the service. Despite iterative intent, IQCOL was perceived by many workers not as a participatory exercise in quality improvement, but as something done to the teams.
The evaluation achieved a high level of participation, and provided an outlet for staff frustration. However, the focus on experience arguably diverted attention from program objectives. An interventionist approach could have brought more successful outcomes. Action research is an iterative process whereby knowledge guides practice, and practice in turn builds new knowledge (Meyer, Reference Meyer, Fulop, Allen, Clarke and Black2001), but this radical departure from neutral enquiry detracts from external validity, and there are limits to responsiveness of a program within fixed resources. The steering committee intended an ongoing cycle of planning, action and review, but the extent of interaction between program and evaluation depended on the frequency of feedback meetings, which was a compromise with the time constraints of managers and practitioners. Program evaluation should provide the right information at the right time to the right people, but as Weiss (Reference Weiss and Alkin1990, p. 83) explained: “evaluation does not aim to replace decision-makers’ experience and judgement, but rather offers systematic evidence that informs experience and judgement”. She soberly remarked that, at best, an evaluation might bear influence, but will rarely be the sole basis of decisions. While IQCOL cannot be described as a fully iterative process, the evaluation alerted management to arising issues, stimulating interventional responses to relieve pressure on the SW team.
Application of the realist evaluation model explained the variable progress of IQCOL across the service, illustrating the effect of differences in work pressure, peer cohesion and viability of teams. However, teams were not isolated entities but interdependent, as shown by the collateral impact of the SW team on its NW counterpart; while this was evident in WES results, interviews elicited the mechanism. Influence on the wider environment should also be considered. CMHTs operate in an open system, interacting with primary care, social services, general hospitals and the community. In a review of organizational change, Burnes (Reference Burnes2005) found a high failure rate, suggesting that change management continues to be understood as a linear process rather than as an input to a complex system. Simultaneous investigation is the norm in program evaluation, but from a systems thinking perspective, organizations are continuously learning and developing, so that an intervention such as IQCOL is an important input to this process, but there is no finite period of change.
Conclusions
Just as the health professions have embraced evidence-based practice, service development should be informed by systematic evaluation. The evaluation described here makes a modest contribution to this goal. Integration of qualitative and quantitative methods produced a candid account of a quality improvement program, providing generalizable lessons for developing mental health services for older people, and useful methodological insights for evaluating complex interventions.
Conflict of interest declaration
Professor Sube Banerjee is clinical director for mental health services for older people in South London and the Maudsley NHS Trust.
Description of authors' roles
Both authors contributed to the study and to the writing of the paper.
Acknowledgments
The program and evaluation were funded by the Charitable Foundation of Guy's and St Thomas’ Hospitals. The authors are grateful to Marisa Silverman, Sue Prior and the staff of the mental health service for older people in Lambeth for their contributions to the study.
