Participation and social policy processes

Participation in the policy process is a key to interest representation. However, participants are not equal in their access to the policy process (Yanow, Reference Yanow2000; Colebatch, Reference Colebatch2002). Approaches to policy management that prioritise participation of citizens intended to benefit from the policy take account of how formal and informal institutions mediate value conflict. They attempt to remedy the tendency of structural interests to exclude citizen and community interests (Rhodes, Reference Rhodes1997; Considine, Reference Considine2005), and are derived from reflexive understandings of the policy process and management (Fischer, Reference Fischer2003).

Specific to the research application of participatory policy management is the implication that citizen participation in policy evaluation methodology changes both the evaluation and the policy process. This approach to evaluation is adopted in constructivist evaluation methods (Lincoln and Guba, Reference Lincoln, Guba and Alkin2004; Taylor, Reference Taylor2005) to address value conflict and unequal power in policy evaluation (Fawcett and Hearn, Reference Fawcett and Hearn2004). While evaluation does not change the power differentials, constructivist evaluation methods at the least highlight exclusionary policy institutions. Participatory evaluation methods challenge the institutions that privilege other interests (Beresford, Reference Beresford2002). This paper argues that participatory research also has the potential to contribute to changing policy implementation through such practices.

These implications of participatory evaluation practice are not reflected in most current government social policy evaluation directives (e.g. HM Treasury, 2007; DOFA, 2003; Fisher, Reference Fisher2009). Although government guidelines and practices can include techniques for data collection about participant experiences, they rarely incorporate techniques for including participant voices in the governance and evaluation design process. Common constraints on evaluating government programs include short timeframes, limited evaluation funding and narrow terms of reference. These conditions encourage non-participatory methods such as data collection by, from and for professionals familiar with evaluation techniques. The further away from a conceptualised ‘general citizen’ the program participants are, the less likely it is that standard evaluation techniques have adequate reach into their experiences.

Participatory policy theory and current government evaluation practice differ due to these constraints. In this context, is it realistic for social policy researchers to attempt to persuade government agencies about the benefits of adopting participatory evaluation principles?

Participation in research about disability policy process

Disability policy legislation now contains aspirational principles for the involvement of people with disability in all levels of decision making (e.g. UNCRPD, 2006). Increasingly, social policy research seeks to actively engage people with disability and people who support them (families, workers and advocates) in research about policy questions and program evaluations (Allen, Reference Allen2005; Badham, Reference Badham2004). This is proving to be a difficult task to do well (Richardson, Reference Richardson2002). The rhetoric that researchers and officials use to support the involvement of people with disability in research on policies that affect their lives often differs from practice (Tregaskis and Goodley, Reference Tregaskis and Goodley2006; Barnes et al., Reference Barnes, Oliver and Barton2002; Clear, Reference Clear2000; Cocks and Cockram, Reference Cocks, Cockram, Ramcharan, Roberts, Grant and Borland1997; Drake, Reference Drake2002; Scott-Hill, Reference Scott-Hill2002). Unreflective uses of the terms ‘participation’ and ‘consultation’ in policy process can hide some of the unstated but influential components of participation that affect the potential for effective involvement of people with disability, particularly people with cognitive disability (Kitchin, Reference Kitchin2001; Cockburn, Reference Cockburn2005).

Past disability policy process privileged participation of officials, practitioners and families, with the effect of framing disability as a medical or individual experience (Priestley, Reference Priestley2000; Woodhams and Corby, Reference Woodhams and Corby2003; Twigg, Reference Twigg2002; Titchkosky, Reference Titchkosky2003). Instead, participatory methodologies reprioritise the voice of people with disability. The intention of the approach is to reflect the experience, needs and expectations of people with disability in the design of research, the policy process, outcomes and the service experience (Davis, Reference Davis2000; Barnes, Reference Barnes1996). The aim of the method is to empower people through the process of constructing their own knowledge, and in doing so to increase the relevance of the research (Balcazar et al., Reference Balcazar, Keys, Kaplan and Suarez-Balcazar1998; Sample, Reference Sample1996).

As well as the potential benefit to policy, practice and participants, participatory evaluation brings with it challenges. It takes longer, may require more dense qualitative approaches in addition to traditional data collection methods to obtain sufficient data to satisfy policy processes, and requires a delicate balance between stakeholders. It relies on a sure methodological and ethical footing from the outset of the research. Checks are required to ensure that a process intended to be empowering does not end up distressing, and that the contribution of people with disability is not stripped of its context in the data analysis.

This paper asks whether social researchers can introduce the method through government commissioned policy evaluation and what impact it has on policy implementation. It addresses these questions through application to the case study of the Resident Support Program in Australia.

Description of the program

The Resident Support Program (RSP) was introduced in the state of Queensland, Australia as a pilot program from 2003 to 2004 and extended thereafter. In the Australian policy context, responsibility for disability policy is shared between federal and state governments. Funding is primarily from federal government, service organisation is primarily by state government, delivery is increasingly managed through contracted services to non-government organisations and private agencies (such as the residential facilities in RSP) and residual services are provided by state or local governments (AIHW, 2006). RSP is a state government initiative, attempting to link together many parts of this disability policy context through an integrated service model.

The RSP provides support services for people with disability living in the private residential services sector. Its aim is to improve their quality of life and access to health and support services in the community, as well as improving coordination of services for clients common to the government departments involved. The program is implemented in identified private sector supported accommodation (hostels), boarding houses and aged rental accommodation facilities. People in hostels have prioritised access to RSP.

The program is jointly funded by Disability Services Queensland, responsible for the needs of people with disability, and Queensland Health, responsible for physical and mental health needs of people with disability. The program began during the government reform of the private residential sector, which included the enactment of new legislation for safety and quality compliance.

Reform of the sector in Queensland was contentious. Facility owners and managers were in open conflict with advocacy groups. Many owners and managers wanted funded support to meet their rising costs. Advocacy groups viewed the private supported accommodation sector as inappropriate for people who have significant support needs, for whom funded support services would better meet their needs. The two groups had mixed views about whether the RSP would support people living in facilities, or whether it would support the residential sector, for example by reducing the facilities’ need to employ as many staff. As such, some stakeholders were concerned that the RSP might serve to bolster an inappropriate form of housing and support for people with disability.

RSP provided three service types to support residents during the pilot. Community linking supported people in community and leisure activities, disability support services assisted people with basic self care, and key support workers linked people to services to improve their health and wellbeing. Non-government organisations were contracted to provide these services.

During the first 12 months of the pilot, 682 people used RSP services and the number of people using them increased with the maturation of the program. Almost two-thirds (63 per cent) of people who received RSP services were male. This reflects the gender balance of residents in private residential facilities. About two-thirds were aged between 33 and 65 years. Thirty-four people (5 per cent) identified as Indigenous (Fisher et al., Reference Fisher, Abelló, Robinson, Siminski and Chenoweth2005). They experienced psychiatric disability (73 per cent), physical disability (55 per cent), neurological and intellectual disability (42 per cent) and multiple disability (64 per cent).

Almost all the people who used the RSP lived in hostels. The largest service type used was community linking (18,148 hours), compared with disability support services (14,482 hours) and key support workers (6,424 hours, plus 1969 transport trips). The program cost up to Aus$546,998 per quarter, equivalent to Aus$5,315 (UK£2,500) per person per year.

The program governance structure was complex because it is a service integration initiative. The multiple agencies included two government agencies, contracted non-government and private providers and disability support organisations. The two government departments, DSQ and Health, manage the program jointly. At a central level, these departments recommend policy decisions to their Boards of Management and have operational responsibility for supporting the regional offices. During the evaluation, regional departmental offices managed the operation of the program, including the operational support and contract management with RSP providers. Local Coordinating Groups of government and RSP providers reported to the RSP Central Working Group (RSPCWG) through the regional DSQ and Home and Community Care representatives. The RSPCWG reported to the Residential Services Sector Coordinating Committee, comprising senior officers from the relevant central agency, service policy and accountability government departments and agencies. A steering committee of representatives from government, industry organisations, service providers and advocacy organisations advised the evaluation process. An internal interdepartmental committee of DSQ and Health representatives managed the evaluation contract.

Evaluation method

The government contracted a university consortium to evaluate the pilot program over 18 months from 2003 to 2005 (design three months, fieldwork ten months and analysis five months). The evaluation adopted a mixed method participatory approach to inform program improvement from process, outcomes and cost-effectiveness evaluation data (Table 1). The evaluation objectives were to evaluate the implementation of this program, the services provided to residents by the contracted support providers, residents’ perceptions of the appropriateness of these services and the impact on their quality of life, health and wellbeing, and the impact on residential facility operators and staff and other human services providers and departments. The evaluation also reviewed the cost effectiveness of the program to inform future resource allocation.

Table 1

Samples and timeframe

Note: *Replacement recruitment.

Quantitative administrative data about service users, service use and cost were supplemented with a longitudinal evaluation cohort study of 36 people who most recently entered RSP at the beginning of the first and second of three waves of data collection. They participated in three semi-structured interviews. This cohort size was chosen to be large enough to test for significant change for the cost-effectiveness analysis, to allow for urban and regional representation and to obtain a range of disability type, gender and age of residents.

Other data sources were interviews with people implementing or affected by the program (Table 1). The qualitative data in the resident and other interviews were analysed thematically to address the evaluation objectives and any other themes that emerged during the longitudinal research process. This article focuses on the participatory processes in designing and implementing the evaluation methods. Details about the methods and analysis of the entire evaluation are available in Abello et al. (Reference Abelló, Fisher, Robinson and Chenoweth2004) and Fisher et al. (Reference Fisher, Abelló, Robinson, Siminski and Chenoweth2005).

Participatory process in the evaluation design

The three-month evaluation design process itself was an important period to familiarise the government agencies with the benefits of participatory research. The design approach also provided clarity for the researchers in guiding methodological choices in the context of a program in which policy participants had conflicting expectations about both the program and the evaluation. As described above, the RSP pilot had complex program management and delivery goals, including integration, coordination and prevention. In addition, the government departments and service providers had conflicting approaches to service planning and delivery. Their service approaches ranged from implementing principles of independence and participation, care and maintenance to business models.

The researchers used the evaluation design process to demonstrate to government officials that an approach that relied on the participation of people who used the program would be a useful policy process for testing the effectiveness and improving the program implementation and outcomes. The three-month design was intended to maximise opportunities for participation in the design and establish relationships as a foundation for the participatory research. Design activities included making contact with key informants in the disability advocacy sector, distributing a design options paper and summary of the design for response from participants, visiting the research sites to establish research relationships and hear from policy participants about what they expected from the evaluation process, distributing a draft plan, receiving feedback and finalising the research plan.

The design phase demonstrated that the researchers had the experience to understand policy process, manage conflicting interests and handle confidential and sensitive material in a way that did not damage government relationships with policy participants. For example, a design question arose in a governance meeting attended by conflicting stakeholders about whether the program was intended to support the people using the program or support the facility owners. The evaluation design explicitly prioritised outcomes for residents. The heated discussion was resolved by reference to acknowledging the evaluation priority and relegating other outcomes to secondary benefits, without offending the owners. Developing this trust was not only important to government, but also allowed the researchers greater access to the ideas of stakeholders whose interests conflicted with those of other stakeholders.

From these design activities, the evaluation adopted a participatory, longitudinal and formative approach described above in the method section and in detail in Abelló et al. (Reference Abelló, Fisher, Robinson and Chenoweth2004). A participatory method was presented at the tender stage of the evaluation proposal, so the government was expecting a research design which included stakeholder involvement. The departmental working group used the researchers’ expertise in participatory methodology as a way to persuade central government agencies (Treasury and Premier's) that the participatory evaluation would be more effective for informing program improvement than more traditional experimental design with a control group and could still be used in conjunction with cost-effectiveness analysis. The government accepted the higher costs of participatory methods by absorbing them within the original budget. As a trade off, other costs such as including a control group were dropped.

Participatory process in the data collection, analysis and dissemination

Participatory mechanisms during the evaluation included formal and informal opportunities to participate in contributing to data, governance and the research process. The data collection and analysis prioritised the experiences of the people using the services. First, the data collection included interviews with people with disability and other people involved in the program. Three repeat interviews over nine months with people using the support captured people's reflections about current experiences, rather than relying on memory, which can be problematic for people with cognitive disability. It enabled comparison of changes in program use, community participation and wellbeing over time. It also privileged the participation of people using the support by ensuring they had multiple opportunities to contribute, that the interviewers went to them, that they were recompensed for their time and expertise and that the interviews were conducted in an informal and friendly manner.

Longitudinal evaluation was important for the research partnership between the researchers and the program participants. Formative evaluation built relationships with participants as the research team could provide concrete examples of program change to indicate to them that their suggestions had been addressed. The formative approach also built trust in the process. The researchers all had experience in applying participatory methods with people with disability and included a researcher with disability.

A challenge in the longitudinal activities was how to maximise participation of people using RSP without risking ethical obligations, including protecting dignity and avoiding further abuse through the research. For example, to avoid potential ethical risks in the feedback processes, choices about the format and detail about the formative findings were decided on the basis of responsiveness to the person's situation. The choices included how the feedback was provided (in person, phone and written) and content (discussion, summary and details). Generally, the feedback process and content were less formal and less detailed the more vulnerable the person, such as talking in person about how their information from the past interview was used and the implications of the formative findings on their personal circumstances. For example, people with intellectual disability who had no social networks living in hostels were not told that their social isolation was measured over the nine months, but they were asked about their relationships. In contrast, detailed feedback was available to consumer advocates on the formal governance structures discussed below.

Second, participation was facilitated through the governance of the research. The three governance groups described above (departmental internal working party, external steering committee and the DSQ Board of Management) contributed to formulating the research design and reacting to the formative research findings. The external committee included people with disability from disability advocacy groups. The iterative research process offered substantial formal and informal feedback to the governance groups after each of the three waves of fieldwork. This facilitated discussion of practices that influenced the effectiveness of the program, sharing innovations between sites and dialogue about incremental service change to better meet the needs of the people it was designed to serve. This approach affected the program implementation. Policy participants could react quickly to resolve problems and share good practice, which resulted in improvements in the experiences of people using the program. It also allowed the departments to respond to opportunities for program improvement. This was preferable to either hearing about problems and conflict in a public arena or waiting until the end of the pilot period before improving the program implementation.

Third, the research process facilitated participation through public opportunities for comment. Initial key informant discussions during the design phase framed the exploratory design discussions. People with disability could contribute through face-to-face interviews and communication via the internet, telephone and written materials. The evaluation process and findings were made available on the internet and distributed to participants. People with disability were encouraged to react to earlier results during the longitudinal process.

Policy outcomes of the program

The lives of the residents who participated in the longitudinal resident interviews at the first contact were characterised by isolation in the community, estrangement from family, detachment from the labour market, poverty and reduced mobility and little prospect that their situation would improve (Fisher et al., Reference Fisher, Abelló, Robinson, Siminski and Chenoweth2005). Through participating in the program, they increased their access to health, welfare and community services and, consequently, their quality of life improved.

The most significant benefits to residents were increased access and effectiveness of health services. Residents’ self-identified health and wellbeing improved substantially across the nine months, approaching population norms. More residents participated in education, training and voluntary activity, but not paid employment. A small number of people moved to more suitable housing with the help of RSP workers. The cost-effectiveness analysis revealed that the benefits were achieved for the relatively low level of investment (Aus$5,300 per person p.a. providing 101 hours support and 4.5 transport trips p.a.).

Community linkage played a major part in improved resident satisfaction with social participation, with most people benefiting from increased social contact and the development of broader interests. Low income and physical access continued to militate against community integration for many residents. The need for ongoing support was a key finding and places a consequent limitation to the number of people who can use the program, due to resource constraints.

Administrative arrangements evolved during the evaluation to respond to the inefficiencies of such a complex model of two government agencies, with central and regional administration, multiple providers and a small program.

The evaluation concluded that the RSP was a successful pilot from the perspective of people with disability and other participants involved. However, the program was limited by context of unsuitable housing for people with complex support needs and a shortage of mainstream and specialist services. As well as informing policy development for continuing RSP, it provided general policy lessons for other programs. The program continued after the pilot, but the model was simplified to improve service delivery.