Background
The isolation of older people is increasingly recognised as a social issue of considerable policy importance in contemporary Western society. While the risk factors and social or health outcomes of isolation in later life are well documented, the evidence regarding the effectiveness of programmes aimed at reducing social isolation in older people remains inconclusive. This is particularly the case for community-based programmes, the majority of which have not included systematic evaluations nor used established outcome measures.
This paper reports on the lessons from an Australian pilot study aimed at administering a formal evaluation of interventions to reduce social isolation among older people. The study involved a series of demonstration projects conducted as part of the Queensland Cross-Government Project to Reduce Social Isolation in Older People (CGPRSIOP), which was led by the Office for Seniors within the Queensland Department of Communities between 2005 and 2007. The three demonstration projects provided a variety of activities and programmes for older people at risk of isolation in different social settings. The selection of interventions was based on comprehensive literature reviews (Findlay Reference Findlay2003) and community consultations conducted in the early phase of the CGRSIOP (Department of Communities 2006, 2008).
Pre- and post-intervention surveys, including social isolation scales, were incorporated into the programmes to evaluate to what extent the interventions produced the intended result. This paper reports on the design as well as the findings of these Queensland programmes, and focuses on the challenges experienced in attempting to include formal evaluation measures in an intervention implemented by service providers rather than trained researchers. As the use of validated psychological measures to evaluate the effectiveness of community interventions is not common practice in government-run programmes, this study presents a useful account of how this was achieved and the challenges involved. Thus, the aim of this paper is to add to existing practice knowledge around how to better implement reliable evaluation techniques in community-based interventions.
The paper starts with a brief review of the evidence base of social isolation in later life. The second section provides details of the three demonstration programmes and study methodology. The third section reports on the findings of these programmes and methodological issues encountered in implementing and evaluating the interventions. In the discussion section, the policy and practice implications of these three demonstration programmes for the prevention of social isolation and loneliness in later life and the implementation of future interventions are outlined along with the challenges encountered in running this project. Finally, the conclusion summarises the main points and observations of the study.
Social isolation in later life
International research findings indicate that social isolation and loneliness are common among older people (Warburton and Lui Reference Warburton and Lui2007). In Australia, a study of veterans found that approximately 10 per cent reported feeling socially isolated with another 12 per cent at risk of isolation (Gardner et al. Reference Gardner, Brooke, Ozanne and Kendig1999). A recent survey of 353 people aged 65 or over in Perth, Western Australia, also found that 7 per cent of the respondents reported severe loneliness (Steed et al. Reference Steed, Boldy, Grenade and Iredell2007). The problem was particularly serious among seniors who were single, those who lived alone and those with poor self-reported health. These findings are similar to research results from the United Kingdom and Europe (Owen Reference Owen2007; Routasalo et al. Reference Routasalo, Savikko, Tilvis, Strandberg and Pitkälä2006; Victor et al. Reference Victor, Scambler, Bowling and Bond2005). As the population aged 65 and over is projected to increase from 13 per cent currently to 24 per cent by 2036 (Australian Institute of Health and Welfare 2007), social isolation will be a major challenge that could affect the wellbeing of many older people in Australia.
The existence of a strong association between social networks, participation and health is well known to researchers (Moren-Cross and Lin Reference Moren-Cross, Lin, Binstock and George2006). Social isolation and loneliness have been shown repeatedly to predict an increase in morbidity and mortality (Berkman and Glass Reference Berkman, Glass, Berkman and Kawachi2000; Bosworth and Schaie Reference Bosworth and Schaie1997), psychological distress, depression and suicide (Cacioppo et al. Reference Cacioppo, Hawkley, Crawford, Ernst, Burleson, Kowaleski, Malarkey, Van Cauter and Berntson2002; Kawachi and Berkman Reference Kawachi and Berkman2001; Roehner Reference Roehner2007), poor health and wellbeing (Chappell and Badger Reference Chappell and Badger1989; Fratiglioni Reference Fratiglioni2000), and decline in cognitive function (Berkman Reference Berkman2000; Glei et al. Reference Glei, Landau, Goldman, Chuang, Rodríguez and Weinstein2005; Yeh and Liu Reference Yeh and Liu2003; Zunzunegui et al. Reference Zunzunegui, Alvarado, Del Ser and Otero2003).
The evidence also indicates that cumulative impact of exposure to disadvantage in the ageing process along with differences in social and cultural backgrounds may subject specific groups of older people to a higher risk of social isolation and loneliness. These vulnerable groups experience ageing differently and are more likely to require services and be excluded by mainstream policies and support. Specifically, groups at particular risk of social isolation include older men living alone (Arber, Davidson and Ginn Reference Arber, Davidson and Ginn2003; de Jong Gierveld Reference de Jong Gierveld, Arber, Davidson and Ginn2003; de Jong Gierveld and Havens Reference de Jong Gierveld and Havens2004; Flood Reference Flood2005), seniors living in remote and rural areas (Becker Reference Becker2003; Upham and Cowling Reference Upham and Cowling2006; Vinson Reference Vinson2007) and older migrants from culturally and linguistically diverse backgrounds (Ip, Lui and Chui Reference Ip, Lui and Chui2007; Rao, Warburton and Bartlett Reference Rao, Warburton and Bartlett2006; Rowland Reference Rowland, Borowski, Encel and Ozanne2007; Yu Reference Yu2000).
Compared with the progress on understanding the causes and outcomes of social isolation, relatively little has been achieved in identifying effective interventions to address the problem. Two systematic reviews on outcomes of interventions on social isolation in later life concluded that although there is widespread belief that interventions can counteract social isolation, the research evidence to support this is ‘almost non-existent’ (Findlay Reference Findlay2003: 655); neither can researchers ‘say with certainty what does not work’ (Cattan et al. Reference Cattan, White, Bond and Learmouth2005: 62). Another recent systematic review identified a few very general characteristics of effective interventions targeting social isolation in older people: having a theoretical basis, using group format in social activity and/or support, and involving older people as active participants (Dickens et al. Reference Dickens, Richards, Greaves and Campbell2011). This review also emphasised the urgent need for well-conducted studies to improve the evidence base regarding the effectiveness of social interventions for alleviating social isolation.
The reasons for the lack of evidence of improvement following interventions are manifold. In particular, many existing initiatives have not been formally evaluated or have employed only opportunistic user satisfaction measures (Warburton and Lui Reference Warburton and Lui2007). Without appropriate methodologies including validated outcome measures, these studies are incapable of assessing comprehensively the effectiveness of specific interventions. In addition, issues like poor programme/research design, high attrition rates of participants, non-representative samples as well as the short timescale of many of the interventions make it very difficult to generalise from the outcomes. In reality, the difficulty of detecting outcomes from social and community-level interventions is a well-recognised issue that concerns researchers and policy makers (Wandersman and Florin Reference Wandersman and Florin2003). Outcomes of specific interventions are hard to establish because the scenario is often very complex, and it is methodologically challenging to try to match outcomes with interventions.
Recent intervention studies continue to have serious methodological limitations, including small sample sizes (Fokkema and Knipscheer Reference Fokkema and Knipscheer2007); participants self-selecting into control or intervention groups (Shapira, Barak and Gal Reference Shapira, Barak and Gal2007); short time periods of three months or less in which to demonstrate an intervention effect (Tse Reference Tse2010; Winningham and Pike Reference Winningham and Pike2007), high attrition rates (Lee, Lee and Woo Reference Lee, Lee and Woo2010); and lack of a clear distinction between control and intervention strategies (Moffatt and Scambler Reference Moffatt and Scambler2008).
Given these ongoing limitations, the literature review and community consultations conducted in the early phase of the CGRSIOP project identified some promising intervention models or practices for tackling social isolation in old age, the key characteristics of which included:
• Building a whole-of-community response.
• Use of existing community resources (hence likely to provide sustainable outcomes).
• Socially and culturally appropriate interventions including matching the needs of specific target groups.
• Involving older people in the planning, implementation and evaluation.
• Provision of training and support for intervention staff.
Regarding programme design and evaluation, the literature review and community consultations also highlighted the importance of well-designed methodologies that allow the clear identification of an intervention effect, and the use of well-validated measures of loneliness and/or social support. Together these observations provided the guidelines for the design and implementation of the demonstration projects that were the focus of this study. The Discussion section of this paper highlights the difficulties that this posed in the current pilot study.
Method
Study design
The CGPRSIOP was established in 2003 by a collaboration of state and commonwealth departments interested in the welfare of older people and led by the Office for Seniors within the Queensland Department of Communities. In 2005 a funding information paper and background paper were released calling for submissions for demonstration pilot project proposals. Fifty-seven proposals were received and five projects were funded. The selected projects either used community development models to build community capacity amongst service providers or focused on expanding social support networks and service delivery that target socially isolated older people, or a combination of both. Of the five funded projects, three successfully completed pre- and post-evaluations and are the focus of the current paper. Details of these three projects are provided in Table 1.
Overview of the three demonstration programmes to prevent social isolation among older people in Queensland
The intervention design, including the selection of the outcome measures, was developed by the research team at the Australasian Centre on Ageing in consultation with the project steering committee which involved representatives from each of the collaborating government departments and the Australasian Centre on Ageing.
Participants
Two of the three pilot projects involved existing services (Hervey Bay; and Culturally Appropriate Volunteer Services: CAVS) so the participants in the project were existing service clients. This was not the case for the rural Greenvale intervention, where a community-based project was established to provide social and personal development opportunities for seniors in the region. The Greenvale project targeted people 55 years and over in the local region, particularly socially isolated mature-aged graziers. Implementing the interventions through existing services (Hervey Bay and CAVS) and community development activities (Greenvale) meant that it was not possible to apply strict inclusion and exclusion criteria in order to select participants in terms of age, level of social isolation, etc.
Table 2 provides detailed descriptive sample statistics for the three programmes, including pre and post sample sizes, mean age and age range, and frequencies for gender, employment status, main income source, living arrangements and amount of contact outside the home each week. The same statistics for each programme are also provided in Table 3 specifically for the subset of participants who completed both the pre and post loneliness and social support indexes, as it is this subset from which the loneliness and social support scores were derived.
Descriptive statistics for all participants
Notes: CAVS: Culturally Appropriate Volunteer Services. SE: standard error.
Descriptive statistics for participants who completed both pre and post survey forms
Notes: CAVS: Culturally Appropriate Volunteer Services. SE: standard error.
A convenience sampling strategy was used with participants recruited through the community organisations. As a result, there were a number of anomalies in the participant samples. These included attrition from pre- to post-programme, unequal gender representation, and relatively small final samples of participants who completed both pre and post survey forms (see Tables 2 and 3).
Instruments
A pre- and post-intervention questionnaire was developed which included the de Jong Gierveld Loneliness Scale (de Jong Gierveld and Kamphuis Reference de Jong Gierveld and Kamphuis1985; de Jong Gierveld and van Tilburg Reference de Jong Gierveld and van Tilburg1999) and an adaptation of the Duke Social Support Index (DSSI) (Koenig et al. Reference Koenig, Westlund, George, Hughes, Blazer and Hybels1993). As these are self-report instruments, the participants were expected to complete the scales themselves with service staff provided with guidance to assist them to do this correctly without interfering with their responses. There are indications, however, that some of the service staff may have breached these guidelines, including the possibility that they completed the scales on behalf of clients. The implications of this are considered in the Discussion section.
The de Jong Gierveld Scale (de Jong Gierveld and van Tilburg Reference de Jong Gierveld and van Tilburg1999) is an 11-item self-report measure of social loneliness. The scale is based on a cognitive theoretical approach to loneliness, where loneliness is seen as a subjective experience and therefore not directly related to situational factors. The importance of social perceptions and evaluations of one's personal relationships is emphasised (van Tilburg and de Leeuw Reference van Tilburg and de Leeuw1991). The scale recognises that loneliness is substantially different from simply being alone, and thus highlights the discrepancy between what one wants in terms of interpersonal affection and intimacy and what one has – the greater the discrepancy, the greater the degree of loneliness (de Jong Gierveld and van Tilburg Reference de Jong Gierveld and van Tilburg1999).
The scale responses were: 1=yes; 2=more or less; and 3=no. Six of the items were negatively worded and five items were positively worded. The responses were scored using the dichotomous procedure recommended by the scale developers (de Jong Gierveld and van Tilburg Reference de Jong Gierveld and van Tilburg1999). From this, aggregate loneliness scores were created to reflect reported levels of loneliness at programme start and at programme finish. The aggregate score ranges from 0=not lonely to 11=very lonely. The dichotomised loneliness scores returned internal reliability coefficients of 0.90 pre-programme and 0.83 post-programme. These findings indicate that this measure of loneliness had good internal reliability.
The 11-item version of the DSSI (Koenig et al. Reference Koenig, Westlund, George, Hughes, Blazer and Hybels1993) used in the current study was derived by Koenig et al. from the longer 35-item scale. The scale is intended to measure two subscales for social interaction and subjective support as well as a composite measure for overall social support. Changes to the wording of the DSSI were made by the funding partner in an attempt to make the survey more acceptable to the service providers and participants. Specifically, these changes were: (1) ‘Number of family members within one hour that subject can depend on or feel close to’ was replaced with ‘Other than members of your family, how many persons in your local area do you feel you can depend on or feel very close to?’; and (2) ‘Do family and friends understand you?’ was revised to read, ‘Does it seem that your family and friends (i.e. people who are important to you) understand you?’ The implications of these modifications to the DSSI are addressed in the Discussion section of this paper.
Responses to questions 1–6 were on a three-point Likert-type scale where 1=hardly ever, 2=some of the time and 3=most of the time. Responses to question 7 were on a three-point Likert-type scale where 1=very dissatisfied, 2=somewhat dissatisfied and 3=satisfied. Responses to questions 8–11 were on a scale where 1=no times, 2=1–2 times and 3=2+ times (related to the number of times participants had social interactions over the preceding week).
Factor analysis was unable to differentiate the two subscales for social interaction and subjective support, so the scale was treated as a single social support construct and all analyses reported on this scale are concerned with the whole scale and not with the subscales. The use of a single social support construct is supported by Koenig et al. (Reference Koenig, Westlund, George, Hughes, Blazer and Hybels1993) who found that the two subscales in the 11-item scale loaded cleanly on to a single factor. In the current study, the 11 items of the social support scale produced internal reliability coefficients of 0.84 and 0.87 for the pre- and post-programme measures, respectively, indicating good reliability of both.
Both the loneliness and social support scales have been extensively used with older populations and have been found to be valid and reliable across a variety of research designs (de Jong Gierveld and van Tilburg Reference de Jong Gierveld and van Tilburg1999; Powers, Goodger and Byles Reference Powers, Goodger and Byles2004; Sansoni et al. Reference Sansoni, Marosszeky, Sansoni and Fleming2010). The adoption of a common tool across the three interventions also facilitates comparison between the programmes.
Procedure
Five locations were selected to run demonstration projects following a call for submissions by the Queensland Department of Communities. The selection was based on a range of criteria, including that older people in these locations were at higher risk of social isolation and loneliness (because of older than average populations, rural or remote locations, and culturally and linguistically diverse communities). Due to incomplete evaluation data from two demonstration sites, this paper focuses only on the evaluation findings from three projects undertaken in Greenvale (a rural town), Hervey Bay (a coastal centre) and the Brisbane central business district (a metropolitan centre). Details of the three pilot projects are provided in Table 1.
Using a community development approach, a wide range of group activities and services were delivered in each location in order to create meaningful social networks and close relationships among older people in the district. The programmes were designed and implemented in partnership with community organisations and councils. Older people in the local district were involved in the process of planning and service delivery through extensive community consultation, and needs analyses to ensure that the resultant programme areas met their needs. Older people were also actively involved in the implementation of the projects, with all three projects providing training for older volunteers to enable them to engage with more isolated people in their community (Hervey Bay and CAVS) or to organise their own community activities (Greenvale).
The different locations for the three interventions were reflected in the type of intervention programme required, with the metropolitan intervention focusing on multicultural issues (CAVS), the coastal centre focusing on service integration (Hervey Bay), and the rural intervention targeting the needs of older graziers (Greenvale). These interventions lasted for six months and took place between 2005 and 2006 with the CAVS project receiving a further six-month extension to February 2007 in order to consolidate partnerships with key organisations (the OzPol Community Association and the Cathay Community Association) and establish volunteer services.
Programme staff members were responsible for recording basic client data and ensuring that the self-report questionnaires were completed correctly. A guide to data collection was prepared by the researchers and relevant programme staff members were briefed by the Department of Communities on their responsibilities.
The evaluation adhered to the Australasian Evaluation Society's Guidelines for the Ethical Conduct of Evaluation (2008). All information collected was non-identifying and was reported at an aggregate level to prevent participant identification. Client confidentiality and privacy were respected and voluntary consent was required from all participants.
Results
For descriptive statistics, see Tables 2 and 3, and for the pre- and post-programme loneliness and social support scores for each project, see Table 4. To correct for small sample sizes and the known methodological issues with these data, α=0.01 was used to determine statistical significance for all inferential analyses. Pre-programme loneliness and social support scores were significantly negatively correlated to a strong degree for Greenvale participants, r(26)=−0.69, p<0.001, indicating that greater loneliness was strongly correlated with lower social support, as would be expected with valid measures of the two constructs. However, these scores were not significantly correlated in the Hervey Bay programme, p=0.514, N=14, or the CAVS programme, p=0.048, N=12. The post-programme loneliness and social support scores were again significantly negatively correlated to a strong degree for Greenvale, r(28)=−0.75, p<0.001, but there was no significant correlation between these scores for Hervey Bay, p=0.406, N=12, or CAVS, p=0.035, N=12.
Pre- and post-programme loneliness and social support scores
Notes: CAVS: Culturally Appropriate Volunteer Services. SE: standard error.
Loneliness
Loneliness was measured at the start of the programme and again at the end of the programme using the de Jong Gierveld Scale, as described above. Pre- and post-programme statistics are presented in Table 4. Independent samples t-tests showed that there were no significant gender differences in pre and post loneliness scores for each programme. There were also no significant correlations between age and the pre and post loneliness scores for each programme.
In considering the impact of each individual programme on loneliness (see Table 4 for means and standard errors (SE)), paired samples t-tests showed that there was no significant difference in loneliness from programme start to programme end for Greenvale participants, p=0.64, N=28, or Hervey Bay participants, p=0.199, N=15. CAVS produced the only significant difference, with greater loneliness reported at programme start, mean=7.7, SE=0.74, than at programme end, mean=5.8, SE=0.65, t(12)=4.71, p=0.001. It should be noted, however, that methodological issues with data collection on the CAVS programme in particular bring into question the validity of this result (this point will be expanded in the Discussion).
Social support
The DSSI, as revised and abbreviated by Koenig et al. (Reference Koenig, Westlund, George, Hughes, Blazer and Hybels1993), and subsequently modified by the Department of Communities, was used to measure social support at the start and end of the programme (see Table 4 for means and standard errors). Independent samples t-tests showed that there were no significant gender differences in social support scores at the start and end of each programme. There were also no significant correlations between age and the pre and post social support scores for each programme. Paired-samples t-tests showed that there was no significant difference in social support between the start and the end of the programme at Greenvale, p=0.205, N=25, or Hervey Bay, p=0.018, N=12. Social support at the end of the CAVS programme, mean=2.70, SE=0.06, was significantly higher than at the start of the programme, mean=2.35, SE=0.12, t(10)=−3.36, p=0.007.
Discussion
The present study sought to pilot an investigation of how participation in three community programmes might affect levels of loneliness and social support in older adults in these locations. The pilot study incorporated a number of key programme design features identified during a review of the literature. One feature was the need for a whole-of-community approach with the three projects outlined here incorporating strong partnerships with local government (such as the Hervey Bay Council) and community stakeholders (like the Greenvale State School Parents and Citizens Association and the Multicultural Development Association). The study also paid special attention to the involvement of older people in the design and delivery of services through extensive community consultations and provision of training for older volunteers. The use of older volunteers in the buddy system of the Hervey Bay project exemplifies how this practice can be incorporated as a component of the project. In addition, the upskilling of Greenvale residents shows how sustainable practice can be embedded in a project.
Another feature identified in the literature and implemented as part of the study was the incorporation of validated evaluation measures in the programme design. Specifically, this study used measures of social support and loneliness in the evaluation to test the effectiveness of the interventions. As mentioned in the introduction, the use of psychometrically validated measures in government-run programmes has not been common practice in the evaluation of community interventions.
This pilot study did not return any sufficiently robust results to demonstrate the effectiveness of community-based interventions in reducing loneliness and increasing social support for older people in the community. In contrast, the qualitative data from the pilot projects reported elsewhere indicate that the participants and service providers felt that the interventions were successful (Department of Communities 2008). As a pilot study of a community intervention in which validated psychological measures were used, however, the research has provided some valuable insights into the methodological problems that can arise in designing and conducting research of this nature, and these need to be taken into account in future research.
First, the attempt of the CGPRSIOP project to incorporate validated psychological measures into the programme is an important achievement, especially given the general lack of robust evaluation in community programmes. In this sense, including validated measures offers an exemplary design for development of strategic initiatives to improve social support and reduce loneliness in later life. It is critical, however, that validated instruments are not modified in any way unless further validation takes place to ensure that the changes have not impaired the reliability and validity of the instrument. In the current project, the changes made to the DSSI (Koenig et al. Reference Koenig, Westlund, George, Hughes, Blazer and Hybels1993) by the funding partners may explain why the two DSSI subscales could not be differentiated in the current analysis.
The second important insight that this pilot evaluation can offer to future researchers in this field is the importance of ensuring that sampling, data collection and intervention strategies are standardised across programmes. Disappointingly, the current evaluation did not return any statistically significant changes in social support and loneliness that can be confidently attributed to the interventions used. However, rather than indicating that the interventions did not work, the lack of significant differences may be largely due to sampling error, changes in sample characteristics from pre- to post-programme, and unstandardised data collection and intervention strategies.
Accessing an appropriate sample of participants in community settings is fraught with difficulty, and reflects the problems that are inherent in any community-based research undertaking. As a result, inconsistent or inappropriate sampling is often a key issue that weakens the validity of community research, and its detrimental effects are illustrated clearly in the current study. In order to conclude with any certainty that engaging in a programme results in improvements in loneliness and/or social support, and to test for differences between programmes, the pre- and post-programme sample sizes need to be large and of similar sizes across programmes. How large the sample needs to be is determined by the types of questions the research is intending to address, and the statistical analyses that will be used to investigate those questions. For simple comparison statistics using t-tests and analysis of variance, sample sizes in excess of 30 are a minimum. For more complex techniques such as factor analysis and regression, sample sizes in excess of 100–150 are preferable, and are guided by how many items and scales participants are completing. In a longer-term project with more carefully standardised sampling and data collection procedures, these issues could be addressed.
The composition of the sample of participants is also important. Ensuring relatively even age ranges and gender proportions is important to minimise age and gender effects. The current research used a convenience sample based largely on the participants who were available through the community organisations. The flexibility required in community-based interventions, including freedom to join or leave at any time, does not fit well with a formal research design and has resulted in high levels of attrition, with a number of people completing the pre- or post-programme evaluation but not both. The resulting changes to the sample characteristics between the two times reduced the possibility of finding any differences, and weakened the validity of any differences that were found, because the post-programme sample was not the same as the pre-programme sample on a number of key characteristics. These included changes in employment status, income source, living arrangements, and the amount of weekly contact that participants had outside the home. Changes in any of these characteristics are highly likely to affect participants' sense of loneliness and will have a direct impact on the availability of social support mechanisms. Changes in the amount of contact participants had outside the home were particularly notable in the Hervey Bay and CAVS programmes, while changes to employment status were particularly prevalent in Greenvale. Thus, any changes in loneliness or social support from programme start to programme finish cannot be confidently attributed to the interventions, nor can they be confidently attributed to other changes in the participants' lives. These changes might not have had such a large impact in this study if the sample sizes had been larger, and therefore able to absorb greater variance. In addition, large sample sizes would allow participants who had substantial life changes between the start and end of the programme to be excluded from analyses, and would also facilitate the use of statistical tests to investigate the impact of changes on these key characteristics on the outcome variables of loneliness and social support. Neither of these strategies can be used when working with small samples.
In terms of unstandardised data collection procedures in the current research, these ranged from minor discrepancies such as differently formatted survey forms being used by different programmes, to more serious issues such as changes to the wording of the DSSI mentioned previously and the inaccurate recording of participant identifiers across the pre- and post-survey forms (and the subsequent need to make a ‘best guess’ as to whether a set of forms related to the same participant or not). In future research, the first issue needs to be addressed at the study design stage, to ensure that every participant receives exactly the same instrument and, in the case of validated measures, the correct wording is used. A potential solution that might alleviate the second problem is to have every participant's full name and date of birth written on a tear-off slip on the bottom of the front page of every survey. Once the unique ID has been matched with the name and date of birth, the slip can be removed, ensuring both accuracy of the unique ID and participant confidentiality across several forms completed at different times. This may raise ethical concerns and would need to be carefully implemented to ensure that participants' confidentiality was maintained.
Also difficult to control in community-based research is the possibility of increasing socially desirable answering because participants have a pre-existing relationship with people collecting the data. There can be a tendency for participants in survey research such as this to respond in ways that they believe will present them in a favourable light to the person collecting the data (Podsakoff et al. Reference Podsakoff, MacKenzie, Lee and Podsakoff2003). This is likely to be particularly problematic when asking people about their levels of loneliness and social support, issues about which most people are likely to feel sensitive. This effect could be further exacerbated if the person who is collecting the survey data is also the person who cares for, entertains or has befriended, the participant.
The remainder of the data collection issues in this study reflect the inexperience of the people collecting the data. Staff in the community programmes showed the best intentions and were highly supportive of the research. The Department of Communities provided guidance and instructional material to assist the community programme staff to collect date. However, collecting data using standardised measures requires a good understanding of issues that can affect the validity of the data collected, and the time, resources and willingness to take steps to ensure that the data collection process is as ‘clean’ as possible. A number of difficulties were encountered in relation to data collection in this study – the problem already mentioned with unique identifiers was one; completing survey forms on behalf of participants was another. In a number of instances in this data collection process, the survey forms were completed by staff at the community organisation on behalf of individual participants. This may have been due to problems with the participant physically completing the form – perhaps because of eyesight or muscular problems. Completing forms on behalf of participants appeared to be particularly prevalent in the CAVS project, where responses to open-ended questions (not reported here) suggested that, on some occasions, staff who completed survey forms on behalf of non-English-speaking clients may have taken the opportunity to express their own opinions rather than those of the participants. However, this practice raises concerns about reliability and validity, particularly in relation to honest responding and social desirability bias, and introduces a problematic bias into the data. For example, if it is clear that the open-ended responses were written from a staff member's perspective rather than a participant's, how is it possible to know that the rest of the survey form was not also completed in this way?
The most obvious way to counteract this is to ensure that data are collected by experienced and anonymous researchers who have no pre-existing relationship with the participants. Another (less expensive) method is to provide comprehensive training to the people who are collecting data, and nominate a supervisor who can check and correct any issues with data collection. It would also be useful to ensure that the person who completes the form is identified on the form as being either the participant or someone else, and to collect information about their relationship with the participant. This information could be used to investigate whether there are any substantial differences between those who complete the form themselves and those who have someone else do it.
This issue also highlights an ongoing concern with surveying population groups with low English-language ability or poor literacy, and again indicates a need to ensure that those who are collecting the data are carefully trained, or preferably that data are collected by trained independent researchers, including those with different language abilities. A related solution is to produce survey forms in different languages. This method comes with its own issues of validity and reliability, particularly in relation to the essential meanings of items changing when translated to a different language, but it is extensively used when research is undertaken across different language groups.
The other fundamental design concern that weakened the current research was unstandardised intervention strategies. Each of the three programmes provided different activities and different levels of interaction with staff, and thus it is not possible to compare outcomes across the three programmes. One of the strengths of pre–post-survey research is directly related to the consistency of an intervention across all groups, and this needs to be addressed in future research. It should be remembered, however, that these diverse interventions are the result of a responsive intervention programme reflecting the needs of the different regions. Another design issue that would have increased the validity of the evaluation study but could impede the aims of the service-based intervention is the inclusion of a control group. While this would help to clearly identify any intervention effects, it would require the exclusion of a group of participants from the intervention strategies – something difficult to manage in an intervention based within a service organisation. This approach may also raise ethical concerns about withholding a potentially beneficial ‘treatment’.
Methodological issues aside, the results of the three programmes are encouraging. The standardised instruments used here provide some indication that could be cautiously interpreted to suggest that interventions of this nature may be beneficial in reducing loneliness and increasing social support for older adults in the community. There is certainly emerging evidence that when properly administered, the use of rigorous evaluation measurements provides stronger results of effectiveness than satisfaction surveys or more qualitative evaluations. Further research is now needed using much larger sample sizes, with carefully standardised and controlled design and administration in order to confirm that this is the case. In addition to the lessons outlined in this paper and in the other CGPRSIOP materials, Craig et al. (Reference Craig, Dieppe, Macintyre, Michie, Nazareth and Petticrew2008) provide useful guidelines on how to develop and evaluate complex interventions such as those covered here.
Conclusions
The present study aimed to report on a pilot study investigating social isolation amongst older people in Queensland. Following promising intervention models and practices identified in literature reviews and community consultations, it provided a range of group-based activities and services to specific groups of at-risk older people in three locations with the aim of building community capacity and social networks. The evaluation results, whilst limited due to methodological issues, generally suggest that there may be benefits in using such a community development model to help reduce social isolation in later life and provide key insights into how future interventions should be undertaken.
Overall, the study reflects many features of good practice in evaluating ways to prevent social isolation and loneliness in old age. The results of the evaluations point to a number of strategies that may reduce the social isolation of older people living in the community. In particular, the incorporation of outcome evaluation in such programmes is essential for planning and designing future initiatives to tackle social isolation and also for the development of a robust evidence base. In addition, the problems encountered in the data collection process of this study highlight the need for a high-quality approach to the training, management and support of intervention staff. A set of best practice guidelines based on the outcomes of the CGPRSIOP has now been produced to assist service providers, government agencies and community groups in designing, implementing and evaluating projects to reduce the social isolation of seniors (Department of Communities 2009). Further projects, with built-in evaluation processes using standardised measures, are needed to build on this promising start.
Acknowledgments
The study was conducted while the first four authors were at the Australasian Centre on Ageing at the University of Queensland and involved a series of demonstration projects conducted as part of the Queensland Cross-Government Project to Reduce Social Isolation in Older People (CGPRSIOP) led by the Office for Seniors within the Queensland Department of Communities. CGPRSIOP was established in 2003 and funded by the Australian Department of Veterans' Affairs and Queensland Department of Health with additional funding from the Queensland Department of Communities. The project was managed through a steering committee comprising representatives from a range of other government agencies, the Queensland Seniors Council and the Australasian Centre on Ageing.
