The Job Retention and Rehabilitation Pilot

The Job Retention and Rehabilitation Pilot grew out of growing concern about the number of people claiming Incapacity Benefit, and the recognition of the importance of early interventions to help people to avoid long spells out of the workplace because of ill-health, injury or disability. The pilot was developed by the Department for Work and Pensions and Department of Health, established as a randomised controlled trial, and ran for two years from April 2003, in six areas of the UK. People who had been on sick leave for between six and 26 weeks were invited to participate, on a voluntary basis. The trial involved a control group and three intervention groups:

• • the health intervention group received health services that were a ‘boost’ to NHS provision, such as medical consultations and diagnoses, treatment, psychological therapy, physiotherapy, complementary therapies and access to exercise facilities

• • the workplace intervention group received help such as employer mediation, ergonomic assessments and occupational therapy

• • the combined intervention group were eligible for all of the above.

The evaluation programme involved a two-wave survey with the intervention groups and control group (Purdon et al., Reference Purdon, Stratford, Taylor, Natarajan, Bell and Wittenburg2006); a longitudinal qualitative panel with participants from the intervention groups which is the focus of this article (Farrell et al., Reference Farrell, Nice, Lewis and Sainsbury2006), qualitative research with people in the control group (ibid.), longitudinal qualitative research with service providers (ibid.), qualitative research with employers (Nice and Thornton, Reference Nice and Thornton2004), and qualitative research with general practitioners (Mowlam and Lewis, Reference Mowlam and Lewis2005).

The longitudinal research with participants combined a panel approach and a repeat cross-sectional approach. Three cohorts of participants were selected with a sample of 12 participants in each cohort spanning the intervention groups – a total of 36 participants across the study. Within each cohort, all participants were interviewed six times. The first interview was face to face, shortly after the person had joined the Pilot; there were then five interviews at roughly monthly intervals, by telephone. The panel element was thus relatively short scale in the context of other qualitative longitudinal studies, although the research overall spanned 23 months of data collection.

Domains of change in evaluation research

Longitudinal qualitative research is of particular value in evaluations because it allows exploration of different domains or areas of change, and the linkages between them. Walker and Leisering Reference Walker, Leisering, Leisering and Walker(1998) and Gershuny Reference Gershuny, Leisering and Walker(1998) emphasise that dynamic research approaches can explore the interplay between individual change and change within institutions and social structures, and between the micro and the macro. In the context of evaluative research this could be seen as involving four key domains of change: the individual, the service, the policy and the structural.

The first domain, the individual domain, brings together the individual's contact with the policy and their broader personal context within which the policy is experienced. People's interaction with a service develops over time, as they experience multiple contacts or the consequences of earlier contact. But their personal lives change too, and it is the relationship between the service and the personal that is often critical in explaining the experience and impact of the policy.

In the Job Retention and Rehabilitation Pilot study, changes occurring in the personal context provided particularly rich material for analysis, showing how sickness absence was experienced over weeks or months and what this meant for people's changing needs of the service. A number of different aspects were critical here.

First, it was essential to be able to analyse change in people's health circumstances. People experienced improvements, deterioration, flux, and the development of new conditions or health problems, and they had contact, at different times, with health services outside the Job Retention and Rehabilitation Pilot. Second, our analysis also focused on changes in work circumstances. People had continuing contact with different workplace representatives, which could move between being supportive and threatening. Their feelings about work and about returning to work changed as they were away from work longer. There were broader changes in the workplace – new line managers, relocation, restructuring. Third, there were important changes to people's financial circumstances: diminishing or ceasing salaries; statutory sick pay, Incapacity Benefit or other benefits; insurance or compensation payments expected, received or refused. Fourth, there were broader changes in people's personal lives: births, bereavement, changes in personal relationships.

A fifth aspect of change in the individual context was people's experience of contact with the Pilot services. Their understanding of what was available increased, or sometimes diminished. Their requirements changed or became more clear. They experienced different assessment procedures, had contact with different members of staff, and used more or different forms of help. A type of intervention which seemed helpful at one stage could seem less so later, or vice versa. Their relationships with individual service members and with the service as a whole changed.

The longitudinal design provided rich material for exploring the interaction between personal lives and use of the service. A key focus of our analysis here was on exploring change framed by the objectives of the policy – sustained returns to work. Our analysis explored how returns to work came about, or why they did not, and whether and how they were sustained, looking at the interplay between the personal and service use to understand each individual case.

The second domain of change is the service domain. Longitudinal research allows observation of the service delivery process from planning to implementation to bedding down to reviewing and changing. Changes in the wider organisational context can also be observed – a service may move from being central to marginal in an organisation's activities, or vice versa. In the case of the Job Retention and Rehabilitation Pilot there were changes in the delivery of the service with different approaches to marketing, liaison with external agents such as employers and GPs, the organisation of case work, the range of providers used and the specific interventions offered. There were some echoes of this in the narratives of individual participants and in comparisons between the three cohorts, although the clearer readings came from the longitudinal work with service providers.

Most of the value of using longitudinal qualitative research in the Job Retention and Rehabilitation Pilot evaluation came from these two domains of change, and particularly from the first. In particular, using longitudinal research gave much more insight into issues such as:

• • how aspects of the continuing experience of sickness absence such as isolation, financial pressures and job loss affected people's plans around returning to work and their needs of the Pilot services;

• • how different agents within the workplace interacted, and how what people needed of their employer changed during their sick leave;

• • the dynamic of engagement with the service: the role of case managers, what led to changes in the degree of engagement, how the locus of control or initiation of contact shifted between service and individual;

• • what worked in terms of sequencing and following up interventions;

• • how changes in service delivery impacted on service users;

• • the sustainability of returns to work and how this was influenced by the Pilot, by the behaviour of employers over time, and by the experience of sickness absence.

Particularly in evaluations which are longer-running, one might also expect to see change in the policy domain. Policies are not usually static. They are reviewed and modified in response to new information, new political imperatives, new options. Social problems are reconstructed, or the ideologies underpinning policies change. Policy developments can cast a different light on issues appearing in the dataset, changing their significance. In the case of the Job Retention and Rehabilitation Pilot, there were a number of new policy documents which emphasised for example the role of employers in supporting returns to work or the role of GPs in actively managing ill-health, and we needed to reflect this in the issues we explored and in our analysis.

Finally, broader structural domain changes can also come into play. They will again be more evident in longer-running longitudinal studies and so were not obviously relevant in the example of the Job Retention and Rehabilitation Pilot. But in a longer study one might expect to see for example changes in the benefit system, employment rights, or the operation and priorities of the NHS; changes in labour markets and local communities; and changes in the prevalence or nature of a social issue such as disability or chronic ill-health, and in its social meaning or construction.

Types of change

The analysis of qualitative longitudinal studies also reveals different types of change, and as an analyst one needs to be alert to them and their different meanings.

The most evident type of change is narrative change – the unfolding of individual stories across time. This will involve both specific events or interactions, and more subjective feelings, hopes, reactions, plans. We use the term ‘narrative’ rather than ‘chronological’ change because stories are not always told in a neatly linear way. We found that participants in the Job Retention and Rehabilitation Pilot study would tell us for example in the third interview about a meeting with the provider that occurred before the second, or would tell us in a later interview about an incident that occurred before they left work which they had not mentioned in the first interview. People forget things, they think they have already mentioned them, they prefer not to volunteer them until the relationship with the researcher has developed over several interviews, or something only becomes relevant in the context of later events. This narrative change was the first focus of our analysis.

The second type of change arises from reinterpretation by the participant – their rethinking or retelling of experiences and feelings described earlier. This is a process that may be explicit or implicit. In the Job Retention and Rehabilitation Pilot study, there were cases where people explicitly revised their interpretation of events or interactions. For example, an employer's regular contact when they first went off sick was initially described as supportive, but reinterpreted as having more sinister intent when the organisation later made attempts to dismiss the participant. Or NHS treatment which had initially been viewed as satisfactory was seen more negatively when contrasted with later help from a Job Retention and Rehabilitation Pilot provider. These explicit rethinkings of experiences are highly illuminating – they tell us more about the mental as well as the chronological journey made.

Implicit reinterpretation was harder to read. There were instances where people reported the same incident in different ways but rejected the idea that what they had said earlier was inconsistent. One woman for example was initially very clear that she wanted to use the Pilot service to justify her sickness absence to her employer. In the second interview, she was equally clear that what she had wanted was their help in finding a new job. It was also common for people during the course of the interviews to describe the service as having helped them to achieve things they could not have done alone – but in the sixth interview, reflecting back, to reframe the role of the service as having been supportive and speeded up or smoothed out progress that they would anyway have made.

These implicit retellings are harder to interpret, but they seem significant. It may be that later events or encounters are leading people to reinterpret earlier events. And perhaps they are about participants wanting to re-present themselves to us, particularly as the research comes to a close – wanting to ensure we understand they are active agents in their own stories.

The third type of change is the researcher's reinterpretation, as we learn more about an individual through a series of interviews and see them more clearly in relation to other research participants. We gain a better – or just a different – understanding of a participant's behaviour or stance, reinterpreting for example what first appears to be resistance to help as people protecting themselves against risk, an indication of vulnerability rather than inflexibility.

So much for change. But the absence of change is also important. Saldana highlights the importance, among other descriptive questions, of asking: ‘What remains constant or consistent through time?’ (2003: 114). This might involve noting the absence of a change that other people have experienced, or that one might have expected given the theory of change underpinning the policy (Pawson and Tilley, Reference Pawson and Tilley1997), or a change that the participant expected, wanted or had been promised.

The absence of change may be a positive as well as a negative finding: it may reflect stability, consistency or maturity. In the case of the Job Retention and Rehabilitation Pilot, since the policy was aimed at sustained returns to work, the absence of change once someone returned to work was usually a positive outcome. Similarly, the absence of change might be the continuation of a positive relationship with a GP or an employer. These instances are opportunities to explore how stability is promoted and supported. But the absence of change may also reflect the fact that progress has not been made, or a desired development has not happened. Understanding the subjective experience of ‘no change’ is therefore essential.

The point here is that there is more than one type of change, and in analysing longitudinal qualitative data one needs to be alert to what type of change is apparent and what this tells us about its possible meaning. Qualitative data rarely speak for themselves and this is particularly true of longitudinal data – a literal reading is unlikely to be sufficient. In qualitative longitudinal research there is more scope to engage the participant in the analysis of change – seeing whether change features in their narrative or their interpretation, reflecting to them the inconsistency or retelling that we think we hear, engaging them in a collaborative interpretation of their account.

Organising data to analyse change

The volume of data is at once the delight and the challenge of qualitative longitudinal analysis: this Job Retention and Rehabilitation Pilot study for example yielded 197 interviews. We were conscious of the danger of being oppressed by this weight of data, losing sight of the individual and the particular.

There is little literature on the application of different data management approaches to longitudinal data, although Saldana Reference Saldana(2003) provides a useful account of how QSR NVivo can be used specifically for the analysis of longitudinal data. In the Job Retention and Rehabilitation Pilot study the method we used was Framework, and we found it suitable to the multiple readings of longitudinal data that are described in the next section.

Framework is a matrix-based method for ordering and summarising data (Ritchie et al., Reference Ritchie, Spencer, O'Connor, Ritchie and Lewis2003; Pope et al., Reference Pope, Ziegland, Mays, Pope and May2006). First, the key themes that need to be drawn out in the analysis are identified and divided into a series of sub-topics, through familiarisation with the data. Then a series of matrices is drawn up, each displaying one theme. The columns represent sub-topics and the rows represent interviews. The matrices can be set up manually or in Excel spreadsheets, and Framework is currently being developed by NatCen as a standalone CAQDAS package. In Excel, each matrix (of which there might be between four and eight in an individual study) is set up in A3 landscape format, replicated on as many pages as is necessary to accommodate all interviews.

Figure 1 illustrates the matrices (or ‘charts’) used in the Job Retention and Rehabilitation Pilot study.

Figure 1.

Organising data to analyse change using Framework

Notes: Four 2-page charts as above were used on the JRRP study for each participant, covering:

Chart 1: Background

Chart 2: Going off sick, entry to JRRP, return to work

Chart 3: Use of JRRP

Chart 4: Impact of JRRP and other activities

In addition a single-paged Chart 5 provided an overall summary, looking at key themes across all interviews.

There were four thematic charts covering: the participant's personal background (chart 1), the process of going off sick, entry into the Pilot and return to work (chart 2), use of the Job Retention and Rehabilitation Pilot (chart 3), and impact of both the Pilot services and activities outside the Pilot (chart 4). On each chart there was one row for each of the six interviews, and each chart was sized to print on two A3 sheets. The first chart also incorporates a summary of the interview for each participant at each of the six interview stages, and the fifth chart was a ‘whole case summary’ completed after the sixth interview.

Each interview was summarised within the framework before the next interview, so that the following interview could build on the narrative and include questions based on our developing interpretation. The analysis at each stage was useful for capturing our understanding of the participant's situation at that point and for recording our own reflections, albeit reflections that we treated as contingent and tentative readings of the data (Thomson and Holland, Reference Thomson and Holland2003). The ‘whole case’ summaries provided an overall narrative within four key themes, which was helpful for comparing between cases, developing hypotheses and signposting us back into the detailed data.

Setting up the analysis in this way retained the temporal structure of the data and brought together data across cases within the same themes, but without losing the structure of each individual case. The thematic framework had to be set up before even the second round of interviewing had taken place, raising the risk that we might miss topics that emerged in later interviews. Therefore, we made the sub-topics broad, and had a spare column on each chart where relevant material not encompassed by a column heading could be fitted. In practice there was always somewhere that material could go. It was important though that the framework could be used in this flexible way, and that in reading data we looked beyond the obvious columns.

Reading the data

There is of course no single established procedure for analysing qualitative data, and the approaches are as varied as the traditions, disciplines, methodologies and schools within which qualitative research takes place. The analysis of qualitative longitudinal data employs the approaches used in cross-sectional studies – exploring diversity, deviant, negative or atypical cases and alternative perspectives; using iterative questions to search deeper into the data; drawing out explicit and implicit explanations; exploring conceptual and thematic linkages; identifying patterns; and constant comparison. But the multi-layered nature of longitudinal qualitative research means that its reading is more complex.

Holland et al. Reference Holland, Thomson and Henderson(2004) describe the analysis process as involving cross-sectional analysis (looking at a moment in time across the sample), repeat cross-sectional analysis (looking for change), and longitudinal analysis (narrative analyses of each case over the duration of the study). This multi-dimensional analysis is critical to making use of longitudinal data. Our reading of the Job Retention and Rehabilitation Pilot study longitudinal data also involved individual case analysis, thematic analysis and comparative analysis. Overall, we identified seven ways of reading the data:

• • Cross-sectional analysis (looking at individual cases at a single point in time) is an important part of the process of building an understanding of the case. In Framework terms it involves reading along a row across the four thematic charts.

  There are two issues to note here. First, the ‘baseline’ in qualitative longitudinal research is not necessarily the first interview, and indeed there need not necessarily be only one baseline. In the Job Retention and Rehabilitation Pilot study, there were differences of several weeks or months in how soon the first interview happened after the person had gone off sick, and differences of days to weeks in how soon the interview happened after their first contact with the Pilot. Therefore, as a baseline, the precise time of the first interview would have been somewhat unstable. We also used the first interview to explore any earlier experiences of sickness absence and, where illness had come gradually, the progression from its first impact on work to the person actually going off sick. When we were looking at issues relating to employers and ill-health, the most useful baseline was usually the point when a conflict between health and work first emerged. For issues relating to GPs and health care outside the Job Retention and Rehabilitation Pilot, it was usually their first consultations. For issues relating to use of the Pilot, it was usually people's first contact with the service, or the point when they first heard about it. In longitudinal qualitative research, the baseline needs to be conceptually as well as chronologically defined.

  The second issue is that the relevance of specific time points in cross-sectional analysis is fluid. The trajectory taken by each participant may be quite different, which may or may not make it illuminating to look at where they are at a particular point in time. In the example of the Job Retention and Rehabilitation Pilot, for example, people stopped using the service, returned to work, went on to long term benefits, got more or less ill at different points, and the service did not follow a fixed time-line in the way in which it was structured and delivered. This meant that no individual stage of interviews had a particular significance across the sample.

• • Repeat cross-sectional analysis (looking at changes between points in time for individual cases) is important to identify changes since earlier interviews, to explore how they come about, to see whether expected or hoped for changes had occurred. It involves looking back not only to the previous interview but also to earlier ones: in Framework terms reading along some or all of the six rows per participant, across the set of four thematic charts. Repeat cross-sectional analysis builds into:

• • Individual case narratives which look across all stages of data collection and the whole case summary, to capture an essence of the journey travelled, to explore how change occurred, the shape it took, how the pace or trajectory changed, and how people experienced the changes, or absence of changes, that had occurred. In Framework terms, the data here have to be read up and down within columns to explore change over time, and left and right across columns to look at how themes are linked at different points in time.

• • As well as this focus on individual cases, thematic analysis enables exploration of key issues across cases. In Framework terms it involves looking up and down a column or series of columns. The temporal aspect of longitudinal research is not always of primary relevance here. The Job Retention and Rehabilitation Pilot data set, for example, provided extensive descriptions of employer practices, service interventions, and contact with the benefit system and the NHS. When these had occurred, in the temporal context of a case, was relevant in understanding change over time. But they also had a relevance in their own right, and told us for example about the diversity of employer approaches or of GP case management practices.

• • Linkages between themes also need to be explored, both at a particular point in time and longitudinally. In Framework terms this involves zigzagging across the charts, reading across, up and down columns. For example, in the Job Retention and Rehabilitation Pilot study, people's changing relationship with their health – their developing conceptualisation or understanding of their condition or experience of it – had implications for what they wanted or needed of the service and of the NHS. Data on changing thoughts about work had to be read against data on financial developments. Relationships between topics thus become more complex than in single-episode research.

• • Between-case comparisons are an important perspective on longitudinal data. Each case is read more acutely, with more depth and texture but also with more questions, as it is seen in comparison with others. The analyst observes differences in the nature of change, in how it is articulated, in how it comes about, in how people respond to it and their evaluation of it. These observations can be taken back into the field in the form of questions to the participant, and taken back into the data. We found it helpful to look at replication and diversity, for example examining cases with apparently similar starting places but different outcomes, or with different outcomes but similar starting places, to get a more nuanced sense of the mechanisms and drivers of change in each case.

• • Between-group comparisons provide the final structural layer of analysis. The definitions of groups will of course be specific to the context of the individual evaluation. In the Job Retention and Rehabilitation Pilot study, three types of groups were relevant. First, there were demographically defined groups: comparisons between people of different ages, with different conditions, working in different roles, occupations or sectors and so on. Second were service-defined groups: the three intervention groups around which the randomised controlled trial was structured. Third were outcome-defined groups: those who returned to work and those who did not – a grouping which is itself defined by reference to change. By exploring what is common within and between groups and what is different, we develop our understanding of the interplay between the personal and the broader contexts.

Asking questions of longitudinal qualitative data in evaluations

Given the volume and complexity of qualitative longitudinal data, clarity about ones questions is a particularly important device in structuring analysis (Pettigrew, Reference Pettigrew, Huber and Van de Ven1995). Saldana Reference Saldana(2003) discusses three types of questions: framing questions (such as ‘when do changes occur through time?’), descriptive questions (such as ‘what increases or emerges through time?’) and analytic or interpretive questions (such as ‘which changes through time oppose or harmonize with natural human development or constructed social processes?’). Pettigrew Reference Pettigrew, Huber and Van de Ven(1995) describes the different facets of change that need to be explored in evaluation studies – speed, quantity, quality and process.

We have found it helpful, in analysing qualitative longitudinal data, to ask questions of eight different types. These are outlined below, and illustrated in relation to one key issue explored – the decision to return to work.

• Questions about description (the type, extent and timing of any change) How firm was the decision to return to work? When was the decision first made and has it been consistently held?

• Questions about location (who, what, where and when) Who changed their minds about returning to work, and in what contexts? Who made decisions earlier or later, and who was still undecided?

• Questions about explanation (the drivers for change) How did the decision to return to work come about and what were the influences? What part was played by the Job Retention and Rehabilitation Pilot, the NHS, the employer, financial pressures or other changes in the person's life? Why did other people make the decision earlier? What strengthened the decision or made it less firm?

• Questions about evaluation (the experience and appraisal of change or lack of change) How did people feel about returning to work, at different stages in their decision making? If they remained firm in their decision, how was this experienced?

• Questions about consequences (further changes, new directions, new trajectories, loss of opportunity) What do people expect to happen next, given their decision? What does it mean for their engagement with or requirements of the Job Retention and Rehabilitation Pilot? What changes to their daily life does it mean and what are the consequences?

• Questions about personal meanings (the perceived importance of the change, the impact on identity, self-construction and intention) How important to the person is the decision to work? Do they see themselves or their future differently?

• Questions about policy meanings (the fit with the underlying theory of change, the way the policy is working) Did the decision to return to work come about as the policy theory would envisage? If so, what components of the Job Retention and Rehabilitation Pilot and what contextual changes were important? Is the decision likely to lead to a sustained return to work?

• Questions about reflections (the interpretation of the data, alternative readings) Am I right in interpreting the decision to return to work as a change? Might I have misread earlier data, or failed to ask appropriate questions? How far does the data reflect the way the person chose to represent themselves in the research?

Concluding comments

Longitudinal qualitative research can make an important contribution to policy evaluation, and to both process evaluations (exploring what works in delivery) and outcome evaluations (exploring what works for whom, how, why, and under what conditions). It is particularly valuable in allowing us to see the shifting relationships between people and services or policies, to see how changes in the personal context shape service use and service requirements. It is an approach which essentially analyses policies in their real social setting, allowing us to see the dynamic interplay between the individual, service, policy and wider structural domains, and through this to understand the mechanisms and conditions that contribute to, or hinder, outcomes and change.

The analysis of longitudinal qualitative data calls for repeated readings, interrogating the data and asking different questions of change, and its absence. We need more discussion and more examples of the practical and the intellectual processes this involves.