The generalisability trap

The issue of generalisability permeates HSR and has its origins in an overly narrow evidence-based medicine (EBM)-type approach, which owes more to observing the tenets of a spurious scientism in order that broad claims can be made for the findings that may satisfy policymakers (Frankford, Reference Frankford1994). However, invariably, HSR involves understanding ‘wicked problems’, which get played out (or not) in complex settings that themselves are highly contextual and not susceptible to generalisation or to producing a neat set of conclusions and recommendations which can be rolled out across whole health systems with similar effect (Rittel and Webber, Reference Rittel and Webber1973). Clinical practice involves uncertainty and meeting patient needs that are invariably heterogeneous and multiple. In such contexts, the certainty offered by science, as Klein noted some time ago in his critique of the ‘new scientism’, will ‘prove elusive’ (Klein, Reference Klein1996). A risk with trying to produce useful generalisations is that they tend to be pitched at such a level of abstraction as to be meaningless when applied to a given context.

Knowledge mobilisation

A national review of health research in the United Kingdom identified what it termed ‘translational gaps’ in the process, particularly when it came to implementing research findings and using evidence to change clinical practice and health care delivery (Cooksey, Reference Chinitz and Rodwin2006). As a result, there is growing interest in, and a burgeoning literature on, related issues around knowledge mobilisation, knowledge transfer, knowledge exchange and knowledge brokerage. Sometimes the terms are slippery, lack precision and are used interchangeably; however, what all have in common, and are seeking to address, is the relationship between those producing research and those consuming it that can often be difficult and not as mutually rewarding as it ought to be. In particular, the interests of research users need to be included at the outset of a study during the conception phase. Although Holland rightly draws attention to such issues, he perhaps underestimates the difficulties of developing research based on co-production principles where researchers, on the one hand, and policymakers and practitioners, on the other, require a better and deeper understanding of their respective worlds. There are implications for training and developing researchers who perhaps need to show a bit more humility when confronted by the messy, complex real world of policy and practice involving politics and power plays.

Ensuring impact

Hardly surprising, then, that a third dilemma that confronts HSR is that, despite the millions spent on it, its impact on the health system is variable. Governments fund the NIHR and the various research councils; but are they getting value for money from their investment? Until recently, that question was rarely raised and certainly not in public. The measure of success lay in having a skilfully crafted research proposal positively peer-reviewed and then funded. Claims about impact and making a difference were rare, if ever pursued, and success was deemed to have been achieved once the final report received approval and entered the public domain. Whether anything happened to it was not something researchers generally worried about or considered to be their business. Consequently, the conclusion reached recently by two observers of health policy and management seems apposite: “Despite decades of policy and management interventions to make health care organisations more effective, efficient, and equitable, ebbs and flows in managed care, and grouping and regroupings of hospitals and physicians, little has changed in the basic arrangements within which physicians practice” (Chinitz and Rodwin, Reference Cooksey2014: 1119).

The reasons for such a state of affairs are complex and multiple. Some are cultural in that, unlike clinicians, few health service managers seem to regard research as being of value to their work. This is not to suggest HSR makes no difference – a review of one of the NIHR’s funding streams demonstrated that research had influenced policy and practice (Peckham et al., Reference Peckham, Willmott, Allen, Anderson and Goodwin2008) – but that impact is not commensurate with the significant investment in research. Observers of the NHS were taken aback when its new chief executive, Simon Stevens, actually mentioned the importance of research and heeding its findings in a recent speech to an audience of NHS managers. No one can recall any previous occupant of that post giving such an explicit endorsement of the value of research.

Thanks to Cooksey and the UK’s Research Excellence Framework 2014, the issue of impact is now high on the research agenda, too high for some academics’ liking who regard the whole business as distasteful, distorting and putting at risk the way research is currently conducted and funded. If not quite a ‘game-changer’, the emphasis now placed on impact and devising impact case studies to demonstrate what has changed as a result of some piece of research being adopted by policymakers and practitioners comes close. Irrespective of the teething problems and shortcomings arising from the approach to assessing impact in its first introductory phase – and there are many – ensuring impact is here to stay. This is not just a consequence of the industry that has rapidly burgeoned on the back of it, adding to universities’ already bloated management overhead, but because the genie is out of the bottle and cannot be stuffed back in again. Conducting research for its own sake is no longer sufficient, especially when it comes to HSR. Of course there are risks with such an instrumental, reductionist view of research, but at the same time there are questions to be asked about how research gets used in practice and if not why not. As things stand, research is being stockpiled often with new studies seeming to replicate earlier ones, which have had limited if any impact. It may be time to interrogate more critically the fate of that body of research and insist that future research is able to articulate, in the new jargon, ‘pathways to impact’.

Of course, as Holland describes, HRS has often included an applied or action component; however, there is now a strong expectation, if not requirement, that there has to be some tangible evidence of the research commissioned having changed something somewhere. A dilemma with this laudable aim is that it is incredibly hard to document with certainty what precisely has changed as a consequence of a particular study. In complex systems, attributing causation to a piece of research will rarely, if ever, be possible or make complete sense. Regrettably, it will not prevent the ‘gaming’ that is already becoming a feature of the attempt to demonstrate unequivocally that some research has significantly changed practice or a policy. At best, research is likely to have an impact if it converges with other factors such as political will, public opinion or some other trigger. Kingdon’s notion of ‘policy windows’ captures this dynamic (Kingdon, Reference Kingdon1984). However, few researchers actively seek to align their research with such policy openings and if there is convergence it is often more a matter of luck rather than design.

The discomforts of researching the policy process

Raising difficult or unpalatable truths can put researchers in an invidious position vis-à-vis policymakers, especially when the majority of funded research comes from government. Various problems, barriers and obstacles can arise, including the following:

1. 1. Grant starvation – the research is not welcome or deemed necessary.

2. 2. Denial of access to conduct field work.

3. 3. Restrictions on, or delay in, publication because the findings are critical of government policy or ‘off message’.

4. 4. An unbalanced focus on the good news, not the bad.

5. 5. Inconvenient findings that are ignored, buried, rubbished.

6. 6. Researchers attacked personally for their motives and/or blacklisted with associated consequences when it comes to accessing research funds.

However, such trials and tribulations tend to be at the extreme end of the fate that can befall some researchers who are labelled arch critics of government policy and risk being blacklisted. More commonly, the complaints are far less extreme or invidious and reflect a general frustration on the part of researchers that they are not listened to or that events move so swiftly that by the time the research delivers the policy and organisational landscape may be transformed (Hunter, Reference Hunter2008). It may be why, as Holland notes, using management consultants holds considerable appeal, especially gold-plated global players such as McKinsey’s. Of course, it is easy for them because they do not have to navigate a hugely elaborate and bureaucratic ethics and research access process, which can both sap the energy and patience of academic researchers and take up to a year or more to complete if multiple sites are involved. And then at the end of the process, before the final report can be published it has to go through elaborate peer review, editorial checks and copy-editing procedures, which may take another year to complete. Although all this process activity may have its merits in terms of assuring quality, given the pace of change in health systems, it can result in important research missing the tide. What might seem timely can rapidly become dated and unfashionable even if the key messages are of enduring value. Timeliness and impact go together.

However, researchers are not blameless in terms of their behaviour and/or a failure to appreciate the way complex organisations work and engage with research. As Raymond Illsley pointed out in his Rock Carling monograph, often researchers are, with understandable reason, so fixated on their careers and the need to publish in respectable peer-reviewed outlets that they pay scant heed to the needs of policymakers and how dissemination of research might best be undertaken to meet their needs (Illsley, Reference Illsley1980). There is an assumption that sound evidence ought to be enough of a pull to get enacted without having to consider whether or not it needs a bit of a push to get it recognised and, hopefully, adopted.