The Rothschild reforms: an overview

Initiated by the Conservative government under Edward Heath, proposals for the future management of publicly funded research were set out in a Green Paper, published in November 1971.Footnote ⁶ The Green Paper bound together two reports, one of which dealt with the organization and management of government research and development (R & D). Lord Victor Rothschild, head of the Central Policy Review Staff (CPRS), was named as the author of this report and took personal responsibility for its preparation.Footnote ⁷ Rothschild's iconoclastic tone contributed to the reaction that followed. So too did the decision to bind his text into the Green Paper alongside a second report of different provenance, less inflammatory content and more conciliatory style – that of a working group of the Council for Scientific Policy (CSP) on the future of the research councils. The inclusion of a short covering memorandum by the government, which clearly favoured the recommendations of Rothschild over those of the CSP, together with a short consultation timetable, set the stage for a notable controversy.Footnote ⁸

Rothschild proposed a scheme with four main components. First, he drew a distinction between basic and applied R & D. He was characteristically impatient with any ‘semantic’ discussion about this distinction, saying only that the goal of applied research was ‘practical application’, which might be a product, process or ‘method of operation’. In contrast, basic research was directed towards the acquisition of knowledge for its own sake, with any practical application as a secondary outcome.Footnote ⁹ Second, Rothschild mandated the customer–contractor principle for all applied research funded by government. ‘Customers’ within government departments would be required to specify their R & D requirements and procure service from willing and competent contractors. Rothschild was adamant that this principle should govern all applied R & D, whilst being inapplicable to basic research. Third, and most contentious, he proposed that funds for the commissioning of applied R & D should be transferred from three research councils to ‘customer’ government departments. Last, Rothschild prescribed specific organizational arrangements, to apply to each of the departments involved. These included the appointment of chief scientists to provide scientific support to the customer function; and ‘controllers R & D’ to oversee both programmes of intramural research and the commissioning of research from extramural providers (including the research councils).Footnote ¹⁰

The ensuing debate was described at the time as ‘unprecedented in the interest it aroused, and unprecedented too in the ferocity with which it was often conducted’.Footnote ¹¹ It was played out publicly in the correspondence section of The Times, which published over 120 letters on the Green Paper, and before the House of Commons Select Committee on Science and Technology. Formal consultation elicited over four hundred submissions, of which 58 per cent were opposed to Rothschild's proposals, with only 26 per cent supportive.Footnote ¹² Regardless of this opposition, a White Paper was published in July 1972. This included a commitment to all four of the principal components of reform, whilst conceding some more minor policy changes and adopting a slightly more conciliatory tone.Footnote ¹³

The three research councils affected by Rothschild's proposals were the Medical Research Council, the Agricultural Research Council (ARC) and the Natural Environment Research Council (NERC). The principal customer departments were to be the DHSS, the Ministry of Agriculture, Fisheries and Food (MAFF) and the Department of the Environment (DoE) respectively.Footnote ¹⁴ The ARC suffered the greatest budget reduction, with a transfer of £10 million or 53 per cent of its budget. The MRC lost the smallest proportion of total budget, at £5.5 million, or just under a quarter. No detailed justification was given for these percentages, which were based on the CPRS assessment of the amount of applied research undertaken by each council. Transfers were to be phased in over three years, beginning in April 1973, and were conditional on departments having established their scientific staff before the start of this transitional period.Footnote ¹⁵ Although no formal conditions were placed upon the use of transferred funds, there was an understanding that these would be used to commission work from the research councils. This would effectively return money to the councils, but with the interjection of accountability through commissioning.Footnote ¹⁶

Medical research: the exceptional case

In November 1979, the DHSS decided that the system for commissioning medical research should be abandoned and funds returned in full to the MRC.Footnote ¹⁷ The sum involved, £13.9 million at 1980 prices, was returned to the science budget in April 1981. This decision went against the grain of national science policy. It was justified with reference to the exceptional nature of medical research, or more specifically, and in a significant linguistic turn, biomedical research, and this justification was seized upon by other government departments to deny the possibility of similar reversal elsewhere.

A pan-government review of the implemented Rothschild system, published as a White Paper in March 1979, had been positive in its assessment, concluding that the reforms ‘appear to have strengthened the government's R & D machinery’. The introduction of the customer–contractor principle, together with the build-up of scientific support, had ensured that government departments were increasingly able to function as ‘enlightened customers’.Footnote ¹⁸ The responsible minister briefed the prime minister that ‘the customer–contractor arrangements are working reasonably well and … no major changes are necessary’.Footnote ¹⁹

No specific measures had been proposed for medical research. A recent simplification of commissioning arrangements was noted as a positive development.Footnote ²⁰ Nothing of what was reported for health in the ‘review’ White Paper was obviously exceptional. Although the MRC and the health departments had found some adjustments necessary, teething pains were also noted in the fourteen non-health departments commissioning applied research from the research councils. The impact on the two other councils affected, the ARC and the NERC, had been greater than that experienced by the MRC, because a larger proportion of their budgets had been transferred. The working out of the customer–contractor relationship had not been straightforward for either the ARC or the NERC, but neither sought a reversal in this period.Footnote ²¹ The NERC had to deal not only with the Department of the Environment, its principal customer, but also with three other departments. Writing before the DHSS took its decision, policy analyst Philip Gummett portrays the MRC relationship as relatively straightforward. After some initial difficulties, he says, ‘the new arrangements seem … to have settled down’.Footnote ²²

Against this background, the decision to reverse the Rothschild system for biomedical research, taken only eight months after the publication of the review, appeared anomalous. Other government departments asserted that this step could only be justified by the unique circumstances of biomedical research. The Cabinet Office sought views from customer departments about the proposals. For MAFF, the permanent secretary replied that his department's policy was to extend, rather than reduce, the scope of the customer–contractor principle and that it would ‘not wish doubt to be cast on Rothschild principles generally’. Because, he added, ‘the considerations relating to biomedical research are peculiar to that field, there should be no difficulty in drawing the necessary distinction’.Footnote ²³ The Departments of Transport and Environment, in a joint response, confirmed that whilst they had no objection to the proposals, they did not want to see any precedent established. Instead they urged that ‘these discussions be confined to the very special relationship between the MRC and the health departments.Footnote ²⁴ Other departments replied in a similar vein.

The official narrative of government was soon adapted, whether consciously or unconsciously, to draw a veil over this awkward departure from policy. Within a year, the Lords Select Committee on Science and Technology had produced a report providing further reassurance that the customer–contractor principle was working well across government and did not need changing. The committee did not see fit to comment on recent exceptional events in health.Footnote ²⁵ A decade later, an updated national strategy for science and technology confirmed that ‘the Government has concluded that the Rothschild principle remains as valid today as it was twenty years ago’.Footnote ²⁶ No explanation was given as to why this had ceased to be true for biomedical research.

The reversal of Rothschild for biomedical research was, then, an exceptional event. The question to be addressed in what follows is how this exceptionalism can be explained. Before offering any answers, it will be helpful to take a closer look at the context for, and implementation of, the Rothschild reforms at the DHSS.

Commissioning health research

Even before Rothschild, the department had gained experience in the commissioning of research, developing its own R & D programme during the 1960s. This was broad in scope, responding to the department's diverse responsibilities in public health, health services, personal social services and social security.Footnote ²⁷ Although never operating under fully integrated management, from 1967 onwards much of the programme came under the direction of a Research and Development Committee, serviced by a Statistics and Research Division. The DHSS never arrived at an all-encompassing descriptor for the research it procured, but ‘health and personal social services research’ (HPSSR) was, by 1971, established usage for the principal streams.Footnote ²⁸ The department had adopted the customer–contractor principle for HPSSR well in advance of Rothschild, with a requirement, introduced in 1967, that internal customers be identified for all research projects.Footnote ²⁹

Once the framework White Paper was published in July 1972, government departments had eight months to establish their infrastructure for scientific support and to enter into commissioning agreements. At the DHSS, this task coincided with a major reorganization, implemented in two phases between 1972 and 1974 and prompted by various considerations, not least the impending reorganization of the NHS. Research management was a relatively minor aspect of this much bigger picture.Footnote ³⁰ The challenges for the department were how to integrate such management into an organizational design shaped by doctrines of ‘planning’, and how to graft the commissioning of biomedical research onto pre-existing structures and processes for HPSSR commissioning. The R & D organization created in response to these challenges soon proved dysfunctional. It depended upon a plethora of committees and control appeared more fragmented than under earlier arrangements. The chief scientist lacked any executive authority or supporting infrastructure and was compelled to rely heavily on a phalanx of scientific advisers. These advisers, who were also sometimes research providers, pursued their own agendas, leading to overdominance of advisory bodies by academic values.Footnote ³¹

By 1977, these shortcomings had been acknowledged. In that year, the Civil Service Department (CSD) instigated a ‘management review’ of the department. On R & D, the preliminary review report concluded that ‘the Department had failed to establish an overall approach to its research expenditure’ and unearthed widespread confusion about how the system was supposed to work.Footnote ³² These preliminary findings prompted a more detailed study, which ran to over one hundred pages. The structures and processes implemented in 1973 were damned with faint praise, being described as ‘a first step in the difficult process of trying to relate research planning to policy priorities’.Footnote ³³

In response, the department set about simplifying committee structures and strengthening the role of the chief scientist. A cull of R & D committees ended with most having been dismantled by 1977. The department also made the chief scientist an executive role, supported by an Office of the Chief Scientist (OCS) with a dedicated research management staff. These changes were agreed in advance with a new incumbent, Professor Arthur Buller, who took up office in August 1978.Footnote ³⁴ Under Buller's leadership, the OCS was to assume responsibility for most of the HPSSR programme, although some strands of research were to remain under the management of specialist branches, such as computing, supplies and buildings. The OCS would also manage the new ‘streamlined’ system for the commissioning of biomedical research.

After a laboured start, the DHSS had thus, by 1978, belatedly assembled an organizational response that was consistent with Rothschild's framework and that appeared fit for purpose. The customer–contractor principle had been institutionalized for applied research. The chief scientist had acquired executive powers and a supporting organization. The OCS had been given a mandate and sufficient resources to manage the core HPSSR, biomedical research and social-security research programmes.Footnote ³⁵ The only notable departure was that no controller R & D had been appointed. This was felt to be unnecessary because, unlike most other civil departments, the DHSS had only two in-house research units.Footnote ³⁶

By this time, concerns about accountability were becoming more salient. R & D commissioning across government was scrutinized by the government's auditor, the comptroller and auditor general (CAG), in 1977.Footnote ³⁷ The CAG found that the MRC still maintained that the customer–contractor principle was inappropriate for most biomedical research; and ‘that by transferring to the Departments a much larger sum than could be used on such projects, the Government intended that the customer/contractor principle be more broadly interpreted’. The report went on to observe that the commissioning arrangements for biomedical research appeared to leave more control in the hands of the MRC as a contractor than in the hands of the health departments as customers. In view of this, the auditor questioned whether these arrangements could be said to truly comply with this principle.

The Committee of Public Accounts (CPA) then decided to investigate the auditor's findings. DHSS permanent secretary Patrick Nairne, Chief Scientist Arthur Buller, and MRC secretary James Gowans, together with representatives of the Department of Education and Science (DES) and the Scottish Home and Health Department (SHHD), gave evidence before the committee in March 1979.Footnote ³⁸ Gowans put up a trenchant critique of the Rothschild reforms, resting on the ‘peculiar difficulties of biomedical research’. Nairne did not respond directly to these criticisms but was instead broadly positive in his assessment of the commissioning system. Although yielding little ground, he confirmed the department's commitment to a review of the simplified arrangements for biomedical research commissioning, promising this for autumn 1979. This pledge originated in Gowans's campaign to secure a return of biomedical funds. He had written to the Secretary of State for Social Services in April 1977, arguing that the system was not working and that funds should be returned to the MRC. At that time, the department was moving towards simplification of the system for commissioning biomedical research, as later implemented in April 1978. The Secretary of State was thus able to argue that such a move would be premature, whilst conceding a review in due course.Footnote ³⁹

The possibility that biomedical funds might be returned to the MRC was included within the scope of this new review from the outset but was talked down by DHSS officials when work began in March 1979. It became an altogether more serious prospect once the CPA reported in September. The committee found that the department ‘had largely ceded to the MRC the customer's normal responsibility for defining the objectives of commissioned research and for controlling the allocation of resources to it’. In the committee's opinion, this represented an unacceptable blurring of accountability. Its members were persuaded by Gowans's arguments that it was unrealistic and impractical to expect the health departments to ever develop sufficient competence for the commissioning of biomedical research. They recommended that the government should ‘give full weight to the possibility of abandoning the formal commissioning arrangements in this field if they add nothing of substance to the guidance and advice which the Health Departments could, in any event, continue to provide through the improved arrangements for consultation and liaison with the MRC’.Footnote ⁴⁰

On 22 November 1979, a meeting between the Secretary of State, the minister for health, Nairne, Buller and a small number of other senior officials was held to consider the courses of action open to the department. Armed with a paper prepared by Buller, Nairne offered three options. The first was continuation of the status quo. This was deemed ‘unacceptable’. The second was to return all or part of the biomedical funds, subject to agreement over new liaison mechanisms and the MRC taking on more of an active role in health services research (HSR).Footnote ⁴¹ The third was essentially a version of the second that included slightly more department control. Ministers opted for the second option.Footnote ⁴² Buller had been in dialogue with Gowans prior to this meeting to ensure that this outcome would be acceptable to the MRC.

Once this decision had been taken, the DHSS notified other departments and sought their views, beginning with the SHHD and the Welsh Office. Both were content with the proposals. The DHSS also sought the agreement of the DES, the Cabinet Office, the CSD and the Treasury. The DES permanent secretary, James Hamilton, pointed out that any return of funds from the DHSS would be to the science vote, whence money had originally come, rather than to the MRC directly. Such funds would then be subject to established procedures for allocation of that vote between the research councils.Footnote ⁴³ Hamilton also worried about the proposal that the MRC should take on more of an active role in HSR, doubting that this was an ‘appropriate responsibility for the science budget’, a concern echoed by CSD. The Treasury's view was that the DHSS should make more effort to improve commissioning arrangements, rather than abandoning them.Footnote ⁴⁴ Nairne had to work hard to overcome these various concerns, invoking the CPA report and the ‘special status’ of biomedical research. The fact that commissioning under the customer–contractor principle would continue for HPSSR also carried weight, and the CSD eventually authorized DHSS proposals in March 1980.

The intention to return biomedical research funds in full and with effect from 1 April 1981 was announced to Parliament by the Secretary of State, Patrick Jenkin, in a written answer on 28 October 1980.Footnote ⁴⁵ Jenkin stated that this move held no implications in other fields of research, where the customer–contractor relationship would continue to apply. The new arrangements, including greater MRC commitment to HSR, were also communicated to the medical profession in a joint letter from Buller and Gowans.Footnote ⁴⁶ The £13.9 million to be returned represented a 20 per cent uplift to the MRC budget. The council had successfully overthrown the customer–contractor principle for biomedical research and conclusively re-established its scientific autonomy. Counterreformation had prevailed, less than a decade after Rothschild.

Conventional explanations for reversal

A prominent line of explanation for these events contrasts the allegedly low levels of competence available at the DHSS and the higher competencies of the MRC. Rothschild himself later spoke of ‘the lower intellectual reputation of the health departments in comparison to the MRC’.Footnote ⁴⁷ Other senior participants may have used more tactful language but were equally willing to talk down DHSS abilities. Douglas Black, the second chief scientist at the DHSS, drew attention in his memoirs to the disadvantages facing medical research managers at the DHSS in comparison with the MRC.Footnote ⁴⁸ His successor, Arthur Buller, has argued that ‘the department simply didn't have the ability to commission biomedical research’.Footnote ⁴⁹

These contentions do not stand up well when subjected to closer scrutiny. The DHSS had steadily built up its capacity to commission research over the decade before Rothschild as required for its fast-growing HPSSR programme, which relied almost entirely on external contractors. As noted, the department adopted the customer–contractor principle four years before the Green Paper. By 1972–1973, the budget for the HPSSR programme had grown to £13.3 million, outstripping the average growth rate for publicly funded civil research and development, so that the health department's share of the total civil R & D budget grew from a tenth of 1 per cent to nearly 4 per cent over the preceding decade.Footnote ⁵⁰ This growth rate had not been achieved without competent medical leadership. The chief medical officer (CMO) in this period, George Godber, recruited the second secretary of the MRC, Richard Cohen, to join the Ministry of Health in 1963. Cohen played a major role in developing the department's R & D programme and served as the first chief scientist at the DHSS for six months, before retiring in 1973.Footnote ⁵¹ He brought with him personal and professional networks and was supported by his colleague James M.G. ‘Max’ Wilson, who had an international reputation in population screening.Footnote ⁵² Had it been so minded, the department could have continued to engage other medical doctors, scientists and administrators of comparable ability to lead the commissioning of biomedical research. Indeed, on the face of it, it did precisely this by recruiting as Cohen's successors eminent researchers: Black, who was chairman of the MRC Clinical Research Board before moving to the DHSS; and Buller, who was chair of the MRC Neurobiology and Mental Health Board. The question remains: why was the DHSS unable to build on its track record in HPSSR commissioning and develop an effective approach to biomedical research under this eminent leadership?

One response to this question is to interpret the relationship between the DHSS and the MRC as being subject to an imbalance of authority, with the advantage resting with the latter. This line of explanation is central to the analysis of Maurice Kogan and colleagues, who studied the chief scientist's organization at the DHSS between 1974 and 1981.Footnote ⁵³ Kogan and Henkel portray MRC authority as stemming from both epistemological and organizational sources. The MRC was supremely self-confident in its ‘internalist’ view of science. In this paradigm, the arbiter of good science must be the community of scientists, and science is best served by the self-governance of that community. Organizationally, the council was able to draw on a reputation for efficient management and the involvement of numerous clinicians and scientists on its panels and committees. In contrast, the DHSS was compelled to advance a less firmly rooted claim to legitimacy, based on administrative criteria and social utility as well as scientific merit. It also possessed less mature institutional arrangements.Footnote ⁵⁴

Kogan and Henkel contend that DHSS officials proved unable to put forward ideas for new biomedical research and see this as symptomatic of the wider inability of the department to assert its authority against the MRC. Of the Panel on Medical Research (PMR), the body established to oversee commissioning of biomedical research, they say that ‘its brief history provided the most severe test of government's ability to influence well-established science’.Footnote ⁵⁵ This deficiency of authority, they argue, proved fatal to the panel, which was dismantled in 1977. This led inexorably, they infer, to the return of biomedical funds. They do not, however, explain this process, providing no details on relevant events between the demise of the PMR and the announcement, three years later, that funds were to be returned. They describe in some detail the PMR's difficulties in arriving at a satisfactory modus operandi. However, such difficulties were not unique. The Health Services Research Board and Personal Social Services Research Board also struggled to find useful roles and were abolished even earlier than the PMR.Footnote ⁵⁶ This suggests that the travails of the panel were as much symptomatic of the dysfunctionality of the entire departmental R & D machinery between 1973 and 1978 as they were of the peculiar difficulties claimed for biomedical research.

Kogan and Henkel argue that ‘arguments such as those of Gowans eventually convinced the health departments to release their hold over MRC funds’.Footnote ⁵⁷ Gowans's performance before the Committee of Public Accounts has been judged especially effective. Nairne, speaking at a witness seminar some years later, recalled that the committee ‘was deeply impressed by all he had to tell them about the way the MRC saw matters’.Footnote ⁵⁸ Gowans's arguments were little different from those used by the MRC when opposing the 1971 Green Paper, repeating claims for the indivisibility of pure and applied medical research and the primacy of curiosity-led research. He repeated the contention, previously made to the CAG, that the government must have intended a loose interpretation of the customer–contractor principle when it transferred funds from the MRC to the DHSS because it did so in the face of the council's arguments in 1972.Footnote ⁵⁹

However, Gowans’s arguments did not, as it may appear with the benefit of hindsight, sweep all before them. A close reading of the transcripts shows that Nairne made no concessions of any substance. Instead, he emphasized the improvement in liaison between the two organizations stimulated by the Rothschild reforms. He described this as ‘mutually beneficial’ and concluded that ‘we are learning the whole time and we think that what we have achieved so far has been of value’. The liaison mechanisms in place were now ‘excellent’.Footnote ⁶⁰ This position was backed by Hamilton of the DES, who said that ‘it would be wrong to think that the customer–contractor relationship has failed or is in some way wanting just because of the peculiar circumstances under which the MRC works’.Footnote ⁶¹ However forceful Gowans's delivery was, at this stage he still faced opposition from both the DHSS and the DES, backed by the conclusions of the review White Paper that no change was necessary.

The argument of ‘superior authority’ is central to Kogan and Henkel's text. However, these researchers, despite their privileged access to DHSS, did not examine critical events that could have supported a more parsimonious explanation. Their formal engagement with the DHSS came to an end in April 1981, yet, as noted, they do not give any account of how the decision to return funds was taken.Footnote ⁶² This was because the researchers were, in this final phase of their investigations, focused on a study of the chief scientist's review of DHSS-funded units, and paid no systematic attention to parallel developments in biomedical research commissioning.Footnote ⁶³

A final established line of explanation stresses difficulties in financial administration. The MRC reacted with alarm to any threat to funding for commissioned research, not least because the department had pledged to protect this budget. In response to difficulties in the public finances, the department looked for economies in its 1977–1978 budget. R & D did not escape, and the budget for commissioning from the MRC was trimmed by 10 per cent, provoking bitter complaint.Footnote ⁶⁴ The MRC subsequently predicted an outturn of around 10 per cent less than the final allocation, which had meanwhile been uplifted for inflation and other adjustments. Not surprisingly, DHSS officials found it illogical that the MRC was predicting a final spend 10 per cent below contract value ‘whilst still complaining vociferously about the “cut” on every possible occasion’.Footnote ⁶⁵ This grievance found its way into the review White Paper, where it was said that the Council had been left with a ‘feeling of insecurity’.Footnote ⁶⁶

A further difficulty claimed by the MRC was that of accounting for commissioned research.Footnote ⁶⁷ The 1972 White Paper had specifically stated that grants-in-aid, which were available to the research councils from the science vote, would be inappropriate for commissioned research.Footnote ⁶⁸ Instead, customer departments were to remain fully accountable, necessitating the provision of detailed project accounting to the department by the MRC.Footnote ⁶⁹ This was not an activity for which the council demonstrated any enthusiasm. A project-costing system was introduced in 1976, but not applied to commissioned research. The explanation for this neglect was that ‘at the project level the activity was … often poorly defined and did not justify the expense of a precise system of accounting’.Footnote ⁷⁰ It is difficult to avoid the conclusion that the MRC talked up the difficulties of both budget security and accounting.

Alternative explanations for reversal

Historians and sociologists of science have drawn attention to the mutable qualities of terms such as ‘basic’, ‘applied’ and ‘fundamental’ research and to their variable usage by actors, depending upon context and intent.Footnote ⁷¹ This should prompt us to pay close attention to the language used in the Rothschild controversy and, more specifically, to how this was used to shift the grounds of the debate. The language initially adopted by the MRC in preparing its response to the Green Paper was that of ‘applied medical research’. However, the term ‘biomedical research’ was subsequently adopted to describe the mainstream of the council's programme. This semantic shift was linked to a claim about the indivisibility of the programme, resting on the axiom that biological discovery leads to medical innovation in ways that are inherently unpredictable. It was this indivisible nature of ‘biomedical’ research, it was argued, that rendered any distinction between pure and applied research impractical. From this chain of reasoning, a bald conclusion followed: ‘the customer–contractor relationship is inappropriate to most biomedical research’.Footnote ⁷²

At first sight, this appears to be a classic ‘biomedical turn’: a critical juncture in a longer-term trend towards ‘biomedicalization’. However, there are grounds for caution about such an interpretation. Despite the post-war emergence and growth of ‘biomedical complexes’, biomedicalization was not a uniform trend and other research traditions relevant to medicine, such as those of social medicine and public-health research, also continued.Footnote ⁷³ The MRC's direction of travel, since the late 1930s, had been towards diversification beyond its initial focus on laboratory science and towards the realization of its historic ambition for a comprehensive programme covering the full spectrum of medical research.Footnote ⁷⁴ In 1953, this goal had been endorsed in the report of a joint working part of the MRC and the health departments, prompted by recognition that the advent of the NHS offered a wealth of new opportunities for clinical research. The working party's vision was that the MRC's programme would, in addition to a continuing commitment to basic research in the laboratory, be extended beyond trials and studies in hospital settings to include ‘field studies in epidemiology and social medicine and observations in general practice’.Footnote ⁷⁵ By the early 1970s, the council had substantially increased its support for clinical research, as well as funding units in fields such as social medicine, applied psychiatry, industrial health and epidemiology.

The emerging programme of the DHSS had, by the late 1960s, led to a substantial overlap with these MRC interests. The HPSSR programme combined medical, social and operational research, with a strong emphasis on multidisciplinary research of practical relevance to the NHS.Footnote ⁷⁶ It included epidemiology, studies of the need and demand for healthcare, evaluation of healthcare technologies, health services research and the evaluation of medical care. The overlap with the MRC's programme was acknowledged by Richard Cohen in a talk given to his former colleagues in 1967.Footnote ⁷⁷ By way of illustration, Cohen spoke about screening for pre-symptomatic disease and ‘trends in medical care’ as falling in the ‘border territory’ between the two organizations. The DHSS was cognizant of the opportunities to commission more applied medical research in this border territory, should more resources become available. When consulted by the CPRS, prior to the Green Paper's finalization, Cohen proposed that half of the MRC's total spend on public health, clinical medicine and social medicine should be transferred to the health departments. He identified several specific areas in which the MRC's existing programme was inadequate, including immunization, nutrition, environmental toxicology, renal dialysis and epidemiology. He argued that ‘the exploitation by applied research of existing fundamental knowledge would pay the quickest dividends at the present time in improved treatment’. This ‘needed shift of perspective’ would be facilitated by increased DHSS influence over the MRC's programmes through the power of the purse.Footnote ⁷⁸

Against this background, the adoption of the term ‘biomedical research’, together with claims for the indivisible nature of such research, was more a defensive and rhetorical act than indicative of an immediate shift of resources towards biomedical research as previously defined (that is, as underpinning research into biological mechanisms).Footnote ⁷⁹ This strategy can be seen in the evolution of the council's response to the Green Paper. An early draft draws a distinction between a ‘central area of research’ and ‘peripheral areas where the research is applied to a variety of ends’. The central area involved ‘much fundamental research, physiological and pathological studies of all parts of the human system and a wide range of clinical research’. The peripheral areas included a range of applied research that overlapped with the DHSS programme. The document acknowledged that work in these areas was ‘the proper sphere of direct influence of government departments’ but then, in the next breath, went on to argue that ‘these areas are so closely related to the Council's own sphere that they cannot be expanded or contracted without having an influence on the central area’.Footnote ⁸⁰ These arguments were not altogether coherent and were rather too blatantly protective of the MRC budget. After three meetings involving the full council a less equivocal final submission emerged, including the uncompromising defence of a single, indivisible programme of ‘biomedical research’, for which the customer–contractor principle was impractical.Footnote ⁸¹ This strategy proved effective in shifting the grounds of the debate. The new, broader meaning of ‘biomedical’ served to differentiate between the programmes of the two organizations, regardless of the obvious overlap. It also fostered a perception that biomedical research operated under special circumstances, even though any critical interrogation might well have asked why such circumstances did not also exist in, for example, agricultural research.

The Rothschild reforms thus presented an unprecedented opportunity for the DHSS to devote more resources to service-oriented medical research. Yet the department entered into formal agreements dealing exclusively with ‘biomedical research’. A DHSS submission of 1973, developed in dialogue with the MRC, sets out how the new arrangements for commissioning would work.Footnote ⁸² The main supporting document begins by rejecting any distinction between basic and applied research for biomedicine, setting out the ‘special characteristics’ of the field: ‘the biological processes with which research is concerned are highly complex, and experience indicates that it is rarely possible to define the course of a research programme in advance’. The submission also restates the claim that the council's close links with the medical profession equipped it to judge the likely relevance of research. ‘This fruitful contact with direct customers’, it stated, should not be damaged by the new arrangements with the DHSS. This preamble amounts to a restatement of familiar objections to Rothschild, as originally articulated by the MRC but voiced, on this occasion, by the DHSS.

These preliminaries can be read as a ‘softening up’ for the detailed proposals that follow, which provide for de facto control of biomedical research to remain with the MRC. All the money transferred to the DHSS was to be spent with the MRC. Biomedical research paid for by the department was to be managed by the MRC. The council would be ‘the final authority for scientific policy’, setting up three new boards to replace its Clinical Research and Biological Research Boards. Each of the new boards ‘would be responsible for a full range of work within a broad clinical field’. They would ‘initiate policy within their fields’ and control funding through grants committees.

The submission proposed both ‘specific’ and ‘broad’ commissions. The former required research requirements to be defined in detail by the department. However, the agreement assumed that broad commissions would be the default, stating that specific commissions would ‘remain open in case the system for broad commissions does not work satisfactorily’. ‘Technically’, the department could refuse to pay for work approved by an MRC board even if it fell within the scope of a broad commission. However, it was noted that the department had no intention of exercising such powers because ‘the essence of the broad commissions is that the interests of the Departments will be best served by allowing the MRC to take certain decisions’. The department also undertook to manage the portfolio of broad commissions in a way that would ensure ‘a reasonably steady level of expenditure and flow of work for the MRC’. Research policy was to be jointly defined ‘without discontinuities between “applied” and “fundamental” research, between clinical and biological research, and between research in the different fields of application’.

An appreciation of this context suggests a simpler explanation for the demise of the PMR than that advanced by Kogan and Henkel. The panel was, from the outset, given no meaningful role. The submission to the Treasury specified its responsibilities. The PMR was to formulate the department's contribution to joint policy for biomedical research. Given the specifics of the MRC–DHSS agreement and the fact that this contribution had to be routed through more senior committees, this amounted to a largely ceremonial duty. It was also tasked with developing specific commissions and undertaking retrospective review of such commissions. However, as has been noted, there was a policy preference for broad commissions. Given this remit, it is unsurprising that the PMR subsequently struggled to find some purpose in its existence and was eventually put out of its misery.

This still leaves the question of why the DHSS agreed to such disempowering terms in the first place. To explain this, one must look at the status of medically qualified officials within the department, and to the values, behaviours and relationships shared by medical elites in both department and research council. Parallel structures for professional and administrative staff existed in most government departments during this period, but the size and status of the dominant specialist class, medical doctors, was unusual at the DHSS.Footnote ⁸³ Status flowed from that of the class's professional head, the chief medical officer. Under agreements originally reached in 1919, the CMO was granted pay and status equal to that of the permanent secretary, including direct access to ministers.Footnote ⁸⁴ Godber, who held office between 1960 and 1973, was committed to a sizeable medical staffing establishment, ensuring that the profession remained able to influence policy, whilst remaining unencumbered by routine administrative tasks. The size and influence of the medical civil service reached a peak in the 1970s, with substantial reductions under Conservative governments after 1979.Footnote ⁸⁵

During the ‘pre-Rothschild’ period, Cohen and Wilson led the development of applied medical research, with accountability to Godber. Other streams of commissioning, including social, operational, computer, building and supplies research, were led by officials in the administrative class. The multidisciplinary nature of the HPSSR programme and the growth of boundary-spanning fields, such as medical sociology, meant that a collaborative approach was required, and Cohen describes the engine of the programme in the 1960s as being an ‘informal team’ working across formal boundaries.Footnote ⁸⁶ During this period, collaboration between specialist and generalist civil servants fostered growth across a broad spectrum of research. The common thread was, in Cohen's words, ‘a precise and practical relevance to the NHS’.Footnote ⁸⁷ Medical officials managed the relationship with the MRC, the formal terms of which had been defined by a ‘concordat’ between the two organizations, first agreed in 1924 and most recently updated in 1949.Footnote ⁸⁸ Management of the relationship codified in this document was facilitated by the shared professional values of the medical elite. Some individuals moved between organizations. The most striking example is Richard Cohen, who worked for the MRC for fourteen years before transferring to the Ministry of Health in 1963. Godber and Cohen were on first-name terms with senior MRC officials. Influential researchers in MRC units, such as Archie Cochrane, were awarded grants by the department and were well connected to both organizations. The maintenance of cordial relations was helped by the shared educational and social background of senior members of the medical profession.Footnote ⁸⁹

The framework Green Paper placed this organizational relationship under unprecedented stress. Godber was alarmed by the damage inflicted by the controversy.Footnote ⁹⁰ For the department's medical leaders, the task of negotiating new arrangements for the commissioning of biomedical research became an exercise in damage limitation – a task that was left almost entirely in the hands of medical officials. Department-wide reorganization meant that the top team had more than enough on its plate without also being asked to implement Rothschild. Participation in negotiations was confined to members of the medical elite, with the administrative side only being informed once agreements had been reached. Specifically, the principles for commissioning biomedical research were agreed between Godber, Cohen, John Gray (secretary of the MRC 1968–1977), Douglas Black and Dr G.K. Matthew, later appointed deputy chief scientist.Footnote ⁹¹

The closing of professional ranks, with the goal of dampening down the Rothschild reforms in medicine and placating the MRC, extended to the appointment of trusted insiders to the role of DHSS chief scientist. Douglas Black, the second incumbent, was a prominent critic of the 1971 Green Paper, signing critical submissions from his clinical research unit in Manchester and appearing for the MRC before the Commons select committee. Yet within a few months of this active opposition he had been appointed chief scientist designate at the DHSS, with Cohen providing interim cover for six months prior to retirement. Black's recruitment was rapid and unadvertised, following consultation with the MRC, the medical royal colleges and the Royal Society. The permanent secretary, by now Philip Rogers, looked to Godber and Cohen for reassurance that this would be an appointment that was acceptable to the profession.Footnote ⁹² Black was able to play an active role as chief scientist designate in the group negotiating biomedical research commissioning arrangements while still serving as chairman of the MRC Clinical Research Board.

In office, Black appears to have done his best to ensure that HPSSR commissioning continued to prosper and to make the most of the dysfunctional system over which he nominally presided. For biomedical research, his approach appears to have been one of allowing the system to fail. In his memoirs, Black acknowledges his vulnerability to the charge that he had not really tried to make the system work.Footnote ⁹³ Elsewhere he has claimed that he did his best, but that ‘the thing was inoperable’.Footnote ⁹⁴ On the fate of the PMR, he claims that the root cause of failure was ‘the inability of anyone in the department, including myself, to come up with specific commissions for the MRC which would even remotely match the £5 m of transferred funds’. He then goes on to claim that this failure drove the DHSS to the ‘somewhat shallow respectability’ of the broad commissions. This explanation appears disingenuous once it is appreciated that Black was a key player in drawing up the agreement that set broad commissions as the default mode. In his private papers he notes (with inaccurate recall of exact timing) that ‘the transfer fund arrangement was happily abandoned around 1978 (my contribution being the characteristic one of showing that it didn't work)’.Footnote ⁹⁵

Black's successor, Arthur Buller, was, by his own admission, an MRC nominee and ultra-loyalist, describing himself as ‘a believer in the MRC’ who ‘felt that the transfer of these funds back from the Department to DES and hence to MRC could be achieved’.Footnote ⁹⁶ In the same interview, he describes himself as a ‘go-between’ for Nairne and Gowans. Buller was consistently dismissive of the DHSS programme and of the OCS, regardless of the additional resources and powers conferred upon the chief scientist in 1978. He wanted to persuade the research councils, and especially the MRC, to take over all the HPSSR activity of the DHSS over time: a total inversion of Rothschild's original proposals. Buller's zeal in this last respect was not shared by either his colleagues in the DHSS, who were fearful of the consequences for HPSSR, or by the MRC governing body, which was nervous about too large a health services research commitment changing the character of the organization.Footnote ⁹⁷

By 1977, the accountability deficit arising from the 1973 commissioning agreement had become a matter of concern for the government's auditor. When the CAG examined the situation, it found that the MRC had accepted commissions from government departments for £10.4 million in total, of which 97 per cent by value had been placed by the health departments. However, only two of these health commissions, totalling £110,000, were specific commissions. In contrast, other customer departments, such as the Home Office, had placed over forty specific commissions.Footnote ⁹⁸ Grants-in-aid had been proposed by the DHSS and the MRC in their 1973 submission to the Treasury, but rejected as being contrary to general guidance and specific statements in the 1972 White Paper.Footnote ⁹⁹ The system of broad commissions meant that a de facto system of grants-in-aid was being operated, as the auditor realized. In an attempt to reconcile medical autonomy and accountability, the DHSS attempted again, in 1977, to persuade the Treasury that grant-in-aid should be formally adopted for commissioned research. This last-ditch attempt was unsuccessful and soon abandoned. Rather than seeking to strengthen the system for specific commissions, which was the Treasury's expectation, Buller was making strong representations to Nairne that biomedical funds should be returned to the MRC, with additional funds transferred from the HPSSR budget for health services research. He restated his view that the department was incapable of developing the necessary competencies for the commissioning of biomedical research. Concerning HPSSR, he returned to his argument that, over the medium term, ‘it would benefit the Department's HPSS programme to establish clearer links with the research councils’.Footnote ¹⁰⁰

Nairne was faced with the realization that the current system could not solve his accountability problem without investing a great deal of effort. It was also evident that his chief scientist would provide no assistance in improving on the system, for example by making a commitment to increase specific commissions, arguing instead for its abandonment. Nairne could also see that the undertakings given to the MRC to maintain funding levels (which he restated at the CPA) would become a problem in times of financial stringency.Footnote ¹⁰¹ With the recent election of a new government committed to significant reductions of public expenditure, reductions in the science budget looked certain.Footnote ¹⁰² Protection of the budget for research commissioned from the MRC would mean disproportionate cuts to the rest of the departmental R & D budget. But, with memories of the 10 per cent ‘cut’ still raw, not to honour the undertakings given would provoke renewed clamour from the MRC. In this situation, it can only be inferred, Nairne decided that his best course of action was to persuade ministers that the system for commissioning biomedical research should be abandoned and funds returned to the MRC. In this he prevailed.

A final factor contributing to the biomedical research reversal was the relative ease with which this could be implemented. The MRC had the largest research council budget before the Rothschild reforms and suffered the smallest loss as a proportion.Footnote ¹⁰³ This relatively limited loss, combined with the mechanism of broad commissions, meant that it was possible to sustain existing programmes without significant disruption. Other government departments, such as MAFF, had substantial in-house research units, in contrast to the DHSS. Disputes arose over the future of both government and research council units, which, in the case of MAFF and the ARC, ended with three units receiving mixed funding.Footnote ¹⁰⁴ Any attempts to unstitch such institutional arrangements would doubtless have led to further organizational complications of a sort that simply never arose in the DHSS–MRC relationship.

After 1981, the MRC and the DHSS reverted to a state of peaceful coexistence. Scope for any tension that might have arisen between the two organizations was reduced by the slow pace and limited perspective adopted by the MRC in developing HSR. A grant-making committee, operating within the paradigm of medically led epidemiology, was not established until 1986. The next critical juncture in the history of health research arose at the end of the 1980s, but this was not because of renewed tensions between the MRC and the health departments. The point at issue was the distinction between the research needs of the health department and those of the National Health Service. Space does not allow for an adequate discussion of this and later institutional and policy developments, but the persistence of underlying tensions through into the present century has been commented on elsewhere and will, it is hoped, be investigated by historians in due course.Footnote ¹⁰⁵