In the future historiography of policy fiascos, the Health and Social Care Act of 2012 is certain to find a place. True, it was not the kind of complete and utter failure classically represented by Mrs. Thatcher's introduction of a poll tax, which ended with the reversal of the policy and the dethroning of the Prime Minister. The 2012 Act is unlikely to be repealed in toto, though it may well be modified at the edges. Its author, as Secretary of State for Health, Andrew Lansley, has been shunted into a low-visibility ministerial position, but the Prime Minister's position has not been directly threatened. It is even conceivable – just – that the restructured National Health Service (NHS) will eventually work as intended. Therefore, it may be premature to classify the Act as an unambiguous exemplar of policy failure. However, indisputably, the passage of the Act, with its attendant political drama, has damaged both the Coalition Government and the NHS. The Government has incurred great political costs by driving through a reform programme that incurred the wrath of NHS professional bodies and trade unions alike and, in so doing, undermined voter confidence in its stewardship of the service. The NHS has been thrown into the turmoil of radical organisational changes, with the attendant morale sapping churning of people and posts, at a time when it is the midst of the most formidable fiscal challenge in its history: that of finding £20 billion of savings out of a total budget of £100 billion.

Come the 2015 General Election, and the NHS will therefore be a vote-losing liability for the Coalition Government. Given the economic pressures on the NHS, it was always likely to be so. However, the Health and Social Care Act has compounded the effect of fiscal pressure, whereas the benefits predicted by its proponents will at best appear in the longer term and, as always, it will be difficult – if not impossible – to disentangle the effects of the policy changes from other factors. If not yet a demonstrable policy fiasco (though many will think it so), the Act is emphatically a political fiasco. What, then, explains the Coalition Government's insistence on pushing it through, an example if ever that was of political self-harm if not suicide? And what does the story of the Act tell us more generally about the policymaking process in the health care arena in England? (Scotland is, of course, a different story). In what follows, we draw on the two books under review to explore the puzzle.

Nicholas Timmins addresses the question of how the 2012 Act came about directly (Timmins, Reference Nicholas2012). His study, drawing on interviews with some 30 participants in the policy process, examines the genesis of the legislation, the coalition of opposition to it and its evolution over the two years that it took to translate the initial proposals into a much amended statute. In doing so, he analyses the various twists and turns of policy as the Coalition Government sought to make the legislation acceptable. It is a dramatic story – for example, last minute concessions were needed to get the Bill through the House of Lords – told with flair and style. In contrast, the study by Robert Hazell and Ben Yong does not have health policy as its focus (Hazell and Yong, Reference Robert and Ben2012). Its aim is to analyse the workings of the coalition government, as illustrated by the first 18 months or so of its existence, and what political cohabitation means for ministers, civil servants and MPs. Again, it draws on extensive interviewing with participants. Therefore, it provides context for the more specific Timmins study, and it features the 2012 Act as one of a number of short case studies. Both books conclude with recommendations for doing things better in future, about which more below.

The simple, Cleopatra's nose explanation of the Coalition Government's decision to dive over the parapet with its radical reshaping of the NHS, and only realising that it was hurtling towards disaster when halfway towards hitting the ground, is that personalities matter in policymaking. If Andrew Lansley had not been appointed Secretary of State for Health, the story would almost certainly have been very different. Lansley was a man with a mission. As the Conservative party's spokesman on health care in the previous Parliament, he had developed his own blue print for the NHS. A conspicuously single-minded man, resistant to any suggestion that his ideas might not match the complex reality of the service or that a less ambitious strategy might produce most of the results while limiting the risks, he brought his blue print with him when appointed Secretary of State for Health. He was determined, as he told Timmins, to “entrench a consistent and coherent structure of reforms” so that the NHS could evolve autonomously and that “things would not just change at the behest of a secretary of state…or a change of government”. It was a vision essentially of a depoliticised and devolved NHS, driven by consumer choices and provider competition, with a Commissioning Board and Monitor interposed between the service and the Secretary of State. And so the story of the 2012 Act began.

However, personalities only matter to the extent that the institutional setting permits. Therefore, we come to the crucial question of why the Coalition Government gave Lansley a free rein in the first place. Reading the 2010 election manifestos of the three parties suggested an emerging consensus about the main elements of the NHS. All three abjured major organisational restructuring; all three were committed, if to varying degrees, to choice, competition and private sector involvement. Nothing in the initial Conservative–Liberal Democrat coalition agreement changed that impression: the only mention of the NHS was that its funding should increase in real terms in every year of the Parliament. And while the subsequent Coalition programme for government had a long list of policy proposals, this was prefaced by the resounding declaration of “We will stop the top-down reorganisations of the NHS that have got in the way of patient care”. It trailered some of Lansley's proposals, notably that for an independent NHS board and a cull of health quangos. Interestingly, however, it also included the Liberal Democrat proposal for directly elected individuals on the boards of primary care trusts (PCTs). No hint here that the PCTs were about to be axed by Lansley and replaced by consortia of GPs. In all, Hazell and Yong calculate that 60% of the programme's health section was based on Conservative policy and 37% came from the Lib Dems, whereas 3% could be found in both manifestos. The result was, as one insider put it to Timmins, “half horse/half donkey”: a political deal, not a coherent plan.

Neither Lansley nor Department of Health civil servants had a direct input into the programme for government; Lansley, Timmins reports, saw a draft and thought it was ‘completely nuts’. In the outcome, however, Lansley simply ignored the programme and steamed ahead with his own agenda. Hence, we come to the heart of the puzzle. Why did the Coalition Government allow Lansley to steam ahead? Just before the publication of the White Paper setting out the Lansley programme, there was a meeting of the Coalition Committee – set up to resolve any tensions between the partners – where some worries were expressed. After that, however, there appears to have been acquiescence.

This is all the more surprising, given that the coalition government believed in cabinet government. David Cameron as Prime Minister saw himself, as The Economist put it, as a chairman rather than chief executive. Therefore, the Lansley proposals were duly considered first by Cabinet and second by the Cabinet's Home Affairs Committee or its Public Health Sub-Committee. Neither Timmins nor Hazell and Yong tell us what went on at these meetings. This suggests that none of the 100 or so policy participants that they interviewed between them reported any significant Cabinet opposition to the Lansley programme. Absence of evidence cannot necessarily be interpreted as absence of criticism. However, given that those interviewed had a strong self-interest in showing themselves to have been prescient about the problems ahead, their silence strongly suggests that the Cabinet did not predict political disaster. Or, if they did (and it would be remarkable if Kenneth Clarke, a former Secretary of State for Health, had no inkling of trouble to come), they apparently kept their doubts to themselves. We will not know for sure until the minutes of the Cabinet meetings become accessible.

The apparently passive role of Cabinet is perhaps not all that surprising, given the circumstances: inexperienced Ministers in a new government were perhaps more anxious to pitch for a place in the legislative programme than to scrutinise proposals that did not touch their own departments directly. Criticism of a colleague might well lead to subsequent retaliation, as anyone who has served on an academic committee knows, so why invite trouble? But there may be a simpler – less cynical – explanation. Conceptually, Lansley's proposals were evolutionary: building on the Labour Government's policies. Only in organisational terms were they in any sense revolutionary. Lansley had persuaded himself – and may have persuaded his colleagues – that he had got the doctors on board. Moreover, many of the most explosive issues raised by the proposals only emerged when they had been translated into the legislation put to Parliament. Even professional and public opposition was something of a slow burner. Therefore, perhaps the failure of Cabinet to anticipate political disaster is neither surprising nor evidence of institutional ineffectiveness.

Both Timmins and Hazell–Yong strongly argue for the importance of another factor: the absence of special political advisers (spads) with relevant health policy experience. The Cameron Premiership started by dismantling the No. 10 policy unit and cutting the number of spads supporting individual ministers, a policy that was subsequently to be reversed. In short, there was no one at the centre to send up alarm signals. Not only did the Cabinet but also the Prime Minister initially played a passive role. This changed as mounting opposition to the legislation and evidence of public anxiety threatened Cameron's project of detoxifying the Conservative Party – in which demonstrating that it could be trusted with the guardianship of the NHS played an important part. Further, when Nick Clegg, the Deputy Prime Minister – pushed by his party rank and file – announced that future Lib Dem support for the legislation was contingent on substantial changes, Cameron took charge. The decision to halt the legislative process and set up what amounted to an instant Royal Commission (the NHS Future Form) was imposed, as Timmins reports, on a humiliated Lansley. And the subsequent adoption of the Forum's recommendations smoothed many of the rough edges of the legislation, even while introducing some cognitive-dissonant complexity: for example, the duty imposed on Monitor to promote not only competition (as initially proposed) but also integration.

But what was the role of the civil service in all this? After all, it is civil servants who are supposed to advise ministers on policy and to warn them about political or administrative dangers. Did they warn the Secretary of State about the “bucket load of trouble” (Lansley's own phrase, quoted by Timmins) that his “consistent, coherent and comprehensive” set of reforms would precipitate? Civil servants, Timmins reports, did indeed warn Lansley that he was walking into a political minefield – and produced papers on alternatives – though accounts varied about how determined the department was in trying to persuade a minister resistant to counsels of caution. Hindsight may, of course, have persuaded those interviewed to exaggerate their willingness to confront a determined Secretary of State, but Timmins is inclined to give them the benefit of the doubt.

The case, however, raises a larger question about the role of civil servants in policymaking. The point was well made by the authors of the inquest into the poll tax fiasco: the professionalism of the modern civil service means that “far from seeking to place a drag on controversial or apparently unworkable policies…places a premium on giving effect to them – once it is clear that they are the will of the government as a whole, and not the whim of a single minister” (Butler et al., Reference Butler, Adonis and Travers1994, p.209). As that was written, successive governments have emphasised even more the responsibility of civil servants for assuring the delivery of policy. Thus, civil servants are in a quandary. Questioning Ministerial policy is apt to be interpreted as disloyalty by their political masters – civil servants pursuing their own agenda as against implementing the government's democratic mandate – although failing to do so may well be interpreted as spinelessness by outside observers and future historians.

The Department of Health is probably an extreme example of a general Whitehall phenomenon in this respect. Nowhere else has there been more emphasis on delivery, unsurprisingly as no other department is directly responsible for a service as large as the NHS. Moreover, the Department is unique in the extent to which its top civil servants have been recruited directly from the service for which it is responsible: the NHS. And although ex-NHS managers have expertise in policy implementation and delivery, it is less obvious that they also have skills in policy formulation and drafting legislation. Whether this explains the sloppy way in which the Health and Social Care legislation was drafted in the first place – not all the 1000 amendments to the Bill during its passage through Parliament were introduced to placate opposition – is not clear. However, certainly the haziness of some of the Bill's initial provisions (did anyone really think that GP consortia could be allowed to determine their governance and size?) does not suggest that the traditional civil service virtues of testing the feasibility and coherence of policy notions and attention to detail are as valued in the Department as they should be.

What of the future? The Lansley fiasco was the only most extreme example of the Coalition Government's tendency to fire off half-baked proposals in its haste to make the most of its five-year tenure: the miscalculation about the cost of student loans and the aborted sell-off of national forests are other instances. Therefore, any recommendations for the future are not necessarily specific to the health care arena. Timmins and Hazell/Yong have largely agreed in the lessons they draw. Should there be another Coalition Government, much more time should be spent in negotiating a coherent action programme. The important function of policy advisers should be recognised, drawing more expertise into government, as should the importance of having a strong unit at No. 10. Further, there is a case – as Timmins argues – for explicitly recognising the case for evolution rather than revolution in policymaking: incremental rather than synoptic policymaking in terms of the old text book debate. Ministers like to present themselves as bold innovators rather than as cautious sock darners, filling in the holes in their predecessors’ policies. Thus, Lansley's insistence on stressing the ground-breaking nature of his proposals, rather than acknowledging continuity, only fuelled the paranoid perception of a plot against the NHS, to which might be added the Government's own recipe for better policymaking, an unwitting and no doubt unintended comment on the Lansley fiasco (HM Government, 2012, p. 14). This is to establish ‘a clear model of open policy making’, encouraging external challenge and insights from external experts into the process of policy formation. Hope springs eternal, of course, but the thought that ministers intent on making their name and with a short life expectancy in office will in future defer to evidence and expertise stretches optimism to a breaking point.