In 1980, the Thatcher government publicly released the rules that guide the behaviour of civil servants when they appear before parliamentary committees. The rules had been around in various formats since at least the 1960s but had been kept strictly internal. The public release of the rules did two things. First, it immortalized the name of the civil servant – Edward Osmotherly – who was most associated with their actual disclosure in 1980.Footnote 1 The rules have been known as the Osmotherly rules ever since, rather than by their much more cumbersome formal modern title of Guidance on departmental evidence and response to select committees.Footnote 2 Secondly, and much more importantly, it allowed MPs and the wider public an insight into the workings of the civil service. The Osmotherly rules did more than set out mundane procedures. They were an attempt by the executive government to formalize what MPs could appropriately ask of civil servants without contravening Westminster conventions on ministerial responsibility. The Osmotherly rules revealed the ways in which civil servants could legitimately avoid questions from MPs that ministers might not want them to answer.
The public release of the rules represented a significant shift in Westminster tradition towards transparency at the expense of the hitherto institutionalized secrecy pact between ministers and civil servants. As one civil servant was to note in correspondence in the late 1970s, making the guidance public ‘would seem to savour of handing over to the prosecution defence counsel's instructions’.Footnote 3 As we enter the twenty-first century, the growing public profile of select committees is continuing to contribute to the growing public profile of the civil servants who appear before them. The Osmotherly rules continue to shape the format that these acts of public accountability can take.
This article analyses the history of what became known as the Osmotherly rules through the eyes of the civil servants involved in their revision and eventual release in the decade from 1970 to 1980. The driving research question for the study is: how did senior civil servants interpret the change towards greater transparency and parliamentary accountability embodied in the new rules? Using a framework of ‘historical interpretivism’, the article argues that civil servants behaved cautiously in publicly releasing internal documents not because of a culture of self-interested secrecy, but because of the need somehow to balance the competing claims of the parliament and the executive on their loyalty. Having been placed by the unwritten constitution under the control of the executive branch of government, civil servants were scrambling to avoid the complexity of becoming accountable to two competing masters – the executive and the legislature.
The significance of the Osmotherly rules as a case-study is that they are emblematic of the tug-of-war between competing visions for the role of the civil service in the years between the Fulton Report of 1968 and the election of Margaret Thatcher in 1979. One constitutionally conservative vision saw the traditional Westminster relationships between civil servants and ministers as being central to the effective governance of Britain. The other vision reflected the view that this kind of secretive, oligarchic partnership between ministers and civil servants excluded the voices of the democratic electorate that they were there to serve. What a close study of events such as the debate over the Osmotherly rules illustrates is that Thatcher's dramatic civil service changes of the 1980s represented an escalation of debates that were already strongly at play in the 1970s. Following the damning Fulton Report assessment of the generalist skills of civil servants, and in the context of the tremendous economic upheavals of the 1970s, debates over the purpose, behaviour, and shape of the civil service were rife, particularly in relation to its level of democratic accountability.Footnote 4 In drawing out the traditions that influenced civil service thinking on the Osmotherly rules, the article exposes some of the wider underlying tensions that Thatcher was determined to resolve through the introduction of New Public Management in the 1980s.
The sections that follow begin by using the ‘new political history’ as a lens through which to examine theoretical debates in political science between interpretivists and those scholars who argue for the primacy of the ‘British Political Tradition’ (BPT). Against the background of those debates, the article then develops a framework for applying an ‘historical interpretivist’ approach that advocates against seeing traditions as detached from the complex contexts within which individuals actually interpret those traditions. This historical interpretivist framework is then applied to the events of 1970–80 to draw out two competing narratives through which civil servants viewed the creation and release of the Osmotherly rules.
The first is a narrative of Westminster constitutionalism, which privileges the centrality of ministerial responsibility and the extent to which secrecy is necessary to maintain the trust between ministers and civil servants that allows the Westminster system to function. The second is a narrative driven by values of democratic accountability that embraces the right of the public to know what their government is doing, and the importance of democratically elected MPs being able publicly to scrutinize the work of un-elected civil servants. The two narratives capture the dual and sometimes competing accountabilities of civil servants to their ministers on the one hand, and the legislature on the other. The two narratives, running side-by-side, contributed to the ‘web of beliefs’ which shaped the views of senior civil servants as they were confronted by this change to long-established practices.Footnote 5
Importantly, these duelling narratives will be shown as having been interpreted through a third ‘intervening’ narrative with the power to trump the other two when civil servants were translating beliefs into action. This third narrative, based on an acute understanding of strategic and tactical imperatives, can be read as civil servants seeing their first duty as being to protect themselves and the government from wider embarrassment. In other words, whatever course of action may have been justified under narratives of civil service tradition and ‘open government’ accountability, it would only be implemented if it did not result in avoidable political embarrassment for ministers or civil servants.
I
Recent historiographical debates have created important new opportunities for bridging the divide between the complementary but frequently dissonant disciplines of political history and political science. In 2002, Pedersen lamented, with some justification, that
In an era of relative cultural pessimism, historians have been more concerned with meaning than causation and have found literary critics and anthropologists more congenial companions than political scientists. And the latter have certainly returned the cold shoulder, jettisoning historical approaches for formal modelling, quantification and theories of rational choice.Footnote 6
Things have changed. Over the past decade, the ‘new political history’ has begun to reconcile the study of history from below with the ‘high politics’ school, whilst simultaneously offering synergies with what we might call the constructivist turn in political science.Footnote 7 Historians have demonstrated a renewed interest in studying individual leaders in their full historical context, complementing an equal willingness within some parts of political science essentially to ‘bring history back in’ to political theory. Whether through the growth of ‘historical institutionalism’ as a means for explaining the seeming stickiness of institutional culture, or through the increasing interest in individual agency as a driver for political change, political science is embracing history as something other than a complicating variable for quantitative investigations of ‘rational’ behaviour.Footnote 8
Nowhere is this embrace more clearly visible than in the kind of interpretative political science advocated by Mark Bevir and Rod Rhodes, which in many ways sees the application to contemporary politics of the kind of perspectives often championed in historical studies of high politics. As David Craig noted in 2010, the work of historians like Maurice Cowling has long suggested that the study of interactions between important individuals is just as central for understanding politics as studying wider systemic cleavages based on class or economic standing.Footnote 9 Following Collingwood and others, Cowling wrote history as experienced through the eyes of influential people, in search of interpretations rather than wider objective truths. High politics as envisaged in Cowling's work is the study of history as seen through the ideas of those who lived it.Footnote 10 In defending Cowling's legacy, Craig persuasively identifies the high politics approach as being a natural relative of the interpretative approach in the social sciences.Footnote 11 But as Craig notes, citing Bevir, individual beliefs must still be situated within the ‘traditions’ and ‘webs of belief’ that help to give institutional shape to a group of people who together build shared beliefs about what it is that they are or should be doing.Footnote 12
The concept of ‘tradition’ has become an important lens through which both political scientists and historians have tried to understand the workings of British government under the Westminster system. There has been considerable debate about how many traditions there actually are, and the degree to which they work to shape the behaviour of individual politicians and civil servants. Advocates of the BPT argue that there is one dominant tradition that has framed British government and governance for decades if not centuries.Footnote 13 The elements of this BPT, and whether it is dominant, remain contested and are drawn from various contributions that have shaped the idea over the past fifty years.Footnote 14 A recent summary of the BPT is provided by Blunkett and Richards who identify the core tenets as support for a ‘liberal notion of representative democracy’, ‘a conservative notion of responsibility’, and ‘an emphasis on the need for responsible government’.Footnote 15 Some supporters of the BPT argue that the influence of the BPT is so strong over time that it has become embedded in the system – that in a sense it provides an ideational frame that can shape how actors will interpret what they should do.Footnote 16 They allow for other traditions, but suggest that the BPT enjoys a power asymmetry over them.Footnote 17
By contrast, the interpretivist approach, best encapsulated in the work of Bevir and Rhodes, has argued that no one single tradition can claim dominance in its influence on how people behave. They nominate at least four underlying traditions – liberalism, socialism, Whiggism, and Toryism – on which individuals may draw when confronted by change in the form of a ‘dilemma’.Footnote 18 For Bevir and Rhodes, agency is paramount as each individual examines their beliefs and desires through the lens of tradition to shape how they will act in any given circumstance. That is why they do not limit the field of possible traditions to the four identified above, but are open to the idea that there may be many and contradictory traditions in operation at any one time, including seeing the civil service tradition itself as drawing on multiple traditions.Footnote 19 The interpretivist approach has come under challenge from realists, and in particular from critical realists who criticize the interpretivist approach to agency by arguing that it is possible to allow for agency without jettisoning the power of institutions to shape behaviour.Footnote 20
This article does not seek to assert that institutions have no capacity to shape behaviour, but rather that people themselves determine the extent to which any combination of institutional structures, traditions, their own beliefs, and situational context impact on how they choose to behave. Using a study of the creation of the Osmotherly rules, I argue that no single tradition holds its intellectual shape sufficiently over time to be isolated and identified as the key motivating tradition in any given circumstance. It is argued here that action and behaviour are always contextual, and so by definition the traditions, beliefs, and institutional structures that help to shape behaviour are also contextually shackled. Traditions and beliefs interact in complex ways with their wider environment, leaving individuals to grasp at whichever components best make sense to them in the particular decision-making context they find themselves in.
The ideas encapsulated in the BPT are undoubtedly powerful and can influence the behaviour of civil servants and their political masters, but only as one line of tradition that is intersecting with multiple other traditions at the given point in time that someone is considering how to act. Individuals simultaneously hold multiple traditions in their head, which interact with their beliefs and the external environment to shape behaviour at each point in time. For example, in this article, a ‘Westminster system-based civil service tradition’ is identified as having an impact on the beliefs of civil servants. This civil service tradition might sometimes interact with a wider BPT, and either one may impact on civil servants as the traditions themselves interact with the specific context in which actors are making a decision.
Tradition is by definition about continuity over time. But one cannot seek to understand traditions, or their force in influencing contemporary decision-making, without locating them contextually in the historical circumstances from which they were derived and in which they operate.Footnote 21 Not every person will interpret traditions in the same way when faced with the same set of circumstances in the same environment because they will each be influenced differently by their interpretation of the interaction between that environment and particular traditions and beliefs. Ontologically, traditions remain constructs that need to be read and interpreted by the actors who give them form through the narratives that they use to encapsulate them.
As the next sections will show, some civil servants and ministers welcomed the greater scrutiny of select committees, and the broader idea of ‘open government’, whereas others viewed both with trepidation. Operating in the same environment, and subject to the same traditions, different people interpreted issues differently. Some saw the Osmotherly rules and select committees through a lens of democratic principles, and some saw them through more conventional lenses of ministerial accountability, under which civil servants should remain anonymous, subsumed for formal purposes into the identity of their minister.
In arguing that traditions remain shackled to context, I mean context in its fullest sense, both narrow and broad. In the narrow sense, the impact of a particular tradition on the view of an individual civil servant will be filtered through the other component parts of the environment in which they are operating. This may include the influence of the views of their colleagues and superiors, the culture within their particular subsection, and the unrelated personal, political, and administrative pressures that they are operating under. Ontologically, it remains very much an ideationally and emotionally constructed form of context, but this makes it no less significant in determining how closely a tradition is adhered to at a given point in time. In the same way, context in its wider form – i.e. outside the micro or local level – will include not just the strength of traditions but also elements of personal judgement, wider government priorities, and questions of timing. For example, whilst there might have been an adherence to traditions of Westminster constitutionalism in the Cabinet Office, the extent to which that tradition determined behaviour could be mediated by other factors, rendering some concession to open government principles as the most viable course of action when dealing with the Osmotherly rules.
So rather than focusing on traditions only as ideas that travel through time, this article argues for seeing traditions as tethered to the historical context in which they are interpreted. In other words, the defining anchor becomes the context rather than the tradition itself. To understand the behaviour of individuals, we need to examine the range of traditions they were exposed to at a particular point in time, and how those traditions interacted with the specific environment in which they found themselves. This type of analysis requires a form of historical interpretivism.
An historical interpretivist approach uses the insights of interpretivism as a way of understanding behaviour, but argues for the primacy of historical context as the key determinant of which particular traditions will shape the ‘web of beliefs’ in which a person operates at a specific point in time. Historical interpretivism argues for a deep examination of the communications between individuals within their historical context as a way for understanding how they interpreted the world, and the particular elements of tradition that they wished to apply to it. Historical interpretivism argues that traditions are not re-shaped each time they are interpreted, but rather that they are only one factor amongst many that shape the temporal decision-making context in which people like civil servants and politicians find themselves. The many other factors in any given case could include the changing nature of the governance environment; government policy towards accountability or transparency; fears of how decisions might look in the media; internal debate with colleagues; public opinion; and the need to avoid public embarrassment. The list is potentially vast, and each factor will interact in a different way with the various traditions that help individuals make sense of their particular circumstances. In other words, tradition is a prisoner of context in its capacity to shape the behaviour of individuals.
Historical interpretivism as a method remains a scholarly construct retrospectively applied to the study of past events from a more beneficial distance. It reads events through the eyes of those experiencing them, but can only recover an imperfect portion of that experience. Actors may have knowingly made strategic calculations weighing the full context in which they found themselves against the background traditions that informed their actions; but equally they may have simply made instinctively prudent judgements at a more subconscious level, utilizing arguments which can only be organized into more systematic narratives by subsequent scholarly analysis. Either way, a historical interpretivist perspective is able to provide an assessment that encompasses the same contextual challenges and observations faced at the time by the actors themselves.
The interpretivist approach has shown over the last two decades that the best way to understand what people think is to look at what they say and write and how they act. The goal of course is to reveal the beliefs, ideas, governing narratives, and motivations that drive behaviour at both an individual and collective level.Footnote 22 As Rhodes's recent work demonstrates, an ethnographic approach allows researchers to watch civil servants in operation to see what their behaviour reveals about the underlying beliefs, traditions, and stories that shape how bureaucratic departments function.Footnote 23
The potential of historical interpretivism is thus already embedded in the interpretivist approach, which stresses the importance of historical factors in shaping traditions. Interpretive scholars stress that beliefs and actions cannot be studied in isolation, but rather must be seen within the full context of the webs of belief that surround people as they try to negotiate their way through the complex world around them. This entails a commitment to what Bevir terms ‘meaning holism’, to seeing how history, tradition, and convention shape the beliefs of individuals at one-and-the-same time as those beliefs are shaping and re-shaping actions.Footnote 24
More importantly, meaning holism implies that social explanation requires us not only to relate actions to beliefs, but then to make sense of these beliefs by locating them in the larger webs of belief that give them their content. It thereby undermines formal and ahistorical explanations, and leads instead to contextualizing and historicist explanations.Footnote 25
In their portrayal of public administration as ‘storytelling’, Bevir and Rhodes emphasize the importance of narratives as being the building blocks that both reflect belief and shape action.Footnote 26 When confronted by proposed changes to established practices – what interpretivists call ‘dilemmas’ – individuals assess competing narratives to revise their beliefs and construct new traditions of practice.Footnote 27 Rhodes argues that a ‘dilemma arises for an individual or group when a new idea stands in opposition to existing beliefs or practices and so forces a reconsideration of the existing beliefs and associated tradition’.Footnote 28
The creation and release of the Osmotherly rules in the 1970s created a classic ‘dilemma’ in this interpretivist sense. It threatened to alter established beliefs and practices that were central to how civil servants operated. The traditional anonymity of civil servants would be compromised by appearing before select committees, and the privileged nature of communications between ministers and civil servants would be threatened with exposure through parliamentary questioning. Rhodes, Wanna, and Weller have demonstrated how governing traditions and narratives are central to establishing what Westminster means to the people who operate within it.Footnote 29 The evidence from the archives presented in following sections will show how senior civil servants and their ministers responded to the ‘dilemma’ of the Osmotherly rules by weighing up duelling narratives in order to reconcile the changes with their underlying beliefs about what Westminster means.
Taking an historical interpretivist approach to primary source historical documents provides for a rich observational experience not dissimilar in some ways to the benefits derived from ethnographic methods. This is particularly so within the vast available resource of civil service paper files. Paper files allow for the study not just of a document's face-value content, but also of the various scribblings and thoughts appended to the file by the many hands through which it passed. It allows for the study of the iterative nature of policy-making, and the degree to which changes are resisted or embraced by the ideas and beliefs of the agents whose job it is to carry out change.
This kind of historical interpretivist approach is particularly useful in seeking to understand the functioning of the senior civil service. The traditional position, to quote Chapman, is that ‘all aspects of the executive work of British central government are largely shrouded in mystery’.Footnote 30 Long renowned as an institution that wields its immense political and administrative power away from the public gaze, the civil service's formal rules and institutionalized structures can act as barriers to effective study of how power at the top actually works. An interpretive approach to historical documents enables an inside view of how senior civil servants really saw their roles and what they really thought of government policy.
Like all methods, historical interpretivism also retains certain drawbacks and limitations. Whilst it allows for in-depth analysis and thick descriptions of particular narratives, traditions, and events, its reliance on interpretations of historical documents does not allow for the kind of triangulation that could otherwise be provided by semi-structured interviews or other similar data sources. Nor can it deliver a definitive interpretation of any one particular event, recognizing that there are as many possible interpretations as there are individual actors, notwithstanding the shared traditions and narratives that shape individual responses. But in revealing the traditions that underpin action, and relating them to the historical context in which those actions occur, historical interpretivism reveals the dynamic nature of human decision-making in areas that at first glance might seem relatively static and path dependent.
Methodologically, the study on which this article is based utilizes tools of qualitative content analysis to reveal the traditions, narratives, and interpretations inherent in the views of civil servants as they confronted potential changes to the Osmotherly rules. Material was examined inductively, coded, and then sorted into categories to reveal the narratives at play. The object of coding in qualitative content analysis is to sort text into categories through a process of ‘selective reduction’ to arrive at categories that are ‘exhaustive’, ‘exclusive’, and ‘enlightening’.Footnote 31 In other words, the categories must capture all components of the data, remain distinctive from each other, and be relevant to the question under study.
For this study, archival records were examined from the Civil Service Department (CSD), the Cabinet Office, and the Prime Minister's Office, including various pieces of correspondence with departments including Treasury, Defence, and the Foreign and Commonwealth Office. The material was originally coded into six thematic categories that emerged inductively. These categories were: (1) accountability and responsibility under Westminster convention; (2) meaning of ‘open government’; (3) protection of civil servants and their relationship with ministers; (4) strategic thinking and decision-making by civil servants; (5) informal or implicit understandings of appropriate behaviour; and (6) internal conflicts of views. These six categories were then collapsed into three broader categories in order to isolate the three overarching narratives reflected in the material, which are discussed in depth in the next section of this article.
The time period for the study – 1970–80 – was chosen as it covers three key stages in the development of the Osmotherly rules. It begins with the internal revision of the still confidential guidance in 1971–2; proceeds to the arguments about whether this guidance should be released to the Procedure Committee of the House of Commons in 1977–8; and finishes with the arguments over its final official release in 1980. The focus on this one crucial decade allows for sufficient breadth to capture changes in interpretations over time, whilst also enabling sufficient depth to apply the level of detailed analysis of correspondence that an historical interpretivist approach demands.
II
The constitutional soul-searching over the proper shape and role of the civil service in the 1970s began in many ways with the publication of the Fulton Report in 1968. Amongst its findings was the perception that civil servants were too remote from the British people, and it recommended that they take on a more visible role: ‘the convention of anonymity should be modified and civil servants, as professional administrators, should be able to go further than now in explaining what their departments are doing’.Footnote 32 It was an appeal for openness and accountability that both politicians and civil servants could not afford to ignore.
This Fulton-inspired drive for change underpinned the first narrative which emerges from the civil service correspondence on the Osmotherly rules, which is one centred on the idea of democratic accountability, encapsulated in what might be called the ‘open government’ agenda. In 1971, the decision was taken to revise the internal guidance document for civil servants appearing before parliamentary committees in order to support a more open and responsive style. When the parliamentary secretary for the CSD – David Howell – was sent a draft of the proposed revised rules in 1971, he was concerned that it did not sufficiently align with the government's public accountability objectives. In a confidential minute, one official noted: ‘This was the situation when we put the draft to the Parliamentary Secretary. He did not like it. His objections were that it was a very anti-open government document. I am bound to say that in MG we all agreed with him but had felt constrained by the existing rules.’Footnote 33
The lord privy seal subsequently wrote to the prime minister, Edward Heath, about the general relationship of the revised guidance to a more open style of government.
I understand that the guidance embodied in the memorandum reflects what I might call the orthodox line which has evolved in the years since the Specialist Select Committees were first established. However, I believe that it would be right and consistent with our philosophy if officials were to adopt a more forthcoming approach in a number of respects. First, I believe that the encouragement we have given to senior civil servants accepting a greater personal responsibility for the execution of policies, which is inherent in the movement towards greater delegation and the concept of departmental agencies, requires us to allow a wider range of official witnesses…Secondly, I believe that we should move away from the rigid doctrine about the extent to which officials should discuss policy matters.Footnote 34
The prime minister responded on 25 April 1972, stating that he was unpersuaded of the desirability of allowing civil servants a wider power to discuss policy with select committees.Footnote 35 Within the CSD, the prime minister's intervention was seen as having decided the matter.
We now have the Prime Minister's reply to the Lord Privy Seal's proposals of 24 March. It is distinctly unfavourable. In effect, we have been turned down on all three of the suggestions which the Lord Privy Seal made. The Prime Minister doesn't even think that any more detailed instruction on the attendance of official witnesses is necessary, and wishes to be consulted before any change even to the existing guidelines is made. In the circumstances, I don't see much point in seeking to contest this decision. The time is obviously not ripe for these changes yet (though I am convinced myself that they will come).Footnote 36
The willingness of CSD officials to align themselves with a more ‘open government’ approach, only to be stymied by the prime minister, is notable. It suggests that any interpretation of the civil service as always instinctively protecting existing procedures in its own self-interest is too narrow. The political agenda of a departmental minister could and did find support amongst civil servants, even if it was ultimately blocked by the prime minister.
Equally, the ultimate unwillingness by Prime Minister Heath to endorse a wider and more open role for civil servants in front of parliamentary committees revealed that the commitment to open government at the top political level did not run deep. A prime minister of long parliamentary and ministerial experience reverted to a traditional view of the need to balance openness with the established precepts of ministerial responsibility. The episode also reveals that the persuasive power of the newly established Civil Service Department was not strong enough to unseat the more protectively conservative views championed in the Cabinet Office. The Fulton-driven exhortation towards openness had not succeeded in displacing the traditional Cabinet Office ability to command the ear of the prime minister on matters of administrative convention.
In the late 1970s, as the Procedure Committee of the House of Commons requested a copy of the thus far confidential guidance document, the same internal debate re-emerged between supporting ‘open government’ and the need to protect the position of civil servants in front of select committees. The outgoing head of the civil service, Sir Douglas Allen, was largely comfortable about sharing the document, suggesting it was consistent with a commitment to open government.
I have shown Sir Douglas Allen your minute of 22 December. He has looked quickly through the relevant guidance GEN 76/78 and at a quick reading he finds it difficult to see anything in this guidance which could not be given to the Select Committee. He is reinforced in this view by our present policy approach towards openness generally. He therefore considers that the right course would be to put to Ministers the proposal that we can see no objection to issuing the guidance in full, and to ask them whether they have any objection.Footnote 37
Allen had throughout his reign as head of the civil service been an advocate for freedom of information and a more open style of government, so his stance is not necessarily surprising.Footnote 38 However, Allen's general inclination towards openness did not immediately sway the strong objections of other civil servants working on the matter, who determined to convince him of the alternative case.Footnote 39
The matter was raised with the cabinet secretary, Sir John Hunt, in January 1978. A minute from the new permanent secretary to the Civil Service Department, also the head of the Home Civil Service, Sir Ian Bancroft, outlined the background to the request from the Procedure Committee, before asking for Hunt's views.
In favour of complete disclosure is our general stance on openness following the line taken by the Prime Minister and Douglas Allen's letter of July 1977 on disclosure of official information…On the other hand, there are arguments against disclosure of the document as it stands. It was not drafted with publication in mind, and therefore contains some infelicitous phrases, apart from generally being written in a tone which just might be regarded as provocative.Footnote 40
Bancroft was quite willing to make clear his own preferences having weighed up those arguments.
Despite the contrary arguments, my own feeling is that I would like to put to Ministers the proposal that we should issue the document as it stands. I accept there are some risks of embarrassment in this course but I do not think any of the embarrassments are great enough to justify going against our general ‘open government’ stance.Footnote 41
Hunt agreed that ‘in itself disclosure would be a disadvantage not a disaster, whereas refusal would be difficult to square with the Government's general stance on open government and could provoke a confrontation’.Footnote 42 Such a conclusion from both Bancroft and Hunt appears to have been ultimately very much a pragmatic one. As Hennessy notes in his history of Whitehall, Bancroft was ‘not an open government man’ and Hunt was a politically astute protector of Cabinet Office power, so neither would have felt driven by philosophical inclination to publish a document of this kind.Footnote 43 A minute from the lord president, Michael Foot, to the prime minister on 23 January 1978 confirmed that the ‘open government’ view had won the day. ‘Despite the difficulties, the meeting agreed that the risks of embarrassment in full disclosure would not be sufficient to justify going against our “open government” stance.’Footnote 44
Foot's willingness to support the kind of openness that Heath had baulked at half a decade before perhaps demonstrates how the strength of the open government agenda had grown over time. What had seemed at the outset of the 1970s to be an uncomfortable and perhaps unworkable ideal could by the end of the 1970s be seen by both Labour and Conservatives as something that could no longer actually be ignored in decision-making without causing confrontation and potential backlash. Certainly, Labour had committed themselves to an ‘open government’ agenda as far back as the Wilson governments of the 1960s, so their support of it in the late 1970s was no longer new nor radical.
On the Conservative side, under Heath the instinctive traditionalism of the prime minister with regards to the civil service was buttressed by wariness from the Cabinet Office about openness, despite CSD support for it.Footnote 45 Once back in opposition under Thatcher, the Conservatives swung behind a policy position advocating for stronger surveillance of the executive by the legislature through select committees. The public release of the Osmotherly rules in 1980 was commensurate with that commitment, with the Leader of the House –Norman St John-Stevas – overseeing the establishment of the new select committee structure.
In summary, the files make clear that both civil servants and ministers were not only aware of the ‘open government’ agenda, but that many were willing to support it even where there was some risk of embarrassment to the government. In other words, the democratic accountability tradition was a powerful component of the decision-making matrix. Equally, individual civil servants voiced clear preferences for and against the document's release, highlighting the degree to which individual views played an important role in internal debates. Civil servants did not demonstrate an instinctively uniform adherence to protecting themselves from public scrutiny, but instead carefully weighed the arguments for and against, based on their interpretation of the government's wider ‘open government’ agenda. Of course, this does not mean that civil servants were ready to jettison the protective aspects of the guidance altogether, as revealed in the second narrative that runs through the correspondence.
The second narrative that emerges from the files centred on the need to defend established Westminster system traditions in order to make government work. The 1971 guidance document picked up on sentiments first set out in 1958 by then Cabinet Secretary Norman Brook, with a focus on the need to avoid ‘political controversy’.Footnote 46
If [sic] is for officials to answer questions of fact, to explain the administrative reasoning behind a policy, and to answer questions in the field between day-to-day administration and high policy which might be called ‘administrative policy’. But if they are asked questions in the field of political controversy, using the term in its widest sense, they should say that this is a matter for Ministers on which they cannot answer.Footnote 47
Set out equally clearly was the need to preserve the conventions of collective ministerial responsibility.
Departmental witnesses, whether in closed or open session, should preserve the collective responsibility of Ministers and their departments by not revealing (except so far as this is implied in what is proposed in para 26 below) the level at which decisions were taken…It should also be borne in mind that decisions taken by Ministers collectively are normally announced and defended by the minister responsible as his own decisions and it is important that no indication should be given of the manner in which a Minister has consulted his colleagues.Footnote 48
Internally, officials debating the potential contents of the 1971 revised guidance were determined that the document should reflect the flows of responsibility and accountability in the Westminster system. ‘As I said in my minute of 1 December, the confidentiality of the advice given to Ministers seems to me to be a fundamental feature of our system. This is not inconsistent with developments towards greater delegation and accountability.’Footnote 49
Concerns about protecting Westminster traditions were strongest within the Cabinet Office, where Cabinet Secretary Burke Trend penned a minute to the prime minister. Whilst he focused his remarks on the undesirability of revealing to select committees the details of cabinet committees, he touched on the core arguments against a more open approach in general.
The reasons why successive Governments have always refused to disclose these details derive, basically, from the almost instinctive attitude of any Government that the Executive should present a single, unified front to the Legislature…the trust and confidence between colleagues, which are essential to the honest discussion of differences of view, could not be expected to survive intact if the means by which those differences are resolved were exposed to public examination.Footnote 50
The right of the executive government to protect its internal workings from parliamentary scrutiny remained a key theme when officials were discussing the potential release of the document to the Procedure Committee in 1977/8. One official reflected on the absolute central part that privacy of advice played in effective government.
Reflecting on the phone conversation we had on this, the argument that looms large in my mind is the point that if all central governmental directions to individual members of the civil service are to be potentially liable to publication, it will become increasingly difficult for any effective instructions to be given in many areas of public administration…It may be argued that instructions to civil servants as to how to conduct themselves when confronted by a committee of Members of Parliament are a special case, but I do not think this is so.Footnote 51
In summary, some senior civil servants interpreted their role as protecting the conventions that underpin the successful workings of the Westminster system. Giving too much power to select committees to question civil servants might undermine the levels of internal trust upon which effective relationships between cabinet ministers, and between ministers and civil servants, relied. In acting as guardians of these conventions, civil servants projected their motivations as not just being about protecting the civil service per se, but protecting the wider ability of cabinet government to function effectively. The fact that publication of the guidance did ultimately occur must be seen in the context of the incoming Thatcher government's wider commitment to a stronger and more rigorous select committee system, championed by Norman St John-Stevas as Leader of the House.
The third narrative, which I characterize here as an ‘intervening narrative’, emphasized that the first duty of senior civil servants was always to protect their ministers and the wider civil service from public embarrassment. This ‘intervening narrative’ could trump both the need to support an open government agenda and the need to maintain the traditions of Westminster governance if either threatened avoidable humiliation of the government. It represented an institutionalized instinct towards self-preservation and the need to avoid what might be termed ‘own goals’.
This intervening narrative shares a similar genealogy to the second narrative, in that it draws on the constitutional tradition that civil servants and ministers are both indivisible components of the executive who must therefore protect each other, but what sets it apart is its sheer pragmatism. Whereas the second narrative around defending Westminster traditions works to block moves towards greater openness, the third intervening narrative allows the embrace of openness if that is what is required to spare embarrassment to ministers or the civil service. It is a narrative that privileges strategic thinking and political awareness over and above any one conception of the relationship that ministers and civil servants should maintain with each other.
In 1977, an initial draft response to the Procedure Committee's request for a copy of the guidance was provided to the CSD's permanent secretary, Sir Douglas Allen, for consideration. The accompanying minute emphasized the dilemmas that would be inherent in releasing the written guidance to the Procedure Committee.
The latter request takes us into very difficult territory indeed. It will be necessary to tread an exceedingly narrow path between demonstrating the readiness of the civil service to be as helpful as possible to House Committees, and making clear that there are a number of kinds of information that really cannot be disclosed, including the CSD guidance circular itself (GEN 76/78).Footnote 52
The official who drafted the minute – Sandy Russell – penned a further minute listing in greater detail his concerns about the potential release of the memorandum. His concerns highlight his interpretation of the difficulties inherent in making civil servants accountable to two masters – the executive and the parliament. It placed them in the awkward position of having to protect information that their ministers might not want released, and as a result face the opprobrium of MPs on committees who would see civil servants as obstructing the right of democratically elected representatives to know what the government was doing. Russell segmented these difficulties into specific points.
(a) It could well make life more difficult for departments when dealing with select committees. Committees would be given ammunition to challenge departments when they were exercising their discretion to refuse information, by placing a different interpretation on the relevant part of the guidance. The detail with which the document necessarily deals with each of the reasons for not disclosing information would provide plenty of toe-holes for committees to create additional difficulties for Civil Service witnesses.Footnote 53
Russell went on to point out that the reason the guidance had been maintained as an internal government document up to this point was that it acted as a training manual for civil servants on how to avoid tactical missteps in front of a select committee.
(b) By its nature the document is partly a tactical guide for Civil Servants on how not to get ‘boxed in’ so that they are required to supply more information than the Government would wish to expose. This is a perfectly legitimate exercise of internal government administration but could be badly misinterpreted if taken out of context.Footnote 54
In further internal correspondence in January 1978, officials discussed problems with specific paragraphs in the guidance and what reaction they might draw if they were made public. Michael Townley in the Cabinet Office wrote:
knowledge of the document would be likely to make the position of official witnesses more difficult by making public the limits to which they can be pushed. These limits, as stated, must inevitably be matters of opinion and judgement. At the moment, it is the witness' private judgement, but if the criteria are publicly known, he seems bound to be pressed further.Footnote 55
Ultimately, ministers decided to release the document to the Procedure Committee. A formal reply was provided under the signature of the lord privy seal (Lord Peart) and hand-delivered to the clerk of the Procedure Committee along with twenty copies of the guidance memorandum. The Procedure Committee went on to append a copy of the guidance to their official report – effectively making it a public document. But in doing so, the committee offered little critique of the rules, meaning the public release did not therefore attract wide media scrutiny.
This became apparent in 1979/80 as, despite the document's effective public release by the Procedure Committee the year before, officials began a fresh internal dialogue of concerns and arguments about the proposed public release of the guidance by the Thatcher government. Some departments were very keen to make changes to the guidance, whilst the CSD was keen to minimize any change to the version released to the Procedure Committee in 1978.Footnote 56
The Department of Defence was very awake to the dangers involved in the greater publicity that would flow to civil servants who appeared before select committees. They lobbied for new wording such as: ‘In giving evidence for broadcasting, witnesses should remember that a slip of the tongue, hesitation or use of a particular tone or emphasis may attract public comment where it otherwise might not.’Footnote 57 In reply, the CSD was willing to allow some statement of warning about avoiding slips of the tongue, but stated that: ‘I would myself expect witnesses to be only too well aware of the danger of conveying unintended nuances when proceedings are being recorded.’Footnote 58
In summary, there were clearly strong concerns within the civil service around how a public release of the guidance document might impact on the protections available to civil servants appearing before select committees. The document was seen as a shield, whose removal could make life much more difficult for civil servants trying to avoid embarrassing themselves in front of MPs. Ironically, the public release of the document by the Procedure Committee in 1978 ultimately attracted such little controversy that some departments were still arguing over whether it should be publicly released in 1979–80, despite its effective release the year before.
This awareness of the need to protect the civil service was coupled with a clear strategic awareness that the best way to ensure their protection was to protect ministers from embarrassment. Individual civil servants displayed an astute strategic and tactical awareness of the realities of politics in a parliamentary system. For example, when the Procedure Committee approached the government in 1977 to procure a copy of the guidance, civil servants were well aware of the political consequences of the various responses they could make. One CSD official – Sandy Russell – whilst against releasing the guidance, was also aware of the problems that would ensue from any attempt instead to sanitize the document before releasing it. ‘We think there would be a great danger that the reissue of a revised document would be seen by the Committee as a rather contrived way of shielding information from the Committee and could well lead to accusations of bad faith.’Footnote 59 There was also the danger that the CSD, as ‘keeper’ of the guidance, might itself get drawn into battles between departments and select committees. Russell's colleague, J. B. Pearce, noted: ‘[W]hether the document itself is released or not, we will need to be careful to avoid any suggestion that the CSD can be drawn into acting as an arbitrator between Select Committees and departments through its “guardianship of the doctrine”. There is obviously a risk of this.’Footnote 60
When revising the document for release by the Thatcher government, a minimalist approach to change on tactical grounds certainly found favour with the relevant officials in the Cabinet Office, especially given the newly ‘public’ nature of the document.
In general, however, I am, as you know, entirely in agreement with the proposed low-key approach to this revision since I suspect that it would be a major tactical error if Departments were to try and define at this stage the range of information, and the conditions for its disclosure, that they were prepared to make available to the expanded select committee system…In any event it may be better for agreements on disclosure to particular committees to be recorded on a private basis, rather than in what must now be regarded as a public memorandum.Footnote 61
A similar tactical awareness was shown in advocating against a proposal to discuss the revision of the document with MPs on the Liaison Committee. Civil servants judged that consultation at this stage would effectively encourage MPs to get involved – which was seen as an undesirable outcome.
I have spoken to Mr Townley about consultation with the Liaison Committee on the Memorandum. He would be inclined to advise Mr St John Stevas not to consult the Committee. He thinks that they would undoubtedly conceive it their duty to subject it to fundamental re-examination, and that they might come up with major proposals which reflected members' individual viewpoints rather than according with the more carefully considered views of the Procedure Committee. If, on the other hand, the Memorandum was not volunteered but simply handed over if and when it was requested, with the comment that this was simply the previous Memorandum slightly revised to bring it up to date, he thought the Government could not fairly be criticised.Footnote 62
This tactical approach was fully endorsed by ministers when the argument was put to them.Footnote 63 Tactics could extend even to such questions as whether to put a cover on the document when it was published. One official noted: ‘I would not myself favour putting a cover on the Memorandum. We want this to look as much as possible like the original Memorandum GEN 76/78, which was published by the Procedure Committee in their report.’Footnote 64
In summary, the files reveal the extent to which civil servants could be both intensely aware of tradition and yet intensely practical at the same time. Whilst some saw it as a necessity that the conventions of Westminster government not be undermined, there was no naivety about how their arguments might be perceived or received by MPs and the wider public. The relevant correspondence abounds with a tactical political awareness of how different options would play out.
III
An historical interpretivist approach to the civil service files shows how senior civil servants confronted the ‘dilemma’ of the Osmotherly rules by weighing up the claims of two competing traditions – the Westminster system-based civil service tradition and the democratic accountability tradition. These traditions revealed themselves as narratives that run through the relevant correspondence in the files. The traditional, constitutionalist narrative of how a Westminster system civil service is supposed to work provided the initial frame of reference. Embedded in that civil service tradition is a narrative about what is ‘properly’ the preserve of ministers and what is properly the role of civil servants. This expressed itself in concerns that select committees might push civil servants into revealing the details of their advice to ministers, and that this would threaten the foundational trust on which the system relies. Existing conventions on ministerial accountability were seen as providing vital protection to civil servants being pushed for answers by MPs.
Officials were clear that any set of rules about how they should behave in front of select committees needed to make sure that civil servants did not become the ‘fall guy’ for questions that should more legitimately have been directed at ministers. But importantly, civil servants did not see themselves as engaging in blocking tactics against the government of the day. Rather, their views reflected their own belief that their role was to work with ministers to protect against the ‘unwestminsterly’ incursions of parliamentary committees into the business of executive government.
Arranged alongside the narrative of Westminster constitutionalism was the democratic oversight narrative about the rights of democratically elected MPs transparently to hold all parts of the executive government to account. This meant that not only should ministers be available to be questioned in parliament and the media, but that civil servants should make themselves available to answer in detail for their stewardship of the public's interests. This narrative manifested itself in a commitment to ideals of ‘open government’ and greater powers for the legislature to scrutinize the executive through an expanded select committee system.
The files show that senior civil servants did not share a uniform interpretation of the meaning of these narratives, or indeed which of the two narratives should be privileged over the other. This is where Bevir's exhortation to adopt a holism approach, informed by historicism, becomes important for fully understanding the context that underpinned these strategic beliefs. For example, many of the bureaucratic supporters of the ‘open government’ agenda were found in the Civil Service Department, which had only been formed in 1968. Its establishment followed from the Fulton Report's view that control of the civil service should be moved away from the Treasury and that the opaque style of civil service workings needed to be updated to meet the needs of the modern world. CSD was therefore on a mission to establish its own identity and contribution to government and had very sound reasons for supporting the political arm of the executive in its pursuit of open government principals. CSD interpretations of the ‘democratic accountability tradition’ were therefore shackled to the particular circumstances and context in which the department found itself.
In contrast, the strongest resistance to the idea of ‘open government’ and of civil servants having frank discussions in front of select committees came from within the Cabinet Office. Long established at the centre of government, with much of its perceived power attached to its proximity to the prime minister and its ability discretely to enforce its power across Whitehall, the Cabinet Office had little reason to embrace a more open and democratic narrative about what it should be doing. Its job was to support ministers, who would in turn do their job and report to parliament as they saw fit. The Cabinet Office saw themselves as the guardians of the ancient tenets of the Westminster system, and that belief governed their caution about embracing change.
Yet these beliefs, and the traditions in which they were grounded, were also mediated further by other contextual factors like the relationship between particular personalities, and the extent to which some leaders were trying to protect or expand their own turf. As Lowe's official history of the civil service lays bare, figures like Sir Douglas Allen, Sir William Armstrong, Sir John Hunt, and others were in a permanent but often unacknowledged tug-of-war for both personal and institutional power.Footnote 65 Personal ambition played its part, as did professional jealousy. These contextual factors must therefore be taken into account in arriving at definitive explanations for why individuals attached themselves to a particular tradition or traditions when deciding on the appropriate course of action.
And yet, whilst underlying beliefs and traditions shaped the views of senior civil servants, they coupled these with a very high level of political and strategic awareness. It re-confirms that senior civil servants did not equate being impartial with being unaware of the potential political consequences of their actions. In the midst of decision-making dilemmas, beliefs and traditions had to be weighed against practical realities such as the possibilities of embarrassing ministers. Adopting a meaning holism approach reveals the complex interplay that occurs in the minds of individuals as they weigh up the various components that influence their decision-making. For example, in 1978, civil servants had the strategic awareness that it might be less damaging if they simply summarized the rules for the Procedure Committee rather than releasing the full document, but also had the political awareness to realize that this could be seen as ‘doctoring’ the document for public consumption.
Civil servants saw themselves as having a legitimate role to play in helping the government to avoid political as well as administrative pitfalls. Regardless of the depth of their attachment to either the Westminster system-based civil service tradition or to ideas of democratic accountability, civil servants allowed them to be trumped by a third narrative grounded in pragmatic assessments of the governance environment, i.e. that it was first and foremost their duty to support the government of the day by avoiding embarrassment to ministers or the civil service. This was both a question of belief about what the fundamental duty of the civil service was, and an understanding that the fate of the civil service was at some level inextricably linked to that of ministers; that if mistakes were made in how they approached questions of public accountability, both would suffer the consequences in terms of public criticism.
There was a strong protective instinct amongst some of the officials involved in both the CSD and the Cabinet Office about the rights of civil servants to have some protective guidance with which to resist the questioning of MPs. Officials showed awareness of the underlying tension between displaying their constitutional loyalty to the government of the day and being seen unduly to obstruct the democratic workings of the parliament. Even in this pre-24/7 media age, officials were aware that MPs had the ability to raise a ‘song and dance’ against the civil service for being seen to stifle the legitimate requests of democratically elected representatives.Footnote 66
Through a close study of these historical files with an interpretivist lens, bureaucracy comes to life not simply as a coherent institutional machine, but as a collection of people with differing interpretations of the competing civil service and democratic traditions. Ultimately, civil servants examined these traditions in this case through the pragmatic lens of the contemporary political context, to guide their actions as they determined how best to respond. The context shaped the weight that each of them was willing to give to each tradition in interpreting their own role.
An historical interpretivist approach of this kind offers potential new methodological avenues and substantive areas for further research. It enables political historians to emphasize the centrality of historical context for understanding political events and institutions, in a way that connects with existing approaches in the political science literature. For political scientists, it holds the promise of grounding theoretical developments more fully within an historical understanding of how traditions have been lived out within British political institutions over time. It demonstrates not just how essential both disciplines are for understanding the political world, but the fruitfulness of pursuing those understandings in tandem rather than in isolation from each other.
