The UK Freedom of Information Act (2000) came into force on 1 January 2005, providing everyone with the right to ask for information from public sector bodies, including central government departments.Footnote 1 Unless covered by one of the exemptions included in the Act, the department concerned has an obligation to provide the information requested within twenty working days. The use of FOI by both politicians and journalists has resulted in a number of high-profile, and sometimes controversial, stories appearing in the national press (to say nothing of its use by those with an interest in UFOs), yet its use by academics appears to have been less marked.Footnote 2 Prior to its enactment, Andrew McDonald predicted that the new legislation would ‘force historians … to think in new ways about access to their sources’. However, writing a few years after the introduction of the Act, Andrew Flinn and Harriet Jones observed that ‘relatively few historians are making regular use of FOI as a tool’.Footnote 3 While no definitive figures detailing the use of FOI by historians are available, data obtained through the use of the Act itself for the purpose of this article appears to support this observation.Footnote 4
For those interested in the history of the British intelligence community, a lack of engagement with FOI can be explained by the fact that the intelligence and security services fall outside its provisions, in marked contrast to the comparable legislation in the United States, where FOI has become an important tool for research into the history of US intelligence.Footnote 5 There are twenty-three areas of ‘Exempt Information’ listed by the Act (Sections 21–4 and 26–44), one of which, Section 23, deals with ‘Information supplied by, or relating to, bodies dealing with security matters.’Footnote 6 These bodies include the Security Service (MI5), the Secret Intelligence Service (MI6), Government Communications Headquarters, Special Forces, and the National Criminal Intelligence Service.Footnote 7 As such, FOI requests made directly to the agencies themselves would unquestionably constitute a waste of the time of both the researcher and the recipient. Yet, this inability to use FOI to access intelligence and security agency-specific material should not preclude FOI requests being made that venture into the wider realm of intelligence and security affairs.
I
When considering the use of FOI in relation to intelligence history, it is useful briefly to chart the trajectory of research in this area over the past three decades. Conducting research into British intelligence and security matters prior to the 1990s was a difficult task. The official secrecy in which the civilian intelligence and security agencies were shrouded was such that they lacked a statutory footing, and their archives remained firmly closed to researchers. Yet although such research was challenging, it was not impossible. Writing in the Historical Journal in 1992, Wesley K. Wark detailed a number of strategies that could help circumvent the continued retention of the records of the agencies themselves. In addition to the use of personal testimony and published memoirs, Wark called upon researchers to:
adopt a more radical definition of the nature of intelligence archives, and to turn our sights from the question of explaining what secret agencies did or do, to how governments think and act. If our task is so defined, the intelligence archive becomes much more capacious and much more resistant to the apparatus of sections 3(4) and 5(1) of the public records act and related measures of censorship. For the intelligence archive becomes the record of all those government departments who receive, incorporate, digest and report on intelligence that comes to them from both secret and overt sources.Footnote 8
Wark's proposed approach was already in evidence in a number of published works that used the accessible official records with a view to gaining some insight into intelligence and security affairs: David Stafford's Britain and the European resistance, widely regarded as a ground-breaking study of the wartime Special Operations Executive (SOE), had been written without access to what survived of the SOE archive. Rather, acknowledging that SOE, while an organization in its own right, ‘was instructed to confine itself to the execution of policy decisions made by others’, Stafford made extensive use of released material found in other file series that were concerned with the work of the organization, specifically ‘the files of the Prime Minister, Chiefs of Staff and related committees, and, to a lesser extent, the Foreign Office’.Footnote 9 A few years later, The missing dimension: governments and intelligence in the twentieth century, edited by Christopher Andrew and David Dilks, further illustrated what could be achieved through the use of material found in the files of non-secret government departments, the editors noting that historians ‘have been far more put off the subject of intelligence than they need have been’, their collection illustrating ‘what can be reliably based on existing archives and published source material’.Footnote 10
At the time of publication of Wark's article, a sea change was taking place in terms of official attitudes towards the secrecy in which the intelligence and security agencies had been traditionally shrouded, which resulted in both their public avowal and changes in archival release policy. The Open Government Initiative, launched by William Waldegrave and Douglas Hurd in 1992, resulted in the release of thousands of documents, previously withheld, into the public domain at the Public Record Office (now the National Archives), including the bulk of the surviving SOE archive.Footnote 11 Within a few years, the Security Service began to release batches of its own historical papers, a practice which continues to the present day, operating outside the remit of FOI.Footnote 12 Such a significant increase in the availability of official archival material inevitably had an impact upon the historical study of intelligence and security issues. The new openness was, however, greeted with caution in some quarters, as concern was expressed over the dangers it posed by way of information control.Footnote 13 Leaving aside concerns about the control exercised by the authorities over what could be released – be that control benign or otherwise – the increased availability of archival intelligence and security agency material also presented the researcher with the temptation to adopt an altogether narrower archival research strategy to their subject than had hitherto been the case. What Aldrich had earlier described as the ‘well worn path to Kew’ was now extended with a new, but quickly well-trodden, path towards certain key file series, including the HS (SOE), HW (GC&CS) and KV (Security Service) collections.Footnote 14 Concern has been expressed about this newfound ability to work extensively with agency material, and with its potential to exacerbate what Aldrich and Ferris have both characterized as ‘trainspotter’-like tendencies in the field.Footnote 15 Echoing the observation made earlier by Andrew and Dilks, Ferris has observed that ‘official records always contained more material on intelligence than was generally realized’. However, pinpointing this material was not necessarily straightforward:
the problem was that finding them took much time and effort. The new releases make intelligence easier to discover, but at a price. Once, when found, such records were embedded in their decision-making context, easing judgement as to what they affected, and how. Now, intelligence is more accessible but stands alone; the problem is to find its context. The situation manifests the tendency to fetishize and to sensationalize intelligence.Footnote 16
The issue is also bound up with a wider, definitional question faced by researchers in the field: what is ‘intelligence’? In one of the earliest academic studies of the subject, Sherman Kent offered three different definitions; intelligence as ‘a kind of knowledge’, the organization ‘which produces the knowledge’, and ‘the activity pursued by the intelligence organization’.Footnote 17 It is clear that these definitions, which continue to be reflected in more recent efforts to pin down the word, would likely favour, and certainly benefit from, the extensive and detailed study of agency material. Yet, such definitions do not satisfy everyone – nor, indeed, does the product of such studies: Aldrich notes the existence of somewhat insular works, that appear ‘content to catalogue operational details and matters of tradecraft, while expending little energy in investigating their broader importance’, while Ferris reminds the reader that intelligence ‘influences decisions but does not make them’.Footnote 18 Such observations can themselves be associated with another definition of intelligence which sees the agencies, their activities, and the ‘product’ they produce as feeding into a wider decision-making process on the part of policy-makers. Acknowledging that some broader studies ‘of intelligence and decision-making’ have appeared, such as Andrew's For the president's eyes only, Aldrich notes that such a ‘contextual approach’ remains ‘the exception rather than the rule’.Footnote 19 It is in the furtherance of such an approach to intelligence that FOI offers a potentially valuable opportunity. Recalling Wark's point that ‘“intelligence” is not simply what the secret services do’ – an observation recently characterized as ‘crucial’ by R. Gerald Hughes and Len Scott – it is worth noting that, while the intelligence and security agencies themselves are exempt from the legislation, those government departments that ‘receive, incorporate, digest and report on intelligence that comes to them’ are not.Footnote 20 As such, while Michael Goodman is entirely correct to note that the exemptions that cover the intelligence and security agencies mean that FOI requests can only be made to ‘the Defence Intelligence Staff … or the Joint Intelligence Committee’, the observation must be considered in a strictly institutional context.Footnote 21 Engaging with FOI offers some opportunity to further the study of intelligence and security history beyond the archives of the agencies themselves.
This opportunity, however, does have clear limitations: in making use of FOI in an effort to locate intelligence and security issues within their wider context, researchers face the relevant exemption – Section 23 – which covers ‘Information supplied by, or relating to, bodies dealing with security matters.’Footnote 22 The phrasing of the exemption could render anything that touched upon the work of the intelligence and security agencies found in the files of Whitehall departments as beyond the right of access provided by the Act. However, guidance issued on the application of Section 23, while acknowledging that the term ‘has the potential to be applied to a wide range of situations’, also recognizes that ‘there will be a point when the connection between the requested information and a security body is too remote to engage the exemption’.Footnote 23 The limitations of the exemption were illustrated by one of the earliest releases added to the Cabinet Office online FOI ‘Reading Room’; information concerning the appointment of the incoming head of SIS, Sir John Scarlett.Footnote 24 Further, while Section 23 is an absolute exemption that does not face a public interest test, this only holds true for more recent material; historical records, when transferred to the National Archives, become subject to a public interest test.Footnote 25 Here, while information ‘may still be sensitive and warrant the use of section 23’, requests ‘should be judged on a case by case basis, consulting as appropriate’.Footnote 26 The remainder of this article will present further examples of intelligence and security-related material that has been brought to light through the use of FOI, before proceeding to offer some practical observations as to its application and pointing to its limitations.
II
In the case of British Cold War special operations, relatively few documents have entered into the public domain and historians have been forced to look further afield for details of ‘covert action’.Footnote 27 However, a successful FOI request to the Foreign and Commonwealth Office (FCO) shed light on a Cold War special operation; Operation Valuable, the joint Anglo-American scheme to penetrate Albania in 1949. Up to this point, details of the operation, unofficially documented in earlier accounts, were absent from the National Archives and it was only in 2010 that its planning and implementation were documented by Keith Jeffery in his authorized history of SIS.Footnote 28 The released material included a briefing note sent to the prime minister, Clement Attlee, in March 1949, by Sir William Strang, permanent under-secretary at the Foreign Office, and two related papers. The first suggested the insertion of ‘Albanian or Albanian-speaking W/T operators’ into the south of the country to create an intelligence network capable of verifying whether conditions were favourable for ‘insurrectionary purposes’ involving the arming of anti-Communist supporters. The plan, Strang wrote, was ‘admittedly … venturesome’ but ‘the prizes to be won are worth the risk involved’.Footnote 29 The second paper by the chief of SIS, Sir Stewart Menzies, warned that it was not worth conducting preliminary operations unless ministers were ‘prepared to follow them up by striking as hard as possible’.Footnote 30 Though extracts from the documents had been quoted by Jeffery, the released material contained a further significant addition that offered an insight into Attlee's attitude towards special operations. On 28 March, two days after Strang had forwarded the proposals, Attlee scribbled on the top of Strang's covering letter: ‘I should like an appreciation of Albanian personalities. Are they not possibly for sale?’Footnote 31 The comment offers a stark contrast to the popular image of Attlee as being apathetic towards special operations.Footnote 32 The next month, ministerial authorisation was given.Footnote 33
FOI has also provided new insights into Britain's domestic Cold War activities, allowing studies of early Cold War Britain to move beyond changes in government vetting procedures and look at offensive anti-Communist counter-measures. Released papers on the Committee on Communism (Home) offer a tantalizing insight into such activities until 1953. Formed two years earlier, the committee was tasked with collecting ‘all available intelligence’ about domestic Communist activities, to make recommendations to ministers, and to co-ordinate any counter-measures subsequently approved. Chaired by the cabinet secretary, Sir Norman Brook, it included representatives from across Whitehall, including MI5, the Home Office, and the Foreign Office.Footnote 34 The papers chart the development and early operations of the ‘Home Desk’ of the Foreign Office's Information Research Department (IRD). Until now, the operations – and even existence – of the section have been absent from accounts of the department, with only Paul Lashmar and James Oliver detailing IRD's domestic activities in any real depth.Footnote 35
While the released papers refute claims that IRD extensively manipulated public opinion, as Peter Weiler argues, they show that the Foreign Office, given the reluctance of Home Office officials to engage in offensive counter-measures, took a leading role in domestic anti-Communist activities, proposing a multi-faceted campaign to educate influential sections of the public on Soviet aims and methods.Footnote 36 Previously, the department had no formal mandate to intervene in domestic affairs, though it had briefed, at varying times, select journalists and organizations, including the Labour party. At first, IRD's contribution to the new campaign was relatively modest; in October 1951, just weeks after its formation, the ‘Home Desk’ numbered just four individuals, including representatives from the ministries of labour and education, though it established useful links and compiled a detailed intelligence picture.Footnote 37 The output of this section was limited to influential groups, including Union officials, the Armed Forces, and hand-picked school inspectors, on account of a fear of Communism in the classroom.Footnote 38 By 1952, IRD was the ‘focus’ for the collection and dissemination of intelligence on domestic Communist activities and produced regular reports using press cuttings and other sources of information for Whitehall consumers.Footnote 39
FOI has also proved to be of value in securing the release of material that illustrates official attitudes towards the secrecy in which the intelligence and security agencies were shrouded for much of the twentieth century.Footnote 40 A request for information concerning the publication of F. W. Winterbotham's The ultra secret in 1974 resulted in the release of a number of documents, including a paper entitled ‘The official history of intelligence in World War II and related subjects’, which was sent to Dick White, a member of the Steering Committee guiding the official history of wartime intelligence that was being written by Professor F. H. Hinsley, on 7 November 1975. The paper illustrated how a number of interrelated issues had fundamentally undermined the traditional policy of non-disclosure towards wartime intelligence activity. No matter how ‘desirable’ the continuation of the traditional stance ‘that HMG does not admit to, or comment on, its intelligence activities’ may have been, the paper conceded that this had ‘been breached so many times … that a new situation has been created’.Footnote 41 This represented a significant shift in the official mindset towards such matters, and was certainly a far cry from the blanket ban adopted by the Joint Intelligence Committee (JIC) in July 1945, at which point it was agreed that wartime ‘special intelligence’ – i.e. Ultra – 'should never be disclosed’.Footnote 42
Publication of The ultra secret was considered by the paper to represent a significant moment in the creation of this ‘new situation’. While the authorities had traditionally ‘attempted to deter authors from writing on UK intelligence organisation and activity’, their ability to do so was shaken by Winterbotham's account of wartime codebreaking activity, which was described as the ‘crunch’ moment.Footnote 43 To a certain extent, this moment was one of the authorities’ own making. Initially ‘deterred from publishing’, Winterbotham had changed his mind in light of the appearance of two publications; Gustave Bertrand's Enigma and J. C. Masterman's The double-cross system, the latter of which, the report admitted, had been ‘published with HMG's tacit approval’. As such, it was considered that a threat of prosecution under the Official Secrets Act was not only ‘unlikely to succeed’, but could also ‘prove counter-productive by arousing undesirable publicity’; a reference, at least in part, to the fact that the revelation of official complicity in the publication of Masterman's account of the double-cross system would likely follow.
The ultra secret ‘aroused considerable interest’, leading others ‘involved in wartime Sigint’ either to write their own accounts, or express their intention to do so, now having recourse to a precedent that the authorities would be hard pressed to challenge. Alongside this continued battle against unofficial authors, which was dealt a severe blow by Winterbotham's actions, the 1967 Public Records Act had replaced the fifty-year rule for the release of official documents with a shorter thirty-year rule, forcing the authorities to consider the question of whether Second World War intelligence material could be released into the public domain. Publication of The ultra secret also proved to have another, altogether unexpected, consequence as it brought the attention of both the authorities and the wider public to the earlier, accidental release of wartime intelligence material. The serialization of Winterbotham's book in the Sunday Telegraph had been followed by a letter to the editor, in which the correspondent noted that Winterbotham's account had ‘clarified the top secret ultra and other references he had found in the Public Record Office’. An investigation followed, which revealed that army and air intelligence records, released by the ministry of defence and available to the public ‘for some time’, had ‘contained a quite substantial proportion of highly classified intelligence’, and ‘revealed a good deal about wartime intelligence sources’. The ‘fiasco’, which saw the material temporarily closed so that the ‘Really sensitive’ material could be withdrawn, led to the review of Second World War Naval intelligence records for release, similarly ‘expunged’, in order to maintain a consistent release policy.
III
While the use of FOI has resulted in the release of new material, we do not wish to exaggerate its effectiveness. Its usefulness is limited, for example, when conducting research on intelligence liaison. Here, in addition to Section 23, historians face the application of further exemptions which can prevent the release of information acquired from foreign governments, as failed attempts to access information on the Commonwealth Security Conferences of 1948 and 1951 illustrate. Although some material relating to both conferences has already entered the public domain, the Cabinet Office, while acknowledging they held the information requested, continued to withhold it.Footnote 44 Alongside Section 23 of the Act, the information was also subject to Section 27(1), which prevents the release of information if its disclosure would, or would be likely to, prejudice ‘relations between the United Kingdom and any other State’ or ‘the interests of the United Kingdom abroad’. Section 27(2), relating to ‘confidential information’ from another state, also applied. Having considered the arguments in favour of public disclosure, the Cabinet Office ruled to retain the information to preserve ‘the trust of our international partners’.Footnote 45 A similar request for access to closed records from Britain's Washington embassy saw the FCO invoke 27(1) (a), fearing that release of the material would prejudice external diplomatic relations; it was felt that disclosure would result in ‘a real and significant risk … to UK relations with the United States’.Footnote 46
It is also important to note that the use of FOI should not be characterized as an ‘easy’ research option. In order to make a successful request, one must know enough about the subject to ensure that it cannot be rejected on the grounds of being too vague, or too vast, to answer within the FOI cost framework.Footnote 47 A request made to the Home Office soon after the Act came into force, asking for ‘all available information’ on Spycatcher fell foul of this criteria; in response, the Home Office noted that ‘to comply with your request would exceed the cost limit beyond which we are not required to supply information’, and suggested narrowing it down: ‘For example, you may like to detail more exactly the piece of correspondence or papers in which you are interested.’Footnote 48 Given the secrecy in which intelligence and security matters are enveloped, it will likely frequently prove difficult to present a detailed request for specific papers. Older material offers the opportunity of spotting ‘gaps’ in the National Archives’ catalogue; the release of information on Operation Valuable, detailed above, resulted from a request to the Foreign and Commonwealth Office for access to FO 800/437, a file listed on the catalogue, but without a description or date. Part of a collection of the working papers of the foreign secretary, Ernest Bevin, the file was sandwiched between entries for ‘Albania’ (1946) and ‘Atomic Energy’ (1945–51), presenting the possibility that it contained details about British policy towards Albania, including Valuable. In response, the FCO confirmed suspicions that the file contained details about the operation, and explained that the contents of the file were considered ‘sensitive’ under Section 23, and would continue to be retained.Footnote 49 Over a fortnight later, and after a further review, the earlier decision was reversed and the file ‘released in full’.Footnote 50
A further significant practical issue that can complicate use of the Act is the time factor, FOI being described by Maurice Frankel, director of the Campaign for Freedom of Information, as a ‘deceptively slow process’.Footnote 51 While stipulating that a public body should respond within twenty working days, the process can take much longer, particularly if the requester has to challenge a decision not to release the material. The release of documents on the Committee on Communism (Home), detailed above, effectively illustrates this point; some nineteen months passed between the lodging of the initial request, in October 2010, and the eventual release of material in May 2012. The Cabinet Office initially refused to disclose the papers, citing Section 23 of the Act, a decision that was upheld following a request for an internal review.Footnote 52 An appeal was subsequently lodged with the Office of the Information Commissioner (ICO), which is tasked with overseeing the implementation of FOI and helping ‘citizens and consumers assert their … right to know’.Footnote 53 The issue was subsequently resolved without formal action. The ICO wrote to the Cabinet Office, asking ‘whether there is scope to resolve this case informally’.Footnote 54 Subsequent discussions prompted the Cabinet Office to agree to release the early papers of the Committee on Communism (Home) in October 2011, though the material only became available to view in May 2012 following consultations with other government departments.Footnote 55
While the use of FOI brings with it new problems and challenges, further engagement with the Act is needed before these can be discussed in any great depth. For example, while efforts are made to draw attention to released material that is likely to be of wider interest via departmental websites, the question of the widespread dissemination of material released to an individual remains problematic, the current situation serving ‘not to release information into the public domain, but rather to create an oligarchy of information-holders where previously there was a monopoly’.Footnote 56 Given that material released from more contemporary files may well be ‘weeded’ prior to its eventual transfer to the National Archives, it is not necessarily simply a question of waiting for a subsequent archival release. Historians of all sub-disciplines who use FOI should therefore give some consideration to the ultimate ‘home’ of material released to them once the relevant research project has been completed, beyond gathering dust in the attic. The question of just how useful ‘fragments’ of history gathered through FOI can be also requires further attention. While the material on the Official history of intelligence during the Second World War, discussed above, provided some useful insights, the papers had been plucked out of their place in the official record, making it difficult to appreciate them fully until they appeared within thick Cabinet Office files released to the National Archives some years later. A further point worthy of consideration concerns the fact that FOI offers access to information; not necessarily to copies of full documents. Rather, a request may well result in the release of a ‘digest’ of relevant information. Both methods of release may raise questions over the nature of the presentation of information provided to the requester.
Such issues underline the fact that FOI is far from a silver bullet; rather, it constitutes a potentially useful facet of a wider research methodology, one which has the potential to impact positively upon other long-standing approaches, such as the use of oral testimony. When used successfully to secure access to material less than thirty years old, the use of FOI has the potential to heighten the significance of conducting interviews with participants in the events documented by the written record, an observation that was made shortly after the introduction of the Act by Anthony Seldon who, while welcoming FOI, stressed the continued importance of oral history.Footnote 57 Interviewing has previously been recognized as being ‘invaluable in clarifying the meaning of a document’, but prior to FOI the opportunities for such interaction were limited on account of the tendency for papers to be released into the public domain ‘just as the last valuable witnesses are shuffling off the mortal coil’.Footnote 58 In addition to providing useful commentary on a released document, an interview held prior to the making of an FOI request could also provide valuable information in order to target the request more effectively.
IV
This article has illustrated how freedom of information can be used in order to cast further light upon British intelligence and security history, despite the exemption of the agencies themselves from the provisions of the Act, and has acknowledged some of the problems that may be encountered along the way. While recognizing its limitations, we nevertheless believe that our efforts illustrate that FOI can be beneficial, a conclusion similarly drawn by those involved in research into nuclear history, another traditionally highly secretive area of the British state.Footnote 59
The use of FOI could, of course, be criticized by those who caution against overreliance on the official record as constituting the archival research equivalent of Oliver Twist, asking for more of the same; that is, more official documents that will be processed – selected, reviewed, and possibly redacted or summarized – prior to their release. Yet, while control over what is, ultimately, released remains firmly in the hands of the authorities, FOI offers researchers the opportunity to prod the dark and mysterious recesses of unreleased material; to question the existing declassification agenda and challenge the decisions that have resulted in its non-disclosure. In some cases, this may result in the release of further useful material; at the very least, the FOI request itself will likely, although not always, result in an acknowledgement of the existence of material, even if it is to remain off limits, challenging the secrecy that surrounds it and potentially reducing the number of ‘unknown unknowns’.Footnote 60 Meanwhile, by detailing the exemptions under which the continued retention of material is justified, FOI can help to foster a greater degree of understanding, if not ultimately agreement, between the official and the historian.
