The cusp of the current decade witnessed a seemingly unprecedented development of the historical study of British intelligence in the publication of officially sanctioned histories of two of the UK's national ‘security and intelligence agencies’ and what one might call an officially indulged albeit not sanctioned and supported account of the third. The UK's three national agencies are, of course, the Secret Intelligence Service (SIS, colloquially known as MI6), the Security Service (MI5) and Government Communications Headquarters (GCHQ). The first two conduct human intelligence (agent-running or HUMINT), physical and technical surveillance operations outside and inside the UK respectively while GCHQ is responsible for ‘signals intelligence’ (SIGINT), that is, interception and analysis of intercepted communications (COMINT) and non-communicative emissions such as radar and vehicle telemetry typically grouped as electronic intelligence (ELINT). The appearance of the two official histories had been keenly looked forward to by intelligence scholars since they were announced and Christopher Andrew and Keith Jeffery selected as their authors. There had been no less interest in Richard Aldrich's even more long-awaited GCHQ history. In the event, however, the official histories have created far less of a stir than might have been expected, while the work on GCHQ rather more. While some of this fizzle rather than bang might be attributed to specific attributes of the works in question, it can also be seen as a consequence of the changing general awareness and understanding of awareness amongst their potential readerships over the last decade or so. This sense of things is made even sharper when placing these scholarly works against the kind of journalistic writing represented by Gordon Thomas's work that a generation ago might have been considered the ‘state of the art’ in writing on intelligence. In short, the official histories are sound but ultimately incremental contributions while Aldrich's work represents something of a step change and Thomas’ work essentially irrelevant.

The steady increase in ‘intelligence literacy’ amongst scholars and the electorate alike has accumulated over the better part of two generations rather than being some brief, steep learning curve since the ‘open government initiative’ of the early 1990s. Although the earliest more or less official discussion of the interagency Joint Intelligence machinery is probably a brief discussion in a survey of British central government conducted by the Royal Institute for Public Administration in 1957,Footnote ¹ the first significant public moment of official openness occurred with the 1963 publication of Lord Denning's report into the ‘Profumo affair’.Footnote ² Denning's report coincided with John Bulloch's (slightly hagiographic) account of the Security Service,Footnote ³ but Denning did more than just provide official acknowledgement of MI5's existence. His report also published the agency's formal mandate, the 1952 Maxwell Fyfe Directive. The Cambridge spy scandals generated a great deal of journalistic exposé and discussion,Footnote ⁴ including Philby's own artfully crafted memoirFootnote ⁵ which was true enough to convince but selective enough to mislead and disrupt.Footnote ⁶ Less well remembered today, although the subject of intense discussion at the time, the mid 1960s also witnessed the first official history of a clandestine organisation. In 1966 the recently deceased and sorely missed M. R. D. Foot published the first of a series of official histories of the wartime sabotage agency called the Special Operations Executive (SOE).Footnote ⁷ The next significant unofficial contributions to our historical understanding of the role and impact of Secret Intelligence came from the 1972 publication of John Masterman's post-war after-action report of the ‘double-cross’ counter-espionage programme run against the Germans,Footnote ⁸ followed in close order by the even more dramatic implications of Frederick Winterbotham's The Ultra Secret in 1974.Footnote ⁹ While Winterbotham's recollections are often faulty – a risk of which he warns the reader, admitting that he was working essentially from memory – the critical impact of this latter unofficial revelation was that SIGINT and especially cryptanalytical successes against the Axis had been so extensive that our basic understanding of the conduct of the Second World War might need to be substantially revised.

Amidst the drama of molehunts in the press and unsanctioned disclosures by intelligence community alumni it is easy to forget that an official history of British intelligence during the SecondWorld War was already in train. Led by Sir Harry Hinsley, himself an intelligence ‘old boy’ of Bletchley Park vintage, Hinsley's team would eventually publish the most extensive and detailed account of UK intelligence to date in a series of five volumes (strictly speaking six with Volume 3 having Parts I and II published roughly two years apart) that appeared over the course of the 1980s.Footnote ¹⁰ It is important to keep in mind that Hinsley's volumes did more than simply recapitulate events 1939–1945.Volumes 1 and 4 reached back to the turn of the century and at points even earlier to provide an account of the origins of the intelligence community. In the meantime, Winterbotham and Hinsley combined prompted the emergence of something of a cottage industry in wartime SIGINT history by independent historians who firstly now knew where to look and what to look for in the archives, and the flow of declassified documents that was prompted by the Hinsley official history project.Footnote ¹¹ It was during this period that David Dilks and Christopher Andrew published what amounted to a programmatic piece for intelligence history The Missing Dimension,Footnote ¹² which Andrew followed with his monolithic and for many years authoritative account of the formative early years of the UK intelligence community leading to the emergence of the Joint Intelligence Organisation.Footnote ¹³ In parallel, Kenneth Robertson generated Britain's first, primarily comparative social scientific efforts in the field, efforts fundamentally reliant on close historical scholarship.Footnote ¹⁴

In the meantime, the Security Commission ticked away in the background generating a succession of official reviews on public intelligence furores such as the Frank Boassard and Percy Allen spy case, Geoffrey Prime in GCHQ and his subsequent defection and David Bettaney's failed attempt to become a ‘walk-in’ source to the KGB London station or ‘residency’.Footnote ¹⁵ Of course, the Security Commission made no mention of the fact that Bettaney's effort failed largely because SIS was running the London Deputy Resident at the time, Oleg Gordievsky. That matter would not come to light until well after Gordievsky's 1985 defection.Footnote ¹⁶ The traditional secrecy surrounding British intelligence was, however, coming under increasing pressure.

It should be apparent at this point that by the early 1980s the British public was already far more ‘intelligence literate’ than it had been little more than a decade earlier. Consequently when Lord Franks conducted his investigation into events leading to the Falkland Islands War and the strategic surprise and therefore intelligence failure that appeared to have taken place, intelligence was not quite the terra incognita it had once been. There was already a narrative of the Joint Intelligence Organisation's origins and its relationship with the national agencies in plain sight and officially sanctioned in the Hinsley official history. It was in this context that Franks elaborated the current workings of the JIC, its analytical subcommittee structure of Current Intelligence Groups and even spoke in broad terms to the role of the ‘Security and Intelligence Agencies’ in that machinery.Footnote ¹⁷ In some respects, Franks represented a significant break with precedent by examining recent or current intelligence arrangements but over the longue durée this was actually a comparatively incremental movement.

Public furore would likewise drive the next major shift in official openness almost exactly a decade later in the form of the Ordtek and Matrix-Churchill affairs and the subsequent judicial inquiry by (then) Sir Richard Scott. Ordtek and Matrix-Churchill were two firms whose officials were prosecuted in the wake of the First Gulf War for selling dual-use technologies to Saddam Hussein's Iraq, putatively in violation of the 1939 Export Control Act. The Matrix-Churchill prosecution collapsed when it transpired that the defendants had been working as volunteer agents or ‘alongsiders’ for SIS and, moreover, that the government in the form of junior Foreign Office Minister Alan Clarke had been aware of their activities and hence implicitly sanctioned them. In response, the government was forced to convene a judicial inquiry headed by Sir Richard Scott.

While the remit of the Scott inquiry might have been another incremental development in public knowledge of British intelligence, the conduct of his inquiry and his final report were not. Senior intelligence officials from the agencies and Cabinet Office gave evidence largely in public, and his final five-volume report cited government correspondence in footnotes and provided a CD of digitised official papers.Footnote ¹⁸ The Scott inquiry was literally unprecedented in the detail and transparency of its inquiry into government operation in general, let alone the intelligence community. Scott set the model for subsequent investigations such as Lord Hutton's voluminous report and detailed verbal, written and documentary evidence on the death of Sir David Kelly after the Second Gulf WarFootnote ¹⁹ and the Scott Baker inquest into the death of Diana, Princess of WalesFootnote ²⁰ More importantly, the Scott inquiry coincided with an additional source of growing momentum for increased openness arising from tension between European institutions and the sovereign practices of member states in the European Union. Britain was not the only state to find its intelligence institutions on the wrong side of judgements made by the European Court of Human Rights.Footnote ²¹ By the late 1980s it was evident that the governance of intelligence was going to have to change. The first step here had been the 1989 Security Service Act governing the activities of the organisation most at risk of running afoul of ECHR ideas of appropriate government. Scott now provided scholars and commentators with a new and rich vein of material to use in their study of intelligence in contemporary British government.

If the first real step change in the public and scholarly understanding of intelligence was provided by Masterman, Winterbotham and Hinsley, then Scott provided the second step while the Matrix-Churchill trial helped promote the next major governmental shift, Prime Minister John Major's ‘Open Government Initiative’, sometimes referred to the Waldegrave Initiative after the minister responsible for articulating and implementing it.Footnote ²² Heralded by a white paper entitled Open Government,Footnote ²³ this initiative involved new public openness at a number of levels of intelligence officialdom, liberalising archive release policy and statutory changes concerning the governance of British intelligence. The first official gesture was a short Cabinet Office handbook on the Joint Intelligence Organisation and intelligence community entitled Central Intelligence MachineryFootnote ²⁴ with the first of nominally three editions of a public information booklet on MI5 coming shortly thereafter.Footnote ²⁵ The first tranche of new intelligence papers concerning intelligence at the turn of the twentieth century were released (the HD series), with Special Operations Executive files (HS series) being released in batches starting in 1995, MI5 papers (KV), GCHQ materials (HW) and a growing volume of JIO papers (mostly in CAB) appearing in due course. Amidst this, and with the embarrassment of Matrix-Churchill and Scott still rankling, SIS and GCHQ were finally placed on a statutory footing like MI5 under the 1994 Intelligence Services Act.

Besides serving as enabling legislation, the 1994 ISA also established the Intelligence and Security Committee. Technically a Prime Minister's ‘committee of Parliamentarians’ rather than a Parliamentary committee as such (and somewhat controversially so as it was not, therefore, a select committee like those for defence or foreign policy), the ISC would prove to be a motherlode of current information on the policy and governance of intelligence over the next decade and a half. Not only did the ISC issue annual reports, it also conducted its case-specific investigations sometimes at the behest of government and occasionally on its own recognisance. This included a report on intelligence on Iraqi ‘weapons of mass destruction’ in 2003,Footnote ²⁶ although this was quickly overshadowed by the subsequent Hutton and Butler inquiries. The ISC also undertook the initial inquiry into any intelligence failure prior to 7 July 2005 terrorist attacks in London,Footnote ²⁷ published well in advance of the wider Home Office review. During the 1990s the recently-established National Audit Office also began to produce published reports on intelligence matters such as the GCHQ move to its new headquarters colloquially known as ‘the doughnut’.Footnote ²⁸ Lord Hutton and Lord ButlerFootnote ²⁹ contributed much to the momentum started by Scott with their reports. In 2005 the volume of archival releases increased again following New Labour's 2004 Freedom of Information Act, reinforced with a more staccato, limited and ad hoc trickle of intelligence-related items via FOIA applications.

2005 also witnessed the publication of Sir Lawrence Freedman's two-volume official history of the Falkland Islands War. Although in principle an operational or campaign history in the mould of Sir James Edmonds’ original, monumental official history of the European campaign in the First World War, Freedman's work added new elements to our understanding of the intelligence picture at the time. Most significantly Freedman demonstrated that while there had been a strategic assessment failure in assessing the degree and rate of deterioration of relations with Argentina there had also been a palpable success in warning intelligence. Far from being surprised by the Argentine landing at Port Stanley, signals intelligence gathered from Royal Navy platforms, processed by GCHQ and assessed by the Defence Intelligence Staff had provided four days’ warning of the attack by detecting the invasion force while it was still at sea. This warning prompted Margaret Thatcher's government to dispatch three nuclear attack submarines to the South Atlantic and begin assembling a task force to reclaim the islands before the Argentines even landed.Footnote ³⁰ Unfortunately the Falklands were at least seven days steaming away from the UK at the submarines’ maximum cruising speed of circa 30 knots so even four days’ warning was not enough to physically intervene in the invasion.

The last five decades have, therefore, witnessed a steady growth in the availability of official and independent scholarship, archival raw materials and current governmental reports and information. One of the effects of this has been what one might call diminishing marginal revelation. Between 1963 and 1993 any fragment of raw information was seized upon like a gold nugget and often valuable chiefly because of the dearth of previous or alternative materiel. Whatever the ideological divisions amongst intelligence scholars might have been, there was also something of an esprit de corps with regard to a common adversary which was simply the scarcity of solid sources with which to work. But the contemporary environment is profoundly different. Blessed in the twenty-first century with source prosperity, the academic discipline of intelligence studies has different requirements and different standards from what once prevailed. And it is in this environment that the value and impact of these new works has to be assessed.

With both official histories, there is something of a sense of opportunities lost. In part this is no doubt because the expectations for any official history today are so much higher than when Foot and Hinsely produced theirs. Much of the lack of impact might also be attributed to the necessary conservatism of both SIS and MI5 as on-going enterprises, unlike SOE, wartime intelligence arrangements (or, for that matter, the British Expeditionary Force when Edmonds provided the original template). Despite the best efforts of both Andrew and Jeffery, the agency convention of ‘neither confirm nor deny’ (NCND) is fundamentally incompatible with the production of official history, which is ultimately about confirming some things and refuting others. One feels throughout that these official histories would have been stronger had more extensive use been made of materials in the publicly available materials in The National Archive, India Office records and personal papers held at various institutions around the UK but perhaps this would have made ‘NCND’ harder to sustain. Defunct entities like SOE or the BEF were hardly likely to see their current and future commitments and activities affected by any historical revelations. This cannot be said of SIS or MI5.

This sense of missed opportunity is all the more acute when the official accounts are placed alongside Richard Aldrich's unofficial history of GCHQ. Aldrich achieves a comparable depth of detail to the official histories when recounting the early history of ‘Q’ and its slow, sometimes piecemeal acquisition of independence from the bureaucratic empire of the Chief of the Secret Intelligence Service or ‘C’. Subsequent operational and administrative detail are also well developed throughout his volume, although like Andrew's account of MI5 the detail necessarily thins as one approaches the present. Secrecy, after all, has merely been attenuated and has not actually evaporated. However, while Aldrich's volume may not be an official history it would not be misleading to describe it as having been officially indulged if not actually countenanced. This is because successive GCHQ staff historians have cultivated close and good relations with British academics. On the strength of a short piece on GCHQ management structure I originally wrote a decade agoFootnote ³¹ the then-GCHQ staff historian took to routinely sending me copies of documents on GCHQ early development for which he had secured release to The National Archive (which, I confess, I never used, leading to some pangs of conscience when the official in question passed away, his generosity having gone to little effect at least in my case). This correspondence was indicative of the kind of outreach on which GCHQ has been engaged since the turn of the last decade. Thus GCHQ's level of secretiveness (as opposed to actual secrecy) can easily be overestimated. And, of course, systematic academic outreach allows an agency to try and put their own case across without having to worry about trying to impose the NCND principle on the final product.

Both Andrew's and Aldrich's volumes have the advantage over Jeffery that they have been able to bring their account as close as possible to the present day. SIS, by contrast, insisted that Jeffery take his work no further than 1949. When the first, hardbound edition was published this drew criticism from reviewers on the grounds that SIS’ chosen cut-off point coincided with the moment when Soviet penetration of the UK government, especially SIS, became a matter of central importance to the nation as well as the agency.Footnote ³² On this the agency has relented somewhat in the paperback edition reviewed here, allowing Jeffery to append a postscript which discusses inter alia H. A. R ‘Kim’ Philby's betrayal of would-be Soviet defector Konstantin Volkov.Footnote ³³ An indication of the fragmentary nature of SIS’ internal archives is the fact that this account was pieced together from Foreign Office papers rather than SIS materials. Nonetheless, Jeffery's work provides some valuable clarifications of our understanding of SIS development and its achievements. On of the most valuable such contributions is a correction to a conventional wisdom that SIS produced little or anything of value during the Second World War and survived as the purveyor of ULTRA code-breaks from Bletchley Park. To the contrary, Jeffery shows us that after initial setbacks the agency began to produce substantial, valuable product from inside Nazi-controlled territory from 1942.This even included an agent in Berlin, run jointly with the exile Polish intelligence service, who had access to ‘contacts’ in the Oberkommando Wermacht and reported reliably for over a year on matters such as ‘“the state of mind” at [OKW] following the defeat of the German Sixth Army at Stalingrad’.Footnote ³⁴

One of the most significant contributions of Andrew's volume is that it extends discussion of MI5 beyond counter-espionage and the more politically fraught realm of counter-subversion to the agency's protective security function. A highly visible feature of today's Security Service through its Centre for the Protection of National Infrastructure, protective security traces its roots to MI5's wartime counter-sabotage function. After the Second World War, formulating cross-government security standards and providing security advice on an interdepartmental basis developed into one of the agency's most significant functions. Such an emphasis is also significant because it sets the context for much of MI5's original role in countering the threat from terrorism. Consequently Andrew gives us a level-headed and balanced narrative of the difficult period of adjustment formulating a counter-terrorism intelligence order of battle to cope with the Troubles in Northern Ireland.Footnote ³⁵ This is a valuable corrective to the somewhat histrionic accusations over the decades of intractable interagency turf-wars in that theatre. This emphasis on MI5's protective security stands in contrast with the absence of a comparable narrative concerning GCHQ's communications security (COMSEC; in today's parlance information assurance) function in Aldrich's volume. Had his publishers been more receptive to including this aspect of the GCHQ story the public would have benefited from a more well rounded understanding of GCHQ, albeit one far more prosaic and more populated with unresolved battles of wits than ripping tales of SIGINT derring-do. It can only be hoped that a second volume on communications security will be forthcoming from Aldrich in due course.

In stark contrast, Thomas’ volume on British intelligence encapsulates all that should be avoided in writing on intelligence and is ultimately something of an intellectual equivalent to a living fossil. The book is confusingly written, delivering its historical narrative through nested flashbacks embedded within contemporary would-be exposé. At points, the flashbacks run two or three deep, meaning that half way through a chapter on contemporary British intelligence activities against terrorism one is first discussing the life history of an agency head, then a few pages later the work of that agency in the Cold War and then a few pages after that an often muddled account of its origins. The result is often bewildering.

Thomas is sloppy in the extreme in matters of detail, consistently referring to the UK's national intelligence budget as the ‘Single Unified Vote’Footnote ³⁶ rather than the Single Intelligence Vote (SIV), and apparently unaware that the SIV had been restructured under New Labour as the Single Intelligence Account (SIA) which it currently remains (he even mislabels that old nineteenth-century standard the Secret Vote as the Secret Service FundFootnote ³⁷). He also seems to subscribe to the frankly delusional notion that SIS receives, holds and collates all of the product of all of the domestic CCTV monitoring,Footnote ³⁸ despite the facts that most CCTV is in private hands, that such activity would be in flagrant violation of both the 1994 Intelligence Services Act (let alone the constitutionally-driven division of labour between SIS and MI5) and the 2000 Regulation of Investigatory and that the scale of manpower required for such a processing and interpretation effort is almost inconceivable let alone unaffordable. Consequently when Thomas does provide a potentially interesting account of SIS turning Ali-Reza Asgari, a senior officer in the Iranian intelligence service VEVAK,Footnote ³⁹ it is impossible to know whether to give credence to any of it or not at all – a problem endemic throughout his entire volume.

Thomas has clearly cobbled this book together from odds, ends and sources he has encountered on earlier projects in his career. Obviously weak and sloppy on British source materials he frequently falls back on second and third-hand reporting from US and Israeli sources. The latter in particular often take the form discursive rabbit-holes that are nothing but self-regarding accounts of Israeli intelligence agencies and activities by officials of Thomas’ acquaintance, both tendentious and irrelevant. In short, Thomas’ volume does not warrant serious consideration by anyone wishing to understand the UK's intelligence system.

There can be no doubt, however, that Andrew, Jeffery and Aldrich have substantially reinforced a historically rigorous understanding of the UK's intelligence community. Of the three, Andrew's appears the more finished work. Jeffery's is at best a first volume in a narrative that badly needs continuation while Aldrich's work leaves the question of COMSEC yet to be addressed. That being said, one could find Aldrich's level of achievement a little disturbing because it prompts the vague, nagging worry that an actual official history of GCHQ work will not now be forthcoming because the agency might take the view there will be so little left to add. Hopefully this will not be the case.

These works also serve to highlight our gaps in understanding British intelligence. We still have no comparable account of imagery intelligence after 1945, nor the work of its main provider, the Joint Air Reconnaissance Intelligence Centre.Footnote ⁴⁰ Likewise, Defence Intelligence (formerly the Defence Intelligence Staff) remains little studied despite its significant national intelligence function and its role in significant events such as assessments on Iraq prior to the 2003 invasion.Footnote ⁴¹ A detailed understanding of the inter-agency machinery of intelligence is also an essential counterpoint to our knowledge of the agencies and still limited mainly to contributions by former practitioners.Footnote ⁴² Aldrich, Andrew and Jeffery are very thorough about placing their respective organisations in the appropriate wider governmental context especially that of the Cabinet Office, but necessarily as background. While I have recently provided my own assessment of the evolution of the Cabinet Office machinery of intelligence in a comparative context set against the US system,Footnote ⁴³ scholars are still awaiting the official history of the Joint Intelligence Organisation being prepared by Michael Goodman and it will be interesting to see how much he focuses on JIO analytical efforts as compared with its pre-2010 governance function as the principal co-ordinating body of the UK intelligence community. In the last analysis, the official histories of SIS and MI5 arrived too late for the great age of exploration in intelligence history when they might have made a more significant contribution. At least in the English-speaking world, the field has already been reduced to bounded and identifiable pockets of wilderness surrounded by charted territory in which intelligence and espionage are ultimately just another routine task of government like building roads or collecting taxes.