INTRODUCTION
The transition of UK Parliamentary, departmental and agency publishing from print to electronic delivery is now virtually complete. The move from print to electronic delivery of information began to gather momentum in the mid-1990s when the government announced its commitment to make information of all kinds available electronically. This strategy covered the whole range of government information, which was to be organised systematically so that publicly available data would be readily accessible electronically in forms that would assist in the improvement of UK competitiveness and promote open governmentFootnote 1.
There followed a proliferation of government websites and a trend towards publishing information in e-format only. As in other sectors, ICT fuelled an information and publishing explosion among government and other official bodies. The Standing Committee on Official Publications (SCOOP) estimated in 2008 that between 400 and 500 new official publications were being published through websites every weekFootnote 2. The consultancy Education for Change (2) calculated that, apart from ministerial departments, about 100,000 e-publications could be available from government advisory and tribunal bodies. A 2007 National Audit Office reportFootnote 3 found that the average government website had 17,000 pages and held “a mixture of information ranging from understandably sizeable documents (often stored in PDF form) reflecting government's wider responsibility for the stewardship and custody of public records through to information on citizen facing services”.
The presence of government publications on the web at that time was transitory due to:
• Websites disappearing following changes to the machinery of government. Many official bodies, particularly advisory bodies, are brought into being for a limited period of time, and many others, including central government ministries, are disbanded or merged due to political change
• Website redesign and upgrade, when non-current publications were removed
• A focus among government webmasters on provision of accurate up-to-date information
Documents removed from the government web estate were lost as there were no comprehensive or effective digital archiving processes in place. Consequently a problem emerged of broken links within official websites. Research by The National Archives and the House of Commons LibraryFootnote 4, Footnote 5 showed that:
• Departments are increasingly citing URLs in answers to Parliamentary Questions, which formed part of the Official Report (including links to non-governmental websites), but 60 percent of links in Hansard to UK government websites, for the period 1997 to 2006, were broken.
• Departments varied considerably. For one, every link worked; for another, every link was broken. This demonstrated that it was possible to maintain links where there was an appreciation of the issues and a will to do so.
The proliferation of large and complex government websites led to difficulties for researchers in finding documents and information (2) due to broken links, site restructuring, older material being removed, poor search engine functionality and design faults.
Then in 2010 a review by UK Digital Champion Martha Lane FoxFootnote 6 led to a decision to reinvent government online publishing. The review pointed out that the fragmentation of the government web presence into hundreds of different sites led to expensive duplication of functions and technology and a highly inconsistent user experience. It argued that it made sense from the user point of view for all central government digital services to reside on a single site. The user should not be expected to navigate the department structure of government to find the service or information they required. The recommendations of the review were accepted by Cabinet Office minister Francis Maude and led to the creation of GOV.UK. This single point of access to all UK Government information and services was launched in beta on 31st January 2012 and at the time of writing hosted the websites of all 24 ministerial departments and 331 other agencies and public bodies.
The problem of broken links due to removal of non-current government information from government websites was tackled by The National Archives (TNA) through a programme to deliver comprehensive archiving of the government web estate. All government websites are crawled and archived at least three to four times a year. Successive gathers are freely available through the UK Government Web Archive at http://www.nationalarchives.gov.uk/webarchive/
UK PARLIAMENT
The transition of UK Parliamentary publishing from print to electronic has proceeded in stages under the Print to Web Publishing programme. Parliament started reducing printing of its publications in May 2010 and the process will be complete by March 2016. Responsibility for publication of Parliamentary papers is divided between the Commons and Lords and The National Archives. TNA, on behalf of government, contracts for the official publication of Command and government House of Commons papers (C&HC papers). To September 2013, the contract was let to The Stationery Office (TSO) and provided for the production of print copies and statutory legal deposit with the British Library and the five other copyright libraries. In addition, government organisations published C&HC papers on their own websites or on GOV.UK. From October 1st 2013 a new model was introduced. The new model was predicated on the C&HC papers' official version being published on GOV.UK, with the government retaining the official publisher role. In addition to the web PDF, the papers' print PDFs was also to be available to download. From October 2013 there was no official print publisher. Anyone, including commercial suppliers, was at liberty to utilise the print PDF and sell print copies. However, for an interim period TNA agreed to supply the six legal deposit libraries with a print set.
The main series of papers published within the control of the Commons and Lords – Select Committee reports and daily Hansard – will also cease to be produced in print from April 2016. Hansard written answers and Commons and Lords general and public bill committee debates have already ceased in print. In fact written questions and answers from June 2014 are only available via a searchable database supplemented by a daily PDF at:
ELECTRONIC LEGAL DEPOSIT
This wholesale transition from print to electronic publication of government information presented the British Library and colleagues in the other five legal deposit libraries with a challenge and an opportunity. The challenge was to develop and implement systems for the collection, storage, discovery of, and access to born digital government and Parliamentary publications. The opportunity was to improve on the traditional systems in place for managing this material and making it available to users. Historically, many UK Parliamentary and government departmental and agency publications were issued in numbered series, e.g. House of Commons papers, Command papers, DWP research reports, Home Office circulars. Because of resource constraints, the British Library recorded such material in its public catalogue at the series level. To take an extreme example, the main reference to the bound set of House of Commons papers and bills in the old British Museum printed catalogue stated that “The blue books are in the State Paper Room”. Users needed to use a range of published and in-house guides and indexes to identify the series in which a document was published and its number within the series. They could then consult the catalogue to find the location of the series and order the document. It is estimated that over 90% of official publications were not catalogued at the level of the individual document. This means that the material is under-used because researchers simply cannot find it and depart empty-handed because they assume the documents they are interested in are not held. Transition to collection of materials in online formats offers a golden opportunity to develop systems that will facilitate creation of metadata with links to full text at the document level and so transform discovery and access at the British Library. The legal deposit libraries have also developed a common digital infrastructure that enables the work of collection, metadata creation and ingest to be shared, cutting out duplication of effort. For example, Scottish parliamentary materials which are only available online are supplied to the National Library of Scotland which will then stream them into the shared digital infrastructure. The documents will be held in the shared digital store and metadata and links exported to the catalogues of the other five legal deposit libraries.
The Legal Deposit Libraries (Non-Print Works) Regulations 2013 (SI 2013/777) made under the Legal Deposit Libraries Act 2003 provides the legal basis on which the six copyright libraries are entitled to delivery of copies of works published online in the UK. There are two options for delivery:
• Documents with metadata may be pushed to the libraries by publishers
• Documents may be pulled from websites in the UK web domain by crawlers
In practice, a mix of pull and push will be used for official material. The downside of the 2013 regulations is that the documents can only be used by one researcher at a time on the premises of the legal deposit libraries. A second, non-legal deposit has to be kept for off-site use. This mirrors the situation with print, where obviously only one person at a time can read a document.
In order to exploit the opportunities offered by this brave new world, the British Library has developed a range of tools for digital collection management on various levels.
THE SOCIAL WELFARE AND MANAGEMENT AND BUSINESS STUDIES PORTALS
The portals are channels which the Library uses to provide access to digital reports, including official publications issued under the Open Government Licence (OGL)/Crown Copyright, to off-site users. All material on the portals is rights-cleared and available for download and re-use for non-commercial purposes. Official documents are gathered and made available under the OGL/crown copyright and are not e-legal deposit copies. If a government body retains copyright of its own publications, formal written permission for their download and re-use on the portals is sought. Government websites in the fields are monitored manually and documents in scope for either of the two portals are selected and downloaded. Copies are held in an access store on the content management system (CMS) and metadata including a detailed abstract is created in the CMS by hand and loaded onto the portal websites. Portal users can find reports by browsing under broad subject headings or publisher names. The metadata is exported to the British Library's main catalogue. Users who find relevant portal material through a catalogue search can click on a link to access the full text of the document. This approach is low-tech as it relies on manual site monitoring and metadata creation. It could not be scaled up to cover the whole universe of government publishing.
THE DIGITAL DOCUMENT HARVESTING AND PROCESSING TOOL (DDHAPT)
DDHAPTFootnote 7 is a pull mechanism which will enable a selector at any of the six legal deposit libraries to set up a list of unique URLs called watched targets to be crawled at set intervals. Frequency of crawl will depend on the volume of publications the organisation produces. After each crawl, the system will present a list of links to documents added to the site since the last visit for selection for full cataloguing and ingest into the digital store for long term preservation. Using guidelines agreed by all six legal deposit libraries, the selector will then create basic metadata for documents in scope. The system presents a copy of the document and a metadata creation form to the selector side by side. Metadata creation is partially automated through use of the MEX (Metadata Extraction) tool developed at the British Library. This enables the selector to highlight data elements such as ISBNs, author names, title, publisher, and publication date in the harvested documents. On mouse up, a menu appears listing ISBN, publisher, corporate author, etc. Clicking on the relevant data element in the list transfers it to the correct box on the form, avoiding transcription errors. Out of scope material such as internal minutes, public information leaflets, summaries, forms and letters will be ignored. The documents will be ingested into the shared digital store and the basic record with hotlink to the full text will be available in the catalogues of all six legal deposit libraries in seven days. We estimate it will take about five minutes to review and create basic metadata for a document. In theory, the basic record will be upgraded in due course by the Digital Cataloguers, but given the volume of material involved this may not be feasible. We will therefore ensure that the basic record will contain enough detail to enable retrieval. End-to-end testing of the system is scheduled for early 2016, and a production version will be rolled out in the spring. The work of monitoring UK central government departments and agencies will be shared among the legal deposit libraries.
It is envisaged that the DDHAPT will be used to crawl and retrieve publications of UK central government departments and agencies published on gov.uk. Discussions with TNA have confirmed that their publishing activities are distributed among departmental sections, units, and working groups and are not centrally co-ordinated. Therefore there is no reliable mechanism under which documents could be pushed to us with accompanying metadata.
THE LEGAL DEPOSIT WEB ARCHIVE (LDWA)
A broad crawl of the UK Web DomainFootnote 8, Footnote 9 is performed annually by the British Library. The scope of the crawl includes all websites that have a UK top-level domain suffix (currently .uk, .scot, .london, .cymru and .wales); websites identified as being in scope by automated hosting checks and websites that have been identified as being in scope by additional manual checks, such as evidence of a UK postal address. In addition, several hundred national and regional online news publications are crawled on a daily or weekly basis. Finally, a list of ‘key sites’ defined and moderated by the Legal Deposit Web Archiving Sub Group are crawled on a regular basis. The domain crawl picks up UK national and local government websites, but in many cases these do not benefit from any form of quality assurance check. However, www.gov.uk is a key site and subject to curatorial quality assurance. Crawl parameters have been adjusted to ensure that most documents are gathered and the site renders correctly. The legal deposit libraries also select a number of themes or events to form the basis of ‘special collections’ each year. Each special collection includes a few hundred websites selected by appropriate curators from the legal deposit libraries. These themed collections include in scope official materials at the national and local level.
The UK web presence comprises more than 4 million websites and continues to grow. The first domain crawl conducted in 2013 contains 1.6 billion URLs from 3.8 million domains, or 31TB compressed data. The second domain crawl, which ran June - December 2014, yielded 52TB of compressed data from 10.3 million domains.
The Annotation Curation Tool (W3ACT) was developed to meet the requirements of subject specialists wishing to curate web content harvested under the e-legal deposit regulations. W3ACT enables users to perform numerous curatorial tasks including the assignation of metadata and crawl schedules to web content, quality assurance and the ability to request permission for open access to selected websites. For obvious reasons, the majority of the websites gathered in the course of the annual domain crawl are not designated key sites and therefore are not actively curated or subject to quality assurance checks.
The gathered websites are held in the Legal Deposit Web Archive which can be accessed on-site at each of the six legal deposit libraries and which includes the themed collections such as Health and Social Care Act 2012 – NHS Reform. In essence the LDWA is a vast unstructured dataset which lends itself to analysis by scholars researching the growth, development and structure of the UK web domain using sophisticated tools such as visualisations and data mining. A recent example of the analysis tools being developed to help researchers exploit web archives is the prototype historical search engine SHINE, which, among other things, can analyse the number of pages a word or phrase appears in a dataset over a given period. However, the LDWA is not well suited for retrieval of individual documents given the search tools currently available.
We are developing the idea of creating quality assured collections within the LDWA to improve access to some categories of government material where we lack resources to create catalogue records at the document level. Instead, we will create records at the collection level and link back from the catalogues of the British Library and the other five legal deposit libraries to the collections in the LDWA. We call this the landing page approach.
THE LANDING PAGE APPROACH
Historically, the British Library never systematically collected local government publications. Any it was sent under legal deposit were accepted, but the Library never actively sought out or claimed local government documents. In order to improve coverage and access, we plan to set up crawls within W3ACT for all English county, borough and district council websites quarterly and in-depth. The gathered sites will be checked carefully using W3ACT functionality to ensure that all documents have been captured and that the site is rendering correctly. The successive quality assured gathers will preserve local authority publications in context and will enable researchers to trace policy evolution over time. We will form successive gathers into structured collections within the LDWA. For example, the collection County Councils will include sub-collections for each English county. Descriptive metadata will be created within W3ACT for each collection and sub-collection and exported to our Library Management System (Aleph) in which master catalogue records are held. Automatic enhancements would be applied at the point of export so that the records met current descriptive cataloguing standards without manual intervention. The enhanced records would then be exported to the British Library's public catalogue and those of the other five legal deposit libraries, possibly using the Z39.50 protocol. A link back from the catalogues to the collections and sub-collections within the LDWA would enable researchers to view and explore successive instances of the sites.
A similar approach can be used for documents in series such as Home Office circulars. Circulars published in a single year share a landing page on gov.uk. For example the 2015 circulars are listed at https://www.gov.uk/government/collections/home-office-circulars-2015. In this case, a historic Home Office circulars collection would be made up of gathers of the completed list for each year through a one-off crawl at the end of December.
To date, the W3ACT system has never interacted with Aleph and development work is needed to link them to enable the transfer and enhancement of the collection level records and the creation of a permanent identifier (URL) for each collection to ensure that the links do not break.
‘PUSH’ MECHANISMS
In the devolved administrations, libraries such as the National Library of Scotland, the National Library of Wales, and Queens University Belfast Library have close collaborative relationships with their legislatures, departments and agencies and are able reach agreements for documents and metadata to be delivered to them by electronic transfer. Queens University Belfast Library acts as the deposit library for Northern Ireland official publications and aims to collect all departmental, agency and Assembly publications.
The Scottish government ceased producing print publications far in advance of the e-legal deposit regulations coming into force in April 2013 and the National Library of Scotland (NLS) reached an agreement, enshrined in a memorandum of understanding, to archive this material as voluntary deposit. This established arrangement allowed a smooth transition when the Scottish Parliament ceased producing any documentation in print in July 2015. The NLS has always worked closely with the Parliament and acts as the official archive for the long term preservation of its electronic publications. Both monographic and serial publications are sent to the NLS by file transfer protocol (FTP). With formal permission from the Scottish Parliament, the NLS creates catalogue records for non-legal deposit duplicates and publishes these in their online catalogue for off-site use, with a hot link to the full text.
Despite protests from the library community, the Scottish Parliament has dropped the use of ISBNs on its one-off publications as they were not needed internally. This presents both the British Library and the National Library of Scotland with a challenge because the ISBN acted as a unique identifier. To compensate for the absence of ISBNs, the NLS has put in place a system for creating consistent file names. However for the British Library to ingest one-off documents without ISBNs and their accompanying metadata into the shared digital infrastructure requires system development to the e-books loader, which was designed for commercially published books which always carry ISBNs. This should be available by March 2016; in the meantime the documents are being held for safekeeping at the NLS, with public access offered through duplicate copies as described above.
The National Library of Wales (NLW) is in discussion with Welsh Government regarding the transfer of their publications via FTP. The main issue is related to linking the metadata and the document at the point of transfer from their library management system (Soutron). In the meantime the NLW is currently manually selecting and downloading material published under the Open Government Licence from other Welsh public body websites such as the Healthcare Inspectorate Wales, creating metadata and making records with hotlinks to the documents available to all via their online catalogue. Uploading the legal deposit copies to the shared digital infrastructure awaits the delivery of the system upgrade that will enable processing of material without ISBNs, i.e. March 2016. The NLW hope to use the Document Harvester to collect legal deposit material from some agency websites.
Queens University Belfast (QUB) Library is creating a DSpace repository to hold its collection of Northern Ireland Assembly, departmental and agency born-electronic publications. The Assembly and some agencies are co-operating by pushing documents to QUB using Dropbox, but departments cannot be relied on to deposit consistently. QUB Library therefore monitors department and agency websites manually and creates lists of links to newly published documents. From September 2015 QUB is downloading the documents to a ‘holding’ file from where they are catalogued and uploaded to DSpace. NI Assembly papers are taking priority, followed by departments and agencies.
QUB Library has asked departments and agencies for formal permission to make material in the repository available remotely. They have either agreed or not replied. QUB has decided to make all material in the repository freely available, taking silence as consent.
In order to save duplication of effort, QUB Library is working with the British Library to develop a workflow to stream documents and their metadata from DSpace into the shared digital infrastructure. These copies would be considered to be the legal deposit copies collected on our behalf by QUB Library acting as our agent. QUB Library is able to export documents and their metadata from DSpace regularly and transfer them to us by FTP. They are also willing to write a transform to convert the Dublin Core metadata produced for DSpace into the standard MARC coding used by the legal deposit libraries. As Northern Ireland government and Assembly documents do not have ISBNs, implementation of this promising plan depends once again on the adaptations to the eBooks loader scheduled for March 2016.
The final ‘push’ mechanism used by some government agencies and the Northern Ireland Assembly is the Publishers' Submission Portal (PSP). This was designed to enable small publishers to deposit both books and serial issues. As regards the books, it was again designed for material carrying an ISBN as a unique identifier. Once a publisher has a login, they can upload documents and are required to complete a basic metadata creation form recording the work's title, author(s), publication year, publisher and, of course, ISBN. For books with ISBNs, the document is ingested into the Digital Library System and the associated metadata into Aleph, the British Library's library management system. From thence it is exported to the catalogues of all six legal deposit libraries, allowing on-site access to the full text after seven days. Unfortunately most government one-off documents do not carry ISBNs and so cannot be processed through the PSP at present. We are once again awaiting the adaptions to the ingest systems due in March 2016 that will resolve this issue. However a workflow is in place currently for uploading serial issues.
CHALLENGES: ISSUE LEVEL SERIALS
A workflow is in place within the British Library to process serial issues deposited via the PSP or sent by FTP. The issues themselves are ingested into the Digital Library System and records for each item are created and are subsequently exported to the catalogues of the legal deposit libraries for onsite access. The solution in place for catalogue access to these ‘issue level serials’ is unsuitable for prolific publications such as the daily UK Hansard, the London Gazette or Statutory Instruments of which 2,000 to 3,000 are published every year. The solution involves a separate record being created in the catalogues for each issue:
Title: London market today
Author:Bannister International Research Organisation
In: London market today; volume 0; Issue 629; 2011–02;
Journal Title: London market today
Publication Details: Bannister International Research Organisation
Over time, browsing to find an individual issue would become impossible. Unfortunately, the only solution we have for prolific official serials at present is to direct users to the live, or to an archived version, of the website.
CHALLENGES: DATABASES
Some official series such as the UK Hansard Written Questions and Answers and the Gazette (formerly London, Belfast and Edinburgh gazettes) have turned into searchable databases on the Internet. For the Gazette at https://www.thegazette.co.uk/, you can search all notices together, or those relating to insolvency, wills and probate, and awards and accreditation separately. Only notices for the past 15 days are available to browse. Web archiving technology currently available cannot harvest and render dynamic databases of this kind, and our only solution at present for online access is to point users to the live site.
CHALLENGES: THE UK PARLIAMENT WEBSITE
The UK Parliament currently owns three associated websites, http://www.parliament.uk, http://services.parliament.uk/ and http://www.publications.parliament.uk/. Publications are scattered over the sites, often behind filters – you have to filter by date or session to find them. Retrieving publications from such complex sites using the Document Harvester would be challenging, and would probably involve setting up multiple targets for different areas – select committees, daily Hansard, research briefings, etc. The Library is taking an alternative approach and is in negotiation with a commercial partner who might be willing to act as an agent for collection and supply of UK Parliament documents. They monitor and download publications from the UK Parliament site and create metadata for their own products. They have expressed willingness in principle to transfer copies of the documents and metadata to the British Library for ingest into the shared digital infrastructure. This approach would save the legal deposit libraries significant staff time and would ensure the preservation of a complete set of documentation.
CONCLUSION
UK official publishing has now effectively transitioned from print to electronic delivery via the free Internet. This article has sought to present an overview of how the legal deposit libraries are working together to develop a set of tools that all can use to identify, store and provide access to the whole universe of government publications. However, government publishing is not static and is moving away from the currently predominant PDF format to use of open document formats and sets of linked html pages. The very concept of the discrete document is fading as multimedia approaches linking text, sound and vision emerge. For example, the Office for National Statistics will no longer publish PDF versions of key series such as the UK National Accounts (the Blue Book) in readiness for the launch of their new website which, once live, will not support these. Instead they are focusing on developing their Excel based tables for presentation of their data. The tools we develop to assure preservation and access will also have to evolve to meet the new challenges as they emerge.
