The work of indexers at the UK Parliament
Parliament's indexing team, mainly based in the Indexing and Data Management section (IDMS) of the House of Commons Library, deals with an extremely diverse range of material, including Parliamentary debates and questions, Early Day Motions and petitions, primary and secondary legislation, European material, Parliamentary papers, Library-produced research and briefing papers, press releases, journal articles and papers deposited with the Library by Members of either house.
The procedural records (the content of Hansard Parliamentary Debates) maintained within the Parliamentary Information Management Services (PIMS) system are notable for their very high degree of granularity; each Parliamentary question, whether oral or written, has its own record, and supplementaries to oral questions are also captured in separate records. Each contribution by a Member, whether a full speech or an interjection (and every ruling by the Speaker), is separately recorded for later retrieval and the extraction of statistics. Each individual contribution is linked to a record for the full debate so that an entire exchange can be viewed in context. The original Bill record is linked to subsequent printings and amendments and the eventual Act, so that progress of legislation through Parliament can be tracked. Where a ministerial correction to a question or statement is produced, this is linked to the original reference.
Until 2007 all indexing was done retrospectively, using an electronic data feed which came through from Hansard, with some records being created manually from the daily Vote pages; this created a large amount of duplication of effort in that Hansard and the Library were using different indexing terms. In early 2007 indexers began to work much more closely with the editors of Hansard, entering terms for written Parliamentary questions directly into Hansard's own reporting system to appear in the daily Written AnswersFootnote 1. This has eliminated what was a considerable backlog of indexing, has substantially increased the currency of material and has led to greater harmonisation of indexing terms between the two organisations. Where differences of practice exist, the transformation of Hansard indexing terms to those used in the Parliamentary databases is handled by a combination of automatic processes in the thesauri and manual quality control by indexers. Questions (generally given no more than four subject terms) are indexed less intensively than debates, Parliamentary papers and statements where an unlimited number of terms can be chosen as appropriate.
The printed index to volumes of Hansard, available in the bound volumes and on the Parliamentary websiteFootnote 2, is also produced by members of the section.
The Parliamentary thesauri
To ensure the consistency of the indexing metadata, the Indexing and Data Management section maintains a series of thesauri and controlled vocabularies for use in indexing material from both Houses of Parliament, the National Assembly for Wales and the Northern Ireland Assembly. The Scottish Parliament has recently begun indexing its Official Report and also takes regular feeds from the UK Parliament subject index thesaurus. The House of Commons Library's catalogue of books and journals, which is maintained on a separate system, also uses the thesauri for indexing. Because potentially any issue can be a matter for Parliamentary discussion or debate, the thesauri are required to cover a very wide range of subjects.
Several thesauri are currently maintained in PIMS:
• The Subject Index Terms (SIT) thesaurus is fully controlled and contains 5,600 terms made up of 3,400 preferred terms and 2,200 non-preferred terms. Because of the nature of the material, a large number of Parliamentary procedural terms are included, both for the UK Parliament at Westminster and for the devolved Parliaments and Assemblies, with scope notes to explain specialist terms. (An example would be the term “Stage 2”, which has the scope note: “Refers to the second stage of detailed consideration in the National Assembly for Wales, by a committee, of the proposed Measure and any amendments tabled – this involves line-by-line examination of the text of the proposed Measure by the committee”)
• The Identifiers (IDS) thesaurus is partially controlled; the controlled part consists of 7,300 terms, made up of 5,500 preferred terms and 1,800 non-preferred terms. Examples of identifiers are the names of specific geographical areas, schemes, prisons, roads, publications etc. The Identifiers metadata field used by indexers also allows the entry of uncontrolled terms; reports of these terms are created regularly, and consideration given as to whether we would want to admit them to the thesauri.
• The Organisations (ORG) thesaurus is partially controlled with the aim of being fully controlled. It consists of approximately 25,000 terms, made up of the names of government departments, public bodies, international bodies, and other organisations which have given evidence to committees or published documents the Library has catalogued.
• Personal names are controlled in a Names thesaurus when there is a potential for confusion in spelling or word order in a name (for example, Aung Sang Suu Kyi).
• The Members' biographical database is structured differently from the other thesauri, but performs a thesaurus-like function in search and retrieval within PIMS. It contains brief personal information and a listing of ministerial posts held where applicable.
The thesauri support the work of the indexing and cataloguing teams and also strengthen searching in the PIMS system to enhance results by making use of the synonyms in the thesauri. A recent example would be the term “Improvised explosive devices” which has synonyms “IEDs” and “Roadside bombs”. Whichever term is used either in the original material or by the searcher, the same results set will be retrieved. Another important use of synonyms is in the area of inquiries and reviews – the current Stern Review is officially known as the Rape Complaints Review, but users will be able to search using the more popular name to retrieve information relating to the review.
Thesaurus development
There is a continuous tension in developing the thesauri, between the need to incorporate new concepts and the need to maintain continuity and consistency. Schemes and organisations come and go, and are frequently renamed before they finally come into being, and we try to keep a balance between providing appropriate terms and making sure that material is grouped similarly across time – so that, for instance, business support in a recession is consistently indexed across more than one recession. We attempt to use current terminology, while being wary of buzzwords which may be incomprehensible to users in a few years' time. This does on occasion mean that re-indexing of records is necessary if we are either over-cautious or over-hasty in adopting new terms.
Indexers discuss terms in an informal weekly meeting where issues and terminology which have come up during the week, or which are likely to become relevant, are discussed and amendments can be made to the thesauri. More formal meetings are held every couple of months, with a wider group of indexers and cataloguers from both Houses; over the last few months we have discussed themed groups of terms, such as all those under “Housing” or “Weapons”, with a view to amending the subject index terms thesaurus where it has proved unfit for purpose or outdated; this is happening alongside a general review and revision of this thesaurus.
Parliament and public information
The focus of the Library's indexing until a few years ago was primarily on serving users with access to the Parliamentary Network (Members, their staff and Parliamentary staff on the Parliamentary Estate or via remote access from constituencies). From 2003 to 2005, an independent commission founded by the Hansard Society and chaired by Lord Puttnam investigated Parliament's presentation of itself to, and communication with, the Public. The report entitled Members Only? Parliament and the Public was published in May 2005 and contained 39 key recommendations covering all aspects of Parliament's connection with the public, including education, visitor services and outreach.
Recommendation 16 stated that “The Parliamentary website should be radically improved. At a minimum, it should be consultative, interactive and easily navigable” and Recommendation 17 adds that “An improved website should engage the widest range of citizens, using well-designed publicity and targeted advertising to help people understand that there is a virtual route through which they have easy access to their Parliament”. It was recognised internally that both the search service and the metadata on the website needed strengthening and that navigability was a problem.
Discussions on a subject-based approach to material on the Parliament website began, and accelerated in January 2008, when the Web Centre and the Library came together with the Education and Outreach teams as part of a new Department of Information Services which was designed to deliver services both to Parliament itself and to the public. The Library had previously maintained a series of A-Z topic pages on the Parliamentary intranet, and it was agreed that these could form a basis for a Topical Issues section of the website containing no more than 300 terms under 20 broad headings. The taxonomy used to maintain these pages had not been updated for some time. The staff involved in constructing the previous version had found the experience rather bruising and recognised that the finished product was far from perfect.
PIMS screenshot.
The revision of the taxonomy took place over 4 months from March to July 2008, during which, as taxonomy editor, I attended meetings with four of the eight research sections in the library, and spoke individually to around 40 researchers and support staff (around 50% of the total number). Discussions did occasionally become lively – the specialist researchers were keen to have as many terms as possible for their areas, while the relatively small number of available terms curtailed this. We also needed a taxonomy which was intuitive enough for the public to use, allowing people to navigate quickly to the information they require, while being very aware of the need to avoid “dumbing down” in dealing with some very specialist material.
One of the key elements of the project was the approval board, which contained senior representatives from research services, information services, the Web Centre and Parliamentary Information and Communications Technology (PICT); the final terms were agreed and endorsed by this board and this lent a great deal of credibility to the project.
Initially, we concentrated on reproducing intranet content including library research briefings, Early Day Motions and reports from the Parliamentary Office of Science and Technology (POST). We tried to anticipate the terms which would be required when we came to add other types of material. The first test for the taxonomy came when Web Centre journalists began to index Parliamentary news pagesFootnote 3 in Summer 2008, using the terms and aided by some feedback from indexing staff. The subject pagesFootnote 4 went live at the beginning of April 2009. The first substantial quantity of new material appeared on the site at the beginning of December 2009 – Committee reports have now been indexed with taxonomy terms and are available on the topic pages. In 2010 it is intended that debates, written statements, Bills and other categories of material will also be available through these pages.
Future developments
The PIMS system, which replaced the previous POLIS system in 2003, is due for replacement by mid-2012 and a series of projects have been undertaken to replace a large, all-purpose system with smaller, more lightweight components which will communicate with each other. In the thesaurus area, this is likely to enable us to achieve some exciting new developments.
We would like to be able to specify more customised relationships – at the moment we support all the standard thesaurus relationships, but would, for instance, as well as the simple “Related term/Related term” relationship, like to create an associative relationship which conveys “is a former name for/supersedes” or “formed by the merger of/merged to form”; and likewise, as well as the “Used for/Use term” relationship be able to create an equivalence relationship “is an acronym for/is the full version of the name of”. Currently, for instance, we have no way of searching across documents produced by the Department of Trade and Industry, the Department for Business, Enterprise and Regulatory Reform or the Department for Business, Innovation and Skills if only one of these organisations is entered in the search box, although users would undoubtedly find this useful if they are unsure what the name of the organisation was at the time a report was produced. If we have different types of associative relationships in the thesaurus, we can use these for configuring the search service to know which are relevant for inclusion in a search.
We would also like to include facets. Parliament has recently procured a search solution which supports faceted navigation, and it is intended that the thesauri should be developed alongside search to assist this. The use of facets would also enable us to create smaller, lightweight vocabularies from our IDs thesaurus, which would be easier to manipulate and analyse (a vocabulary for constituencies, say, or other geographical areas); one disadvantage of having non-specialist thesauri is that they are sometimes difficult to navigate – creating smaller vocabularies of diseases, chemical names or government schemes would enable better maintenance and greater awareness among indexers of which terms are available to them.
