Introduction
Despite the success of search engines, human-created indexing remains important to enhance access to the most relevant information for searchers' needs. This article covers a range of access methods, focussing mainly on the areas in which human input is most significant, including website indexes (back-of-book style A–Z indexes to websites and web documents), metadata creation, social bookmarking, and classification, followed by a brief discussion of automated indexing. Searches for ‘website indexing’ also retrieve information about webwide search engines, but they are not discussed here.
1. A to Z indexes
Website indexing became important in the 1990s, as indexers, librarians and web managers experimented with different approaches for making the information they were providing on the internet more accessible. The tools for creating A–Z indexes have changed over time, from simple HTML coding to HTML Indexer and other specialist software. On the downside, website indexes appear to be an especially vulnerable part of a website, and many quality indexes that have been created over the last decade no longer exist.
A–Z indexes on the web range from simple alphabetical lists, often of names, to complex, carefully constructed book-style indexes that provide access to selected, important information. They often refer to the content of one site only, but may be used to provide access to a number of websites on related topics.
The advantages of using book-style indexes on the web is that they are familiar to users, they are easily browsable, and they provide guidance in the form of subheadings, cross references and so on. The problems with book-style indexes include the cost involved in their creation, and the need for continual updating if the content being referred to is changed. Research has also found that people do not always know what to expect from a website index, and often do not choose to use one (Barnum et al., Reference Barnum2004), so clear naming of the link (eg, ‘A–Z index’) or having the main index links on every page can be useful.
Creators of A–Z style website indexes employ a range of software products to create them. These include HTML IndexerFootnote 1, which uses embedded indexing to create updateable website indexes and Help files; XRefHT32Footnote 2, which is free, open source software that can be linked to a thesaurus and HTML/PrepFootnote 3, which is used to create website indexes from existing indexes. Indexers also use dedicated indexing software packages such as SKY Index, CINDEX and Macrex to create indexes which they can then format as appropriate. Unfortunately there is no generally-available single sourcing indexing program that makes it possible to create the index and output it in a variety of formats.Footnote 4 A–Z website indexes can be implemented with a Content Management System (Leise, Reference Leise2002) but because this is not always straightforward, it may have contributed to the decline in the use of indexes on websites.
The following sections provide examples of A–Z indexes on the web.
Indexes not linked to online material
Indexes may be mounted on the web without providing links to the content being indexed. These indexes are functionally equivalent to print indexes, with the advantage of providing broad availability.
One example is the Dust Diseases Tribunal ‘DDT Subject Matter Index’Footnote 5. The Lawlink: Law Reform Commission New South Wales: Report 83 (1997) Partial Defences to Murder: Provocation and Infanticide indexFootnote 6 also has no direct linking from index entries, but there are links to chapters of the book at the bottom of the page.
The University of Auckland library has created the first index to festschriften on common lawFootnote 7. Festschriften are collections of articles written to honour a person or event. This index is not linked to full text content, but is linked to catalogue records at the University of Auckland library.
Websites that are simple lists
Website indexes that are simple lists include two created by the New South Wales Parliament:
• NSW Legislation WebsiteFootnote 8.
• Parliament of New South Wales: Former Members Index A–ZFootnote 9.
Lawlink has created a number of list-style indexes, including some that provide access to the same information from different viewpoints. For example, you can access enterprise agreements through the name of the agreement (Lawlink 2009a) and also through the names of the participants to the agreements (Lawlink 2009b).
Some simple lists are alphabetised by a word that occurs in the middle of the title, eg, in the Lawlink Equal Time index (Lawlink 2009c). This can be problematic when more than one keyword in the article is relevant, eg, ‘Homosexual discrimination – Discrimination and older gays; surviving aged care’ is indexed at ‘H’ but not at ‘D’ or ‘A’.
Manually created subject indexes to websites
Manually created subject indexes require more attention than simple lists, during both creation and maintenance. The website index of the Australian and New Zealand Society of IndexersFootnote 10 won the ASI Web Indexing SIG award in 2009Footnote 11. It features an alphabet bar on every page for immediate access to the index, and is created using plain HTML.
Special challenges occur with bilingual sites, but can be managed, e.g., the Indexing Society of Canada (ISC) has a bilingual site indexFootnote 12 with a column each for English and French entries.
The Province of British Columbia Hansard provides a subject index, members index, and business index for each session of parliamentFootnote 13. Indexes from previous sessions are available separately.
The Australian Law Librarians' Association (South Australian Division) has collaboratively created a subject index to South Australian LegislationFootnote 14 (originally funded by The Law Foundation of South Australia). Links are provided from the index to the legislation on the SA Attorney-General's websiteFootnote 15. Index terms are based on a thesaurus which was compiled using terms from existing indexes such as the Index to Subject Matter of Victorian Legislation.
The BNA (Bureau of National Affairs) Labor Relations Reporter Index includes the LRR General Index and a Table of Cases for LRR ManualsFootnote 16. It offers progressive display of the index, i.e., when you start examining the index only the main headings are displayed, but as soon as you select one of the main headings, the subheadings are opened up for display. This means you can quickly search for a main heading before browsing in more detail, especially important for legal and similar indexes which use classification and long lists of subheadings.
Manually created subject indexes to books
Online books can also have their indexes transferred online. These are straightforward to maintain as the content does not continually change, although the links have to be created initially. One example is the index to the book Orders of MagnitudeFootnote 17 which has been online for over a decade.
In some cases structure from the manual index does not transfer to the online version, although the direct linking ameliorates problems from this, e.g., the index to Search User Interfaces by Marti HearstFootnote 18. Periodicals can be indexed in a similar way to online books although, if they are ongoing publications, the index will require regular updating. Examples of long-term periodical indexes include Rochester History Footnote 19 and Writer's Block Footnote 20.
As well as standard alphabetical order, people have experimented with the use of maps to lead into content, e.g., the index to the Handbook of Western Australian Aboriginal languages south of the Kimberley Region Footnote 21.
2. Metadata
Metadata refers to data about data. It is a broad term, which in the context of indexing the web usually refers to the provision of keywords which reflect the subject content of web pages and make that content easier for searchers to find. Metadata is usually hidden ‘behind the scenes’ and works to enhance search without users necessarily knowing that it exists, although it can also be displayed to users for browsing. Document metadata is also important to lawyers. It refers to the metadata stored automatically or added by writers to word processed and other documents and is not discussed hereFootnote 22.
Metadata keywords can be easier to create and maintain than a handcrafted A-Z index. This is because each metadata keyword can be selected in isolation from other keywords, and does not need to be changed to fit a specific index. Book-style indexing, on the other hand, is always fluid, and terms may need to be changed to better reflect the nature of a topic in context with the other topics being indexed. This is especially true with subheadings, which are all worded individually.
Metadata keywords are usually taken from a thesaurus (or other controlled vocabulary), so decisions about the terms to be used have already been made, although indexers need the skills to select the most appropriate terms. Cross references are created outside the system. The use of a controlled vocabulary helps to ensure consistent retrieval no matter what words are used in the pages (ie, it improves recall). Metadata also helps ensure that the most relevant pages rank highly (ie, it improves precision). It therefore overcomes some of the problems of free-text search. Creating a thesaurus from scratch is a time-consuming, specialist task. Pre-existing thesauri can be purchased or, in some cases, freely downloaded from the web. There is a comprehensive list at the Taxonomy WarehouseFootnote 23.
Because metadata is hidden, it is not always possible to point to sites that are using metadata, nor to examine what difference the metadata is making. There is, however, some research on the topic. For example, the Attorney General's Department of New South Wales has published a report (Lawzone 2009) on the impact of metadata on resource discovery on the internet.
There are a number of general and specialist metadata standards in use, including the Dublin Core (most widely used) and government standards such as AGLS (Australian Government Locator Service), which is based on Dublin Core. The Attorney General's Department of New South Wales publishes the Justice Sector Metadata Standard, which is based on AGLS, thus promoting compatibility with other services. Some of the issues specific to the creation of metadata for legal materials are the need to include jurisdiction; to make reference to the Act or Regulation covered by the secondary material; and to indicate whether the information is primary (Acts, case law) or secondary (guides to the law). Details of the Justice Sector Metadata Standard, along with an example, are provided on their websiteFootnote 24.
A good example of browsable metadata is Montague Institute Review. The A–Z index to the siteFootnote 25 is dynamically updated from metadata derived from a thesaurus and stored in a database.
The Flare Index to TreatiesFootnote 26 is a searchable database of basic information on over 1,500 of the most significant multilateral treaties from 1856 to the present, with print information and, if available, a link to the online publication. The index is published by the Institute of Advanced Legal Studies, University of London. Technical information on the index database is available in an article by Whittle and Clinch (Reference Whittle and Clinch2009).
The Australian Domestic & Family Violence Clearinghouse Good Practice databaseFootnote 27 provides a central access point for shared information about domestic violence interventions. The database is searchable by keyword, program title, and record ID. In addition, you can browse by sub-categories of Target Group, Service Types and Interventions. When you select an entry for a project it shows the keywords that have been allocated to that program. This helps with search refinement, as the user can then browse the additional keywords in programs that have been retrieved in return for their original search. Hits are sorted by jurisdiction.
3. Classification, categorisation, portals and taxonomies
As well as indexes and metadata which give access to specific topics, many content providers use classification or categorisation to allow users to browse through collections of related information.
In the early days of the internet a number of websites attempted to provide classified access using library classifications such as the Dewey Decimal Classification. Most of these projects have not continued, but BUBL LINKFootnote 28 still provides access to topics by classification number (as well as alphabetically). Information on Law is available from 340 to 349.
Portals/directories appear to be relatively common in the legal field. Some examples are:
• InfoLaw (UK)Footnote 29
• Lawlinks (UK)Footnote 30
• Legal Resources in the UK and Ireland maintained by Delia VenablesFootnote 31
• Find Legal Answers (New South Wales)Footnote 32
• FreeAdvice (US)Footnote 33.
The Public Library of Law (PLoL) is the largest free law library in the world, and allows searching for information that is scattered across many different websitesFootnote 34. There are also many national websites that provide access to legal materials from around the worldFootnote 35 and from specific countriesFootnote 36. Davis (Reference Davis2001) has written on the processes involved in the development of AustLII.
4. Social bookmarking/Folksonomies
Social bookmarking (tagging) brings the power of the people to indexing the web, as people tag sites of interest to them, which may help other users find those sites. The lists of tags are somewhat like an index, but lack features such as cross-references and subheadings, and often lack consistency. I tag website indexes I find with the tags website_indexes, website_indexes_names and so on (see del.icio.us.com/GlendaIndexer/website_indexes).
Other social bookmarking sites include TechnoratiFootnote 37 (for blogs) and CiteulikeFootnote 38 (for articles). When searching for content relevant to indexing, I have had occasional serendipitous finds, but have not yet found them a good way to do a thorough search.
5. Automated indexing
Automated indexing has been used alongside and instead of manual indexing for a number of years. It involves the analysis of text using computer algorithms. Analysis of terms in their context enables the identification of concepts as well as simple keywords. Losses of work for indexers in some areas have been replaced by newer roles in other areas, so it is difficult to predict the balance for the future, although high labour costs and increasing technological sophistication suggest that the role of manual indexing may decrease.
Automated indexing using software such as packages from Autonomy is used by many legal companies for the management of unstructured data (Autonomy 2009), including automated taxonomy generation. Recommind is another program used in law firms for enterprise search to index email, people finding and so on, as well as straight document retrieval (Alber Reference Alber2007).
Automated book indexing
While automated indexing has been used with large-scale collections such as intranets, it has generally proven ineffective for standalone products such as single books, where much smaller ‘content chunks’ are being indexed. Kluwer Belgium has taken a new approach to this problem, dividing their content into small, reusable ‘content units’ and classifying content at acquisition (i.e., when it is created as a ‘chunk’ of content) rather than when all of the content has been gathered into one product. Indexing keywords are derived from a multilingual French and Flemish thesaurus covering terms for laws, regulations, jurisprudence, and commentFootnote 39.
Conclusion
The Declaration on Free Access to Law made by legal information institutes of the world (http://www.worldlii.org/worldlii/declaration) affirms the importance of access to legal information for the promotion of justice and the rule of law. The range of legal indexes on the web enhances access to this material.
