The development of the Cochrane Collaboration has been described elsewhere (Reference Allen and Clarke1;Reference Allen, Clarke and Tharyan2;Reference Bero and Rennie6;Reference Chalmers11–Reference Chalmers and Haynes13;Reference Chalmers, Sackett, Silagy, Maynard and Chalmers17;Reference Clarke and Langhorne22;Reference Dickersin and Manheimer27). In this article, we describe the origins and evolution over the past 20 years of the Collaboration's principal product, that is, The Cochrane Database of Systematic Reviews (CDSR), and some of the features that set it apart from more traditional publications.
In 1972, Archie Cochrane's book Effectiveness and Efficiency drew attention to the need to obtain better evidence to inform the development of health services, and emphasized the important role of evidence derived from randomized controlled trials (Reference Cochrane23). One consequence of this was that, encouraged by Cochrane himself, one of his readers (I. C.) began to assemble a collection of reports of randomized trials done to assess the effects of care during pregnancy, childbirth, and early infancy (the perinatal period).
In a publication a few years later, Cochrane rated obstetrics and gynecology the medical specialty most guilty of ignoring the need to base clinical practice on reliable research evidence and issued a call to all specialties to assemble “a critical summary, adapted periodically, of all . . . relevant randomized controlled trials” (Reference Cochrane24). In response to Cochrane's challenges, staff at the recently established National Perinatal Epidemiology Unit in Oxford (Reference Chalmers10) (i) developed a computerized register of controlled trials from the collection of reports of trials in perinatal care mentioned above (Reference Chalmers8;Reference Grant and Chalmers34;Reference Mugford, Grant, Chalmers, Lindberg and Reichertz43;44), (ii) demonstrated how data from similar studies could be combined to create overall estimates of effects (Reference Chalmers, Thalhammer, Baumgarten and Pollak7;Reference Saunders, Dick, Brown, McPherson and Chalmers50), and (iii) began to establish the foundations for an international collaboration to prepare “critical summaries” of the studies included in the register (Reference Enkin and Chalmers29).
By the mid-1980 s, following the publication of other documents reinforcing Cochrane's messages, such as the U. S. Office of Technology Assessment's report on technology assessment in health care (45), the stage was set for the National Perinatal Epidemiology Unit in Oxford to coordinate the creation of a body of systematic reviews of interventions in pregnancy and childbirth. More than a hundred contributors prepared a two-volume book—Effective Care in Pregnancy and Childbirth (Reference Chalmers, Enkin and Keirse15)—and a companion volume—Effective Care of the Newborn Infant (Reference Sinclair and Bracken54).
THE OXFORD DATABASE OF PERINATAL TRIALS
The contributors to both these volumes used explicit methods to prepare systematic reviews of controlled trials, using statistical synthesis (meta-analysis) when appropriate and possible. This work drew heavily on the perinatal trials register. This had been further developed to create the Oxford Database of Perinatal Trials—known to many affectionately as “Odd Pot” (ODPT). ODPT went beyond storing bibliographic information on the clinical trials that formed the material for systematic reviews. In an innovative step, it was expanded to include raw data from the clinical trials and meta-analyses, summarizing the results across trials. The tables and graphs for both books were generated from the database. A software program was written to step through the database and produce the graphs and figures for delivery to the publisher of the books.
The data published in the books represented a “time-slice” or “snap-shot” of the state of research evidence at the time of submission for publication, so that the analyses initially published in the books could be kept up to date as new evidence emerged, the Oxford Database of Perinatal Trials was conceptualized as a publication in its own right (Reference Chalmers9). Published twice yearly, first on 5¼ inch and then on 3½ inch diskettes, ODPT was distributed on a subscription basis and provided the opportunity to circulate newsletters highlighting new or substantially revised reviews (see Spring Newsletter 1992; Figure 1).
First page of the Spring 1992 Newsletter of Oxford Database of Perinatal Trials.
ODPT was developed using a popular microcomputer database program. The reviews were not held as documents per se. Instead, a relational database was used, and the data were stored in a highly structured form. All references, for example, were held in one file, while all authors' names and affiliations were held in another. Similarly, the titles of all comparisons in the analyses were held together, as were the data extracted from each of the included studies. As a consequence, the systematic reviews themselves were highly structured, and were all in the same format.
The systematic reviews in ODPT were known as “overviews” to distinguish them from records of the ongoing, planned, unpublished and published trials which were also included in the database. As the term suggests, overviews were summaries of groups of studies evaluating similar interventions. Each overview had a designated “editor”, who provided a structured textual summary termed the “Editorial Commentary”.
The overviews in ODPT were data-orientated. Data on different outcomes following the interventions studied were considered “Parts” of the overview, and within each Part, where appropriate and possible, the software would display the data from each included study, and produce a statistical summary, or meta-analysis. The analyses were based on the notion of calculating odds ratios from the outcome data of the individual trials, and then pooling the odds ratios to get an overall estimate of effect (Reference Saunders, Dick, Brown, McPherson and Chalmers50).
This highly structured approach to presentation gave the reader an important advantage: once one understood how to interpret one graph, one could apply that understanding to all overviews in the database. This removed one of the barriers between research and practice, namely the effort required to figure out how the researchers had analyzed the data and had reached their conclusions. The process was transparent and consistent for the reviews presented in ODPT.
Unlike more traditional printed articles, the ODPT database system allowed the raw data used in the meta-analyses to be stored with the article, which in turn meant that statistics could be calculated and figures drawn in “real time,” that is, when the figure was displayed. As new studies were added to the database, and new data became available, they were incorporated in the analyses. These analyses were presented as “forest plots” (Reference Lewis and Clarke39), showing the result and the contribution of each study, as well as the pooled estimate.
One interesting feature when displaying meta-analyses in ODPT was that one could watch the pooled effect estimate and confidence interval change as each trial was added into the analysis, the ordering being based either on the assessed quality of each study, or on its statistical power, or on its year of publication.
THE COCHRANE COLLABORATION PREGNANCY AND CHILDBIRTH DATABASE
By 1992, many policy makers, practitioners, and consumers had come to recognize the importance of systematic reviews for making decisions about health care. Effective Care in Pregnancy and Childbirth (Reference Chalmers, Enkin and Keirse15) and Effective Care of the Newborn Infant (Reference Sinclair and Bracken54) were acknowledged as important achievements (41), and A Guide to Effective Care in Pregnancy and Childbirth (Reference Enkin, Keirse and Chalmers30), a paperback summary of the evidence in Effective Care in Pregnancy and Childbirth, had proved popular with professionals and nonprofessionals alike.
The Oxford Database of Perinatal Trials was being maintained at the National Perinatal Epidemiology Unit, and an international team of contributors was updating the systematic reviews. As the editors of ODPT recounted at the time: “If we had the opportunity to begin again from the beginning, we might well have decided that, before any attempt was made to prepare either of the books, all of the systematic reviews on which they were based should have been completed, with structured reports prepared, and held in electronic form.” (Reference Chalmers, Enkin and Keirse16).
Although the books had proved popular, the publisher—Oxford University Press—found the electronic publication costly to maintain, and decided to discontinue ODPT as a commercial product in 1992. It was still early days in the history of electronic publishing (Tables 1, 2, and 3). The World Wide Web had yet to gain popularity, and attempts to interest other publishers in taking over the publication of ODPT were not successful. The most likely successor to ODPT at the time was an ambitious project known as “Medical Knowledge,” which was to be a publication under the auspices of the then recently formed Maxwell Electronic Publishing. This project ended following Robert Maxwell's death in 1991 (http://news.bbc.co.uk/1/hi/business/1249739.stm).
Pre-history of Electronic Publication of Systematic Reviews
Pilot experience in Perinatal Care: I. The Oxford Database of Perinatal Trials and The Cochrane Pregnancy and Childbirth Database
The Cochrane Database of Systematic Reviews
In the United Kingdom, the example of the work in the perinatal field led to a recognition that support for the preparation of systematic reviews was a legitimate use of research and development money (Reference Peckham48). This led to the recently inaugurated National Health Service Research and Development Program to fund some members of the perinatal team to establish “a Cochrane Centre,” to promulgate across all of health care the methods they had used, as envisaged by Archie Cochrane in 1979.
It had been hoped that central funding would also be obtained to support the dissemination of The Oxford Database of Perinatal Trials. This was not to be, however, partly because plans were being made to create a National Health Service Centre for Reviews and Dissemination at the University of York (Reference Sheldon and Chalmers53), so commercial models were again pursued. In the early 1990s, personal computer technology was changing quickly, and access to desktop computers was becoming commonplace. In 1992, to make the information more accessible to the average computer user, Update Software Ltd. worked with the editorial team overseeing ODPT and staff at the Cochrane Centre to bring the systematic reviews to the forefront in a new product. In April 1993, Update Software reissued the systematic reviews contained in The Oxford Database of Perinatal Trials as an electronic publication entitled The Cochrane Pregnancy and Childbirth Database (CCPC). The launch of CCPC thus coincided both with the opening of the UK Cochrane Centre in October 1992, and with planning for the inauguration of the international Cochrane Collaboration in October 1993.
CCPC was designed in part as a pilot to show how Cochrane reviews in all areas of health care could be published electronically. The original title of the publication was the Cochrane Collaboration Pregnancy and Childbirth database, hence the double “C”s in CCPC. The reviews in CCPC were organized under topics following the natural progression of pregnancy and childbirth, from “Care during pregnancy” to “Unhappiness after childbirth.” Each review included a summary displaying the pooled effect measure for each outcome included in the review, displayed on a single screen, generated directly from the stored data, which summarized the review (Table 4). The software allowed the user to start at the beginning of the database, and, using the PageDown key on the computer keyboard, display each summary graph in turn until the end of the database was reached. By 1994, Issue 2, there were 615 reviews included in CCPC (Reference Enkin, Keirse, Renfrew and Neilson32).
Effect of prophylactic surfactant vs surfactant treatment of established RDS
Summary of Data on All Outcomes from Trials Comparing Prophylactic versus “Rescue” Surfactant in Neonates, Oxford Database of Perinatal Trials, Spring 1992.
The highly structured nature of the reviews in CCPC allowed for very specific search and retrieval facilities. It was possible, for example, to search on the entry characteristics for the studies reviewed (for example, women who had had Caesarean sections), the intervention reviewed (for example, Caesarean section), and/or the outcome measured (for example, the number of women having Caesarean sections after induction of labor).
When the second edition of A Guide to Effective Care in Pregnancy and Childbirth (GECPC) (Reference Enkin, Keirse, Renfrew and Neilson31) was commissioned, the editors were able to start at the top of the topic list and step through the database summarizing the evidence available under each topic. One immediate benefit of the relationship between the database and the book was that it allowed the individual reviews to be linked to text from the relevant chapters. That is, while reading a review on screen, a single keystroke allowed the reader to display the relevant chapter of the book. For example, the review “Upright versus recumbent position during the second stage of labour” was linked to the GECPC Chapter 2, “The Second Stage of Labour”, which includes a discussion of position during labor.
CCPC proved very popular and sold over 1,000 copies in the first year, which was remarkable given that it was still early days in electronic publishing. It was especially popular among midwives and others who did not have ready access to the research literature. Including the text from GECPC in the electronic publication was a significant contribution, partly because it provided those who were unfamiliar with the research literature with a context in which it could be interpreted.
THE COCHRANE DATABASE OF SYSTEMATIC REVIEWS (CDSR), 1995-PRESENT
Background
The Cochrane Collaboration was inaugurated in October 1993, and the first Cochrane Review Groups were established to oversee the preparation and maintenance of systematic reviews in a variety of areas of health care. The first of these was the Cochrane Pregnancy and Childbirth Group, which had editorial responsibility for the reviews in CCPC. However, it continued to be difficult to interest publishers in an exclusively electronic publication. The most promising discussions were with the American Association for the Advancement of Science (AAAS). AAAS had just launched an electronic journal named the Online Journal of Current Clinical Trials (hosted by OCLC—Online Computer Library Center, Inc; see http://www.aegis.com/pubs/atn/1992/ATN15403.html), and the plan was to create a separate section for Cochrane reviews in the new journal. It was proposed that authors of Cochrane Reviews would prepare the documents electronically, and, once these had been approved by the appropriate Cochrane Review Group, the reviews would be published on the OCLC system. These discussions ultimately failed to progress, in part because of the legal implications of AAAS and OCLC publishing “sight unseen” material from international sources over which they had no editorial control.
The Cochrane Collaboration is based primarily on the enthusiasm of people wishing to contribute to it and does not have an easily defined membership. In 2007, more than 17,000 participants were listed by Cochrane Review Groups, living in more than 100 countries (Reference Allen, Clarke and Tharyan2), and the current sources of funding for the organization's work are summarized in Box 1. The Collaboration was, and continues to be, a loose-knit organization, although steps were taken from its inception to reduce duplication of effort. For example, collaborative searching (including hand-searching) for and pooling of randomized trial reports was organized, a task that could never have been undertaken by individuals working in isolation on individual reviews, and title of proposed reviews were registered to try to ensure that a specific question was only addressed once in a Cochrane review.
Funding the Cochrane Collaboration
Update Software continued to develop a system for managing and disseminating registers of trials and systematic reviews, and by October 1994, at the second Cochrane Colloquium in Hamilton, Ontario, Canada, the company demonstrated a prototype of a new CD-ROM publication—The Cochrane Database of Systematic Reviews (CDSR). The CD-ROM publication provided a means of communication among those interested in the work of the Collaboration, as well as an outlet for its work. The CD-ROM listed contact details of all groups in the Collaboration, including geographically based Cochrane Centres, which provide support for people from all countries in the world; Fields, which cut across broad areas of health; Methods Groups, which develop the methodology for systematic reviews; and the Cochrane Consumer Network, which encourages the involvement of patients and their carers in the work of the Collaboration. The CD-ROM also included listings of titles of planned reviews, and protocols for reviews in preparation.
CDSR continued the tradition established with CCPC, in which authors retained copyright and were encouraged to publish articles in print journals based on the reviews held in electronic form. This encouragement continues, with agreements with several journals willing to co-publish versions of Cochrane Reviews. Furthermore, in keeping with the spirit of Collaboration, and as an incentive to publish, the contact author for each review was given a complimentary subscription to the CD-ROM publication. The contact author of each Cochrane Review continues to receive a complimentary subscription but, to encourage the periodic updating of reviews, the subscription lasts for 2 years from the date of each update of their review.
Authoring Tools: ODMAN, RevMan, and ModMan
The original information management system supporting the Oxford Database of Perinatal Trials was a program named the Oxford Database Manager (ODMAN). This started the “manager” naming convention that has persisted to this day for the software used to prepare Cochrane Reviews. ODMAN was maintained at the National Perinatal Epidemiology Unit in Oxford, UK, and all data were entered and reviews created on this one system.
With the development of the Cochrane Collaboration, it became clear that the resources were not available to create and operate a centralized management system. “Let a hundred flowers bloom” was the sentiment of the time. The Cochrane Collaboration was committed to building on the enthusiasm of individuals. In response to this new organization, Update Software introduced a review authoring tool, Review Manager (RevMan), to allow individual authors to produce the highly structured reviews that featured in ODPT and CCPC. The information management system was designed to mirror the structure of the Collaboration. Thus, systematic reviews were prepared by individual authors in RevMan, and then sent to the editorial bases of Cochrane Review Groups, where they were assembled into modules of reviews using a Module Management program (ModMan). These modules were then sent to Update Software, where they were assembled to form a Parent Database, from which publications could be derived. RevMan was accompanied by a Handbook describing how to conduct a Cochrane review, which is now in its fifth edition. It is available within RevMan and on the Internet (http://www.cochrane.org/resources/handbook/) (Reference Higgins and Green36) and was first published in book form in 2008.
This system remained intact for 10 years, with the software programs supporting the system remaining virtually unchanged. RevMan was first released in July 1993 (RevMan 1.03 for DOS) followed by versions 2 and 3 in November 1995 and October 1996, respectively. Interim versions were released subsequently but it was recognized that a period of stability was desirable and so no major changes were made for a couple of years.
In 1997, Update Software gave the Information Management System to the Collaboration, and the Nordic Cochrane Centre assumed responsibility for maintaining RevMan, and, in 2001, for developing ModMan as well. The Nordic Cochrane Centre released version 4 of RevMan in 1999. This included the addition of several new features in Cochrane reviews, including a synopsis for each review for lay readers (this section was re-named the “plain language summary” in 2006). Once again, RevMan 4 was followed by a period of stability and the next major release of the software came in 2008. This incorporated two major attempts to make the content of Cochrane reviews more useful to readers: Summary of Findings tables, and Risk of Bias tables, to provide more transparent accounts of the assessments of the studies included in the reviews.
As well as adding new features to the review writing software, RevMan 5 also built on other developments with the Collaboration's software programs, which had been developed by the Nordic Cochrane Centre following a Collaboration-wide software needs assessment survey in 2002. This had highlighted the need to reduce the number of standalone software tools used within Cochrane Review Groups, and the resulting duplication of effort, for example, in storing and editing contact details of authors. This led to the creation of a new, integrated Information Management System (IMS). The IMS includes a central computer server (called Archie) on which all Cochrane reviews are now stored, along with their earlier versions and the contact details for people working within the Collaboration. This makes it easier for authors and Cochrane Review Groups to keep track of Cochrane Reviews as they are prepared and maintained. The RevMan software has always been freely available for noncommercial use and has been used widely to produce systematic reviews outside the Cochrane Collaboration. It can be downloaded from http://www.cc-ims.net/RevMan.
The great advantage of using the authoring tool RevMan has been that all documents have the same structure and the different parts of the document can be readily identified and independently manipulated. This allows them to be transformed into different formats for different publishing systems. Over the years, Cochrane reviews have appeared as ASCII text, in MARC formats (www.loc.gov/marc), using Standardized General Markup Language (SGML), HyperText Markup Language (HTML), and Extensible Markup Language (XML; see http://www.w3.org for information on standard technologies for Web publishing), as well as in proprietary formats for specialized products. The use of standard software for preparing Cochrane Reviews and for conducting the analyses within them has also made it much easier to provide training and support to the increasing number of authors of Cochrane reviews—more than 9000 by 2007 (Reference Allen, Clarke and Tharyan2).
MetaView
The Cochrane Database of Systematic Reviews used a novel approach to presenting figures and graphs, and further developed the strategy of separating the data themselves from displays of the data. In most scientific publications, figures and graphs are stored as images. The Cochrane Database of Systematic Reviews broke with this tradition and continued the strategy first used in ODPT—storing raw data with the Cochrane reviews and generating figures and graphs in “real time,” that is, at the time when they were displayed. In the case of The Cochrane Database of Systematic Reviews, this was accomplished by developing a specialized program named MetaView, which acted as a “data viewer.” MetaView was included with the authoring tool RevMan to allow the review authors to calculate statistics and view figures and graphs. MetaView was also included with the CD-ROM and later Internet versions of The Cochrane Database of Systematic Reviews, which meant not only that readers of reviews could view the same figures and graphs as the authors, but they could select different statistics and manipulate the data as well. In 2003, MetaView was replaced within RevMan by analysis and display software developed by the Nordic Cochrane Centre, which allowed the user to generate the same variety of analyses as that available to the authors of Cochrane reviews.
Technically, this was a very efficient method of publishing figures and graphs. By 2003 there were approximately 35,000 graphs with over 100,000 trial outcomes with numerical data to be displayed. The numerical data that were needed to create a graph could be transmitted far more rapidly than the graph images themselves, and the space saved by storing numerical data allowed the database to be published on CD-ROM.
This approach was criticized by some statisticians (Reference Senn51), who argued that the average reader would not know which statistics were appropriate to a given data set (e.g., odds ratio or relative risk; fixed effect or random effect analysis). In recent years, reviewers have had the option of setting a default analysis rather than having to view the default that was set for all reviews. This allowed the user of The Cochrane Database of Systematic Reviews to see the analyses that the authors had focused on when writing their review. With the transition to RevMan 5 and its new analysis software, and in light of data on usage revealing that users very rarely took advantage of this feature to conduct their own analyses, the Cochrane Collaboration decided to include the graphical images in The Cochrane Library, but to make the raw data from the review available for download. This increases the flexibility available to those users who wish to conduct their own analyses, because it allows them to use RevMan or other statistical software to analyze the data.
Comments and Criticisms
Each completed review and review protocol appearing in The Cochrane Database of Systematic Reviews includes contact details for both the reviewer(s) and the Cochrane Review Group responsible for the review. Readers have been encouraged from the outset to use these contacts if they had criticisms or suggestions for how the review might be improved. Comments on review protocols provide a form of “pre-publication” feedback on how the reviewers plan to conduct the review. Comments made “post-publication” provide a mechanism for ongoing feedback on such topics as the methods used, the questions addressed, and the conclusions reached. Reviewers and review group editors take these comments into account when updating the reviews. By updating reviews in response to comments and criticisms, mistakes can be corrected and the overall quality of reviews will continue to improve.
In 1996, Update Software launched a formal “Comments and Criticisms” system for Cochrane reviews at the 4th Cochrane Colloquium in Adelaide, Australia. This developed into a system allowing readers to post comments to a Web site from links in the reviews, the abstracts of reviews, or directly. Once posted, a copy of the comment was automatically forwarded to the Cochrane Review Group responsible for the review and to the Criticism Editor within that Group, who in turn passed the comment or a summary of the comment to the appropriate reviewer. When the review was updated, either the comment or the Criticism Editor's summary of the comment, along with the reviewer's reply, were published as part of the updated review. Reviewers, editors, and others could also reply to the comment directly on the Web site.
In recent years, following the move from Update Software to Wiley-Blackwell, the system was replaced by one in which feedback continues to be sent to the Cochrane Review Group and authors for response and incorporation into the Cochrane review, but the comments themselves are not published on a Web site. This decision was taken by the Collaboration's Steering Group in part because of the work involved in ensuring that all comments met the house rules before they were posted on the Web site, and in part because of the potential for confusion between feedback on the Web site, feedback within the review itself, and the updated version of the review. The continuing desire to improve the feedback system, to encourage more readers of Cochrane reviews to comment upon them to improve their content and interpretation, means that this important issue is being kept under review by the Feedback Management Advisory Group of the Cochrane Collaboration. At the time of writing, a new system is envisaged through which readers will be able to submit comments more comparable to “letters to the editor,” rather than simply suggestions for improving reviews.
Recent Developments
Although Cochrane reviews are intended to help people to make well informed decisions, it can take over 12 hours for someone to extract from a Cochrane review key information needed to inform decisions. In 2008, summary of findings tables were introduced in RevMan 5 to address several limitations of Cochrane and other systematic reviews, including problems with specifying the main outcomes, reporting adverse effects, and dealing with continuous outcomes (Reference Glenton, Underland, Kho and Oxman33;Reference Oxman47).
Summary of findings tables address these limitations by presenting the key information needed by people making decisions. They present, in a structured and consistent format, absolute and relative effect estimates, the number of studies and participants, and the quality of evidence for each main outcome. The quality of evidence for each outcome (i.e., the extent of confidence that the estimate of effect is correct) is based on judgments made by the review authors in a systematic and transparent way, using the approach developed by the GRADE Working Group (Reference Guyatt, Oxman and Kunz35). Results of user testing and evaluations of the summary of findings tables found that they improved by approximately two-thirds understanding of the results of reviews and reduced the time it took to find information such as the risk of an outcome with and without an intervention. Users agreed that Cochrane reviews should include summary of findings tables (Reference Rosenbaum, Glenton, Nylund and Oxman49).
Other new features introduced with RevMan 5 in 2008 included a “risk of bias tool,” to address inconsistencies in how the risk of bias has been assessed across reviews. The risk of bias is routinely assessed in Cochrane and other systematic reviews, but this has typically been done across outcomes for each study rather than across studies for each important outcome. The “risk of bias tool” provides standard criteria and guidance, the empirical evidence for these recommendations, optional criteria for specific situations, standard responses, and, most importantly, transparent descriptions of the basis for each judgment (Reference Altman, Antes, Gøtzsche, Higgins and Green3).
Another recently added feature provides “overviews of reviews.” Cochrane overviews of reviews summarize multiple Cochrane reviews addressing the effects of two or more potential interventions for a single condition or health problem (Reference Becker, Oxman, Higgins and Green5). In the absence of a relevant Cochrane review, they may additionally include systematic reviews published elsewhere. Overviews have a similar structure to reviews, but include reviews rather than primary studies, and an “overview of reviews” table designed to reflect the summary of findings tables in Cochrane reviews.
DISSEMINATION OF CDSR AND THE COCHRANE LIBRARY
From January 1997 to March 2003, Update Software distributed The Cochrane Database of Systematic Reviews through The Cochrane Library. In 1995, Update Software had convened an advisory group that recommended creating a library of information sources to inform decision making in healthcare, and to help in the production of systematic reviews. David Sackett, then Chair of the Cochrane Collaboration, welcomed the suggestion and authorized its implementation by Update Software. Thus it was that, in April 1996, Update Software presented the first issue of The Cochrane Library, which incorporated not only regularly updated systematic reviews and protocols for reviews in preparation in the Cochrane Database of Systematic Reviews, but also the Database of Abstracts of Reviews of Effectiveness (now, Effects) (DARE). DARE was assembled by the UK National Health Service Centre for Reviews and Dissemination at the University of York, and included appraisals of reviews published elsewhere, and health technology assessments from around the world.
In addition, The Cochrane Library contained the Cochrane Controlled Trials Register (assembled by Update Software from registers submitted by Cochrane Centres and Cochrane Review Groups, together with entries downloaded from MEDLINE and more recently Embase) (Reference Dickersin, Manheimer and Wieland28); a register of articles on the science of reviewing evidence (contributed by the Norwegian branch of the Nordic Cochrane Centre); and information about the Cochrane Collaboration. The register of trials is now called the Cochrane Central Register of Controlled Trials, or CENTRAL. There were three issues of The Cochrane Library in 1996, and this changed to the current four issues per year in 1997.
The Cochrane Library thus contained elements of a hierarchy of information, ranging from regularly updated full-text reviews, to structured summaries of high-quality reviews published elsewhere, to citations and abstracts of individual controlled trials. The intention was to extend this range to include planned and ongoing trials (Reference Chalmers, Hetherington and Newdick14) and topic summaries such as those from the Guide to Effective Care in Pregnancy and Childbirth (cf. CCPC).
The Cochrane Library was accessed using specially written software that allowed rapid searching of all documents in the collection, while distinguishing between types of documents when reporting the results of the search. Thus, after entering search criteria, the user was shown the number of regularly updated systematic reviews that met the criteria, followed by the number of reviews published elsewhere, then the number of trial reports, and so forth. This design emphasized the differences between the types of documents retrieved by a search strategy.
The dissemination strategy, as agreed with the Cochrane Collaboration, was to make The Cochrane Library available directly to subscribers, and to prepare the reviews and other resources for those wishing to disseminate the material on other platforms. With the growth in the Internet as a means for disseminating electronic documents, the focus for maximizing accessibility to Cochrane reviews and The Cochrane Library as a whole shifted from CD-ROMs to online publication. In September 1996, for example, The Cochrane Database of Systematic Reviews was made available on the World Wide Web in partnership with Synapse Publishing Inc. Update Software worked with Ovid Technologies to prepare data for use in Evidence Based Medicine Reviews, launched in October 1998. The World Health Organization sponsored the WHO Reproductive Health Library. This highly successful disk-based product, which has recently become an online publication, disseminates a selection of Cochrane Reviews relevant to developing countries, together with practitioners' comments on the applicability of the reviews in different practice settings. Evidence-based medicine guidelines presented treatment recommendations backed by evidence summaries and the full text of Cochrane reviews. This allows a “drill-down” approach where readers can examine for themselves the evidence on which treatment recommendations had been based.
By 2003, Cochrane reviews were available from most major information providers, including Dialog and DataStar, EBSCO, SilverPlatter, and Ovid. In 2002, the first national licenses for The Cochrane Library were implemented, giving free at the point of use access to everyone in a particular country or region over the Internet. This is achieved through computer-to-computer recognition of the location of the user's connection to the Internet, bypassing the need for users to login with a username and password. The first of these licenses were activated on the island of Ireland in February 2002 and were followed shortly by similar arrangements for several countries, including Australia, Finland, Norway, and the United Kingdom. The Cochrane Library was also made available throughout Latin America and the Caribbean through the BIREME system and to all low- and low-middle countries as defined by the World Bank, through the HINARI, INASP, and TALC programs.
In 2003, under a new publishing arrangement with the Cochrane Collaboration, publication of The Cochrane Library moved from Update Software to John Wiley & Sons Ltd (now Wiley-Blackwell), and The Cochrane Library was included in Wiley InterScience, alongside the other journals published by Wiley. In addition to including the online versions of Cochrane reviews and other content in The Cochrane Library, this allowed users to download a pdf version of each Cochrane review.
In 2007, a national license was purchased for India (Allen et al. 2007), and, to improve access and overcome limitations of the HINARI scheme, direct access to The Cochrane Library was made freely available to all countries in Band 1 of the HINARI scheme, through a no-cost agreement between Wiley-Blackwell and the Cochrane Collaboration. In 2008, Evidence Pods—short audio podcasts for selected new and updated Cochrane reviews—were introduced on the Cochrane Collaboration's Web site (http://www.cochrane.org/podcasts).
IMPACT AND CONTINUING CHALLENGES
Cochrane reviews published in The Cochrane Database of Systematic Reviews remain distinctive because of their prepublished protocols and their electronic publication. The latter allows three key features that are difficult to achieve with traditional print media: (i) because there are no practical constraints on space, systematic reviews published electronically can include more transparently details of background, materials and methods, data presentation, and analysis, which would often have to be omitted in print documents. (ii) Reviews published electronically can be updated as new information becomes available and when mistakes or other ways of improving them are identified. This makes it possible to produce a reference work that is continually improving in content and quality. (iii) Reviews can cross link to other, related reviews and sources of relevant information, allowing the reader to be little more than a click away from finding additional related evidence.
Although there are a great number of electronic publications available on CD-ROM and over the Internet today, most of these contain “papers” originally intended for print journals. In general, the electronic versions of the major medical journals have been judged to be very much like their print counterparts, and in many respects have failed to live up to expectations (Reference Delamothe26). The Cochrane Database of Systematic Reviews differs from traditional publications in that it was conceived as an electronic publication from the outset, and was designed to take advantage of features unique to electronic publishing. It illustrates how an electronic publication was developed in its own right to improve the quality and relevance of published reports of scientific information.
Cochrane reviews are now widely regarded as being of better quality, on average, than their counterparts in print journals (Reference Jadad, Cook and Jones38;Reference Moher, Tetzlaff, Tricco, Sampson and Altman42;Reference Shea, Boers, Grimshaw, Hamel and Bouter52), they are now widely cited in policy documents and practice guidelines (Reference Cumpston, Clark, Spithoff and Ohlsson25;Reference Holmes, Lusher and Chalmers37), and they have been shown to be a rich source of suggestions for new research (Reference Clarke, Clarke and Clarke20). Hopefully, they will also be used increasingly to improve the quality of Discussion sections in reports of new research on the effects of health care (Reference Chalmers8;Reference Clarke and Chalmers18;Reference Clarke, Alderson and Chalmers19;Reference Clarke, Hopewell and Chalmers21). In these ways, The Cochrane Database of Systematic Reviews has helped and should continue to help people making decisions about healthcare interventions to do so in light of the most reliable evidence and, thereby, help ensure that these interventions do more good than harm. As noted recently in an op-ed by Billy Beane, Newt Gingrich, and John Kerry in the New York Times: “a health care system that is driven by robust comparative clinical evidence will save lives and money. One success story is [the] Cochrane Collaboration, a nonprofit group that evaluates medical research. Cochrane performs systematic, evidence-based reviews of medical literature” (Reference Beane, Gingrich and Kerry4).
In 2008, The Cochrane Database of Systematic Reviews (CDSR) received its first official impact factor. This was based on citations in 2007 to new and updated Cochrane Reviews published in 2005 and 2006. At 4.564, CDSR was placed fourteenth in the list of general medical publications. The sheer number of citations to Cochrane Reviews meant that the Database ranked seventh in terms of citation frequency among general medical publications.
CDSR has grown considerably in size: by the end of 2008, there were 3600 full Cochrane Reviews, 2000 protocols for reviews in preparation, and systematic reviews of methodology, produced by the Cochrane Methodology Review Group. From 2009, it will include the first Cochrane reviews of studies of diagnostic test accuracy and Cochrane overviews of reviews.
In addition to CDSR, The Cochrane Library contains approximately 550,000 reports of controlled trials in CENTRAL, 8,000 records for other systematic review in DARE, and 11,000 records in the Cochrane Methodology Register relating to research into the methods of systematic reviews and other evaluations of health and social care. Other elements of The Cochrane Library include a database of health technology assessments (7,500 records) and economic evaluations (24,000 records), both of which are contributed by the Centre for Reviews and Dissemination in York.
Ten years ago, one of us (A.O.) identified ten challenges that had to be confronted if the Cochrane Collaboration was to achieve its principal aim—to help people make well informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions (Reference Oxman, Egger, Davey and Altman46). These included ethical, social, logistical, and methodological challenges (Box 2). The success of The Cochrane Database of Systematic Reviews up to now indicates that the Collaboration has, to an extent, succeeded in addressing these challenges. However, its future depends on ongoing efforts to continue to address them. For example, finding an appropriate balance between building on the enthusiasm of individuals and avoiding duplication of effort has been a challenge for the Collaboration from its inception. The Collaboration has been successful in developing fifty-two groups of people with common interests (drawing on their enthusiasm), each responsible for undertaking reviews within their areas of interest, and there is little conflict between groups and little duplication of effort within or across groups.
Ten challenges and one reason why they must be met
That said, there is undesirable variation in the editorial processes used across these groups, and unnecessary duplication of effort in developing and maintaining good editorial processes. To harmonize and improve the quality of editorial processes, and through these, improve the quality of Cochrane Reviews and the Cochrane Database of Systematic Reviews, an editorial board (consisting of the coordinating editor of each review group) has been formed, and an editor-in-chief appointed. An important function of the editor-in-chief is to lead the development of a vision for the future of the Cochrane Database of Systematic Reviews and how to achieve that vision.
Substantial efforts are currently being made to identify priorities for reviews and focus resources on these, with somewhat less reliance on the enthusiasm of individuals in choosing topics for reviews. Further developments may include reconsidering how many review groups are needed and how best to organize these groups, beyond basing them on the enthusiasm of the individuals that formed them.
Similarly, ongoing efforts are going into minimizing bias in Cochrane Reviews, and these, too, need to be balanced against the principle of building on the enthusiasm of individuals, as well as the principle of inclusiveness. There are concerns that new innovations and demands on review authors, such as implementing summary of findings tables, may deter participation, particularly for people in low resource settings.
Promoting access to The Cochrane Database of Systematic Reviews while ensuring continuity, ensuring sustainability, accommodating diversity across increasing numbers of contributors, updating searches for studies and reviews, and broadening the types of studies that are included, particularly for evidence of adverse effects, all continue to be challenges.
A VISION FOR THE FUTURE OF THE COCHRANE DATABASE OF SYSTEMATIC REVIEWS
It was clear from the inception of the Cochrane Collaboration in 1993 that it would be many years before the majority of reliable research studies assessing the effects of healthcare interventions could be placed in the context of a systematic review (Reference Mallett and Clarke40). It was also clear that the Cochrane Collaboration was not the only group producing high-quality reviews. A challenge for the continuing development and success of The Cochrane Database of Systematic Reviews will be to develop a shared vision for its future and a plan for achieving that vision. We end this chapter on the history of CDSR by outlining what we believe are likely to be key elements of such a vision, drawing on discussions among the editors responsible for coordinating Cochrane Review groups.
A continued and reinvigorated focus on quality is essential. This includes ensuring that reviews address sensible, relevant questions for decision makers, that the risk of bias is minimized through adherence to the guidance provided in the Cochrane Handbook (Reference Higgins and Green36), and that reviews are accessible (well written, concise, and readable), up-to-date, and timely.
CDSR must be user friendly. This will require a more flexible interface (adaptable for different types of users), making it easier to find and use reviews (for example, through overviews of reviews and linkages between reviews); improving links from guidelines and other resources to reviews in CDSR; licensing the use of CDSR to others; and collaborating with others to develop derivative products to meet the needs of different user groups.
The coverage of CDSR needs to be broad. It should include systematic reviews for all questions about the effects of health care that are important to people making decisions.
CDSR should be widely used. Cochrane Reviews are and should become even more widely used in debates, in journals, in resources like UpToDate and Clinical Evidence, by guideline developers, and by health technology assessment agencies.
CDSR must be dynamic and responsive. It should continue to evolve and respond to new developments and to the demands and expectations of its users.
Finally, CDSR must also be author-friendly. To avoid turning review authors or potential authors away, bureaucracy and administrative hurdles need to be minimized, and concerted efforts made to attract review authors. Without them, it will not be possible to achieve this vision or the aims of the Cochrane Collaboration.
CONTACT INFORMATION
Mark Starr, PhD (mstarr@update.co.uk), Update Software, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK
Iain Chalmers, DSc (ichalmers@jameslindlibrary.org), Editor, James Lind Library, James Lind Initiative, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK
Mike Clarke, PhD (mclarke@cochrane.co.uk), UK Cochrane Centre, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK
Andy Oxman, MD (oxman@online.no), Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway
