DEFINING SEARCH COMPONENTS ACCORDING TO PICO

A translation of the research question into a searchable question should be presented using the PICO structure (Patients, Intervention, Comparator, Outcome) of clinical questions (4), see Table 2. Because the final search components are seldom identical to predefined inclusion criteria applied in trial (screening) searches, presenting information retrieval specifications will promote understanding. The list of components can easily be extended but not all components are mandatory for information retrieval.

Table 2.

PICO Definitions of the Example “Effectiveness of Human Insulin in Diabetic Patients Aged <18 Years”

DEFINING THE SEARCH MODEL

Graphical presentation of the search model or models is still an uncommon practice in reporting information retrieval processes and their results. This information would, however, be very helpful to researchers and other non-information specialists. The more complex the search question, the more helpful the graphical presentation. If a search model is presented, Venn diagrams are preferred (see Figure 1). In Boolean logic, the search model of our example reads as follows: (search component “patient”) AND (search component “problem”) AND (search component “intervention”) AND (search component “study design”).

Figure 1.

Venn diagram and Info-Crystal development on the example “Effectiveness of human insulin in diabetic patients aged <18 years.”

The sample space usually covers the hits of interest. If the search model is more complex and further subsets are of interest, for example, the intersection of the search components “intervention” and “problem,” an Info-Crystal (Reference Spoerri, Card, MacKinlay and Shneiderman17) might be more suitable. Info-Crystals are enhanced Venn diagrams where each intersection is defined as a sample space of its own and could thus be addressed separately (see Figure 1). Info-Crystals are often appropriate to presentations of ethical or legal issues or, in some cases, economic evaluations of health technologies.

As Venn diagrams are restricted to three or four search components, search models with more components require an alternative presentation method. One alternative could be derived from the Unified Modeling Language (UML) (7). A UML-derived model on our example is presented as Supplementary Figure 1, which can be viewed online at www.journals.cambridge.org/thc2010030.

Appendix/Additional Materials Transcribing the Information Retrieval. Lately, documentation of the information retrieval process including a full transcription of search histories has become more common and is regarded as state-of-the-art in HTA. However, reporting practices are still unsatisfactory. Incomplete reporting, lack of detail and the missing peer-reviewed procedures render evaluation and repetition of search strategies difficult. Accurate reporting is also needed for updating. However, when updating searches, merely reprocessing existing strategies is insufficient, as it is necessary to check for changed or new indexing of medical terms etc. Hence, updating purposes are a concern, but not a primary one.

Several studies on the completeness and accuracy of information retrieval processes and results have found poor reporting of search processes in published systematic reviews and HTAs. Various authors have called for improvements in reporting or demanded an international consensus on reporting standards; some suggest peer-reviewing as improvement (3;Reference Golder, Loke and McIntosh8;Reference Liberati, Altman and Tetzlaff11–Reference Sampson, McGowan, Tetzlaff, Cogo and Moher14;Reference Yoshii, Plaut, McGraw, Anderson and Wellik23). Few publications transcribe the full search in an appendix of the report or “additional material” on Web pages.

While PRISMA (Reference Liberati, Altman and Tetzlaff11) recommends reporting the full search strategy “for at least one major database . . .” (not including number of records per query line), state-of-the-art practice in HTA is to report all full search strategies, including number of records. The INAHTA checklist (9) suggests providing detailed materials optionally, “on request.” This option results in less transparency and impedes evaluating the methodology and credibility of search strategies. Hence, this option is not recommended in publications (such as HTA reports) where page restrictions are less important.

In the context of this common—suboptimal—practice, we developed templates to present the information retrieval workflow, conceptualization process (defining the question using the PICO scheme and additional components), retrieval model, database selection and other sources of information, additional hand-searching, and templates for structured presentation of complex and lengthy search histories. No single template fits all conceptualizations, but a few can be applied to most of the common retrieval processes. Our templates proved to be practical in selected research questions and more easily understandable than common practice today.

RECOMMENDATIONS FOR REPORTING INFORMATION RETRIEVAL PROCESSES IN DETAIL

The whole information retrieval process should be presented in a comprehensive manner (cf. Appendix 3 in CRD's guidance [5]). For better understanding of the proposed reporting and presentation, a complete set of transcriptions on autologous stem cell transplantation in breast cancer is added to this article as an example (Supplementary Figure 2, which can be viewed online at www.journals.cambridge.org/thc2010030).

According to our concept, the steps in the retrieval process include defining search components and models, selecting information sources, and transcribing search strategies.

1. Defining search components using PICO and additional components. If the search components are not documented in the text, they should be presented at this stage as described earlier in detail.

2. Defining the search model(s). Likewise the search model(s), if not documented yet, should be presented at this stage. If different models were processed this should also be stated, for example, different models for primary studies and secondary literature, different languages, adverse events, quality of life, ethical or legal issues, or economic evaluation.

3. Selection of information sources. All selected information sources should be listed in appropriate detail which varies by source as shown below.

Databases. Searched bibliographic and full-text databases, including gray literature and library catalogues, should be presented using their full name, including the name of the provider and search interface. If appropriate, the time range covered by the database should also be specified. A template for presenting database selection is shown in Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2010030.

Registers. All searched registers should be listed. A template for describing these searches is shown in Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2010030.

Hand Searching. A list of all journals, conference proceedings, Internet search engines or other Internet sources, as well as any other information resources used in the retrieval process should be compiled. A template for describing hand searching sources is shown in Supplementary Table 4, which can be viewed online at www.journals.cambridge.org/thc2010030.

Additional searches of reference lists in publications should be explicitly mentioned. This could be achieved by marking all publications retrieved in this manner. If all publications in an assessment are hand searched, a separate list of resulting new references could be provided.”

Regulatory Institutions. Searches for regulations or guidelines should be listed and described. A template for documenting relevant regulations and guidelines is shown in Supplementary Table 5, which can be viewed online at www.journals.cambridge.org/thc2010030.

Routine Collected Statistics. If general statistics sources were included, they should be listed. A template for describing sources of statistical data is shown in Supplementary Table 6, which can be viewed online at www.journals.cambridge.org/thc2010030.

Other Sources of Information. Any other sources of information used should also be listed. These may include contacts to manufacturers, patient groups, researchers etc. A template for describing such sources is shown in Supplementary Table 7, which can be viewed online at www.journals.cambridge.org/thc2010030.

If Web pages of a group of institutions (such as HTA units) were searched, a list of the institutions’ names should be presented.

Original Primary Data. If the researchers collected primary data that are not published elsewhere or already described within the text, at least the method and results should be stated in a separate appendix, for example. If a survey was performed, the questionnaire should similarly be presented as an appendix.

4. Transcripts of search strategies. For all searched databases and registers, full search strategies should be reported including additional information on the following items: The searcher; Citation of sources used; Name of the database (if metafiles were searched, the names of all databases included in the search process should be listed) Interface used; Sampling strategy (comprehensive sampling, selective sampling, purposive sampling); Standard search filters used; Time period restrictions (even though defined by the search component); Any further limits, for example, language restrictions (which are not state-of-the-art for information retrieval, but if they were introduced, they should be mentioned at this stage); and Date of searching. The search strategies should be reported according to the defined search components and the search model in all applications where possible. A structured conduct of search queries is a prerequisite for structured and complete reporting. The full search protocol should include a running number for each query line, the number of records identified, and the query text line by line.

The search histories as transcribed by the provider should be corrected as little as possible. If relevant terms resulted in no records, these lines should remain in the search history—except where this was caused by typing errors. In the interest of readability, the number of records identified should be reported in a separate column left of the query text, and the index terms searched and Boolean operators used should be written in capitals while all other query text should be written in lower cases.

A template search on “Effectiveness of human insulin in diabetic patients aged <18 years” was designed as shown in Table 3. This design is simple in structure and does not fit all search strategies, but further strategies can be designed along these lines.

Table 3.

Full Search Strategy Reporting of the Example “Effectiveness of Human Insulin in Diabetic Patients aged <18 Years”

REPORTING UPDATING INFORMATION RETRIEVAL

As information retrieval is conducted at the beginning of the assessment process, it is usually necessary to update the search to identify relevant publications that were published or indexed after the initial search date. No common practice exists for reporting updates. State-of-the-art could be as follows: (i) If the information retrieval update used identical search strategies with the primary search, it suffices to report the date of the update and the number of additional records (overall and by information source). (ii) If the update introduced one or two new or modified query lines while all other lines were identical, reporting additional or changed query line(s) is sufficient. The date of the update should be reported by adding the new or modified query line using the format in which initial strategies were reported. A detailed description of the location of newly introduced text is required and modified query lines should be named. (iii) If more than one or two lines were inserted or modified, the whole search strategy should be presented including the date of update.

PROCESS-BASED VERSUS ENTRY-BASED REPORTING OF SEARCH STRATEGIES

The application of published search strategies to different database user interfaces yields markedly different results. In addition to incomplete or faulty documentation, all commonly used search engines merely provide documentation of the exact query string input entered. Database interface providers do not readily document the queries actually processed. PubMed® (http://www.ncbi.nlm.nih.gov/sites/entrez) does, however, provide a special frame called “Details” showing the actual search query as processed by PubMed® algorithms. Thus, in PubMed® the huge difference between queries entered and those processed can be observed. Because background algorithmic transformations can change frequently, the reader of a published search history cannot necessarily determine what was actually searched. In consequence, state-of-the-art in reporting search strategies should be, wherever possible, to transcribe the queries processed instead of those entered. Copying each query from “Details” in PubMed® is more laborious, but adhering to these details ensures transparency and facilitates evaluation of the information retrieval.

An example on the different records between queries entered and those processed is shown in PubMed® when entering “information retrieval” in the query line. The search history in PubMed® Advanced Search reports this search term only. The “Details” window reports the query translation processed: “INFORMATION STORAGE AND RETRIEVAL”[MeSH Terms] OR (“information”[All Fields] AND “storage”[All Fields] AND “retrieval”[All Fields]) OR “information storage and retrieval”[All Fields] OR (“information”[All Fields] AND “retrieval”[All Fields]) OR “information retrieval”[All Fields].

QUALITY CONTROL IN REPORTING INFORMATION RETRIEVAL PROCESSES

PRISMA authors and others ask for a peer review of search strategies (e.g., Reference Liberati, Altman and Tetzlaff11;Reference Patrick, Demiris and Folk12). Not only the strategies themselves but also their reporting should be checked for quality purposes. This can be done by applying the Peer Review of Electronic Search Strategies (PRESS) checklist® (3, Appendix G). This validated instrument is available for evaluating the accuracy and completeness of reported search strategies, for example, checking for insufficient translation of the search question, missing thesaurus terms, misused Boolean operators, spelling variants, and typing errors.