Introduction
Region Västra Götaland, the second largest healthcare region in Sweden, has had a regional hospital-based health technology assessment (HB-HTA) center since 2007 (HTA-centrum). Assessments are performed by clinicians supported by HTA methodologists (for details, see Jivegård et al. (Reference Jivegård, Bergh, Kindblom, Samuelsson, Sjögren, Sjövall, Sampietro-Colom and Martin1)). Two major goals of HTA-centrum are to support decision making at different levels in the organization and to train hospital staff in evidence-based medicine by letting clinicians be responsible for HTA production. The participation of the Sahlgrenska University Hospital medical library is considered an important success factor. Librarians conduct systematic literature searches for the assessments, screen abstracts, and make first selections of full-text articles. Furthermore, they report on literature searches in the HTA reports and conduct administrative work, such as ordering full-text articles.
Background
HTA-centrum Region Västra Götaland
Region Västra Götaland serves a population of approximately 1.7 million. There are several hospitals in the region, with the Sahlgrenska University Hospital being the largest with 16,000 employees and 1,950 beds. HTA-centrum, which was established in 2007 and has been permanently funded since 2011, is located at the university hospital but supports the entire region. The unit has twelve employees, all working part time: eight HTA methodologists, one statistician, one health economist, one project coordinator, and one HTA librarian. The HTA methodologists comprise five physicians, one dentist, one physiotherapist, and one psychologist. Most employees have a PhD degree and many of them are also professors.
Initially, the assessments were modified versions of the Danish Mini-HTA, a form of checklist with questions about the health technology at hand (see Kidholm (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck2) for more information about the Danish Mini-HTA), but it has evolved toward a more comprehensive quality assured assessment. The present approach is similar to a systematic review (see the HTA report by Nilsson Ek et al. (Reference Nilsson Ek, Kovács, Linderholm, Olofsson Bagge, Stadig and Svanberg3) for an example of a recent assessment). Although most requests for assessments come from units at the Sahlgrenska University Hospital, other hospitals in the region and the regional health authorities also make requests.
A healthcare professional or manager from Region Västra Götaland, most often a clinician or a head of department, nominates a question. If the question is accepted for assessment, the head of the nominating department is then required to appoint clinical staff to do the assessment in collaboration with staff from HTA-centrum. The process ranges in length from 3 months to 2 years with a normal time frame of 6–8 months. There is an extensive quality assurance process with two external reviewers for every report, and all reports are discussed at a meeting of a regional quality assurance board. See Figure 1 for a complete description of the process.
Process. From nomination to publication.
Hospital-Based Health Technology Assessment
The Adopting Hospital-based Health Technology Assessment (AdHopHTA) Project has defined HB-HTA as “performing HTA activities tailored to the hospital context to inform managerial decisions on different types of health technologies. It includes the processes and methods used to produce HTA reports in and for hospitals” (Reference Sampietro Colom, Lach, Haro, Sroka, Cicchetti and Marchetti4, p. 20).
The scope and effort of HB-HTA can vary widely, from a mini-HTA in the form of a checklist (as in, for example, the Danish model described by Kidholm et al. (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck2)), to a full HTA report with systematic literature searches, critical appraisal of the literature, and GRADE assessments of the certainties of evidence.
Sampietro-Colom et al. (Reference Sampietro Colom, Lach, Haro, Sroka, Cicchetti and Marchetti4) describe four different types of HB-HTA units based on the formalization of procedures, the degree of specialization, and the level of integration with other organizations: (i) independent groups, (ii) integrated-essential HB-HTA units, (iii) stand-alone HB-HTA units, and (iv) integrated-specialized HB-HTA units. These categories should be considered ideal types; there are a number of different solutions for implementing HTA functions within hospitals. Even within these categories, there are many differences (Reference Sampietro Colom, Lach, Haro, Sroka, Cicchetti and Marchetti4;Reference Cicchetti, Iacopino, Coretti, Fiore, Marchetti and Sampietro-Colom5). In this model, HTA-centrum is best described as a stand-alone HB-HTA unit, a formalized and specialized HB-HTA unit, independent of the Swedish national HTA organization SBU (Swedish Agency for Health Technology Assessment and Assessment of Social Services). According to Sampietro-Colom et al. (Reference Sampietro Colom, Lach, Haro, Sroka, Cicchetti and Marchetti4), stand-alone HB-HTA units mainly act within hospitals, that is, the assessments are conducted internally by teams of hospital professionals (e.g., clinicians and HB-HTA unit) and lead to managerial decisions regarding health technologies. As a regional HB-HTA unit, HTA-centrum focuses on processes within hospitals, primary care and dental care across Region Västra Götaland, not in a single hospital. Nevertheless, it fits Sampietro-Colom's definition of acting internally. The assessment processes are carried out internally by hospital professionals from the nominating units (supported by HTA methodologists) with the aim of leading to managerial decisions, even if this is not always the case (Reference Wallerstedt, Bernhardsson, Grip, Hjalmar, Jivegård and Sjöwall6).
Literature Review of Information Retrieval in Hospital-Based Health Technology Assessment
The PubMed and LISTA [Library, Information Science & Technology Abstracts (EBSCO)] databases were searched in February 2020, using the search terms hospital-based health technology assessment OR hospital-based HTA OR HB-HTA OR HBHTA. A total of 268 abstracts were screened for relevance to HB-HTA and information retrieval, and 23 of them were read in full text. In addition, the books, Hospital-based Health Technology Assessment: The next frontier for Health Technology Assessment by Sampietro-Colom et al. (Reference Sampietro-Colom and Martin7) and The AdHopHTA handbook: A handbook of hospital-based Health Technology Assessment (HB-HTA) by Sampietro-Colom et al. (Reference Sampietro Colom, Lach, Haro, Sroka, Cicchetti and Marchetti4), were read. Reference lists of relevant articles and book chapters were scrutinized, finding two more articles. However, only one of them (Reference Fure, Ormstad, Vang, Lauvrak, Frønsdal and Skår8) mentioned information retrieval. In all, eight articles, four book chapters, and one report were included in the review (Reference Jivegård, Bergh, Kindblom, Samuelsson, Sjögren, Sjövall, Sampietro-Colom and Martin1;Reference Cicchetti, Iacopino, Coretti, Fiore, Marchetti and Sampietro-Colom5;Reference Fure, Ormstad, Vang, Lauvrak, Frønsdal and Skår8–Reference Tal, Booch and Bar-Yehuda18).
Most of the retrieved abstracts and articles did not mention libraries, librarians, information specialists, or information retrieval at all. Even in the included literature, there were limited descriptions of the roles of librarians and information specialists in different HB-HTA units. Many authors did not mention the profession at all, but a few HB-HTA units stated having access to library services. In some cases, the HB-HTA unit had a librarian or information specialist among its staff. Such units included the Center for Evidence-Based Practice at the University of Pennsylvania Health System in Philadelphia (Reference Gagnon9), Fondazione Policlinico “A. Gemelli” at the Catholic University of Rome (Reference Cicchetti, Iacopino, Coretti, Fiore, Marchetti and Sampietro-Colom5), Sainte-Justine University Hospital in Montreal (Reference Martelli, Lelong, Prognon and Pineau14), and HTA-centrum (Reference Jivegård, Bergh, Kindblom, Samuelsson, Sjögren, Sjövall, Sampietro-Colom and Martin1). Other HB-HTA units collaborated with a library or an information specialist, with collaborations taking a variety of forms and described in varying ways in the literature. Some authors described in detail the roles of librarians or information specialists. Fure et al. (Reference Fure, Ormstad, Vang, Lauvrak, Frønsdal and Skår8) specifically stated the importance of access to library services so that mini-HTA (mini-metodevurdering) in Norway could be successful. He suggested that literature searching could be best carried out by librarians in hospitals, because the Norwegian hospital librarians were specialists in systematic literature search. Ormstad et al. (Reference Ormstad, Graff and Norderhaug15) wrote about HTA-centrum, among other HB-HTA units, and commented on the library not only performing searches, but also excluding and including publications. Jivegård et al. (Reference Jivegård, Bergh, Kindblom, Samuelsson, Sjögren, Sjövall, Sampietro-Colom and Martin1) described the role of librarians at HTA-centrum and concluded that the librarians’ screening and initial selection of full-text articles saved a lot of time for the clinicians; librarians were also important when formulating answerable clinical questions. Halmesmäki et al. (Reference Halmesmäki, Pasternack and Roine10) described a case study in which Finnish hospitals collaborated with the national HTA unit: a typical assessment involved an information specialist from the national unit, providing expertise for the literature search. Also in Finland, Roine and Pasternack (Reference Roine, Pasternack, Sampietro-Colom and Martin16) wrote about the Helsinki University Hospital where literature searches were performed in collaboration with the medical library of the University of Helsinki and the Helsinki University Hospital. At Odense University Hospital in Denmark, the library assisted the HTA unit with literature searches (Reference Kidholm, Ølholm, Sampietro-Colom and Martin12).
Some authors described collaboration with libraries without specifying the role of the librarian or information specialist. Sampietro-Colom et al. (Reference Sampietro-Colom, Lach, Pasternack, Wasserfallen, Cicchetti and Marchetti17) stated that most HB-HTA units collaborated with, among others, university libraries, to cover unmet technical needs, without presenting examples of either HTA units or of the kind of technical needs referred to. Elsewhere, Sampietro-Colom et al. (Reference Sampietro-Colom and Martin7) gave the example of Hospital Sant Joan de Dèu in Barcelona, where a librarian was a member of the HTA committee. Lepanto (Reference Lepanto, Sampietro-Colom and Martin13) wrote that the HB-HTA unit at the Centre Hospitalier de l'Université de Montréal, Canada, had strong links with the university library and sought collaboration when expertise was needed. However, Lepanto did not present examples of collaborations or how they might be practiced. Juzwishin et al. (Reference Juzwishin, Olmstead and Menon11) described a technology assessment working group at a multihospital structure in Victoria, Canada. Among its members, the working group had a medical library. The exact role of the library (expertise in information retrieval, access to resources, or something else) was not specified. Finally, Tal et al. (Reference Tal, Booch and Bar-Yehuda18) mentioned an HB-HTA unit at the Shamir Medical Center in Israel that had the support of an information specialist, but they did not describe how this support was designed.
Assessing the literature about HB-HTA and information retrieval, limited descriptions of the roles that librarians and information specialists play in different HB-HTA units were found. When addressed at all, the focus was on literature search. However, in the authors’ own unit, HTA-centrum, the librarians contribute much more than this. This will be described in detail in the next section.
HTA-centrum and the Role of Librarians
The medical library at the Sahlgrenska University Hospital consists of three units, one at each hospital site: Östra Hospital, Mölndal Hospital, and Sahlgrenska Hospital. Between them, they cater to the information needs of all 16,000 hospital employees and perform traditional medical library tasks, such as information searches (for regional medical guidelines, systematic reviews, and research), interlibrary lending, subscription management, teaching, and book loans. There are eight librarians and one library manager.
The library was involved in building HTA-centrum from the first steps in an establishment project during 2006–7, and has taken an active role ever since. The former library manager played a vital part during the first years, contributing with extensive knowledge and interest in the role of medical libraries in evidence-based medicine. The library is organized within HTA-centrum in the organizational structure of the Sahlgrenska University Hospital, under the functional area of Research, Development, Education, and Innovation. A majority of the librarians (currently five out of eight) are active in HTA projects, and the appointed HTA librarian is responsible for coordination, maintaining, and developing work routines together with the other HTA-centrum staff. The HTA librarian is also a member of the regional quality assurance board. All library employees have a degree in library and information science. In order to participate in an HTA project, experience from systematic literature searching is needed. This is gained by conducting searches for hospital employees and by taking part in various courses. Their first HTA projects are conducted in collaboration with the HTA librarian to ensure that they get the support needed. The librarians involved in HTA projects have attended several HTAi meetings and Cochrane colloquia. The HTA librarian is involved in three to six (median 4) HTA reports per year. The other librarians participate in zero to four reports per person per year, but the number varies from the most active librarian (one to four, median 3, reports per year) to the least active librarian (zero to one, median 1, report per year). Librarians can often be involved in several projects at the same time, but as far as possible, the projects will be at different stages: one project might be in the time-consuming phase of searching and screening, whereas another project might have reached the stage where the librarians are waiting for the report to be ready for the addition of reference lists, and so on and so forth.
The process begins when a question is nominated for review. One librarian then performs a scoping search to see how much, if anything, is published. It is also of interest to see what types of study have been published. Are there a lot of systematic reviews? Are there any randomized controlled trials or large case studies? The scoping search usually includes the Cochrane Library and PubMed. If nothing is found, Embase (Ovid), the Allied and Complementary Medicine Database (AMED, EBSCO), and/or APA PsycInfo (EBSCO) are searched. In addition, the Web sites of the Swedish national HTA organization (SBU), the Swedish National Board of Health and Welfare (Socialstyrelsen), the Norwegian Institute of Public Health (Folkehelseinstituttet), and a number of Swedish regional HTA units are browsed. Depending on the number and types of studies that have been published, HTA-centrum will produce one of three kinds of reports:
• a full clinician-based HTA report;
• a shorter HTA report without any involvement of clinicians (Rapid HTA);
• a statement saying that a systematic search has been done and has found nothing has been published.
Reasons for not doing a report of any kind could be that there are recent national guidelines evaluating the question.
In most cases, a full clinician-based HTA report is produced. A project group is appointed, consisting of clinicians from the nominating unit, two HTA methodologists, and two librarians, and a start-up meeting is called. Prior to this meeting, the librarians scan for published literature by going through the search hits from the scoping search and try to locate possible queries. If necessary and if time permits, the librarians send the HTA methodologists a short summary of the populations, comparisons, follow-up times, and other salient features present in the literature. If needed (e.g., if there is a recent systematic review the librarians want to raise), articles will be sent to the whole project group. The start-up meeting marks the beginning of the project. The participating clinicians are introduced to HTA methodology, and most importantly, a preliminary PICO (Patient, Intervention, Comparison, Outcome) is discussed and formulated. After the meeting, the PICO is finalized by HTA-centrum in cooperation with the clinicians.
The two librarians together perform searches in relevant databases, after having prepared separate search strategies. Any disagreements and discrepancies are resolved by consensus. The librarians do not regularly use a formal peer-review checklist such as PRESS (Peer Review of Electronic Search Strategies) (Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre19), but searches are always validated by discussion. The search minimally includes PubMed/MEDLINE, Embase, and the Cochrane Library. Other databases are included when deemed necessary. The librarians also again browse the Web sites of the SBU and Folkehelseinstituttet for relevant reviews. Limitations in study design are always considered at the start-up meeting, but normally everything, from case series to randomized controlled trials, are included. Many of the health technologies that HTA-centrum studies are in their early phase of development. Randomized controlled trials are, in these cases, usually few or absent, and, therefore, nonrandomized controlled trials are considered to contribute to the evidence. Even large case series may be important to include to cover the aspects of complications, in which randomized controlled trials often lack power. This approach means that no specific search filters can be used (such as The Cochrane Highly Sensitive Search Strategies for identifying randomized trials (Reference Higgins, Thomas, Chandler, Cumpston, Li and Page20)).
The librarians then independently screen all retrieved abstracts, either on paper or in a screening tool such as the systematic reviews tool Rayyan QCRI (Reference Ouzzani, Hammady, Fedorowicz and Elmagarmid21). Which method they use is up to the participating librarians. After screening the abstracts, the librarians hold a consensus meeting to make a first selection of full-text articles. They then read the selected full-text articles, and at a second consensus meeting, exclude publications that do not fulfill PICO criteria. At this stage in the process, the librarians also scrutinize the reference lists of relevant articles to possibly identify articles not identified in the searches.
All articles fulfilling PICO criteria or whose relevance is uncertain are sent to the rest of the project group for final inclusion or exclusion. The librarians send the included articles either as PDFs or as printed copies (depending on the copyright conditions for each article), together with a list of the included articles indicating if each is printed or electronic. No other information is added to this list. A reference list of articles excluded by the librarians is also attached, specifying why each was excluded.
The final decision regarding inclusion for the report is taken at a consensus meeting where clinicians and HTA methodologists participate.
Between 2011 and 2019, HTA-centrum published a total of seventy-eight reports. The number of abstracts screened for each ranged from 54 to 4,639 (median 844). The number of full-text articles sent to the rest of the project group for inclusion/exclusion ranged from 12 to 165 (median 38), and the number of included articles ranged from 2 to 70 (median 13).
When the project group has made its the final inclusions and exclusions, the librarians search PubMed for comments on or errata in the included articles and send these to the rest of the project group. They also search ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform portal for ongoing trials. The search results are sent to the rest of the project group, which screens the results for any relevant ongoing studies that should be commented on in the report.
Administrative work such as downloading and ordering articles, keeping track of the EndNote library, and updating literature lists is done by the librarians, who are also responsible for describing the literature search in the report and creating an appendix. This appendix contains a flowchart, reference lists, and search strategies.
A Rapid HTA report is produced when the scoping search reveals few or no published studies. This process normally involves only one librarian and one HTA methodologist. The literature search is as systematic and thorough as for a full report, but the number of relevant studies is normally too few to require the involvement of clinicians, a second librarian, and a second HTA methodologist.
There are also cases where the initial scoping search does not identify any studies at all. In these cases, one librarian performs a thorough systematic literature search, to ensure that there really is nothing relevant published. An HTA methodologist and the librarian write a statement saying that a systematic search has not identified any published studies.
Discussion
When HTA-centrum was set up, it was decided that each project should always involve two librarians, partly as a quality assurance for the literature searches, but mainly to facilitate clinicians taking part in the projects. Librarians conduct the abstract screening and most of the full-text screening; however, this work was deemed too daunting and time-consuming for clinicians, thus deterring them from taking part in it.
Initially, there were concerns that nonclinicians who were not subject matter experts would not be able to read and understand the literature involved. Through the years, however, we have seen that this procedure works properly and also noted some advantages with it. In addition to the time-saving aspect for participating clinicians, it is the experience of the HTA-centrum staff (unpublished observation) that clinicians deeply involved in, and interested in, a subject tend to deviate from PICO. This sometimes includes wanting to read interesting articles not necessarily relevant for the question at issue. Experienced screeners are also able quickly to identify methodological flaws. The complexity of some questions means the librarians sometimes need to send abstracts to project group clinicians for second opinion.
In our literature review, we had hoped to find literature on the pros and cons of librarians as screeners, but we found information on only one HB-HTA unit where the librarians included and excluded publications: our own unit HTA-centrum, as described by Ormstad et al. (Reference Ormstad, Graff and Norderhaug15) and Jivegård et al. (Reference Jivegård, Bergh, Kindblom, Samuelsson, Sjögren, Sjövall, Sampietro-Colom and Martin1). Only Jivegård et al. described the benefits of this approach, with focus on the time-savings for clinicians.
In our unit, we have observed that screening abstracts and articles means librarians improve their skills at literature searching. Seeing and reading the results of searches makes it easier to understand what happens with different search strategies. This is advantageous not only in working with health technology assessments, but also in helping regular library users with searches and when teaching.
There are other advantages to involving librarians in HTA projects. Librarians get a better understanding of the process of writing a systematic review. They evolve their understanding of the complexities in formulating a question or PICO that is both specific and at the same time does not risk excluding relevant publications. They learn how to run a critical appraisal of literature and how to assess the certainty of evidence (even though this is not one of their tasks in the projects). They continuously attend meetings where clinicians and HTA methodologists discuss various issues. All these insights might be useful in assisting researchers in other contexts. Meanwhile, participation in HTA projects means clinicians get to know the librarians and the library services. This is favorable for the library, which needs to market itself to justify its existence and maintain sufficient resources to maintain its services.
During the years we have found that having two librarians involved in each project reduces the risk of errors in searching and screening (unpublished observation).
At HTA-centrum, librarians play an important role, not just in literature searching but also in screening. This does not seem to be standard procedure. With a few exceptions among Swedish colleagues, we are not aware of any other HB-HTA unit where librarians or information specialists are responsible for screening. In our review of the literature on HB-HTA, we found very little about librarians, information specialists, or literature searching in general. We are, of course, slightly biased, but we find this quite surprising. The European Network for Health Technology Assessment (EUnetHTA) methodological guidelines (22) present evidence that literature searches are of a higher quality when performed by information specialists compared with nonspecialists. Search strategies developed by information specialists are conducted more comprehensively and contain fewer consequential errors. The guidelines recommend that information specialists be involved in HTA and systematic reviews and form an integral part of the assessment team from the beginning of each HTA project (22). The Cochrane Handbook for Systematic Reviews of Interventions (Reference Higgins, Thomas, Chandler, Cumpston, Li and Page20) recommends that review authors seek guidance from a medical librarian or information specialist with experience in supporting systematic reviews. On the subject of literature searches for systematic reviews in general, there is, thus, evidence for the benefits of information specialists, but when it comes to the involvement of information specialists or librarians in HB-HTA, there is rarely any literature at all. Hopefully, this does not mean that librarians or information specialists are not involved in HB-HTA work. Our experience tells us that there are many advantages to librarians being involved at HTA-centrum, and we hope that other librarians and HB-HTA units will experience the same benefits.
Conclusion
In our brief literature review on the topic of information retrieval in HB-HTA, we found limited descriptions of the role librarians and information specialists play in different units. When the profession is addressed at all, the focus was on literature searching. Librarians at the authors’ own unit, HTA-centrum, not only perform the literature searches, but also conduct the abstract screening and make first selections of full-text articles. Furthermore, they participate in meetings where PICO is discussed, report on literature searches in the HTA reports, and conduct administrative tasks.
We believe that other HB-HTA units could learn from HTA-centrum regarding librarian involvement. We have experienced the following advantages:
• Involvement of two librarians in each project serves as quality assurance for literature searches.
• Screening conducted by librarians saves time for participating clinicians, who do not have to perform this screening themselves. This also increases the probability of clinician participation in projects.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
