Introduction
Evidence syntheses of all types, including health technology assessment (HTA) and systematic reviews, have their foundation in literature searches. Literature searching is developing in line with the growing use of evidence synthesis and is also informed, as a field of work, by the spirit of being evidence-based. This means increasing numbers of research papers about information retrieval are being published (Figure 1). As well as “mainstream” literature searching issues such as strategy development, search filters, and information resource content and overlaps, searching to inform evidence synthesis is also spreading into more fields such as machine learning, text mining, and citation analysis. Keeping up to date with the latest developments in this ever more wide-ranging field is demanding. To assist with meeting this challenge, the Health Technology Assessment international (HTAi) Information Retrieval Group (IRG) (https://htai.org/interest-groups/information-retrieval/) has produced a Web site resource: Summarized Research in Information Retrieval for HTA (SuRe Info) (1). This Web site (http://www.sure-info.org/) summarizes research-based information on effective and efficient evidence identification for the different aspects of HTA and evidence synthesis, such as identifying economic evidence. It is a unique resource, supporting the timely uptake of potential efficiencies arising from new evidence that may be incorporated into the evidence identification processes of HTA organizations.
Figure 1.PubMed search results (13 November 2020).
This paper describes the rationale, processes, and challenges of producing SuRe Info in the context of fast-moving pace of development in the field of evidence-based information retrieval. The paper also provides scenarios suggesting how SuRe Info can help searchers in their daily work and with specific questions.
Why SuRe Info?
Literature searching involves the identification of research using a variety of methods. It typically has two components (Reference Glanville, Cooper, Hedges and Valentine2). One component is the resources to be searched, which may be databases, Web pages, or hard copy publications. One of the challenges of evidence synthesis is to know how many resources should be searched and which resources are the best for which questions. The other component is the methods and approaches to be used to search the resources such as search strategies or techniques such as citation analysis or text mining. These techniques involve a detailed knowledge of database content and search features as well as knowledge of the search interfaces offered by different information sources.
Because evidence syntheses tend to emphasize the efficient but comprehensive identification of relevant research to answer specific questions, literature searching is one of the crucial early stages of the production of any evidence synthesis product. Literature searching for evidence synthesis typically seeks to identify as much relevant research as possible while minimizing the recall of irrelevant research (precision) (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3;4). Usually, literature searching is a series of compromises balancing the need to be sensitive with the need to account for the variabilities of the terms used within titles, abstracts, and indexing. The research on literature searching tends to address the ramifications of these issues from a variety of perspectives and to seek to generate efficiencies in searching that do not result in the loss of sensitivity. In the search for better precision, literature searching is moving into newer fields such as text mining and machine learning. It is also rediscovering existing tools such as citation analysis which allow searchers to break free from subject searching to use cocitation to unlock networks of publications (Reference Janssens, Gwinn, Brockman, Powell and Goodman5–Reference Hinde and Spackman7).
Given that literature searching is a fast-moving field and it is challenging to stay on top of the number and range of new developments, librarians and information specialists in HTA have long been aware of the need for regular update syntheses. For many years, the needs of information specialists in evidence synthesis were met by the guidelines produced by national and international groups such as Cochrane (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3;Reference Higgins, Thomas, Chandler, Cumpston, Li and Page8), the Centre for Reviews and Dissemination (CRD) (9), the Institute of Medicine (IOM) (10), and the EUnetHTA (4). However, these guidelines tend to focus mainly on effects and adverse effects, and the subject requirements of HTA are more extensive. Also, these guidance documents are produced by major undertakings, which cannot be updated as frequently as we might hope. In light of these challenges, in 2011, a group of members of the HTAi IRG started to consider how to meet the continuing challenge of staying up to date on the latest developments within the field of information retrieval. After considering the requirements and the best form to meet those requirements, the SuRe Info project was launched in November 2011.
What Is SuRe Info?
To meet the challenges of many topics and the need for currency, the SuRe Info authors developed an open access Web resource organized into chapters. SuRe Info currently comprises seventeen chapters, falling into two categories:
1. chapters about general search methods that are common across all health technologies such as which sources to search, how to design search strategies, using search filters, peer-reviewing of search strategies, and how to document and report the search process.
2. chapters summarizing the methods to use when searching for specific aspects of HTA such as economic evaluations or qualitative evidence (as defined in the European Network for Health Technology Assessment (EUnetHTA) HTA Core Model® (11).
SuRe Info chapters are drafted by teams of experienced information specialists, many of whom are also researchers and acknowledged experts in their specific areas of information retrieval. The resource is managed and coordinated by an editorial team of information specialists.
Information to update the chapters is obtained by running topic-specific searches in selected relevant databases to identify information retrieval methods publications that report new research on finding evidence. Chapter authors write a structured abstract for any eligible publications containing a brief critical appraisal of the publication. The authors synthesize information from the new published research with already available information, by considering the robustness of the research design of the new study, the generalizability of the new evidence, and the overall picture of the evidence to date. For example, a search filter that has been developed using a large gold standard set of relevant records and validated on a second large set of records will be considered more rigorous than a filter developed using more informal methods and will be more likely to be recommended in a SuRe Info chapter. If current advice for a procedure is based on many papers, the publication of one contradictory paper will not usually be considered adequate information to change the current advice, although the advice may become qualified to acknowledge the fact that not all evidence points in the same direction. Strong evidence from a research paper in a focused subject area may not be considered generalizable enough to change the current recommendations in a chapter, but the chapter authors may acknowledge that the recommendations may need to be adapted in specific subject areas.
SuRe Info is published as a part of an already existing Web resource, the HTAi Vortal, that seeks to collect and make available in one place information of interest to HTA teams. Chapters are updated every 6 months. To avoid duplication of effort, some chapters are summaries of published guidance. For example, the clinical effectiveness chapter is a summary of the Cochrane Handbook chapter on searching for studies (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3). These chapters use the published guidance as a baseline and then update them with new evidence as new publications are identified. SuRe Info currently has seventeen chapters (Table 1).
Table 1.SuRe Info chapters
The key challenges around updating each chapter lie in screening the results of the update searches, assessing the eligibility of the publications to the chapter, deciding whether the research is robust enough to justify changing current recommendations, and deciding whether the research is generalizable enough to justify changing current recommendations. Surprisingly, the volume of publications that needs to be included in each update is generally not high, although the number of records screened tends to be substantial.
What SuRe Info Is Not
SuRe Info is not a resource list, because the resources listed in each chapter have to be supported by research evidence or recommendations of other guidance. This means that when a new resource is announced, it does not automatically get added to its relevant chapters, because usually there is no research on its yield, features, quality, or overlap with other resources. To fill this gap, it has been proposed to develop separate resource lists without requiring research evidence to support their inclusions. For example, the Cochrane Handbook technical supplement (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page12) fulfills some of this need for clinical effects questions.
SuRe Info is not formally endorsed by any organization, and it does not reflect the endorsement of HTAi. It is not a comprehensive bibliography of all published research evidence within information retrieval because it focuses on health technology assessment issues.
It is not a detailed “how to” guide—we have focused on high-level brief summaries that are quick and easy to read, but for more in-depth information, we anticipate that readers will access formal detailed guidance or read the primary studies. The SuRe Info resource is intended as a summary and perhaps a map to best evidence, rather than repeating work already done well by other organizations.
How Can We Use SuRe Info in Our Daily Work?
Scenario 1: I have been asked to support a review of diagnostic test accuracy studies, and this is the first time I have been asked to do this. Is there a chapter to help me?
Yes there is. In the chapter Diagnostic accuracy you will find information on sources to search and how to design search strategies.
Scenario 2: The last time I did a search for economic evaluations, I searched HEED and NHS EED. Are these still the gold standard databases to search?
SuRe Info contains a chapter Costs and economic evaluation. In the section Sources to search, the chapter authors note that HEED is no longer available and NHS EED has ceased updating and is available only as a closed database. A list of information sources that should be considered when searching for economic evaluations and health economic studies is provided.
Scenario 3: I have just read a paper that says we should replace database searches with cocitation analysis. Should I change my practice?
Single studies in most topics are rarely significant enough in themselves to change practice and it would be best to read the paper within the context of the current research evidence. In the SuRe Info chapter Value of using different search approaches, the section on citation searching summarizes evidence on cocitation searching. Two studies found that cocitation analysis (instead of database searches) is quite highly sensitive but not 100 percent sensitive. This suggests that reading the new paper (if it is not one of those cited in the chapter) in conjunction with the findings listed in the SuRe Info chapter would be advisable, to judge whether the weight of research evidence is increasing in favor of cocitation analysis and for what subjects/topics.
Scenario 4: I know that my search strategies for a major health technology assessment to which I am contributing should be checked or peer-reviewed before I run them, but I work alone as an information specialist. How can I find someone to help with peer review?
In the Sure Info chapter Peer-reviewing search strategies, the authors describe the PressForum project which is a Web-based service that provides a peer review of search strategies. If you use the service, you are expected to reciprocate and peer-review strategies produced by other information specialists.
Scenario 5: I have to provide a report of my searches to be incorporated into the HTA report. I know that I have to write information to go into different sections of the report, but I am not sure exactly what to write for each section.
The SuRe Info chapter Documenting and reporting the search process is generally useful on this topic. It also has a table, at the bottom of the text, that synthesizes information from reporting standards and research articles on what information is typically reported in each section of a report, and this should help indicate the level of detail that you should provide.
Scenario 6: I have done many searches to support HTAs that have relied on RCTs and economic evaluations, but now I have to do searches to inform a review of qualitative research. I realize that PICO may not be the best question formulation structure for such a review and its search strategies. What other question structures could I explore to find one relevant to the current question?
Synthesis of qualitative research can be very different in terms of the search approaches used. The SuRe Info chapter on Qualitative research suggests a range of different question structures that could be explored to find the best fit for the question you are trying to capture.
Scenario 7: My manager has encouraged me to do some research to enhance my professional portfolio. I don't know where to start, to identify questions to research.
SuRe Info is a map of the research that needs to be done in information retrieval relating to HTA. Each chapter tells us what research has been done, what guidelines are recommending, and, often by implication, where evidence may be lacking or uncertainty remains. The absence of research publications within a section will often be an indication that practice is based on expert opinion or tried and tested approaches, and there may well be issues that could be investigated and informed by research. Obviously, not all activities need to be underpinned by research—some are just common sense or the only way to achieve something. However, many of our daily practices might benefit from research and SuRe Info chapters show us key areas where evidence is sparse, uncertain, or contradictory. The challenge for the researcher is choosing the question that will have most value to their own situation and also be as generalizable as possible, to help others.
Because SuRe Info is research based, it is also a useful place to identify research methods to inform the design of any new research project.
Plans for the Future
Although SuRe Info started almost a decade ago and is regularly updated, there is always room to improve it and develop it further. Since it was established, we have actively promoted SuRe Info in conferences and training events via leaflets and presentations. Increased marketing activities via, for example, e-mail lists are planned to raise awareness and uptake of the resource. SuRe Info is published as a part of an already existing Web resource, the HTAi Vortal. Work is under way to update the Vortal, and the appearance of SuRe Info will change along with the changes made to the Vortal itself. Feedback forms will be added to enable users to leave questions, comments, and suggestions. SuRe Info might also benefit from a more active social media presence. For example, when chapters are updated or new chapters are added, social media would be one method of alerting users to changes.
SuRe Info focuses on research-based guidance, which means that we do not typically include lists of resources unless the resources have been assessed and researched perhaps in comparison with other resources. However, many resources never receive evaluations or formal comparisons with other resources. Because it is still important to know that resources exist, even if we do not know information on their comparative value or performance, the SuRe Info team is considering building resource lists, or as noted above, referring (from within chapters) to published resource lists (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page12).
As described above, the SuRe Info team regularly synthesizes evidence to make judgments about the volume of evidence relating to specific questions and whether there is adequate volume to recommend new approaches or change practices. The authors do not have an agreed standard approach and would benefit from more formal guidance on how to synthesize new evidence into the current evidence base. Readers would benefit from a clearer description of the methods used to create SuRe Info chapters and how authors reach their recommendations. The chapters and the appraisals provide some information on the basis for recommendations, but the process could be made more transparent.
The SuRe Info team welcomes feedback and notification of new studies, but the Web site does not state this clearly. We hope to include feedback options in the future.
The SuRe Info team is always seeking new contributors to maintain its currency, develop new topics, and assist with marketing and editorial work. Identifying new ways to reach potential volunteers and involve them in SuRe Info is a continual challenge for a group of volunteers.
Conclusion
Over time, many organizations producing evidence syntheses have developed their own policies with respect to information retrieval and management. SuRe Info is not a substitute for methods handbooks and has not been adopted as formal policy by any organization as yet, but we believe it provides an overview of best evidence-based practice that can be used by any searcher and can also be used when creating and updating search guidance in handbooks. We suggest that it might be particularly useful for those developing searching guidance in new topic areas beyond its main focus of health, because many of the principles of searching can cross disciplines. SuRe Info is not a comprehensive bibliography of all published research evidence within information retrieval because it focuses on HTA issues. However, by providing an overview of current research evidence for major issues in information retrieval in HTA, SuRe Info helps searchers in this field to keep abreast of the latest developments and offers research-based information about searching issues that are relevant beyond the confines of evidence synthesis. SuRe Info is made possible through the continued dedication of its volunteer authors and editors.
You have
Access
Introduction
Evidence syntheses of all types, including health technology assessment (HTA) and systematic reviews, have their foundation in literature searches. Literature searching is developing in line with the growing use of evidence synthesis and is also informed, as a field of work, by the spirit of being evidence-based. This means increasing numbers of research papers about information retrieval are being published (Figure 1). As well as “mainstream” literature searching issues such as strategy development, search filters, and information resource content and overlaps, searching to inform evidence synthesis is also spreading into more fields such as machine learning, text mining, and citation analysis. Keeping up to date with the latest developments in this ever more wide-ranging field is demanding. To assist with meeting this challenge, the Health Technology Assessment international (HTAi) Information Retrieval Group (IRG) (https://htai.org/interest-groups/information-retrieval/) has produced a Web site resource: Summarized Research in Information Retrieval for HTA (SuRe Info) (1). This Web site (http://www.sure-info.org/) summarizes research-based information on effective and efficient evidence identification for the different aspects of HTA and evidence synthesis, such as identifying economic evidence. It is a unique resource, supporting the timely uptake of potential efficiencies arising from new evidence that may be incorporated into the evidence identification processes of HTA organizations.
PubMed search results (13 November 2020).
This paper describes the rationale, processes, and challenges of producing SuRe Info in the context of fast-moving pace of development in the field of evidence-based information retrieval. The paper also provides scenarios suggesting how SuRe Info can help searchers in their daily work and with specific questions.
Why SuRe Info?
Literature searching involves the identification of research using a variety of methods. It typically has two components (Reference Glanville, Cooper, Hedges and Valentine2). One component is the resources to be searched, which may be databases, Web pages, or hard copy publications. One of the challenges of evidence synthesis is to know how many resources should be searched and which resources are the best for which questions. The other component is the methods and approaches to be used to search the resources such as search strategies or techniques such as citation analysis or text mining. These techniques involve a detailed knowledge of database content and search features as well as knowledge of the search interfaces offered by different information sources.
Because evidence syntheses tend to emphasize the efficient but comprehensive identification of relevant research to answer specific questions, literature searching is one of the crucial early stages of the production of any evidence synthesis product. Literature searching for evidence synthesis typically seeks to identify as much relevant research as possible while minimizing the recall of irrelevant research (precision) (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3;4). Usually, literature searching is a series of compromises balancing the need to be sensitive with the need to account for the variabilities of the terms used within titles, abstracts, and indexing. The research on literature searching tends to address the ramifications of these issues from a variety of perspectives and to seek to generate efficiencies in searching that do not result in the loss of sensitivity. In the search for better precision, literature searching is moving into newer fields such as text mining and machine learning. It is also rediscovering existing tools such as citation analysis which allow searchers to break free from subject searching to use cocitation to unlock networks of publications (Reference Janssens, Gwinn, Brockman, Powell and Goodman5–Reference Hinde and Spackman7).
Given that literature searching is a fast-moving field and it is challenging to stay on top of the number and range of new developments, librarians and information specialists in HTA have long been aware of the need for regular update syntheses. For many years, the needs of information specialists in evidence synthesis were met by the guidelines produced by national and international groups such as Cochrane (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3;Reference Higgins, Thomas, Chandler, Cumpston, Li and Page8), the Centre for Reviews and Dissemination (CRD) (9), the Institute of Medicine (IOM) (10), and the EUnetHTA (4). However, these guidelines tend to focus mainly on effects and adverse effects, and the subject requirements of HTA are more extensive. Also, these guidance documents are produced by major undertakings, which cannot be updated as frequently as we might hope. In light of these challenges, in 2011, a group of members of the HTAi IRG started to consider how to meet the continuing challenge of staying up to date on the latest developments within the field of information retrieval. After considering the requirements and the best form to meet those requirements, the SuRe Info project was launched in November 2011.
What Is SuRe Info?
To meet the challenges of many topics and the need for currency, the SuRe Info authors developed an open access Web resource organized into chapters. SuRe Info currently comprises seventeen chapters, falling into two categories:
1. chapters about general search methods that are common across all health technologies such as which sources to search, how to design search strategies, using search filters, peer-reviewing of search strategies, and how to document and report the search process.
2. chapters summarizing the methods to use when searching for specific aspects of HTA such as economic evaluations or qualitative evidence (as defined in the European Network for Health Technology Assessment (EUnetHTA) HTA Core Model® (11).
SuRe Info chapters are drafted by teams of experienced information specialists, many of whom are also researchers and acknowledged experts in their specific areas of information retrieval. The resource is managed and coordinated by an editorial team of information specialists.
Information to update the chapters is obtained by running topic-specific searches in selected relevant databases to identify information retrieval methods publications that report new research on finding evidence. Chapter authors write a structured abstract for any eligible publications containing a brief critical appraisal of the publication. The authors synthesize information from the new published research with already available information, by considering the robustness of the research design of the new study, the generalizability of the new evidence, and the overall picture of the evidence to date. For example, a search filter that has been developed using a large gold standard set of relevant records and validated on a second large set of records will be considered more rigorous than a filter developed using more informal methods and will be more likely to be recommended in a SuRe Info chapter. If current advice for a procedure is based on many papers, the publication of one contradictory paper will not usually be considered adequate information to change the current advice, although the advice may become qualified to acknowledge the fact that not all evidence points in the same direction. Strong evidence from a research paper in a focused subject area may not be considered generalizable enough to change the current recommendations in a chapter, but the chapter authors may acknowledge that the recommendations may need to be adapted in specific subject areas.
SuRe Info is published as a part of an already existing Web resource, the HTAi Vortal, that seeks to collect and make available in one place information of interest to HTA teams. Chapters are updated every 6 months. To avoid duplication of effort, some chapters are summaries of published guidance. For example, the clinical effectiveness chapter is a summary of the Cochrane Handbook chapter on searching for studies (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page3). These chapters use the published guidance as a baseline and then update them with new evidence as new publications are identified. SuRe Info currently has seventeen chapters (Table 1).
SuRe Info chapters
The key challenges around updating each chapter lie in screening the results of the update searches, assessing the eligibility of the publications to the chapter, deciding whether the research is robust enough to justify changing current recommendations, and deciding whether the research is generalizable enough to justify changing current recommendations. Surprisingly, the volume of publications that needs to be included in each update is generally not high, although the number of records screened tends to be substantial.
What SuRe Info Is Not
SuRe Info is not a resource list, because the resources listed in each chapter have to be supported by research evidence or recommendations of other guidance. This means that when a new resource is announced, it does not automatically get added to its relevant chapters, because usually there is no research on its yield, features, quality, or overlap with other resources. To fill this gap, it has been proposed to develop separate resource lists without requiring research evidence to support their inclusions. For example, the Cochrane Handbook technical supplement (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page12) fulfills some of this need for clinical effects questions.
SuRe Info is not formally endorsed by any organization, and it does not reflect the endorsement of HTAi. It is not a comprehensive bibliography of all published research evidence within information retrieval because it focuses on health technology assessment issues.
It is not a detailed “how to” guide—we have focused on high-level brief summaries that are quick and easy to read, but for more in-depth information, we anticipate that readers will access formal detailed guidance or read the primary studies. The SuRe Info resource is intended as a summary and perhaps a map to best evidence, rather than repeating work already done well by other organizations.
How Can We Use SuRe Info in Our Daily Work?
Scenario 1: I have been asked to support a review of diagnostic test accuracy studies, and this is the first time I have been asked to do this. Is there a chapter to help me?
Yes there is. In the chapter Diagnostic accuracy you will find information on sources to search and how to design search strategies.
Scenario 2: The last time I did a search for economic evaluations, I searched HEED and NHS EED. Are these still the gold standard databases to search?
SuRe Info contains a chapter Costs and economic evaluation. In the section Sources to search, the chapter authors note that HEED is no longer available and NHS EED has ceased updating and is available only as a closed database. A list of information sources that should be considered when searching for economic evaluations and health economic studies is provided.
Scenario 3: I have just read a paper that says we should replace database searches with cocitation analysis. Should I change my practice?
Single studies in most topics are rarely significant enough in themselves to change practice and it would be best to read the paper within the context of the current research evidence. In the SuRe Info chapter Value of using different search approaches, the section on citation searching summarizes evidence on cocitation searching. Two studies found that cocitation analysis (instead of database searches) is quite highly sensitive but not 100 percent sensitive. This suggests that reading the new paper (if it is not one of those cited in the chapter) in conjunction with the findings listed in the SuRe Info chapter would be advisable, to judge whether the weight of research evidence is increasing in favor of cocitation analysis and for what subjects/topics.
Scenario 4: I know that my search strategies for a major health technology assessment to which I am contributing should be checked or peer-reviewed before I run them, but I work alone as an information specialist. How can I find someone to help with peer review?
In the Sure Info chapter Peer-reviewing search strategies, the authors describe the PressForum project which is a Web-based service that provides a peer review of search strategies. If you use the service, you are expected to reciprocate and peer-review strategies produced by other information specialists.
Scenario 5: I have to provide a report of my searches to be incorporated into the HTA report. I know that I have to write information to go into different sections of the report, but I am not sure exactly what to write for each section.
The SuRe Info chapter Documenting and reporting the search process is generally useful on this topic. It also has a table, at the bottom of the text, that synthesizes information from reporting standards and research articles on what information is typically reported in each section of a report, and this should help indicate the level of detail that you should provide.
Scenario 6: I have done many searches to support HTAs that have relied on RCTs and economic evaluations, but now I have to do searches to inform a review of qualitative research. I realize that PICO may not be the best question formulation structure for such a review and its search strategies. What other question structures could I explore to find one relevant to the current question?
Synthesis of qualitative research can be very different in terms of the search approaches used. The SuRe Info chapter on Qualitative research suggests a range of different question structures that could be explored to find the best fit for the question you are trying to capture.
Scenario 7: My manager has encouraged me to do some research to enhance my professional portfolio. I don't know where to start, to identify questions to research.
SuRe Info is a map of the research that needs to be done in information retrieval relating to HTA. Each chapter tells us what research has been done, what guidelines are recommending, and, often by implication, where evidence may be lacking or uncertainty remains. The absence of research publications within a section will often be an indication that practice is based on expert opinion or tried and tested approaches, and there may well be issues that could be investigated and informed by research. Obviously, not all activities need to be underpinned by research—some are just common sense or the only way to achieve something. However, many of our daily practices might benefit from research and SuRe Info chapters show us key areas where evidence is sparse, uncertain, or contradictory. The challenge for the researcher is choosing the question that will have most value to their own situation and also be as generalizable as possible, to help others.
Because SuRe Info is research based, it is also a useful place to identify research methods to inform the design of any new research project.
Plans for the Future
Although SuRe Info started almost a decade ago and is regularly updated, there is always room to improve it and develop it further. Since it was established, we have actively promoted SuRe Info in conferences and training events via leaflets and presentations. Increased marketing activities via, for example, e-mail lists are planned to raise awareness and uptake of the resource. SuRe Info is published as a part of an already existing Web resource, the HTAi Vortal. Work is under way to update the Vortal, and the appearance of SuRe Info will change along with the changes made to the Vortal itself. Feedback forms will be added to enable users to leave questions, comments, and suggestions. SuRe Info might also benefit from a more active social media presence. For example, when chapters are updated or new chapters are added, social media would be one method of alerting users to changes.
SuRe Info focuses on research-based guidance, which means that we do not typically include lists of resources unless the resources have been assessed and researched perhaps in comparison with other resources. However, many resources never receive evaluations or formal comparisons with other resources. Because it is still important to know that resources exist, even if we do not know information on their comparative value or performance, the SuRe Info team is considering building resource lists, or as noted above, referring (from within chapters) to published resource lists (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page12).
As described above, the SuRe Info team regularly synthesizes evidence to make judgments about the volume of evidence relating to specific questions and whether there is adequate volume to recommend new approaches or change practices. The authors do not have an agreed standard approach and would benefit from more formal guidance on how to synthesize new evidence into the current evidence base. Readers would benefit from a clearer description of the methods used to create SuRe Info chapters and how authors reach their recommendations. The chapters and the appraisals provide some information on the basis for recommendations, but the process could be made more transparent.
The SuRe Info team welcomes feedback and notification of new studies, but the Web site does not state this clearly. We hope to include feedback options in the future.
The SuRe Info team is always seeking new contributors to maintain its currency, develop new topics, and assist with marketing and editorial work. Identifying new ways to reach potential volunteers and involve them in SuRe Info is a continual challenge for a group of volunteers.
Conclusion
Over time, many organizations producing evidence syntheses have developed their own policies with respect to information retrieval and management. SuRe Info is not a substitute for methods handbooks and has not been adopted as formal policy by any organization as yet, but we believe it provides an overview of best evidence-based practice that can be used by any searcher and can also be used when creating and updating search guidance in handbooks. We suggest that it might be particularly useful for those developing searching guidance in new topic areas beyond its main focus of health, because many of the principles of searching can cross disciplines. SuRe Info is not a comprehensive bibliography of all published research evidence within information retrieval because it focuses on HTA issues. However, by providing an overview of current research evidence for major issues in information retrieval in HTA, SuRe Info helps searchers in this field to keep abreast of the latest developments and offers research-based information about searching issues that are relevant beyond the confines of evidence synthesis. SuRe Info is made possible through the continued dedication of its volunteer authors and editors.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflict of Interest
Both authors report having received small annual fees from HTAi, paid to their organizations as a contribution toward their time allocated to produce SuRe Info chapters. Both authors report conducting training in information retrieval, which is the focus of SuRe Info chapters.