Effective methods for identifying research and guidance priorities are essential to inform health policy and practice. In the UK policy on the use of health technologies is underpinned by the work of several agencies, including the Health Technology Assessment (HTA) Program and the National Institute for Health and Clinical Excellence (NICE).
Although there are established systems for identifying research and guidance priorities in the United Kingdom, it has become increasingly difficult to engage the wider National Health Service (NHS) in identifying important topics. For example, the NHS Research & Development (R&D) HTA program operates four advisory panels in areas such as diagnostics and screening, pharmaceuticals, public health, and therapeutics. The panels identify and prioritize topics that are commissioned as primary and secondary research by the program. This system is regarded to be successful in terms of capturing the range of technologies relevant to the health service, and at scoping whether or not evidence for effectiveness is required. However, there have been few attempts to evaluate innovative methods of identification and prioritization in specific topic areas, particularly capable of engaging all relevant stakeholders.
Specialty mapping, one such innovation, is a concept whereby appraisal guidance on the use of specific technologies in specific conditions; clinical guidelines on the management of diseases and conditions, and evaluation research on the effectiveness of health interventions are described and mapped to identify gaps in the evidence base. The concept has been discussed over many years, and piloted in a handful of specialties (3;4;13), but is not widely implemented in the United Kingdom.
We undertook a pilot project to evaluate the methodology for specialty mapping in the area of child and adolescent health, an area defined as a priority by the Department of Health (DH) in England (6). The primary aim was to develop and evaluate the methodology and to disseminate the lessons learned so that it might be refined and implemented. This was to be achieved through two pilot specialty maps (i) the prevention and management of sexually transmitted infections (STIs) in teenagers, and (ii) the recognition, assessment, and management of acute pain in children and young people (ages 28 days to 19 years). A secondary aim was to identify and prioritize topics for the NHS R&D HTA program, NICE and, where relevant, other R&D programs.
In this study, we use the two pilot specialty maps as contrasting case studies to describe: (i) How we developed the methodology, and (ii) How we evaluated the methodology of specialty mapping, in order that a transparent methodological framework could be provided and disseminated.
METHODS
Overview
After consideration of various models of participation and the need to solicit topic suggestions from a broad spectrum of agencies a “stakeholder model” of participation was chosen. Relevant stakeholders included policy leads from the DH, specialists in child health, health service users, and voluntary and charitable organizations. Stakeholders were involved throughout the project.
The outline methodology for the pilot project was specified in advance (see Figure 1). The key stages were identifying the topic area and setting the scope, developing a care pathway, searching for evidence and guidance/guidelines, synthesis and mapping, and prioritizing the topics with stakeholders. Although we began with aims, objectives, and an outline methodology, the project was designed to be flexible and iterative. In keeping with this strategy, we reviewed and adapted the approach as time went on, a process akin to action research (18;20).
Outline methodology for specialty mapping.
The pilot project was guided by a steering group with representatives from the National Coordinating Centre for Health Technology Assessment (NCCHTA) and the National Horizon Scanning Centre (NHSC). The steering group convened four times over the 6 months of the pilot.
We worked closely with the Child Health Advisory Group (CHAG) at the DH in identifying clear priorities for the pilot project. The CHAG were initially involved in highlighting priority areas of child health where guidance was necessary. They also identified relevant child health practitioners who could help with identifying or designing a generic care pathway for the topic under study. The mapping process itself was to adhere as closely as possible to established systematic review methodology, with systematic literature searching, data extraction, critical appraisal, and synthesis (12;14).
Developing the Care Pathway
A framework was necessary to ensure that, where gaps existed, they could be identified across all of the relevant prevention and management strategies in a given area. It was envisaged that, for each pilot topic, a care pathway would be used as a framework within which to map the literature (1;7).
We searched for published care pathways that might be adapted for our purposes. If none were available, a pathway would be devised and subjected to clinical face validation by practitioners from the topic area or experts in child health.
Literature Searching
Literature searching was undertaken at the NCCHTA by an experienced information scientist. Two types of literature were sought: (i) evaluations of the clinical and cost-effectiveness of health technologies (e.g., randomized controlled trials and systematic reviews), and (ii) policy literature published between 1999 and 2004 (clinical guidelines and guidance on the use of health technologies from technology appraisals). Several key electronic bibliographic databases were searched for completed evaluation studies (e.g., MEDLINE, The Cochrane Library), and studies in progress (e.g., National Research Register). Policy literature was sought from databases of clinical guidelines/guidance (e.g., the National Electronic Library for Health “Guidelines Finder Specialist Library”) as well as Internet sites of relevant organizations (e.g., Scottish Intercollegiate Guidelines Network [SIGN]).
Inclusion and exclusion criteria were clearly defined at the outset for each topic area as a result of discussion with child health experts and the steering group. The results of the searches were filtered by reviewing individual outputs for information pertinent to the age group under study.
Synthesis and Mapping
The included literature was read, assimilated, and mapped to the appropriate nodes of the care pathway. This process was performed by one researcher and checked by a second. Additional areas for intervention along the care pathway were identified during this assimilation process. Validation of this process was undertaken by an independent reviewer. The quality of the guidelines was appraised narratively using the AGREE appraisal tool (19). Analysis of the populated care pathway and narrative map (the framework based on the care pathway) facilitated the identification of gaps in research evidence and policy literature. The gaps were identified by (i) analyzing those areas of the care pathway lacking appropriate guidelines/guidance or evaluation research, (ii) noting gaps cited within existing guidelines/guidance or evaluation research, and (iii) identifying poor-quality and out of date studies and guidelines/guidance.
Prioritization of Gaps
Prioritization of the identified gaps was undertaken in a “virtual” workshop for each pilot topic using e-mail and the Delphi consensus technique (11). The CHAG, the DH, and our steering group identified policy leads, healthcare professionals, charitable and voluntary groups, and services users who were invited to join the Delphi workshop. Each gap identified by the mapping was translated into a specific research or policy question, and a questionnaire containing all of the questions constructed and piloted before use in the Delphi exercise.
Two rounds of the Delphi process were undertaken to reach consensus. In the first round, participants were asked to independently rank the research or policy questions according to their perceived level of importance. The results were collated for each topic area with the research and policy questions ranked according to the level of importance identified in the first round. In the second round, participants were asked to score their highest priorities with a score of two and their next most important priorities with a score of one. The participants were supplied with standard criteria to help them judge the importance of the topics to policy and practice, as used by the HTA Program for routine prioritization of topics.
RESULTS
Case Study 1: Prevention and Management of STIs in Teenagers
Scope
The prevention and management of STIs in teenagers was chosen in consultation with the DH and the CHAG for this pilot project due to the broad nature of the topic and its relevance to current health policy. The project focused on the five most prevalent acute STIs in the United Kingdom (Chlamydia trachomatis, genital warts, Neisseria gonnorhoea, genital herpes, and infectious syphilis), as well as HIV/AIDS (21).
Inclusion/exclusion criteria were (i) Age range: young people ages 11 to 19 years; (ii) Interventions: prevention and management of STIs, except specific treatment strategies; (iii) Country: UK literature was prioritized, although international literature was also mapped where relevant.
Care Pathway
Literature searching of electronic sources (e.g., Web sites of professional organizations) did not yield any published care pathways that encompassed the prevention and management of STIs with young people. A bespoke pathway was, therefore, devised and validated both clinically (through discussion with health professionals in the area of sexual health and young people) and operationally (based on examples of currently used interventions from research databases; Figure 2). The pathway encompasses a wide range of strategies to promote and maintain good sexual health among young people at a variety of stages of sexual development. At the start of the pathway, interventions such as sexual health education are proposed for those not yet sexually active. For those who are sexually active, there is a set of strategies based on whether or not an STI is acquired, and whether symptomatic or not. Strategies include population and opportunistic screening, further investigation, treatment, and partner notification, where appropriate.
Care pathway for the prevention and management of sexually transmitted infections (STIs) in teenagers. GP, general practitioner.
Literature Searching
The search strategy was devised using nationally recognized terms and validated by the experts responsible for identifying the generic care pathways. In addition to the sources of literature mentioned earlier, several topic-specific sources were searched. These included Web sites of organizations such as the Health Development Agency (now the Centre for Public Health Excellence at NICE), and charitable trusts (e.g., the National AIDS Trust, the Terrence Higgins Trust). Evidence for the safety and efficacy of new and emerging interventions and technologies currently undergoing evaluation was sought from emerging technology briefings on STI from international horizon scanning collaborators, from news alert services and NHSC internal databases of technologies in development.
Given the limited time available a pragmatic decision was made not to conduct a specific search for evaluation evidence in the health promotion literature but to make extensive use of the two “evidence briefings” of STI prevention (2004) (9) and HIV prevention (2003) (8) available. Additional systematic reviews known to the authors of this report were included in the pilot map (8;9;10;15).
Mapping and Synthesis: Results
A total of fourteen sets of guidelines/guidance, four systematic reviews, and twelve primary evaluations were identified. Figure 2 shows the policy literature and research evidence mapped to the various nodes of the care pathway. For each node (the boxes shaded in gray) the unshaded boxes at either side of the figure show specific intervention strategies that relate to the node, and the volume of relevant guidelines/guidance and evaluation evidence identified. Some guidelines/guidance and evaluation studies were broad in scope and, therefore, applicable to more than one node of the pathway. In some areas, there were relatively more examples mapped than other areas, suggesting variability in the extent to which the sexual healthcare pathway had been covered. For example, among sexually active teenagers (who had not necessarily acquired infection) we mapped two clinical guidelines and five published evaluation studies addressing strategies for sexual health education and prevention of STIs. The guidelines were a DH “toolkit” for health professionals to promote good sexual health and prevent HIV (5), and Scottish guidelines of the management of chlamydia infection (16) emphasizing, among other things, the importance of primary prevention. Of the five studies, four were systematic reviews of the effectiveness of interventions to prevent STIs (8;9;10;15). Two of these five were tertiary evidence briefing reviews, which themselves included systematic reviews of effectiveness of prevention. The fifth study was an on-going evaluation of a school- and community-based initiative to promote safer sex and chlamydia testing. Areas of the pathway where fewer examples of guidelines/guidance or evidence were identified and mapped included management of acute and long-term complications.
Prioritization
The gaps that were identified were translated into fifty-four specific topic questions to be considered by a panel of stakeholders in the Delphi exercise. Of the fifty-four, forty-two were research questions, seven were topics suitable for potential clinical guidelines/guidance, and five suitable for potential technology appraisals, underpinning guidance to the health service. The topics spanned the entire care pathway, although many of the research questions related to the formative nodes, notably primary prevention of STIs.
The stakeholders were asked to identify and score up to a maximum of ten of the forty-two research questions, and a maximum of six of twelve guideline/guidance suggestions. Response rates for participation in the Delphi consensus process were 10/19 (53 percent) and 13/19 (68 percent) for the first and second rounds, respectively.
Outputs
Five guideline/guidance topics were prioritized, and these have been discussed at prioritization committees for potential commissioning. Of the forty-two research topics, twenty-three were within the scope of the HTA Program, whereas nineteen were relevant to other DH research programs. Of the 23 HTA research topics, 14 came under the remit of the Program's Disease Prevention Panel and underwent a further round of prioritization by the panel in March 2006. After discussion by the panel, four of these were prioritized and advertised as part of a call for proposals in December 2006 (two primary research, and two secondary research).
Case Study 2: Recognition, Assessment, and Management of Acute Pain in Children and Young People (Ages 28 Days to 19 Years)
Scope
As with case study 1, the topic of the second case study was chosen for its broad nature and importance to policy. Pain is an area of health care associated with variations in clinical practice, high healthcare costs, and significant morbidity for patients.
Inclusion/exclusion criteria were (i) Topic area: acute pain defined as “pain associated with actual and potential tissue damage, which is of recent onset and probable limited duration” to exclude predicted pain; (ii) Age range: age 28 days to 19 years; (iii) Interventions: recognition, assessment, and management (pharmacological and nonpharmacological interventions); (iv) Country: Any. Due to resource and time limitations, only policy literature was reviewed in this case study.
Care pathway
In common with the STI pilot, a published care pathway was not available from the literature. We developed a simplified generic pathway of care for a child experiencing acute pain that was clinically validated. The pathway sets out the various options for the management of acute pain in children and young people by health and other professionals. As with the STI pilot, the pathway was extensive, ranging from the initial recognition of pain of varying severity (e.g., through visual analogue and rating scales) to formal assessment, pharmacological management, and nonpharmacological interventions (e.g., cooling for burns or temperature, distraction therapy, splinting, psychological therapies) in either primary or secondary care. Effective management is defined as using these different interventions together/in the correct sequence, where appropriate.
Literature Search
As well as the literature sources outlined above, several topic-specific databases and Internet sites were searched, including The British Pain Society, The Joint Royal Colleges Ambulance Liaison Committee, and the London Emergency Medicine Paediatric Interest Group.
Mapping and Synthesis: Results
A total of sixteen sets of clinical guidelines/guidance were included: eight from the United Kingdom, four from the United States, and one each from France, New Zealand, Australia, and Canada. Not all of the documents included and reviewed described themselves as clinical guidelines. For example, one document was described as a “position statement” on the management of acute pain, whereas another was an evaluation report of a nursing project to reach consensus on guidelines for the implementation of standardized pediatric pain assessment in hospital settings. The documents varied in length from very short summaries of key recommendations with little supporting documentation to large reports with numerous appendices. The overall methodological quality was generally poor. There was little description of the background and methods used, and few were developed according to established guideline/guidance methodology. Many are now out of date, being over 5 years old at the time of the project.
Figure 3 shows the care pathway and the results of the mapping. Some nodes of the pathway appeared to be well covered by the literature, such as assessment of pain in prehospital and hospital settings, and pharmacological management. Other areas were more neglected, such as training and screening tools on recognition of pain.
Care pathway for the recognition, assessment, and management of acute pain in children.
In terms of “gaps” in the guidelines/guidance literature two levels of need were identified. There was a general need for up to date child-focused clinical guidelines/guidance covering all aspects of acute pain care, developed according to rigorous and transparent methodology. Several more specific needs were identified from the guidelines that were identified including evidence-based recommendations for the recognition and assessment of pain (to include training), appropriate pharmacological and nonpharmacological interventions, the safety and effectiveness of both nonpharmacological and pharmacological interventions, duration of benefit of pain control measures and the frequency of re-assessment, and the choice of treatment in relation to severity. These need to be age-specific (infant, child, adolescent) and setting-specific (primary care, paramedic care, and hospital emergency and in-patient), with particular attention to the paramedic setting.
Prioritization and Outputs
After mapping, eight guideline/guidance topics were identified for the Delphi virtual workshop. These topics included separate guidelines for pharmacological and nonpharmacological management of acute pain in, in hospital and prehospital settings, training and screening tools for the recognition of acute pain in children and young people, and validated and effective acute pain assessment tools for use with children and young people in the primary and secondary (to include A&E) care and paramedic settings. The stakeholders were asked to identify and score up a maximum of four of the eight guideline/guidance suggestions. Three of these were prioritized by the workshop and have been discussed at prioritization committees for potential commissioning. Response rates for participation in the Delphi consensus process were 7/14 (50 percent) and 11/14 (79 percent) for the first and second rounds, respectively.
DISCUSSION
This study is one of the few published examples of the evaluation of a mapping technique to set the policy and research agenda in health (17;21). This pilot evaluation has provided valuable feedback on one particular method for mapping research and policy priorities in health technology assessment.
We believe that, by applying the principles of systematic review methodology, we increased the transparency of the process, reduced the potential for bias, and generally enhanced the credibility of the approach. This was particularly important for ensuring that the outcomes (policy and research questions) were appropriate and that the process is accountable. Application of these rigorous methods in this pilot required adequate staff capacity with input from experienced researchers and a trained information scientist.
The pilot was ultimately successful in that we identified a large number of research and policy gaps and suggestions for clinical guidelines. These findings have been fed subsequently into policy prioritization committees, and primary and secondary research topics have been scoped, prioritized, and advertised for commissioning.
A key finding from this study is that a framework reflecting current policy and practice in the particular specialty is essential. Previous mapping exercises have used National Service Frameworks (3;4) or a cancer specific “Common Service Outline” (13), or decision trees (2) as an operational framework. To our knowledge, this study is the first to use a care pathway. This strategy had several advantages and provided an invaluable infrastructure. However, “off the shelf” published pathways were not available and construction of a bespoke pathway was not straightforward, despite helpful input from stakeholders. That said, for the purpose of specialty mapping, it does not necessarily have to be a “gold standard” care pathway covering every possible management strategy in detail. The level of detail required ultimately depends on the aims and scope of the map and the needs of end users.
As with all methodologies, there are limitations. The process was time consuming, and although it was systematic, it was not always exhaustive. Adequate time is needed to develop the care pathway, conduct literature searches, screen literature for inclusion, appraise and synthesize the literature, map to the care pathway, and translate gaps into clear and specific research or guidance questions. The process of securing input from stakeholders, all of whom had busy schedules, was also time consuming. Use of the “virtual” prioritization process probably helped to ensure the wider involvement of both clinical, nonclinical, and patients participants, and in greater numbers. The time and energy required to solicit involvement of stakeholders should not be underestimated.
Based on our experience in this pilot, we believe that mapping may be particularly useful where there is a need for a whole topic area to be prioritized, in addition to the routine identification and prioritization of available technologies across a spectrum of conditions. In this sense, it can contribute to “joined-up” policy making, ensuring consistency and reducing the potential for duplication.
CONCLUSION
Specialty mapping has the potential to generate topics for research and policy by identifying gaps in evaluation evidence and clinical guidance. The implications for research and policy are numerous. The identification of gaps in the policy literature can inform effective policy decisions. Depending on the number of evidence gaps identified there is the potential to commission high-quality research. Research capacity building will need to be maintained if this approach is to be used more widely to ensure demand is met.
In terms of the way forward, the methodology should be refined in the light of the lessons learned from this pilot exercise. It will be important to assess its value across a variety of topic areas. It should continue to be as participative and as inclusive as possible in the spirit of involving all in setting the agenda for health technology assessment.
CONTACT INFORMATION
Jonathan Shepherd, MPhil (jps@soton.ac.uk), Principal Research Fellow, Jackie Briggs, MBBS, MSc (jps@soton.ac.uk), Liz Payne, PG Dip Lib, MCLIP (eapayne@f2s.com), Independent Information Specialist, National Co-ordinating Centre for Health Technology Assessment (NCCHTA), Wessex Institute for Health Research and Development, University of Southampton, Mailpoint 728, Boldrewood, Bassett Crescent East, Southampton, SO16 7PX, United Kingdom
Claire Packer, BM, BS (c.packer@bham.ac.uk), Senior Clinical Lecturer in Public Health, Department of Public Health and Epidemiology, The University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom
Lynn Kerridge, MSc (lk1@soton.ac.uk), Chief Executive Officer, NCCHTA, National Co-ordinating Centre for Health Technology Assessment (NCCHTA), Wessex institute for Health Research and Development, University of Southampton, Mailpoint 728, Boldrewood, Bassett Crescent East, Southampton, SO16 7PX, United Kingdom
Martin Ashton-Key, MFPHM (martin.ashton-key@southcentral.nhs.uk), Consultant in Public Health Medicine, Department of Public Health, South Central Strategic Health Authority, First Floor, Rivergate House, Newbury Business Park, London Road, Newbury RG14 2PZ, United Kingdom
