Appropriate antimicrobial use is associated with improved patient outcomes and decreased risk of adverse events, including development of antimicrobial resistance and Clostridium difficile infection.Reference Davey, Brown and Charani 1 – Reference Kaki, Elligsen, Walker, Simor, Palmay and Daneman 3 To this aim, many healthcare and professional organizations advocate for coordinated programs that implement activities to ensure appropriate antimicrobial prescribing. 4 – Reference Ashiru-Oredope, Sharland, Charani, McNulty and Cooke 7 Such programs are referred to as antimicrobial stewardship programs (ASPs), or simply antibiotic stewardship. Antimicrobial stewardship is a key prevention strategy to address the global concern of increasing antimicrobial resistance.Reference Canton and Bryan 8 , Reference Agarwal and Schwartz 9 Preservation of effective antimicrobial agents is an international public health issue; therefore, implementation of ASPs in hospitals is a focus of healthcare systems and governments in many countries.Reference Dellit, Owens and McGowan 5 , Reference Huttner, Harbarth and Carlet 10 , Reference Tacconelli, Cataldo and Dancer 11
The Transatlantic Taskforce on Antimicrobial Resistance (TATFAR) was established by a 2009 US-EU summit declaration to enhance cooperation to address antimicrobial resistance. 12 , 13 One key focus area of this collaboration is the appropriate therapeutic use of antimicrobial drugs. Recognizing that a common way to assess ASPs is needed to understand and promote effective antimicrobial stewardship, in a 2011 report, TATFAR recommended the development of common structure and process indicators for hospital ASPs. 14 Using common indicators, EU member states and the United States could meaningfully characterize and compare antimicrobial stewardship efforts among different countries and healthcare systems.
Assessments of antimicrobial stewardship practices have been conducted in EU member statesReference Amadeo, Dumartin, Parneix, Fourrier-Reglat and Rogues 15 – Reference Morris, Brener and Dresser 18 and in the United StatesReference Abbo, Lo and Sinkowitz-Cochran 19 – Reference Trivedi and Rosenberg 21 but vary in terms of focus, length, and applicability to other health systems. The objective of this effort was to learn from and build upon these previous assessments in order to create a manageable number of ASP indicators that are relevant and feasible for comparison. To meet this objective, experts in hospital antimicrobial stewardship developed a set of indicators to describe the structure and functions of ASPs across healthcare systems.
METHODS
Participants
Antimicrobial stewardship experts from the European Union and the United States were recruited to participate in a modified Delphi process. Purposive sampling was used to ensure participants (experts) had the necessary expertise and experience for the development of the indicators, knowledge about clinical and public health practice, and diversity in geography and healthcare systems. All 20 invited experts participated in the rating of indicators, including the project coordinators from the European Centre for Disease Prevention and Control (ECDC) and the US Centers for Disease Control and Prevention (CDC). The panel had multidisciplinary representation from 9 EU member states and 6 US states (Table 1 and list of members at end of text).
Characteristics of 20 Experts Who Participated in the Modified Delphi Process to Develop Antimicrobial Stewardship Program Structure and Process Indicators
NOTE. The experts had a mean of 17 years of experience in antimicrobial stewardship.
a Austria, Belgium, England, France, Germany, Greece, Italy, Netherlands, Scotland (2), Slovenia, Sweden (2).
Consensus Process
A modified Delphi process using the RAND/UCLA appropriateness method was followed to build consensus using self-administered questionnaires with email exchange among the experts.Reference Fitch, Bernstein and Agular 22 An initial list of 53 indicators was developed from the Cochrane systematic review of interventions to improve antibiotic prescribing practices for hospital inpatients by Davey et alReference Davey, Brown and Charani 1 as well as previously developed structure and process indicators, antimicrobial stewardship surveys, and antimicrobial stewardship guidelines in the European Union and United States.Reference Amadeo, Dumartin, Parneix, Fourrier-Reglat and Rogues 15 – Reference Trivedi and Rosenberg 21 In addition, the project coordinators conducted a comprehensive, nonsystematic review of antimicrobial stewardship literature published from 2006 to 2013.
In the first rating round, experts were asked whether to “retain” or “remove” each indicator on its ability and feasibility to characterize ASPs. Indicators for which more than 60% of experts marked “retain” continued on to a second round. Next, the experts rated each of the structure and process indicators on a Likert scale from 1 to 9, where 1 indicated “Strongly disagree” and 9 indicated “Strongly agree,” according to 3 criteria: feasibility (ie, it would be possible to implement and measure this indicator at the facility level); clinical importance (ie, this indicator is important for optimizing the appropriateness of antimicrobial prescribing); and relevance to minimizing antimicrobial resistance (ie, this indicator is relevant to reducing the development of antimicrobial resistance). The median values of responses for feasibility and clinical importance were summed and indicators with a combined median score of 14 or greater were retained. Based on the RAND/UCLA appropriateness methods, agreement in ratings among experts was measured using an agreement score defined as the interpercentile range adjusted for symmetry minus the interpercentile range (ie, interpercentile range adjusted for symmetry − interpercentile range).Reference Fitch, Bernstein and Agular 22 A 90/10 interpercentile range was used for better discriminating power and agreement was defined as an agreement score greater than 0. The results were analyzed using an electronic spreadsheet (Excel; Microsoft). After the first and second rounds, members of the expert panel received feedback on the rating and agreement for each indicator, then participated in a conference call with a trained facilitator who moderated a structured discussion focusing on indicators with low agreement scores among the ratings, borderline combined median scores (12–14), and low median ratings for clinical importance. In a third and final round, experts re-rated the structure and process indicators. The changes in median and agreement score from the second to the third round were determined for each indicator. Also, agreement scores after the second and third round were compared by the Wilcoxon signed rank test to assess consensus. The experts were also asked to indicate whether each indicator should be included as either a “core” indicator (necessary to characterize ASPs) or a “supplemental” indicator (not necessary but desirable); or removed.
An in-person expert consensus meeting moderated by the project coordinators took place following the third round in Stockholm, Sweden, on June 18, 2014. At this meeting, experts reviewed and discussed the indicators that at least 20% of experts marked for removal in the third round. If consensus about removal was not achieved, the indicator would be carried over to the supplemental indicators group. Next, experts examined the supplemental indicators and discussed whether any should be reclassified as core or be removed. Finally, the indicators that emerged as core and supplemental during the meeting were reexamined with attention on their utility to comprehensively describe hospital ASPs and be understood in a multinational context. Final decisions regarding classification and wording of indicators were made by consensus.
RESULTS
In March 2014, 20 invited experts (100%) responded to the first round. Of the 53 proposed structure and process indicators, 36 were retained (10 with revision) and 17 were removed (Table 2). In response to feedback that the proposed indicators focused on ASP staffing and activities (structure), but did not capture the extent of activity performance (process), 8 process indicators were added to the second round of rating (Table 3). Two separate indicators, having an infection preventionist and a hospital epidemiologist on the antimicrobial stewardship team, were merged.
Summary of Results From the Modified Delphi Process
a In the third round, experts were asked whether indicators should be consider “core” (ie, the indicator should be included in a set of indicators for ALL hospitals to be asked) or “supplemental” (ie, hospitals can include if the indicator is of interest).
Results From the Modified Delphi Process to Develop Structure and Process Indicators for Hospital Antimicrobial Stewardship Programs: Rating and AgreementFootnote a on Feasibility and Clinical Importance With Final Decision on Inclusion as Core or Supplemental Indicator
a A higher number indicates higher agreement in the ratings among the experts.
b The numbers correspond to the number of the indicator at the start of the second round. Bolded numbers indicate revisions between the second and third rounds (strikethrough for deleted text; added text is italicized).
c The wording of many core indicators was revised in accordance with feedback from the in-person meeting. For this indicator, 3 separate indicators on condition-specific treatment guidelines were merged into a more general phrase, “…for common clinical conditions.”
Nineteen experts (95%) rated the 44 indicators carried forward to the second round. The mean (range) scores for feasibility and clinical importance for all indicators combined were 7.5 (5–9) and 7.6 (6–9), respectively. The experts rated indicators related to having a cumulative antimicrobial susceptibility report and guidelines high for both feasibility and clinical importance; whereas those related to governance and management and policies were generally rated higher for feasibility than for clinical importance; and indicators related to activities and interventions were generally rated higher for clinical importance than for feasibility (Table 2). For relevance to minimizing antimicrobial resistance, no indicator had a median score higher than 7 and there was low agreement on the scoring of the indicators among the experts (data not shown), so rating of relevance to minimizing antimicrobial resistance was not repeated in the third round. Following the second round, 14 experts (70%) participated in a group call on which many remarked that agreement or disagreement in feasibility ratings might be related to differences in healthcare settings and systems more than to discordant expert opinion. Variation in the information technology (IT) capacity among healthcare systems (eg, technical equipment, electronic systems) led to discussion on whether to remove the IT indicators or refocus the domain in more general terms. Some experts expressed that IT indicators are drivers for improvement and should be retained to track growth in the future, even if some countries or systems may not yet have advanced IT capacity for ASPs.
After analysis of the second round ratings and input from the expert panel call, 37 of the 44 proposed structure and process indicators were retained. An indicator that combined assessment of compliance with community-acquired pneumonia and urinary tract infection guidelines was divided into 2 separate indicators, making a total of 38 indicators advancing to third round (Tables 2 and 3). Six indicators were removed: 2 human resources indicators (involvement of IT staff and quality improvement staff); 2 indicators related to IT (presence of an IT system for prescribing and its application to clinical decision support for antimicrobial prescribing); the indicator on routine use of antimicrobial order forms and the indicator about dedicated time for clinical teams to review antimicrobial orders. For the proposed process indicators, the experts expressed that 95% compliance with guidelines was too stringent and recommended that the threshold change to 80%.
The response rate for the third round was 95% (19/20 experts). The mean (range) scores for feasibility and clinical importance were 7.5 (6–9) and 7.8 (6–9), respectively. The indicators with highest agreement for feasibility among experts were the identification of a defined ASP, formulary, and surgical prophylaxis guidelines; whereas the indicators with lowest agreement on feasibility were process indicators that assessed whether “>80% of sampled cases” had a documented indication or followed facility-specific guidelines. For clinical importance, the indicators on physician and pharmacist leadership, IT capability, facility-specific treatment guidelines, and postprescription review and feedback had high agreement in scoring among experts; whereas the human resources indicators not related to leadership, the discontinuation of specified antimicrobial prescriptions after a predefined duration, and the capture of indication for treatment in the medical record had the lowest agreement scores (Table 3). The mean agreement score increased from 1.8 to 2.9 for feasibility (P=.007) and from 2.9 to 3.3 for clinical significance (P=.087).
Thirteen experts (65%), with balanced representation from the European Union and United States, attended the in-person expert consensus meeting. There was consensus that indicators that at least 70% of experts rated as “core” would be core indicators, with minor revisions to clarify definitions. Most experts at the meeting recommended removal of the indicators that assessed compliance with facility-specific community-acquired pneumonia and urinary tract infection guidelines in at least 80% of sampled cases for the following reasons: there were concerns about feasibility; collection of such data would increase workload; accurate quantification would be challenging; and the indicator may not reflect appropriateness of non–guideline-concordant clinical decisions. Similarly, documentation of an indication for treatment in at least 80% of sampled cases was rejected. Some experts noted that monitoring other aspects of antimicrobial prescribing rather than attempting to quantify compliance may be a more effective use of the time of those responsible for antimicrobial stewardship. Experts also recommended removal of “a current susceptibility report has been distributed to prescribers” because this indicator does not assess the application of this information to patient care nor the ability of prescribers to interpret it; and “does your facility have a defined formulary of antimicrobial agents?” because the term formulary was found to have different interpretations, and diversity in prescribing may be considered as an approach to prevention of antimicrobial resistance.Reference van Duijn and Bonten 23 , Reference Martinez, Nicolas and Marco 24 In contrast to recommending removing the assessment of compliance with community-acquired pneumonia and urinary tract infection guidelines, EU experts strongly advocated for the monitoring of surgical prophylaxis as a core indicator. The feasibility of this indicator is supported by current measurement in the United States through the Surgical Care Improvement Project and in the European Union by the Healthcare-Associated Infections Network as well as longstanding experience at national level.Reference Bratzler and Hunt 25 – Reference Kurz, Mertens and Ronveaux 28 Therefore, review of surgical prophylaxis was reclassified from a supplemental to a core indicator and the quantification of guideline concordance became a supplemental indicator. Experts who were not able to participate in the in-person meeting were provided all materials before the meeting and a summary. Each expert had the opportunity to give input into the final indicators and report developed after the meeting. At the conclusion, there were 17 core indicators (Tables 3 and 4) and 16 supplemental indicators (Table 3).
Core Structure and Process Indicators for Hospital Antimicrobial Stewardship Programs (Final Set)
a The wording of core indicators may have been revised in accordance with feedback from the in-person meeting.
b These numbers are provided as reference to better understand the changes that happened during the Delphi process.
DISCUSSION
A multinational panel of experts in antimicrobial stewardship developed a set of indicators to characterize the infrastructure and activities of hospital ASPs using standardized methods to address the TATFAR recommendation to develop common structure and process indicators for hospital ASPs. The modified Delphi process reflected the input of experienced clinical and public health professionals with diverse perspectives and fostered an exchange of best practices to address antimicrobial resistance and appropriate use of antimicrobial agents among diverse health systems. The high ratings of feasibility and clinical importance of each indicator along with convergence of ratings between multiple rounds demonstrate that the final indicators are likely to be practical in diverse settings and meaningful to quality of care.
Themes that emerged during the expert consensus meeting were the necessity of support and accountability for antimicrobial stewardship activities. Salary support was specified to differentiate a higher level of support than mere inclusion of responsibilities for antimicrobial stewardship in job duties without salary support. Participants acknowledged the importance of multidisciplinary involvement, but given that the specific composition of teams was highly variable among healthcare systems and facilities within the same system, indicators that asked about specific staff roles were retained as supplemental rather than core indicators. Participants noted that active feedback was more effective in changing prescribing practices compared with passive education of prescribers; therefore, direct communication of antimicrobial audits or reviews to prescribers was deemed core and education on improving prescribing was removed. An annual report on antimicrobial stewardship and an indicator related to IT capability remained as core indicators because they were seen as “reach” goals—that is, indicators that may be advanced for the current state of ASPs in most facilities but could, in the future, differentiate ASPs and set a target for achievement.
The proposed indicators build upon similar efforts to assess the hospital ASPs in France, Germany, the United Kingdom, the United States, and the European Commission–sponsored Antibiotic Strategy International.Reference Dellit, Owens and McGowan 5 , Reference Amadeo, Dumartin, Parneix, Fourrier-Reglat and Rogues 15 – Reference Morris, Brener and Dresser 18 , Reference Buyle, Metz-Gercek and Mechtler 29 , Reference Thern, de With, Strauss, Steib-Bauert, Weber and Kern 30 Experts involved in the development, implementation, and analysis of many of these ASP assessment efforts participated in this process. The TATFAR ASP indicators align and could be compared with many questions in these previous assessments provided that an acceptable balance between flexibility and consistency in translation is achieved. By design, the TATFAR core indicators are a smaller number of items suitable for public health surveillance or integration with special studies, such as the European Surveillance of Antimicrobial Consumption Network coordinated by ECDC or point prevalence surveys of antimicrobial use conducted by the ECDC or the CDC or within large health systems. The CDC has developed a guidance document, Core Elements of Hospital Antimicrobial Stewardship, that aligns with the TATFAR ASP indicators and has incorporated antimicrobial stewardship questions based on the indicators into the 2015 National Healthcare Safety Network Annual Hospital Survey of more than 4,000 US facilities. 31 , 32 Similarly, the ECDC is planning to include questions based on the TATFAR ASP indicators into its second point prevalence survey of healthcare-associated infections and antimicrobial use in European acute care hospitals that will take place in 2016–2017.
Most of the final indicators are structure rather than process indicators, reflecting the critical importance of staffing, baseline capacity, and support for ASPs as well as the variability of ASP implementation in healthcare. The establishment of a well-supported, multidisciplinary ASP infrastructure ensures an ASP is sustainably integrated into facility practices rather than dependent on a single person. The evidence base for any individual structure and process indicator is limited and challenging to establish because antimicrobial stewardship activities are multifaceted and involve a combination of efforts to attain results. Hopefully, as guidelines for ASPs are developed and adopted, metrics of appropriate antimicrobial use will also be developed as accurate process indicators of ASPs.Reference van den Bosch, Geerlings, Natsch, Prins and Hulscher 33 Although minimizing antimicrobial resistance is a primary goal of TATFAR and was initially included as a rating criterion in this process, the ratings of these indicators were low and highly divergent. This result reflects the challenge of establishing the impact of ASPs on antimicrobial resistance.Reference Schulz, Fox and Polk 34 Other metrics of ASP success such as antimicrobial use and patient-focused outcomes are critical to demonstrate the impact of ASPs. Outcome indicators for ASPs were not evaluated as part of this modified Delphi process and expert consensus because standards for measuring antimicrobial use in hospital settings are the objective of a separate TATFAR recommendation. 35
The modified Delphi process is a widely used, standardized method to develop healthcare quality indicators that ensures equal representation among participants and allows for input and collective consensus across diverse geographic locations. This method provides multiple opportunities for clarification and revision throughout the iterative process of rating and soliciting comments. Additionally, the expert panel group call and in-person consensus meeting contributed greatly to developing a common understanding and reaching final consensus, and included input from the 7 experts who were unable to travel to participate in the final consensus meeting. Potential limitations were that individual ratings by the experts were self-reported and subject to personal perspectives or experiences. A formal systematic review was not within the scope of this project.Reference Jackson and Waters 36 The project did not include representation from countries other than Europe and the United States owing to scope established in the 2009 US-EU Summit declaration. The entire process was conducted in English, which may have created a cultural bias or limited the participation of those who did not feel most comfortable exchanging views in English. Although the final indicators could offer a starting point for developing international ASP indicators, their relevance and applicability to Latin American, African, Asian, and other countries is unknown.
In conclusion, core and supplemental indicators of ASPs deemed to be essential elements to ensuring appropriate use of antimicrobial agents in the hospital setting were developed by a multinational group of experts through a modified Delphi process and consensus meeting. The collaborative development of these structure and process indicators for TATFAR contributed to mutual understanding of the capacity and vision for hospital ASPs in EU member states and in the United States. Implementation of these TATFAR-developed core indicators in multiple countries could contribute to a comprehensive, comparative description of infrastructure, policies, and practices of antimicrobial stewardship internationally. These findings could, in turn, lead to an understanding of best practices of ASPs through further investigation into the relation of different antimicrobial stewardship approaches to antimicrobial use and resistance. We believe that these indicators define clear expectations for hospital ASPs and, through piloting, implementation, and evaluation, will contribute to the understanding of ASP practices in hospitals and improve antimicrobial prescribing. A full report with a more detailed summary of the process and results of the ratings by rounds is available from the TATFAR Secretariat (TATFAR_Secretariat@cdc.gov).
Members of the TATFAR Expert Panel on Stewardship Structure and Process Indicators
Anastasia Antoniadou (University Hospital Attikon, Athens, Greece); Bojana Beović (University Medical Centre, Ljubljana, Slovenia); Franky Buyle (Ghent University Hospital, Ghent, Belgium); Sara Cosgrove (Johns Hopkins Medical Institutions, Baltimore, Maryland); Peter Davey (Medical Research Institute, Dundee, United Kingdom); Elizabeth S. Dodds Ashley (University of Rochester Medical Center, Rochester, New York); Catherine Dumartin (Bordeaux University Hospital, Bordeaux, France); Alison Holmes (Department of Medicine, Imperial College London, London, United Kingdom); Winfried Kern (University of Freiburg Medical Centre, Freiburg, Germany); Maria Luisa Moro (Regional Agency for Health and Social Care of Emilia-Romagna, Bologna, Italy); Dilip Nathwani (Department of Medicine, University of Dundee, Dundee, United Kingdom); Jeanne Negley (Georgia Department of Public Health, Atlanta, Georgia; Melinda M. Neuhauser (VHA Pharmacy Benefits Management Services, Hines, Illinois, and University of Illinois at Chicago College of Pharmacy, Chicago, Illinois); Christopher A. Ohl (Wake Forest University School of Medicine, Winston-Salem, North Carolina); Jeroen Schouten (Scientific Institute for Quality of Healthcare, Nijmegen, Netherlands); Ed Septimus (HCA Health Care System, Nashville, Tennessee, and Texas A&M Health Science Center College of Medicine, Houston, Texas); Agnes Wechsler-Fördös (Wiener Krankenanstaltenverbund, Vienna, Austria).
ACKNOWLEDGMENTS
Financial support. This work was performed for TATFAR as part of the routine work of the ECDC and the CDC. ECDC provided funding for participants to attend the Expert Consensus Meeting that took place on June 18, 2014, in Stockholm, Sweden.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the European Centre for Disease Prevention and Control, US Centers for Disease Control and Prevention, or US Department of Veterans Affairs.
