Order sets are groups of medical orders that work to standardize diagnosis and treatment following pre-established clinical guidelines or protocols. A typical order set would include orders for medications, therapies, and/or diagnostic tests for a specific situation (e.g., general admission) or a specific patient condition (e.g., asthma, diabetes). Examples of order sets are provided in Supplementary Figures 1 to 3, which can be viewed online at www.journals.cambridge.org/thc2012021). Physicians would prescribe by selecting orders from the list of relevant orders on the order set, instead of writing or entering the orders manually one by one. Order sets have been used in hospitals since the 1980s (Reference Thorn, Sexton and Semav23), and there is a general consensus that well-designed order sets are beneficial to both providers and patients (Reference Bobb, Payne and Gross3;Reference Rosenal, Fantin and Sinnarajah20). Many believe that grouping relevant medical orders together can make ordering more efficient, decrease variation in care and enhance compliance with treatment guidelines (Reference Bobb, Payne and Gross3;Reference Rosenal, Fantin and Sinnarajah20).
Order sets commonly exist in three formats: paper-based, standalone electronic, and computerized physician order entry (CPOE)-embedded. Paper-based order sets are typically a preprinted form with a list of orderable items (e.g., diagnostic tests, medications, therapies). While they are simple and straightforward to use, it is difficult to ensure that supplies of the order forms are always conveniently available in all patient care areas and that the forms are of the latest version (Reference Bobb, Payne and Gross3). Standalone electronic order sets allow users to fill-in a Web-based form on the computer, and thus may be more accessible to clinicians. However, they often do not have the capability to communicate with other electronic systems. Order sets can also be embedded in CPOE systems. CPOE-embedded order sets provide the added benefits of order checking and electronic communication of orders to ancillary services. However, they are not always accepted by clinicians due to usability issues (Reference Campbell, Sittig and Ash5). Moreover, a major challenge exists with all three formats of order sets to keep them up-to-date and to monitor them for compliance with ever changing practice guidelines. If inadequately maintained, they can become “templates for efficiently practicing outdated medicine on a widespread basis” (Reference Bobb, Payne and Gross3).
Because there were no published systematic reviews of the potential benefits and shortcomings of standardized order sets, the authors responded to a request from the Ontario Health Technology Advisory Committee (OHTAC) to conduct this review (13). This review aimed to investigate the impact of order sets implementation on guideline adherence, processes of care, diagnosis/treatment outcomes, user efficiency, and costs. The secondary aim of this review was to examine strategies to enable hospitals to derive optimal benefit from the development and implementation of order sets.
METHOD
Literature Search
A comprehensive literature search was performed in OVID MEDLINE, MEDLINE In-Process and Other Non-Indexed Citations, EMBASE, CINAHL, The Cochrane Library for studies published between January 1, 1990, and April 18, 2009. Keywords such as order set, preprinted order, preselected order, standard order, order form, order sheet, etc., along with topic headings were used. In addition, bibliographies of relevant papers were searched for additional references that may have been missed. Abstracts were screened and the full text of studies meeting inclusion criteria were obtained. Studies were then assessed by two independent reviewers. Where there were disagreements over study inclusion, the reviewers discussed until an agreement was reached. A search was also conducted for publications in the gray literature, including the Gartner reports and studies registered in clinicaltrial.gov.
Inclusion and Exclusion Criteria
We included studies that evaluated order sets as the primary intervention for hospitalized patients and reported impacts on guideline adherence, processes of care, diagnosis/treatment outcomes, user efficiency, or costs. Studies that used order sets as the only tool to implement a new guideline or clinical pathway were included, provided that the primary focus was on the order set. We excluded studies in which order sets represented just one component of a multifaceted intervention (e.g., with educational interventions, other changes to clinical charts or clinician work flow) because it would not be possible to evaluate the individual impact of order sets on patient outcomes. We also excluded studies that did not report a sample size or statistical analysis. Eligible study designs consisted of randomized controlled trials, nonrandomized controlled trials, and observational studies, including simple before-after studies. We excluded studies with no predefined control group (e.g., if cases in which physicians had chosen not to use the order sets were counted as controls). We restricted our focus to studies published in the past 20 years to emphasize relevance to contemporary practice and technology. Furthermore, only English-language studies were considered.
Quality Assessment
The quality of each individual study was assessed with criteria based on key factors that were relevant to this review, as well as the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) checklist (25) because a majority of the included studies were observational studies. Examples of these quality factors included adoption rate of the order set, study design, sample size, and difference between the before and after group. Two evaluators assessed each included study independently and discussed their findings until an agreement was reached. The overall quality of the body of evidence was graded according to the GRADE Working Group criteria (Reference Guyatt, Oxman and Kunz12). The consistency, directness, and precision of evidence, as well as any study limitations and publication bias were evaluated to assign a quality rating.
Synthesis of Findings
The outcome measures of the included studies were summarized qualitatively. A meta-analysis was not possible due to heterogeneity in study populations, intervention type and outcome measures. Aside from determining the effectiveness of order sets, common challenges and strategies for the implementation of order sets were derived from the included studies.
RESULTS
Results of the literature search are presented in Figure 1. Eighteen studies met inclusion criteria and were included in this review (Reference Asaro, Sheldahl and Char1;Reference Biviano, Rabbani and Paultre2;4;Reference Chapman, Curtas and Meguid6–Reference Garrelts, Meister and Bond11;Reference Hughes, Smith and Tecklenburg14;Reference Koplan, Regan and Goldszer15;Reference Miller, North and Anderson17–Reference O'Connor, Adhikari and DeCaire19;Reference Theilen, Gritzke and Knutsen21;Reference Thiel, Asghar and Micek22;Reference Treece, Engelberg and Crowley24;Reference Webb, Kuykendall and Zeiger26). It should be noted that the study by Micek et al. (Reference Micek, Roubinian and Heuring16) was initially identified as a eligible study, however, it was found to be a pilot study of that conducted by Thiel et al. (Reference Thiel, Asghar and Micek22) with the same order set. Hence, this study was used to supplement the findings of Thiel et al., but was not included as a separate study in this review.
Search results of literature review.
The included studies were published between 1990 and 2009, with fourteen of them published after the year 2000. The study design, setting, population, intervention characteristics, adoption rate of the intervention, and study limitations of the included studies are summarized in Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012022).
Descriptive Findings
Summary of Study Design
Of the eighteen studies, sixteen were observational before-after studies with a historical control group. One study was an observational before-after study with both a historical control group from the same site and a control group from other hospitals (Reference Debrix, Tilleul and Milleron9).
The controlled trial by Noschese et al. (Reference Noschese, Donihi and Koerbel18) was the only study with a concurrent control group. However, due to physician crossover between the intervention and control group, the order set was also used in some of the patients in the control group, thus limiting study conclusions.
The studies conducted by Elsasser et al. (Reference Elsasser, Goodman and Destache10), Chisolm et al. (Reference Chisolm, McAlearney and Veneris7), and Biviano et al. (Reference Biviano, Rabbani and Paultre2), did not compare all patients in the intervention group with the historical controls. Rather, the intervention group was retrospectively divided into a group with the order set used, and a group with no order set used. Comparison was made between the intervention group with order set used and the historical controls.
Please refer to Supplementary Table 1 for further details on study designs.
Summary of Study Population
Combining participants from control and intervention groups, the study sample sizes ranged from 52 to 34,554 (median, 244), and altogether 44,529 patients were involved. The conditions targeted by the order sets included cancer (Reference Debrix, Tilleul and Milleron9), diabetes (Reference Noschese, Donihi and Koerbel18;Reference Theilen, Gritzke and Knutsen21), acute coronary disease (Reference Asaro, Sheldahl and Char1;8;Reference Koplan, Regan and Goldszer15), ischemic stroke (4), and chest pain (Reference Biviano, Rabbani and Paultre2), pneumonia (Reference Koplan, Regan and Goldszer15;Reference O'Connor, Adhikari and DeCaire19), chronic obstructive pulmonary disease (Reference O'Connor, Adhikari and DeCaire19), asthma and status asthmaticus (Reference Chisolm, McAlearney and Veneris7;Reference Webb, Kuykendall and Zeiger26), sepsis (Reference Thiel, Asghar and Micek22), febrile neutropenia (Reference O'Connor, Adhikari and DeCaire19), soft tissue infection (Reference O'Connor, Adhikari and DeCaire19), urinary tract infection (Reference O'Connor, Adhikari and DeCaire19), upper gastrointestinal bleeding (Reference O'Connor, Adhikari and DeCaire19), and anemia (Reference Garrelts, Meister and Bond11). Studies also involved patients requiring withdrawal of life support (Reference Treece, Engelberg and Crowley24), receiving mechanical ventilation (Reference Hughes, Smith and Tecklenburg14), or requiring enteral or parenteral nutritional support (Reference Chapman, Curtas and Meguid6;Reference Miller, North and Anderson17). One study used general order sets that targeted all patients (Reference O'Connor, Adhikari and DeCaire19). Most studies did not exclude patients based on age. One study included only adult patients (i.e., >18 years) (Reference Elsasser, Goodman and Destache10), and two only pediatric patients (Reference Chisolm, McAlearney and Veneris7;Reference Webb, Kuykendall and Zeiger26). Study settings varied from academic hospitals (Reference Chapman, Curtas and Meguid6;Reference Theilen, Gritzke and Knutsen21;Reference Thiel, Asghar and Micek22), community hospitals (Reference O'Connor, Adhikari and DeCaire19), to tertiary care centers (Reference Noschese, Donihi and Koerbel18;Reference Treece, Engelberg and Crowley24). Most studies were conducted in North America, and one was conducted in Europe (Reference Debrix, Tilleul and Milleron9). Please refer to Supplementary Table 1 for further details on study populations.
Summary of Intervention Characteristics
Most studies investigated paper-based order sets while two investigated CPOE-embedded order sets (Reference Chisolm, McAlearney and Veneris7;Reference Koplan, Regan and Goldszer15). None of the included studies investigated electronic stand-alone order sets. One study evaluated both the paper-based and CPOE-embedded versions of the order sets (Reference Asaro, Sheldahl and Char1).
Only two studies clearly stated that the order set was applied to all patients in the intervention arm (Reference Debrix, Tilleul and Milleron9;Reference Theilen, Gritzke and Knutsen21). In most studies, order set usage was optional and monitored, but several of the studies neither mandated nor measured adoption (4;Reference Chapman, Curtas and Meguid6;8;Reference Hughes, Smith and Tecklenburg14;Reference Miller, North and Anderson17;Reference Thiel, Asghar and Micek22).
Please refer to Supplementary Table 1 for further details on intervention characteristics.
Summary of Control Characteristics
With the exception of Asaro et al. (Reference Asaro, Sheldahl and Char1), all of the control groups consisted of patients cared for without order sets. Asaro et al. evaluated the paper-based and CPOE-embedded versions consecutively.
Summary of Outcome Characteristics
The outcome measures of the included studies varied considerably. The only commonly reported outcome was the rate of adherence to guideline. Other measures included treatment outcomes, changes in processes of care, cost, and efficiency. Because of variability in populations and patient conditions, comparisons across studies are not valid.
Main Findings
The outcomes measured and main findings of each study are shown in Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2012022. While most of the studies showed positive outcomes from the use of order sets, two studies showed mixed outcomes (Reference Noschese, Donihi and Koerbel18;Reference O'Connor, Adhikari and DeCaire19). The following sections detail the findings from studies in terms of (i) adherence to guidelines, (ii) other outcomes, and (iii) undesirable outcomes.
Adherence to Guidelines
An outcome measured by the studies was adherence to recommended guidelines. Thiel et al. (Reference Thiel, Asghar and Micek22) reported a significant increase in sepsis patients who received appropriate initial antibiotic therapy. Garrelts et al. (Reference Garrelts, Meister and Bond11) found a significant increase in the guideline-supported uses of epoetin alfa, but no significant difference in patients receiving the recommended dosage. The study conducted by the California Acute Stroke Pilot Registry (4) found a significant increase in patients receiving optimal treatment for ischemic stroke. Debrix et al. (Reference Debrix, Tilleul and Milleron9) reported a significance increase in proportions of prescriptions in compliance with guideline for patients receiving colony-stimulating factors as part of cancer treatment. Webb et al. (Reference Webb, Kuykendall and Zeiger26) also reported significantly more patients being treated with various elements of their status asthmaticus protocol.
Two other studies compared historical controls with the intervention group with order set used and found positive effects on guideline adherence as well (Reference Biviano, Rabbani and Paultre2;Reference Elsasser, Goodman and Destache10); however, the impact of any outcomes reported by these two studies may be inflated because they did not compare the control group with the overall intervention group (i.e., whether the order sets were used on all patients or not). These outcomes are indicated with a superscript “b” in Supplementary Table 2.
On the other hand, Noschese et al. (Reference Noschese, Donihi and Koerbel18) did not find any significant difference in the portion of orders that are appropriate between the control and intervention groups of diabetic patients. Asaro et al. (Reference Asaro, Sheldahl and Char1), whose study evaluated both a paper-based version and CPOE-embedded version of an acute coronary syndrome order set, also did not report any improvement in compliance with guidelines for either order set. However, both of these studies were conducted with serious limitations such as physician crossover (i.e., the order set was used in some of the patients in the control group) and not comparing sample characteristics between the control and intervention groups.
In general, the studies suggested that the use of order sets could have a positive effect on the adherence to recommended guidelines. There were no important inconsistencies between the results reported by studies involving the type of order sets, whether they be paper-based or CPOE-embedded.
Other Outcomes
Four studies reported significant improvements in treatment outcomes with the use of order sets (Reference Chapman, Curtas and Meguid6;Reference Theilen, Gritzke and Knutsen21;Reference Thiel, Asghar and Micek22). Thiel et al. (Reference Thiel, Asghar and Micek22) reported a significant reduction in in-hospital mortality rate with the use of a bacteremic severe sepsis order set, from 55.0 percent in the before group (n = 200) to 39.5 percent in the after group (n = 200; p < .01). Although patients in the before group were significantly more ill, the authors showed that the reduction in hospital mortality was independent of the illness severity. Thiel et al. also reported a significant reduction in length of stay and occurrence of renal and cardiovascular failures.
Another study by Theilen et al. (Reference Theilen, Gritzke and Knutsen21) reported a significant decrease in mean blood glucose level and number of hypoglycemia incidences, and a significant increase in the number of patients that stayed within their target blood glucose range with the use of a glycemic control order set. Chapman et al. (Reference Chapman, Curtas and Meguid6) also found a significant decrease in the time for patients to meet their caloric goals after implementation of their nutrition order forms. In contrast to these positive results, four studies found no significant difference in patients’ treatment outcomes before and after the use of order sets (Reference Chisolm, McAlearney and Veneris7;8;Reference Miller, North and Anderson17;Reference Treece, Engelberg and Crowley24).
Eight studies also reported significant improvements in various processes of care (Reference Chisolm, McAlearney and Veneris7;Reference Elsasser, Goodman and Destache10;Reference Garrelts, Meister and Bond11;Reference Koplan, Regan and Goldszer15;Reference Noschese, Donihi and Koerbel18;Reference O'Connor, Adhikari and DeCaire19;Reference Thiel, Asghar and Micek22;Reference Treece, Engelberg and Crowley24), and one study found a significant decrease in the time taken to provide appropriate care to patients (Reference Thiel, Asghar and Micek22).
Two studies measured the economic impact of using order sets. Garrelts et al. reported a 37 percent reduction in the annual costs of a specific medication with the use of order sets (Reference Garrelts, Meister and Bond11). Hughes et al. also reported estimated savings that resulted from a decrease in resources used (Reference Hughes, Smith and Tecklenburg14). However, one study reported no significant changes in cost (Reference Chisolm, McAlearney and Veneris7).
Undesirable Outcomes
Only two studies reported any undesirable outcomes associated with the use of order sets. O'Connor et al. (Reference O'Connor, Adhikari and DeCaire19) found that order set use was associated with a significant decrease in orders that were completed with a date and an unintended increased in night-time sedation orders. Noschese et al. (Reference Noschese, Donihi and Koerbel18) reported a higher frequency of corticosteroid therapy orders, resulting in a significant increase in diabetic patients experiencing hyperglycemia on the intervention unit than the control unit.
Quality of Evidence
The overall GRADE quality of the body of evidence was graded as very low for all outcomes, as shown in Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2012022. Because most of the included studies were observational studies, the overall quality of the evidence would be considered low. However, most of the studies were found to have serious limitations, thus the overall quality was downgraded to very low. These limitations include not measuring adoption rate of the intervention, having a very small sample size, and not comparing the population characteristics of intervention and control group.
DISCUSSION
The eighteen studies included in this review generally suggested the use of order sets could positively influence guideline adherence, treatment outcomes, processes of care, efficiency and cost. However, the overall quality of the evidence was found to be very low based on the GRADE assessment.
While order sets were associated with very few negative outcomes, two studies demonstrated the consequences of not designing an order set carefully. Specifically, in the studies by O'Conner et al. (Reference O'Connor, Adhikari and DeCaire19) and Noschese et al. (Reference Noschese, Donihi and Koerbel18), reminders for night-time sedation or glycemic control on the order set led to an unintended increase of certain orders. Hence, careful consideration must be given when deciding what items should be included in an order set and how they should be presented to ensure its use would not result in a widespread of nonoptimal care.
The included studies also gave insights to the challenges and strategies of implementing order sets. The major challenge that many studies have reported is the lack of motivation from physicians to adopt this tool. As stated by Noschese et al. (Reference Noschese, Donihi and Koerbel18), “The use and success of [an] order set depends on medical staff being aware of it, understanding it and liking it enough to use it correctly and consistently.” Without support from physicians, it would be very difficult to realize the benefits of order sets. Based on the included studies, some of the key factors that can greatly affect physicians’ adoption of an order set include the quality of the order set, efficiency in using the order set, awareness of the order set, and accessibility of the order set.
To ensure that these four key factors are addressed, many of the included studies have used similar strategies in the development and implementation process. First, many studies involved the formation of a multidisciplinary team when developing and implementing an order set (Reference Asaro, Sheldahl and Char1;Reference Biviano, Rabbani and Paultre2;Reference Debrix, Tilleul and Milleron9;Reference Hughes, Smith and Tecklenburg14;Reference Noschese, Donihi and Koerbel18;Reference Thiel, Asghar and Micek22). These multidisciplinary teams typically included at the very least physicians, nurses, and pharmacists. By engaging different healthcare professionals, the needs and concerns that are unique to each group may be taken into consideration. Physicians are also more willing to support the order set if their peers are involved in the development and implementation process. Several studies also engaged physicians by establishing a feedback system to address issues concerning the order set (Reference Chisolm, McAlearney and Veneris7;Reference Noschese, Donihi and Koerbel18;Reference O'Connor, Adhikari and DeCaire19). This ensures that individual concerns from physicians can be heard, and more importantly, feedback from physicians can help improve the order set. Finally, several studies acknowledged that the implementation of order sets requires an on-going effort (Reference Hughes, Smith and Tecklenburg14;Reference O'Connor, Adhikari and DeCaire19). Because order sets are mostly based on existing evidence or best practice guidelines, they should be evaluated and updated regularly to include the most current evidence/guidelines. It is also essential to continually monitor the treatment outcomes and adoption rate associated with each order set to determine the sustainability of the improvements and the need for any modifications (Reference O'Connor, Adhikari and DeCaire19).
Successful implementation of order sets is not a simple task. The guidelines presented in this report serves as a high-level overview, and are certainly not exhaustive. Each institution should take measures to engage its staff in the development and implementation process, address key factors that affect their decisions to adopt order sets, customize order sets based on institutional needs and goals, put in place methods to ensure that their order sets reflect current evidence from the medical literature, and assess the extent to which implemented order sets are complied with in clinical practice.
Limitations and Strengths of Studies
The studies included in this review were greatly limited by their quality. Hence, based on the GRADE assessment, the overall quality of the evidence was very low. However, due to the nature of the intervention, it would be difficult to conduct true randomized controlled trials. Issues such as blinding would be very difficult to address. It should also be mentioned that in other areas such as aviation and the nuclear industry, the methodical use of checklists has been considered a standard safety measure, and in part, this general assumption about the intrinsic benefits of checklists, or order sets, explains why more rigorous studies have not been undertaken in the clinical setting. Nevertheless, while a true randomized controlled trial would be difficult to conduct, further research of higher quality (e.g., studies with larger sample size, studies with a concurrent controlled group, studies across multiple sites) would be beneficial to understand the use of order sets.
Limitations and Strengths of Review
Although a thorough search strategy was used, there may be some literature on order sets that was not identified for this review. Specifically, we did not examine all types of gray literature or any articles that were not published in English. Also, this review might not have identified studies of CPOE systems that included outcome measures related to the use of order sets (e.g., speed of order entry). However, a majority of these studies focused primarily on the use of CPOE systems. It would be difficult to assess the true effects of order sets that were not confounded by external factors (i.e., implementation of CPOE system).
In addition, due to the variations in study population, intervention and outcome measures, it is difficult to perform a meta-analysis and draw conclusions on the magnitude of each effect. Despite these limitations, this is the first literature review, to our knowledge, that examines the effectiveness of order sets.
CONCLUSION
Published studies generally suggested that order set implementation could result in positive outcomes in terms of guideline adherence, treatment outcomes, processes of care, efficiency and cost, but the studies were typically of low quality, with simple before-after designs, small study sizes and other methodological limitations. While the value of order sets remains intuitively plausible, the existing literature does not provide solid support for its benefits. Further investigations that formally evaluate the effectiveness of order sets would be highly valuable. As hospitals move toward a paperless environment, high quality studies on electronic order sets would be particularly beneficial. Future research that focuses on determining the types of order sets most likely to be of benefit, the outcomes most likely to improve, and predictors of greater adoption, would also be very informative.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1
Supplementary Figure 2
Supplementary Figure 3
www.journals.cambridge.org/thc2012021
Supplementary Table 1
Supplementary Table 2
Supplementary Table 3
CONTACT INFORMATION
Alvita J. Chan, MHSc, Healthcare Human Factors, University Health Network, Toronto, Canada Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, Canada
Julie Chan, MHSc, Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, Canada
Joseph A. Cafazzo, PhD, PEng Healthcare Human Factors, University Health Network, Toronto, Canada Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, Canada Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
Peter G. Rossos, MD, MBA, FRCP(C), FACPCentre for Global eHealth Innovation, University Health Network, Toronto, Canada Department of Medicine, University Health Network and University of Toronto, Toronto, Canada Tim Tripp, MLIS Centre for Global eHealth Innovation, University Health Network, Toronto, Canada
Kaveh Shojania, MD, Department of Medicine Sunnybrook Health Sciences Centre, Toronto, Canada Centre for Patient Safety, University of Toronto, Toronto, Canada
Tanya Khan, Medical Advisory Secretariat, Ontario Ministry of Health and Long-Term Care, Toronto, Canada
Anthony C. Easty, PhD, PEng, CCEHealth Technology Safety Research Team, University Health Network, Toronto, Canada, Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, Canada
CONFLICT OF INTEREST
All authors report their institution receives a grant from the Ministry of Health and Long-term Care, Ontario, Canada. Tanya Khan is an employee of the Ministry of Health and Long-term Care.
