The objective of the Evidence-based Practice Center (EPC) program, created within the Agency for Healthcare Research and Quality (AHRQ), is to provide technical reports for clinical and public health policy-makers. The principal products of the EPCs are called “evidence reports” or “technology assessments.” These products are requested by organizations representing clinicians, policy-makers, and other decision-makers (“partners” in EPC parlance). The applications of these reports can be specific, such as providing source material for a professional society guideline, or more general, such as serving as background for future research funding. Furthermore, the EPC program provides resources to other evidence-based initiatives in the United States and abroad.
Given the impact of the EPC program, it is crucial to continuously assess and improve the program's processes. One such effort was initiated by the Duke EPC, focusing on the “statement of work” (SOW). As a contractual document, the SOW (or its equivalent for EPC projects, the “Task Order”) is the tangible manifestation of the objectives of each EPC project, serving as a foundation for interaction between involved parties, including technical details such as budget, timeline, and deliverables. Our focus on the SOW is based on evidence that the single most significant cause of project failure is the lack of a clearly defined and detailed SOW (13).
Preliminary to the current report, we pursued two formative activities. First, we interviewed participants in the EPC process to establish a deeper understanding of the current SOW process, focusing on the definition and determinants of a successful report. We also solicited ideas about solutions, including strategies for producing a more useful SOW. Second, we reviewed the technical contracting literature related to formulating SOWs for the production of policy reports in areas other than health care, as well as literature related to the determinants of a successful policy analysis. This step was intended to complement the interviews and focused on identifying existing concrete solutions to problems noted in the interviews. We particularly explored the strategies that had been used in government contracting (the National Aeronautics and Space Administration (NASA), and the Department of Defense (DOD) handbooks that describe the process of developing an SOW (5;17). Considering the relevance of creating reports that are significant for decision making in health policy and practice, this investigation into EPC methods should be important in several other non-EPC contexts in the United States and in the international arena.
THEORY OF CONSTRAINTS
The foundations of this project are the methods of the Theory of Constraints (TOC). TOC is a well-established, explicit, and reproducible means of identifying and addressing complex problems in a systematic and comprehensive manner (6). TOC is similar to traditional process improvement techniques, with the exception that it is designed to accommodate complex and non-linear processes involving multiple participants. TOC was developed to specifically address the fact that processes do not function in isolation but are part of a larger, intertwined system. Therefore, this methodology is especially suited to the present purpose because the EPC process is part of a larger system involving the partners' interests and the AHRQ's mission (9;15;18;19).
TOC starts with a focus on the range of undesirable effects (UDEs) of a complex system of processes. In general, one UDE is dominant; in the case of the EPCs, interviews suggested the most fundamental UDE in the final EPC product was not used in any demonstrable way. TOC suggests that addressing root causes by understanding any single process and addressing the root cause of failure of that one process does not result in sustainable solutions. TOC demands a much more involved investigation into all the processes involved in a system and all the UDEs that are produced. TOC then provides techniques to identify several potential root causes of these UDEs. TOC techniques force the strategic thinker to explore the one root cause that leads to most UDEs and recognizes it as the core problem. The core problem is labeled the “constraint” in TOC terminology. This constraint prevents an organization (e.g., an EPC) from reaching its goal (namely, developing a “useful” evidence report).
After identifying the constraint, TOC next focuses our attention on this core problem to devise solutions to it so that the UDE is transformed into a desirable effect, i.e., development of a “useful” evidence report. Because the TOC thinking strategy has considered other processes in a system that can affect the root causes of problems, its change strategies (termed “injections” in TOC parlance) can be considered in light of the range of possible effects. If potential injections survive this scrutiny, then they are much more likely to produce sustainable results.
TOC proposes that this entire, detailed investigation into UDEs, root causes, constraints, and injections can be best performed through five systematic exercises. Each of these exercises requires construction of corresponding logic “trees.” Most of these logic trees can be used as stand-alone tools, depending upon the nature of the questions under consideration.
Our goal in this project was to apply TOC to identify strategies for improving the EPC process through concrete changes in the SOW that would result in “useful” evidence reports. As detailed below, we used two of the five trees—the Current Reality Tree (CRT) and the Future Reality Tree (FRT) for this purpose.
METHODS
We accomplished our objectives in the following steps.
Step 1
(i) Identify all the UDEs in the SOW process and explore the relationship between them so a fundamental UDE could be identified. For this purpose, we synthesized the insights provided by interviews with representatives of the three parties participating in the SOW process (i.e., EPC researchers, AHRQ representatives, and partners).
(ii) List a set of root causes that led to them and investigated the relationship between these root causes and the UDEs identified in (i). The interviews and the literature search allowed us to explore the relationship as it applied to problems in the SOW process in non-EPC situations (4;5;7;10;14).
(iii) Develop a representation of the current SOW process, which was the draft CRT: To construct the CRT as related to the SOW process, we used a standard template similar in principle to the one depicted in Figure 1. It represents the UDEs associated with the production and use of evidence reports, as well as a series of hypothesized causal relationships depicted by the arrows and the blank rectangles. At the top of the tree is the most important UDE, the UDE that is fundamental to the entire process. At the bottom of the tree is the core problem, the constraint that leads to the fundamental UDE. The arrows represent the relationship between the constraint, other root causes, and UDEs.
Reality tree template.
(iv) Present the draft CRT in the 2004 EPC directors' meeting: At the 2004 annual EPC meeting, we presented the work from the interviews and the literature search that served as a basis for the current project, as well as the draft CRT. We solicited written feedback regarding the CRT, and the participants' opinion regarding promising change strategies.
(v) Finalize the CRT.
Step 2
(i) Explore, and list, possible change strategies, promising solutions to address the core constraint identified in Step 1. We reviewed the information collected in the interviews and literature review in the context of the TOC exercise to identify strategies applied successfully in non–health care situations (1–5;8;10–12;14;16). These change strategies are called “injections” in TOC terminology (i.e., what should the EPCs, partners, and/or the partners do to change existing reality so that the UDEs are transformed to desirable effects—DEs). We supplemented this list of solutions/injections from the feedback obtained during the 2004 EPC meeting,
(ii) Select the most promising strategy: This selection was accomplished by a brainstorming session.
(iii) Construct an FRT: This tree depicted a desired future, an illustration of the impact of the proposed solutions on the EPC's SOW process (what should the SOW process change to). For this purpose, we used a template of the FRT similar in principle to the one illustrated in Figure 1. The FRT is one of essential logic trees that TOC proposes. As the template illustrates, the UDE from the CRT is the desired effect (DE) of the FRT, and it results from one or more proposed injections.
RESULTS
Step 1
When we studied the results of the interviews, we discovered that there were three characteristics of an EPC report that were undesirable to all participants of the report (the UDEs of the system): (i) The report is late, (ii) The report does not meet the partners' needs, and (iii) The report is not used.
When we explored the relationship between these three UDEs, we concluded that, when a report was late or when it did not meet the partners' needs, it would not be used. Therefore, we concluded that: A report is not used is, among all the UDEs, deemed fundamentally undesirable. This effect is undesirable from all three perspectives: the partner has sunk cost on an ineffective report, the AHRQ has not met its mission of “providing policy-makers and other health care leaders with information needed to make critical health care decisions”. The EPC researchers thought that their report was “unsuccessful” when the partners failed to use the report and when they complained that the report did not meet their requirements. They also acknowledged that a continued trend of producing “unsuccessful” reports could threaten the financial viability of the EPCs.
When we investigated the causes of all these UDEs, it appeared that a late report and an unused report or one that did not meet the partners' needs resulted because of one or more of the following root causes: (i) Partner does not know how to conceptualize and articulate needs, objectives, and specifications required for the desired report (resulting in report does not meet the partners' needs); (ii) Specifications for a report are not initially sufficient (resulting in report does not meet the partners' needs); (iii) Specifications for a report change “mid-stream” (resulting in report does not meet the partners' needs and report is late); (iv) Partners' initial objectives regarding the final report are clarified late in the process (resulting in report does not meet the partners' needs); (v) Partners' needs change late in the process (resulting in report does not meet the partners' needs and report is late); (vi) Volume of anticipated work required for the completion of the report is initially underestimated by the EPCs (resulting in report does not meet the partners' needs), (vii) True cost and time requirement of the project are underestimated (resulting in report is late); (viii) Project cannot be accomplished within the timeframe allotted because new costs emerge later during the course of the project (results in report does not meet the partners' needs), and (ix) Budget and timelines cannot be revised (resulting in report does not meet the partners' needs and report is late).
We developed a draft CRT by laying down the relationship between the three listed UDEs themselves and subsequently between the UDEs and the nine root causes, so that one or two core constraints to the current SOW could be identified. This CRT represented the current SOW process.
We formulated the CRT (Figure 2) working from the fundamental UDE (report is not used by partner) at the top through proximate causes and, finally, root causes. For this purpose, we made two basic assumptions: (i) The partner had some potential use for a report for which they could articulate needs. This assumption precludes the possibility that a report is requested for no reason at all; and (ii) EPCs have expertise sufficient to produce a technically perfect report given adequate time, resources, and specifications. This assumption derives from the vetting process used by the AHRQ in selecting EPCs for the program.
Current reality tree for Evidence-Based Practice Center.
With these assumptions in mind, and from the list of root causes that led to the fundamental UDE, we explored the relationship between these root causes. We identified two major problems that lead to the rest and ultimately result in a report not meeting the partners' needs and, therefore, not being used (Figure 1). These root causes would fall in one of the two categories: (i) resource-related (budget or time) and (ii) specifications-related (stipulations of content or process for development of content).
Whether a resource issue relates to budget or to time schedule, process failure can result from underestimation of resource requirements at the outset of the project, or because unanticipated requirements emerge later. Given the assumption about the capabilities of the EPCs, the proximate cause is insufficiency of the initial specifications of a report.
The specifications can be insufficient because they do not provide an accurate guide to the ultimate volume or nature of work (e.g., number of studies to evaluate or specific techniques of synthesis to use) at the outset, or because the specifications change after the project is well under way. The latter problem is the result of partners stating objectives at the outset that do not actually reflect their needs, or the partners' needs change due to external forces (e.g., new science, new political, or other practical considerations). Insufficiency of initial specifications and scope creep are linked to the inability of the partners to conceptualize and articulate their needs, objectives, and specifications (referred to as the “conceptual framework”).
From this list and from the CRT, we identified four basic root causes for the three UDEs: (i) Partner does not know how to conceptualize and articulate needs, objectives, and specifications (the SOW), (ii) Partners' needs change late in the process, (iii) Insufficiency of specifications is never appreciated, (iv) Budget and timeline cannot be changed in response to down-line changes in project specifications.
Of these, the first, partner does not know how to conceptualize and articulate needs, objectives, and specifications, was rated as a core problem or the constraint that prevents an EPC from achieving its goal because it affects most of the other problems (this is the reason why specifications are insufficient in the first place, which leads to an underestimation of the volume of work, results in new cost and time requirements that lead to delays in the report, and finally results in the report not being used). If this one problem is solved, it is expected that most other issues can be resolved. [Partners' needs change late in the process is isolated from the rest of the CRT, because resolving this root cause is beyond the span of control of the EPC. TOC suggests that we eliminate it from our logical thinking process.]
The result of the CRT exercise was the identification of the core constraint, i.e., partner does not know how to conceptualize and articulate needs, objectives, and specifications. [Note that this was presumed not to be based on some cognitive deficit by the partner, but rather a lack of familiarity with the process of linking a preliminary request into a fully formed request, including specifications sufficient for a contractor to work with maximum efficiency.] The interviews, literature search, and the draft CRT were presented in the EPC directors' meeting. From the input during these meetings, we finalized the CRT (Figure 2).
Step 2
Our literature search allowed us to identify solutions that other industries had used to improve their SOW process. Our interviewees had also suggested some change strategies. In addition, our EPC directors provided us with some valuable insights. From all these resources, we identified several potential injections/change strategies that could address the root causes. These possible change strategies, and the root causes they could potentially address, are listed in Table 1.
Of all these, the two injections establish an ongoing relationship between partners, EPC, and AHRQ early in the process and use a changes clause were deemed to be the most practical means to exert impact on the SOW process by addressing the core constraint, partner does not know how to conceptualize and articulate needs, objectives and specs of the SOW.
With these concepts of UDEs, root causes, and injections that lead to converting a UDE to a DE, we constructed the FRT for the EPC-SOW process (Figure 3). The UDE from the CRT (e.g. “Report is not used”) is the DE of the FRT (“Report is used”), and it results from one or more proposed injections (e.g., “establish on ongoing relationship between partner, EPC, and AHRQ starting early on”).
Future reality tree for Evidence-Based Practice Center.
The FRT that we developed illustrates the following aspects of an improved SOW process for the EPCs: The injections (change strategies) lead to the ability of a partner to conceptualize and articulate needs, objectives, and specifications of the SOW (the core constraint is addressed by the injection). This strategy ultimately results in the production of a successful EPC report that meets partner needs and, therefore, will be used (i.e., we can effectively convert the UDE into its opposite, the DE).
DISCUSSION
EPCs are a relatively new resource for health care decision making. As with any complex enterprise, the EPCs need to engage in self-reflection and active improvement to maximize its value. In that spirit, the current effort draws on an approach used with success in other industries—the TOC—and led to a series of actionable suggestions.
TOC is a process-improvement technique especially suited for circumstances in which multiple partners are involved and their roles are complex and may change over time. The development of technical reports intended to inform decision making represents such a complicated area, and using the TOC permitted us to understand process failures in a systems context—how does one process failure lead to another. TOC also allowed us to identify key constraints that, through influence on several processes, prevented an EPC from producing reports that met the partners' needs.
Our results can be used by most EPCs and by any organization involved in the process of creating reports relevant for decision-makers. To develop a report that is intellectually satisfying to the EPC (or researcher), fulfills the mission of the AHRQ (or any other intermediary), and results in a report that meets the partners' needs, an effective SOW should be created. For an effective SOW to be created, two conditions appear to be important.
First, a close relationship between the participants in the process should be established early on and should be ongoing throughout the report development process. The relevance of this ongoing partnership is emphasized in non-EPC situations (4;5;8;13).
Second, a changes clause should be part of the contracting process. Once again, this clause has always been part of non-EPC contracting situations (10). These two solutions are solutions that AHRQ (or any intermediary) can implement without any significant change in current processes.
Another relevant strategy (but one that will require more legwork and entail budgetary changes) is the involvement of a knowledge transfer expert, one that has been successfully used in public policy making and decision making (1; 3;8;11;12;16). In fact, according to a 2003 survey of 265 directors of applied research organizations in Canada, 63 percent of organizations had research transfer staff, and an average of 14 percent of the organizational budget went to research transfer activities (12).
Our exercise has resulted in some actionable solutions for the complex task of developing reports that meet partners' needs and are intellectually satisfying for researchers as well. The TOC technique that we have used also illustrates a useful methodology by using a systems perspective for quality improvement or process improvement for any complex environment involving multiple partners.
POLICY IMPLICATIONS
Health care decision making in the context of formation of policies is a daunting task. Basing these decisions on evidence is clearly ideal. Therefore, improving the process of developing evidence-based technical reports that truly inform decisions assumes great importance for the EPCs as well as policy-makers. Using the TOC, we have explored the network of processes and have devised some ways in which EPC reports can be of greater value to policy-makers. Increasingly, health care policy makers are demanding scientific evidence in support of their decisions. The U.S. network of Evidence-Based Practice Centers are well-experienced with developing technical reports in service to this mandate. To enhance the quality and value of future reports, we applied the techniques of the Theory of Constraints to sort out the network of effects that interfere with an excellent and useful technical analysis and to identify practical strategies to overcome these constraints.
CONTACT INFORMATION
Meenal B. Patwardhan, MD, MHSA (Meenal.P@Duke.edu), Assistant Research Professor, Department of Medicine, Duke University Medical Center, Duke University, Durham, NC 27708; Director of Operations, Evidence-Based Practice Center, Duke Center for Clinical Health Policy Research, 2200 West Main Street, Suite 220, Durham, NC 27705
Antonio Sarría-Santamera, MD, PhD (asarria@isciii.es), Senior Health Services Researcher, Spanish Agency for Health Technology Assessment, Sinesio Delgado 4, 28029 Madrid, Spain
David B. Matchar, MD (match001@mc.duke.edu), Professor, Department of General Internal Medicine, Duke University Medical Center, Duke University, Durham, NC 27708; Director, Duke Center for Clinical Health Policy Research, Duke University Medical Center, 2200 West Main Street, Suite 220, Durham, NC 27705
The study received financial support under contract 290-02-0025 from the Agency for Healthcare Research and Quality (AHRQ). The authors of this article are responsible for its contents. No statement in this article should be construed as an official position of the Agency for Healthcare Research and Quality or of the U.S. Department of Health and Human Services.
