1 Introduction
Politicians, practitioners, and the general public tend to regard – or at least portray – scientific research and technical data as means by which to resolve policy disputes. However, technical means cannot always, or perhaps even usually, resolve political controversies or value-based disagreements (Sarewitz, Reference Sarewitz2004). While the benefit-cost analysis (BCA) literature places great emphasis on the theory and methodology of BCA, the interactive – and often emergent – role of such analysis within the decision-making process is often neglected. In what follows, we use the case of flood control in the Chehalis River Basin of Washington State to explore how BCA can influence the decision-making process, and how, in turn, the decision-making process can come to shape BCA.
Our research begins with a simple question, “why has BCA failed to alleviate policy gridlock and controversy in the Chehalis Basin?” Using a historical analytic approach, we find that BCA increased rather than decreased controversy in the Chehalis Basin, and we provide suggestions as to how practitioners and policy-makers can avoid BCA-inspired controversy in the future. While BCAs are often viewed as neutral informational input regarding costs and benefits of policy alternatives, and are ideally conducted by objective technicians (Weimer & Vining, Reference Weimer and Vining2010), we find that in practice BCAs are highly contextual. That is, BCAs are generated out of and interpreted within a political, economic, or social context that influences both the counting of benefits and costs and the interpretation of methodology and results. We label this contextual version of BCA as “Bureaucratic BCA,” in keeping with Boardman, Vining and Waters (Reference Boardman, Vining and Waters1993), who describe the role that bureaucratic lenses play in shaping the results of BCA. We emphasize that the influence of bureaucratic lenses are not limited to the analysis stage, but influence the entire process, from the generation of project ideas through to the choice of a winning alternative.
In what follows, we demonstrate how the failure of BCA to alleviate policy controversy in the Chehalis Basin is not because of technical incompetency or badly done BCA, but rather because the textbook model of BCA quickly runs aground on the practical realities of time and resource limitations, institutional constraints, and legitimate value-based disputes that BCA itself cannot resolve. Accordingly, we identify current shortcomings in the development, presentation, and use of BCA that have significant implications for BCA practitioners and policy-makers. We do not, in making this observation, intend to detract from the value of BCA. Instead, we believe that BCA scholars and practitioners can enhance the value of BCA by paying attention to the processes leading to studies as well as the situations under which they will be interpreted. By acknowledging the existence of Bureaucratic BCA, and viewing it as a process in and of itself rather than a failure to implement the ideal BCA framework, we believe BCA can be used as a tool to alleviate conflict and improve public sector decisions. In this spirit, we conclude with specific recommendations for practitioners that may enable a more effective application of BCA in public policy decision-making, and ultimately more efficient public policy.
2 Rationale
BCA is an informational tool intended to foster economically efficient bureaucratic decision-making. In applied settings, benefit-cost reports and data are often used not by trained university economists or statisticians but instead by city council members, farmers, civil service members, and local businessmen. Broader and more diverse involvement introduces significant variability in terms of interpretation and application. Accordingly, even the best applied methods and most theoretically sound BCAs can be misapplied (Hahn & Sunstein, Reference Hahn and Sunstein2002). Whether due to the influence of analysts who are required to make judgments about how to code data and use them in models, institutional constraints on how or what can be analyzed, or simply due to a lack of understanding of the underlying economic and statistical theories, in practice BCA is used to serve myriad purposes and interests other than efficiency (from a welfare economic perspective).
Previous scholarship has emphasized this point. Boardman, Greenberg, Vining and Weimer (Reference Boardman, Greenberg, Vining and Weimer2011) posit that bureaucrats act as “analysts,” “spenders,” or “guardians,” and the perspective of the bureaucrat conducting a BCA can have tremendous impact on the outcomes of the analysis. Analysts, for example, emphasize objectivity and methodological rigor. Guardians try to limit spending and are wary of underestimates to cost and are likely to see possible benefits as costs. BCA for guardians is thus a check on potential actions. Spenders may systematically underestimate costs, and overestimate benefits. Spenders tend to embrace a “pull through effect,” in which BCA is used to justify a preferred project (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011). Yet, we contend that this bureaucratic lens of BCA does not end at the conclusion of the study, but instead extends into the decision process – thus shaping not only the conduct of BCA, but its use and interpretation. Theodore Porter argues that the role of BCA in decision-making processes has driven the development of BCA methodology and states that the “history of cost-benefit analysis in the United States bureaucracy from the 1920s until about 1960… is not a story of academic research, but of political pressure and administrative conflict” (Porter, Reference Porter1995, 149). Thus, the very formation of BCA practice is rooted not in efficiency but in bureaucratic goals.
Without question, methodology should remain a focus of benefit-cost scholars, as it is this theoretical rigor and advanced methodology that grant BCA its predictive power. However, methodological improvements that focus purely on efficiency and do not consider the interactive role of BCA within bureaucratic processes will likely have limited impact on policy decision-making. Because those conducting BCAs are usually different individuals than those charged with deciding whether or not to implement a project, the bureaucratic lens of the policy-maker(s) charged with assessing relevant information and making the final decision about project implementation are every bit as important as the methodological approach used in the BCA. This bureaucratic lens is not limited solely to how a bureaucrat may value benefits and costs and thus the conduct of BCA, but also how a bureaucrat may utilize a BCA to structure or influence the decision-making process itself. Thus, it is critical that those conducting BCAs understand the empirical value of their results and the role of BCA in broader policy processes.
Accordingly, Bureaucratic BCA does not refer to BCA conducted by bureaucrats, per say, but rather to the realities of conducting economic policy analysis in complex decision contexts. In other words, the issue is not that policy-makers and analysts are not performing adequately, but rather that the existing model for BCA use in practice does not perform adequately in the face of time and resource constraints, rancorous policy debates, and messy and incomplete data. Moreover, in practice, BCA inevitably serves to structure choice sets, make inherent value judgments, embody particular risk preferences, and justify the preferences of some stakeholders over others. Thus, BCA plays an interactive and even emergent role in policy deliberations rather than simply providing informational input. We advance the concept of Bureaucratic BCA to explicitly acknowledge this more complex reality. In the next section, we describe the case we use to explore Bureaucratic BCA.
3 Background: A problem in the Chehalis Basin
A historical view of the role of BCAs in the flood-reduction projects of Washington State’s Chehalis River Basin reveals that while BCA in the ideal ex-ante form is meant to provide for the “efficient allocation of society’s resources” (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011), the influence of “spenders” and “guardians” within local bureaucracy undermines the very decision-making power of the BCA process. In what follows, we examine how this phenomenon occurs, particularly emphasizing how the implementation and presentation of BCA relates contribute to this problem.
The Chehalis River is located in southwestern Washington State approximately eighty miles south of the city of Seattle and almost an equal distance north of Portland. It is the second-largest river basin in Washington State (trailing only the massive Columbia River). When the Chehalis River topped its banks on December 3rd, 2007, the social and economic toll (the flood caused an estimated$450 million in damages (Cowlitz-Wahkiakum Council Governments, 2009)) reinvigorated a century-long policy debate about how to keep the Chehalis River from inundating the towns of the basin. In the aftermath of the flood, Washington State and the federal government declared the basin a disaster area and pledged to provide funding to prevent a flood from occurring again (Postman, Reference Postman2007). However, just two years later, the river again overtopped its banks, flooding the same areas and again causing significant damage. The aftermath saw the reinvigoration of flood control debates in the Chehalis Basin and resumption of BCA studies – continuing a trend of debate and study that has existed for the latter half of the twentieth century.
The Chehalis Basin is a “typical case” (Seawright & Gerring, Reference Seawright and Gerring2008), in that it is not unlike many river basins or other regions in which BCA has been a key component and driver of decision-making. In other words, the Chehalis Basin is exemplary, not extraordinary. We use the history of BCA development and application in the Chehalis Basin as a lens for understanding how BCA can play an interactive role in policy controversies. The story of the Chehalis Basin demonstrates key implications regarding Bureaucratic BCA. By considering the role that BCA has played in policy deliberations within the basis, we develop three practical tenets for practitioners and policy-makers in order to make better use of BCA in policy decision-making.
4 Bureaucratic BCA in the Basin
4.1 Institutional purposes vs. policy goals
In 1927, motivated in part by the Mississippi Flood (Barry, Reference Barry2007), Congress passed a new Rivers and Harbor Act that tasked the Army Corps of Engineers (ACE) with surveying US rivers for the development of hydropower, irrigation, and flood control infrastructure, a transition from the navigation-oriented development of prior years. This led to the publishing of the 308 (named after the bill, not the number of completed assessments) reports inventorying rivers nationwide in terms of the benefits and costs associated with building dams and other waterway infrastructure. The report on the Chehalis was published by the ACE in 1931 and concluded that “further improvement of the Chehalis River for navigation… was not justified” (Hurley, Reference Hurley1931). According to the report, the Chehalis Basin was “not well adapted to the development of waterpower.” However, the 1931 report also concluded that, “there are no important flood problems in the basin” (Hurley, Reference Hurley1931, 6).
In 1933, a large flood in the Chehalis basin prompted Congress to authorize a new examination of flood control possibilities that which was released eleven years later, in 1944 (Anderson, Reference Anderson1948). In the interim, the Flood Control Act of 1936 defined the Corps ACE’s mandate to build flood control structures, dredge rivers, and create new floodways. Other methods of flood control were put outside ACE’s jurisdiction (Cech, Reference Cech2009). As a result, the ACE investigated four dam sites in the upper Chehalis Basin, and originally settled on two for potential sites. However, the report showed that project costs were “153 percent of estimated annual benefits, including liberal allowances for the value of such storage for irrigation development and abatement of pollution” (Secretary of War, 1944, 30). There was no inclusion of alternative projects that may have helped to alleviate the flooding (an issue which we argue continues in current BCA as applied in the basin).
In this early use of BCA in the Chehalis Basin, we see how bureaucratic mandates and missions can result in a policy process that uses BCA to analyze a specific alternative rather than to efficiently address policy goals. The analysis of the Chehalis Basin was not about how to efficiently address flooding, but instead whether building a dam in the Basin would create social benefits that would outweigh construction and maintenance costs of the facility. Even before the 1936 Flood Control Act, when the ACE wrote that “there are no important flood problems in the basin,” what this meant was that flood problems were not significant enough to justify the massive expenditure of dam construction, dredging, or other large-scale waterway improvements (Hurley, Reference Hurley1931). Of course, there are many other alternatives for flood mitigation that do not involve massive infrastructure investments, such as floodplain management and zoning practices, building codes, and protection of natural flood mitigation resources such as wetlands and riparian areas. However, BCA in this case occurred within a specific agency context that was not able to take the holistic, social welfare economic approach that is typically associated with ideal BCA.
Proposition 1. Bureaucratic BCA focuses on the efficiency of a specific alternative (or limited set of alternatives), rather than how to efficiently achieve a policy goal.
4.2 What gets measured gets counted
The 1944 report also emphasizes difficulties in the quantification of values in BCA. Although almost 80 years ago, this reflects a fact that continues in the present. Given limited resources, benefits and costs will tend to be counted in accordance to how easy they are to count. The ACE BCA of 1944 focussed solely on changes to the agricultural capabilities of the Chehalis Basin. For example, the 1944 BCA classified damages based on damages accrued by farms. The Corps recognized that the value of the region was chiefly agricultural and thus improving the river’s flood control features would not necessarily improve its function as an agricultural region, though it ignored the fact the makeup of the valley might not remain this way. The valuation of damages also did not consider costs due to “loss of business to merchants, delays in transportation, increased shoaling of navigation channels” because there was no adequate data on such topics (Secretary of War, 1944, 2). Theoretically, inclusion of costs and benefits should be based on economic standing and significance. In practice, however, we find in this and other Chehalis BCAs that data availability governs inclusion. Agricultural benefits were relatively easy for the ACE to estimate, as damages to crops, implements, and infrastructure are readily monetized. For instance, crop prices are easily available.
To be certain, valuation techniques in modern BCA have advanced far beyond what was done in 1944. There are many advanced techniques available for analyzing hard-to-count environmental goods and services (Lipton et al., Reference Lipton, Lew, Wallmo, Wiley and Dvarskas2015), and the ACE, in particular, is highly advanced in the use of nonmonetary valuation and other techniques (Zerbe et al., Reference Zerbe, Davis, Garland, Scott, Scott Farrow and Zerbe2013). However, while the development of advanced valuation techniques pushes the envelope of what – and how – goods and services are counted, it does not solve the inherent problem at the heart of this issue.
A theoretical rule-of-thumb is that information for BCA should be subject to its own benefit-cost test, in that resource expenditure toward data collection, estimate, and analysis regarding a particular flow of benefits or costs should be commensurate to the “value” of said information (see Zerbe et al., Reference Zerbe, Davis, Garland, Scott, Scott Farrow and Zerbe2013). Obviously, this rule is not and cannot be strictly adhered to in practice, but we generally assume that an informal approximation of this standard is followed. In the Chehalis example, however, we observe that the ACE counted what was readily available. Giving priority to already available metrics occurred in 1944, but also in 2013. Because there has been considerable attention paid to the economic valuation of Pacific salmon stocks in the in light of Endangered Species Act listings, hydropower facility operations, and water resource pressures, analysts have access to tools and techniques for valuing salmon stocks (Loomis, Reference Loomis, Fisher and Nijkamp2014; Keeler, Reference Keeler2016). Accordingly, BCAs used within the 2013 deliberations incorporated the estimated costs and benefits associated with how particular project alternatives would affect salmon stocks in the Chehalis River, but did not incorporate other costs and benefits associated with ecosystem goods and services provided by a free-flowing river (EES Consulting, 2011).
Again, the point here is not that analysts did a poor job; rather, in practice BCAs are always subject to time and resource limitations, and subject to bounded rationality (as opposed to truly being able to subject all information to its own benefit-cost test). Moreover, analysts do regularly incorporate qualitative benefits and costs into reports when they are difficult or prohibitive to measure and there are established best practices for doing so (Boardman, Greenberg, Vining & Weimer, Reference Boardman, Greenberg, Vining and Weimer2006; Farrow & Zerbe, Reference Farrow and Zerbe2013). However, in real-world policy deliberations, quantified costs and benefits can crowd out qualitative costs and benefits because these “hard” numbers are readily incorporated into summary metrics (Zerbe & Scott, Reference Zerbe, Scott, Fantuzzo and Culhane2015). Inevitably some costs and benefits – whether monetary, nonmonetary, or moral goods – will be left out of a quantitative assessment, and this complicates the discussion and use of BCA. This is not a fatal flaw of BCA, but rather an inherent reality that can become a lightning rod for controversy when BCA is rhetorically or structurally positioned as a final arbiter of the merits of each policy alternative.
Proposition 2. Bureaucratic BCA favors costs and benefits that are more readily countable and are not necessarily representative of economic standing or economically significant cost or benefit flows.
4.3 Serial processing of parallel projects
Returning to the progression of BCA in the Chehalis over the 20th century, later of studies and deliberations continually failed to give rise to a policy solution. A hallmark of these later studies is inclusion of projects other than dams. In 1982 the ACE published another BCA that reflected numerous advances over the prior decades in terms of BCA methodology, correcting some previously discussed problems. The 1982 report included both structural and nonstructural alternatives – adding nondam alternatives such as channel clearing, excavation, levees, land use regulation, relocation, insurance and, watershed management (Hintz, Reference Hintz1982, 11). After this broad evaluation, the Corps identified two viable alternatives: modifications to the existing Skookumchuck dam which were later found infeasible, and no action (Hintz, Reference Hintz1982, 11). Of the new dam sites considered, none had a benefit-cost ratio greater than 0.4 (Hintz, Reference Hintz1982, D2–2). While the dams did provide significant reduction in overbank flood protection, the report concluded that the dams would increase development pressures while causing a loss of habitat for fish and wildlife.
The return of “100-year” floods again in 1990 and 1996 prompted local civic leaders to again call for action (Poor, Lindquist & Wendt, Reference Poor, Lindquist and Wendt2008). In 1998, Lewis County requested that the ACE study be reopened yet again to see if additional measures could be added to increase flood protection (US Army Corps of Engineers, 2003). Concurrently, the Washington State Department of Ecology decided that collaboration among local governments was critical in establishing actual programs to fight flooding. Seeking better coordination, in 1998 local and regional governments in the basin signed an “Intergovernmental Agreement” forming the Chehalis Basin Council (Partnership, Chehalis Basin, 2009). The Chehalis Basin Council (now Partnership) intended “… to collaboratively pursue strategies within the Chehalis River Basin which include the key elements of flood reduction, fisheries, recreation, water quality and water quantity” (Chehalis Basin Partnership, 2008).
On contract from the Council, Tetra Tech and Triangle Associates conducted a multipurpose water storage assessment, which focussed not only on water storage options, but programmatic alternatives as well (Tetra Tech/KCM, Inc, and Triangle Associates, 2003). For storage opportunities, the report looked at both building new reservoirs as well as modifying old storage facilities. Due to the low probability of success and the amount of further study needed for any of the proposed sites, the group recommended that surface storage come under no further consideration as a flood control method. Instead, Tetra Tech and the Chehalis Basin Partnership recommended a suite of projects including wetland creation and riparian revegetation as structural projects, and beaver reintroduction, forest conservation, low-impact development, and the blocking of agricultural drainages as programmatic solutions (Tetra Tech/KCM, Inc, and Triangle Associates, 2003). Ultimately, very little came of these reports. In particular, the programmatic options such as beaver reintroduction or revegetation largely vanished from discussion in subsequent years.
A primary reason why smaller mitigation measures and various minor retention alternatives fell by the wayside was (and continues to be) a bureaucratic lens that takes a project-by-project approach to the use of BCA. One problem created by this failure to consider projects as an investment portfolio is that small policy measures or minor infrastructure improvements have little effect on issues such as flooding when considered in isolation. Whereas a dam has a very large marginal effect on flood control, restoring one acre of wetlands obviously has a very small marginal effect. On a project-by-project basis then, wetlands restoration is likely to be an economic “loser” no matter how cheaply it can be done. However, numerous small policy measures will have an additive or even synergistic effect that might achieve results that are in total comparable to that of a dam, possibly at less cost.
Adequately addressing such alternatives requires a focus on overall outcomes and potential project alternative portfolios rather than specific projects, since it might take numerous small projects to reach a threshold that would substantively reduce flooding. For instance, a portfolio of alternatives might involve restoring a few hundred acres of wetlands, replanting a couple miles of streamside vegetation, removing several floodway obstructions, and incentivizing farmers to temporarily store water on their fields during high water events. On their own, each project would effectively achieve nothing, but in concert the portfolio of projects might achieve benefits that outweigh the total cost of these smaller measures. Viewing projects on a strictly case-by-case basis effectively favors large projects, since large-scale projects are typically the only ones capable of generating a significant marginal effect on their own. In this manner, Bureaucratic BCA echoes a more general struggle within the policy process toward serial information processing as an issue becomes particularly salient (Baumgartner, Jones & Mortensen, Reference Baumgartner, Jones and Mortensen2014; Jones, Reference Jones2003); because of increased issue salience and attention, there is an inevitable tendency to focus on a given major question (such as whether or not to build a dam) before proceeding to other policy issues.
Proposition 3. Bureaucratic BCA takes a project-by-project approach rather than a portfolio approach, rendering small projects that only work in concert with other alternatives to be nonviable.
4.4 Simple metrics and complicated realities
Following the devastating floods in the late 2000s, the state’s primary objective for flood projects was that they be “cost-effective” (Lewis County Public Utility District, 2009). This would presumably have ruled out a large-scale water retention facility given that such projects had already been considered several times by the ACE and had been rejected based on their lack of cost-effectiveness. Undeterred however, in post-2007 deliberations, basin stakeholders again pursued building a dam. In January of 2008, the Lewis County Public Utility District (PUD) released a concept paper titled, “Discussion of Chehalis Basin Flooding and Solutions” (Lewis County Public Utility District, 2008).
In their report, the Lewis County PUD wrote that “because of extensive damage to both public and private properties in the western Lewis County communities… it may be the time to revisit storage possibilities” with the purpose of not only providing flood control but also “domestic water supply, hydroelectric generation, and enhanced fisheries habitat” (Lewis County Public Utility District, 2008, 2). This report also stated that “levees have been historically deficient in protecting against flooding as they fail or overtop in major floods” (Lewis County Public Utility District, 2008, 1) and called for “a broader solution for reduced flooding for the entire Chehalis River Basin” (Lewis County Public Utility District, 2008, 2). The PUD believed such a project could bring multipurpose benefits, including domestic water supply, hydroelectric generation, and enhanced fish habitat. The PUD justified these claims with a PUD Phase I study concerning a retention facility presented to the Chehalis Basin Flood AuthorityFootnote 1 in which the benefit-cost results were described as “incredible” (Chehalis Basin Flood Authority, 2008, no. 1/19/2009).
The resulting debate and controversy regarding the Phase I study serves to illustrate an additional challenge to the ideal BCA paradigm posed by the use of BCA within local bureaucracies: the shortcomings of summary metrics. Contracted from EES Consulting, the Phase I study examined two potential water retention sites, one on the South Fork of the Chehalis and one on the mainstem (Lewis County Public Utility District, 2009). The combined benefit-cost ratio was put at 2.02 for the 30-year net-present value, and at 1.64 with leveled benefits. With a regional analysis, the ratio rose to 3.58. While the positive ratios appeared to merit further studies, a more careful reading of the report indicated that without a $300 million credit to the dams for not requiring the WSDOT to raise Interstate 5 (I-5), the projects failed to present adequate benefits to outweigh costs. Without the WSDOT project cancelation benefit, the dams would have had a B/C ratio far below the break-even point.Footnote 2 This was despite the fact that the dams were not predicted to protect I-5 without an estimated $100 million in levees and other flood mitigation projects on top of the project. By granting the dam such cost-avoidance benefits, the feasibility study gave itself monetary credit for a project that from a theoretical perspective should have been considered as an alternative, especially considering that protecting I-5 was the major goal of the project from the state’s perspective.
While a neutral assessment would likely conclude that the Phase I Study was as much an effort to engender support for the dam as it was to objectively evaluate the merits of the project, the more relevant issue for the current discussion is the intensive focus on summary metrics, particularly benefit-cost ratios. The intention of codifying summary metric criterion in various rules and regulations was to ensure that only economically efficient projects were selected. For instance, for many decades the ACE used benefit-cost ratios as their trigger for project approval: if a project had a benefit-cost ratio greater than one, then it was approved. The unintended effect of this well-intentioned approach, however, is to increase controversy, as project supporters count benefits to push the B/C ratio above one and detractors count costs to push the B/C ratio below one.
While the ACE itself has greatly revised this practice, a general focus on summary metrics remains quite strong within the bureaucratic use of BCA more generally. Simple “point-estimate” summary statistics offer a quick heuristic by which to judge projects. Accordingly, there is an inherent tendency for BCA as practiced within bureaucratic decision-making processes to focus on basic summary metrics (given time demands, attention limitations, and political constraints). Thus, rather than an informational tool for efficient policy decision-making, bureaucratic BCA becomes a battle to justify, or sink, a project due to its benefit-cost ratio.
Benefit-cost theory speaks to the need for probabilistic simulations, sensitivity analyses, and the presentation of ranges of outcomes in order to demonstrate potential project or policy outcomes (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011; Zerbe et al., Reference Zerbe, Davis, Garland, Scott, Scott Farrow and Zerbe2013). In practice, whether due to time and resource constraints or willful avoidance, public officials and political advocates simply do not do this. When a policy decision is driven by a simple summary metric, this is a recipe for controversy, since not everyone will agree on what should be counted as costs and benefits. While point estimates give policy-makers a highly limited view of uncertainty and assumptions (Merrifield, Reference Merrifield1997), they are easy to understand and lend cover for political decision-making.
Proposition 4. Attention constraints and political considerations incentivize reliance on basic summary metrics as a way to justify a desired policy or project within Bureaucratic BCA.
4.5 Data are measured, findings are constructed
While it is easy to think that BCA controversy is driven by “bad” analysis or politically motivated actors, this overlooks a more fundamental challenge of bureaucratic BCA. A technical, rational choice-oriented perspective of BCA tends to assume that, even if BCA does not foster consensus amongst all stakeholders about the correct course of action, it will nonetheless identify the most efficient policy alternative. This overlooks the extent to which policy knowledge is constructed not just by data but by values and preferences (Stone, Reference Stone2002). Rational, objectively minded individuals – let alone those with a political agenda – can view the same information and reach divergent but equally legitimate conclusions (Sarewitz, Reference Sarewitz2004). Given the natural tendency toward confirmation bias in the way individuals make judgments about and interpret information (Nickerson, Reference Nickerson1998; Gilovich, Griffin & Kahneman, Reference Gilovich, Griffin and Kahneman2002; Kahneman, Reference Kahneman2011), Bureaucratic BCA should inherently exhibit a strong correlation with the preferences and beliefs of those conducting or commissioning the analysis. More recent developments in the Chehalis Basin speak to the difficulties of this issue.
A month after the presentation of the Phase I Study, the Flood Authority voted to approve the funding of Phase II of the PUD study with a continuing contract to show the feasibility of a water retention project. Phase II was released in November 2009, with a period of public comments to follow. Phase II sparked much greater controversy, led to the authors being made aware of this case, and serves as yet another example of the role BCA can play in public policy debates.
Whereas large Federal agencies (e.g., the ACE) have the internal capacity to conduct BCAs, local government agencies typically lack sufficient resources and expertise to conduct an internal analysis. Thus, local or regional administrative bodies typically develop a project proposal, and then solicit a BCA to evaluate the said proposal. Following this model, the Flood Authority hired EES Consulting to conduct the feasibility study. Seeking to ensure that their entire budget was not spent on evaluating a project that could prove infeasible (thereby leaving no additional funding for other analysis), the contract structure divided Phase II into A and B subphases: Phase IIA paid EES $250,000 to conduct an engineering and structural study as well as broad economic analysis; EES would then receive a contract to complete Phase IIB for an additional $230,000 if no “fatal flaws” were discovered in Phase IIA (Chehalis Basin Flood Authority, 2009; Schwartz, Reference Schwartz2009; The Chronicle, 2009).
However, consider the resulting incentive structure: EES stood to receive the full $480,000 contract only if the initial analysis failed to show that the study should not go forward. Since the proposed retention facility would be a large earthen dam, one of the most prominent Phase IIA “fatal flaw” questions was where fill materials would come from. There was no definitive evidence that sufficient materials were available onsite, but there also was no definitive evidence that sufficient materials were not available either, and so analysts were required to make a judgment about the availability of suitable fill materials at the construction site on the basis of limited, incomplete evidence (see reports Phase IIA and Phase IIB). EES Consulting could have chosen to assume that materials would need to be transported to the construction site and added this cost to the estimate. Based upon the cost of similar actions in other dam construction projects, this likely would have resulted in projected costs greatly exceeding projected benefits. Instead, EES elected to assume that the materials would be available onsite. Since Phase IIA did not then identify any “fatal flaws,” Phase IIB soon followed. In light of this ambiguity, EES was certainly justified in assuming fill materials would be present. However, opponents met this judgment with contentions that materials would need to be transported to the site and that these costs should be incorporated into the BCA.
While it would be easy to conclude that project supporters simply contracted to get the result they desired, such a conclusion would be erroneous. Rather, this situation represents an endemic issue posed by bureaucratic BCA. The irony of bureaucratic BCA is that while BCA is conceptually regarded as causing belief convergence and fostering policy consensus, in practice it is often the source or at least the focal point of policy controversy. Bureaucratic BCA tends to find positive net benefits in the eyes of project supporters and negative net benefits in the eyes of project detractors – focussing controversy onto issues such as where material from a dam could come from as proxies for a debate about best flood control methods. Both sides can focus on areas where the BCA is potentially lacking their preferred assumptions or counting methods, leading to debates not about alternatives, but about the quality of the BCA.
This debate about quality of analysis was observed time and again in the case of the Chehalis Basin at a two-day Flood Authority summit in 2012 where the Phase I, IIA, and IIB benefit-cost studies dominated discussion and deliberation throughout both days. Arguments regarding Phases IIA and IIB split the Flood Authority between those who supported a dam and those who did not. This split not only hindered collaboration, but led to the beginnings of dissolution of the group. To be certain, this is not an occurrence unique to BCA; similar issues arise in numerous policy contexts. However, what makes bureaucratic BCA particularly interesting is that this occurrence is not necessarily due to purposefully self-interested behavior. Disagreements often arise not through willful misrepresentation but rather more subtle, systematic variations in the way different stakeholders compile and interpret available evidence. In other words, controversy arises even amongst analysts and policy-makers acting in good faith.
BCA is not simply a counting exercise. Public and private analysts are required to exercise informed judgment and discretion concerning what types of benefits and costs to include, to whom such cash flows should be attributed, and how benefits and costs should be counted. Given the role of confirmation bias, framing effects, and other human judgment and decision-making heuristics (e.g., see Kahneman, Reference Kahneman2011), it is not surprising that the results of bureaucratic BCA tend to strongly correlate with the sentiment of the group conducting or commission the analysis. Due to the uncertainty and discretion required for BCA, it is a prime example of how “science supplies contesting parties with their own bodies of relevant, legitimated facts… chosen in part because they help make sense of, and are made sensible by, particular interests and normative frameworks” (Sarewitz, Reference Sarewitz2004, 385).
Proposition 5. Bureaucratic BCA exhibits an inherent correlation between the preferences of those conducting or commissioning analysis and analysis results, even when analysts act in good faith and conclusions are supported by legitimate bodies of credible evidence.
5 Discussion
As the story of BCA in the Chehalis Basin illustrates, the application of BCA is generally much messier and more context dependent than the concept of ideal BCA would suggest. The overarching implication of Bureaucratic BCA refutes the common misconception about BCA as voiced by Louis Wolcher that:
“Once its numerical data on preferences is collected, CBA’s [BCA’s] calculations lead directly to administration, bypassing the processes that are constantly socializing and individuating the people whose lives will be affected by the decisions that administration makes” (Wolcher, Reference Wolcher2007, 106).
Bureaucratic BCA most certainly does not obviate political processes by providing “cold calculations.” Instead it falls fully within the realm of political debate. Policy scholars and analysts have long recognized professional roles for analysts that go far beyond conducting BCA (and other forms of policy analysis) as purely technical exercises (Jenkins-Smith, Reference Jenkins-Smith1982). After all, “science, for all its important contributions, does not deliver decisions” (Gregory, Failing, Ohlson & Mcdaniels, Reference Gregory, Failing, Ohlson and Mcdaniels2006, 718). In applied settings, however, this message is often lost (or ignored) and BCA is mislabeled as a purely scientific endeavor. When policy-makers and public officials ignore the critical subjective, political, and historic components of BCA, this serves to increase, rather than decrease, policy discord.
The authors first became aware of the ongoing saga of BCA in the Chehalis Basin because we were asked by the William D. Ruckelshaus CenterFootnote 3 to analyze the BCAs produced concerning various proposed alternatives for mitigating severe flooding in the Chehalis River basin. Our final product was a written report and an oral presentation given to the Flood Authority that emphasized strategies and tools that could: (1) better enable the Flood Authority and involved stakeholders to use BCA to make inter-project comparisons; (2) provide a framework for a coherent discussion of values and preferences; and (3) prevent BCA from becoming the focal point of policy controversy and thereby detract from its utility. All of our points relate to the implications of Bureaucratic BCA described above.
Ultimately, one might characterize our presentation to the Flood Authority as primarily related to the presentation of BCA. The presentation of BCA findings is critical to the useful application of BCA in policy-making. In many cases, the consequences of a policy decision result directly from the quality of methods used to display and assess quantitative data (Tufte & Weise Moeller, Reference Tufte and Weise Moeller1997). Whereas countless textbooks and methodological critiques concern the various theoretical assumptions and analytical techniques employed during the data analysis stage, the overarching implication of Bureaucratic BCA is that all of the methodological rigor in the world can go to waste when BCA encounters real people and real policy situations. In reviewing the history and implications of BCA in the Chehalis, we find that presentation of BCA is critical. In what follows, we present three overarching guidelines for Bureaucratic BCA that are variations of our recommendations given to the Flood Authority. These guidelines are intended to help practitioners and analysts carefully consider the manner in which they conduct and present BCA, in order that it can have a positive impact on policy making. They focus on the concepts of transparency, accessibility, and clarity (Tufte & Weise Moeller, Reference Tufte and Weise Moeller1997; Caulkins, Reference Caulkins2002).
5.1 Move beyond the summary metrics
While generating estimates of net benefits for different policy options is the fundamental output of BCA, the lesson of Bureaucratic BCA is that analysts should not overly emphasize benefit-cost ratios, net-present benefits, and other “point-estimate” summaries (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011). Without context, net benefit estimates lack meaning and can result in ill-informed decision-making. Each component of the analysis, from data gathering to modeling processes, potentially generates highly relevant policy information that deserves equal footing alongside model results in presentation. Summary metrics tend to crowd out other information in empirical policy debates and become the point of policy contestation, leading stakeholders to debate whether the true benefit-cost ratio is greater or less than one, rather than using BCA results as a basis for constructive deliberation.
While there is an understandable tendency for policy-makers to simply look for a single point estimate, BCA practitioners should resist this pressure to “just provide a number.” Information and findings garnered throughout the analysis, such as model distributions, sensitivity analyses, and current data deficiencies, provide context and depth to model outputs that greatly enhance the overall usefulness of BCA for policy decision-making. Caulkins (Reference Caulkins2002), Morgan, Henrion and Small (Reference Morgan, Henrion and Small1992) and Krupnick et al. (Reference Krupnick, Morgenstern, Batz, Nelson, Burtraw, Shih and McWilliams2006) and others provide guidance as to presenting probabilistic information to decision-makers. Similarly, qualitative effects must be purposefully presented so that they are not “lost” to the policy process. In Bureaucratic BCA, “hard” data tends to drive out “soft” data because they provide a concrete rhetorical reference for decision-makers. The presentation of BCA should strive to provide the best information possible to policy-makers regarding the risks, implications, and assumptions underlying estimates so that they can be fully understood and utilized; this includes whatever quantitative or qualitative information is relevant to the decision.
5.2 Provide a final product that is tractable and informative
The archival documents we analyzed for the Chehalis Basin demonstrate the bimodal tendency of Bureaucratic BCA, in that it seems to exist in either: (1) voluminous, largely undecipherable multivolume reports; (2) or oversimplified shorthand references (US Army Corps of Engineers, 2012; EES Consulting, 2011; Chehalis Basin Flood Authority, 2008, nos. 2009–2012). This is not a coincidence. The ACE’s 2012 Closeout Report on Chehalis River levees checked in at a 960 pages. Meanwhile, the public comment record, Flood Authority meeting transcripts, and other documents are rife with brief references to the Closeout Report that overly simplify its conclusions and draw no information into deliberation. While the report might serve to satisfy regulatory requirements or legal mandates, it most certainly does not, on its own, serve to enhance decision-making or substantively inform the policy debate.
To help address this problem, there should be a brief supplement for decision-makers describing analytical process, results, and implications. Although the actual report will, of course, be longer, especially given technical appendices, a project summary should also be composed that concisely details the analytical process, results, and implications. The overarching goal should be a document that gives those without economic training and with other jobs and responsibilities an opportunity to gain a comprehensive picture of the analysis and its findings. The summary should not oversimplify the analysis or overgeneralize findings, but it should be worded in nontechnical language that makes the report accessible to nonexperts and policy-makers lacking economic expertise.
A related point is that data and model information should be made available to facilitate replication, revision, and refinement of the analysis. Whether provided in a technical appendix, supporting documents, or made available digitally, every effort should be made to create a “living, breathing” model that allows policy-makers to explore alternatives and changes so that BCA is not simply a report to be discarded but a decision tool that can be used throughout the deliberation process. We observed in the Chehalis case that while stakeholders continued to argue about the results of the analyses, the reports themselves were essentially fixed; there was no effort or ability to use the information provided to gauge new ideas or examine the implications of questions posed by participants. Making the data and model publicly available would allow other analysts and interested parties to conduct their own sensitivity analysis, “what if ” analysis, or Monte Carlo analysis. For nontechnical users, one method that can help demonstrate the implications of analysis decisions and communicate policy risks is to create interactive models that are accessible to policy-makers (Cook, Reference Cook, Scott Farrow and Zerbe2013). Models may be designed so that policy-makers may change point estimates of parameters to observe the impact on the BCA outcome. This method can be used to improve communication of estimates with individuals less familiar with probabilistic statistical modeling techniques.
5.3 Do not provide false certainty
The lack of a fully comprehensive, let alone perfect, analysis should not prevent completion of a useful analysis (Farrow & Viscusi, Reference Farrow and Viscusi2011). However, giving overly precise estimates obscures more complicated realities, namely the fact that a BCA might be the best possible prediction of future events but remains only a prediction nonetheless. At Flood Authority meetings, in public comments, and newspaper editorials, stakeholders discuss alternatives as if they are certain: “The dam will actually help fish,” or “the dam will cost $
$\times$
amount.” EES’s Phase IIB report provided a construction cost sensitivity analysis, estimating low, medium, and high potential construction costs which then correspond to high, medium, and low net benefit estimates. However, no other sources of uncertainty were considered. The ACE’s Draft Close-out analysis on Chehalis River levees gives only a single benefit and cost estimate (US Army Corps of Engineers, 2012, 16).
The treatment of uncertainty (or perhaps more accurately, lack of treatment) in the Chehalis-related analyses is especially troubling given the high variability associated with flood events. The analyses we examined use an expected-value framework for valuing uncertain outcomes. This method is widely used, since it offers a tractable means by which to develop a single monetary estimate of the value of an uncertain payout.Footnote 4 However, not only is the year-to-year occurrence of flood events highly variable (as evidenced by the rash of severe floods in recent decades) but the nature of flood events themselves varies greatly as well. For instance, flood damages are dependent not just on the amount of precipitation, but also on the location of the precipitation and preceding conditions. Thus, the empirical outcome that we know with absolute certainty will never occur is that the benefits of flood prevention and mitigation will simply accrue as an even, yearly benefit flow.
Further, the expected-value framework effectively assumes dependence between years within a project’s lifespan by modeling outcomes such that a 10-year level flood, for instance, can possibly occur only five times in the life of a 50-year project. In reality, an event of this magnitude could occur any number of times within a 50-year period. Fixing the number and distribution of flood events projects a degree of certainty into project evaluation that does not adequately inform decision-makers about the likely economic outcome of flood mitigation or prevention policies. Given that policy-makers are not generally able to devote the time and attention to parsing the expected value in terms of these potential outcomes, using expected values simply causes this uncertainty to be lost in translation.
The most informative way of presenting benefit-cost uncertainty is to use a probabilistic analysis method (often referred to as Monte Carlo simulation).Footnote 5 Monte Carlo simulations should be the standard means by which to assess net policy benefits (Vining & Weimer, Reference Vining and Weimer2010). A Monte Carlo simulation entails specifying probability distributions for the various model parameters and then running the model a large number of times (typically 10,000). Each model “run” selects from the distribution of each parameter to get a specific value for use in that particular iteration of the model. The resultant distribution of model results can provide valuable information about overall model variability and sensitivity to changes in parameter values.
The value of a Monte Carlo simulation, as opposed to a sensitivity analysis, is that it provides significantly more information to policy-makers. Should a project actually be implemented, the accuracy of prior estimates is likely to prove highly varied. A Monte Carlo simulation treats each assumption and estimate as uncertain, and thus over the course of thousands of model iterations the simulation can develop a picture of policy outcomes as a function of this representative assortment of estimates. While this is not a new proposal, the lack of simple Monte Carlo-type simulations or other modern means of addressing uncertainty continues to hinder the role of BCA in bureaucratic decision-making processes.
6 Conclusion
Implementing each of these recommended guidelines requires significant effort; moreover, combatting the inherent tendencies of Bureaucratic BCA will not be easy. Nonetheless, we believe that if BCA is to be a force for efficient, effective policy-making, such steps simply must be taken. Despite its myriad critics, the theory and methodology of BCA is well-founded and “healthy.” As demonstrated by the case of the Chehalis Basin, however, the empirical application of these methods in local and regional policy-making is a far cry from this rigorous standard. Scholars and practitioners would do well to consider the implementation component of BCA just as strongly as they do the theoretical and methodological components. To do otherwise risks having even the most informative, rigorously conducted BCA be at best wasted and at worst contribute to, rather than alleviate, policy controversy.
