1 Introduction
On February 17, 1981 President Reagan signed Executive Order 12291 requiring agencies to conduct a benefit-cost analysis (BCA) of every major proposed regulation. Although the details of this order have evolved over the years, BCA has remained a crucial component of the federal policymaking process. The validity of a BCA hinges, in part, upon the ability of analysts to both identify and value every impact of the policy under consideration on individuals considered to have standing. The literature surrounding BCA has devoted significant attention to theoretical and empirical issues surrounding the valuation of policy impacts. Far less attention, however, has been given to the process of identifying the impacts to value.Footnote 1 Consequently, analysts often draw upon on the frameworks of values identified in the published literature, prior policy analyses, or on the accumulated wisdom from current policy actors. Absent such sources, the analyst may be left to rely on dialogue with stakeholders who are invested in specific policy alternatives, or categorization of statements collected from comment periods. Although all of these are important and useful practices, none are sufficiently systematic, transparent, and replicable to assure the inclusion of the full range of significant potential impacts, thus potentially limiting the value of the BCA to policymakers. Indeed, these approaches to identifying valued impacts have led some to argue that the utility of BCA – and policy analysis more generally – is fully realized only when multiple competing analyses that reflect the values of a wide range of affected stakeholder groups can be provided to decision-makers (Jenkins-Smith, Reference Jenkins-Smith1990). Such views may be understandable given the current political environment, but we contend that any well-executed BCA should aspire to be inclusive of all significant valued impacts, regardless of the degree of stakeholder engagement, political contention, or any other factor.
In this paper, we develop and apply a systematic approach for identifying relevant policy impacts for valuation in a BCA.Footnote 2 This approach involves drawing upon the universe of Congressional hearings on the policy topic of interest and analyzing the values being expressed in statements from each witness in the set of hearings. Hearings are one of the primary formal mechanisms by which information on policy impacts – communicated in the form of problems, concerns, and values – is communicated to legislators. While legislators may or may not act on the information provided in hearings, the content of testimony is a critical source of information on the impacts of a given policy or set of policies (Jones & Baumgartner, Reference Jones and Baumgartner2005). In using the content of hearings, our approach is designed to increase the likelihood that all impacts valued by policymakers, stakeholders, and the public at large are accounted for in the BCA.Footnote 3 Moreover, by using Congressional hearings as the basis for our approach to identifying potential policy impacts, we are using information provided through the very process that BCA is ultimately intended to inform (Workman, Reference Workman2015).
Although the hearings-based approach we propose holds the promise of transparently identifying a more comprehensive set of impacts for inclusion in BCAs, it is undeniably a resource-intensive endeavor and would be burdensome for agency personnel to implement, thereby increasing inefficiency in the generation of policy decisions. Thus, we believe the approach we propose may be best suited for initial application in the academic realm, where time constraints and procedural requirements are looser than in the formal regulatory process. The results of such academic applications – the identified set of potential impacts in a given policy area – could then be made available to agency personnel for potential inclusion in agency-conducted BCAs. Such a division of labor could reap the benefits of increased transparency in identifying the set of policy impacts for inclusion in a BCA without saddling agency personnel with an additional regulatory requirement. Of course, relying primarily on academic applications means that it will take time to develop a body of work covering the range of policy areas in which our approach might usefully be applied.
We demonstrate the utility of our hearings-based approach to identifying policy impacts through an application to proposals for altering operations of the Glen Canyon Dam (GCD). This application is based on Congressional hearings data from the U.S. Policy Agendas Project, which maintains a dataset containing information on every U.S. Congressional hearing, including an abstract from Congressional Information Services (CISs) and a major- and sub-topic classification that denotes the subject of the hearing (i.e., energy, water resources). From the universe of Congressional hearings, we identify 34 hearings relevant to proposals for changing operations of the GCD. In this set of hearings, 344 separate witnesses provided a total of 409 statements. We code all 25,916 sentences in each of the 409 statements using a systematic impact classification scheme, which we detail in Section 4 of the paper. Our analysis of the resulting data reveals 35 categories of potential impacts that should be considered for inclusion in a BCA. These conclusions differ from other studies of operational changes to the GCD, which exclude several of these potentially valued policy impacts from their analysis. We begin by outlining the theoretical considerations that necessitate – from a validity standpoint – comprehensive inclusion of policy impacts in a BCA. We then review the relatively small body of existing work that provides guidance on identifying potential policy impacts before outlining our approach that makes use of Congressional hearings. In outlining our approach, we compare it with techniques commonly used in existing analyses and discuss the conditions under which Congressional hearings can serve as a valid source of information about the full set of policy impacts. We then apply our approach to identifying potential policy impacts using a specific case: proposals to alter operations of the GCD on the Colorado River. In this application, we begin by providing background information on policies governing GCD operations before describing our techniques for identifying the set of Congressional hearings relevant to this topic and coding and analyzing those data. We then present the results of our analysis, discuss the implications for policy and practice, and highlight some of the key limitations of our approach.
2 Identifying potentially valued policy impacts for inclusion in BCA
The primary goal of BCA is to provide a valid, money-metric estimate of the net social benefits of a given policy or project. In pursuit of this goal, analysts must identify all potentially valued impacts – the potential costs and benefits – of the policy under consideration for individuals considered to have standing. Failure to identify the full set of valued impacts can undermine the validity of the BCA and result in a biased estimate of the net social benefits. To illustrate why this is the case we turn briefly to the conceptual foundations of BCA.
The concept of Pareto efficiency – and its connection to individual utility – serves as the theoretical foundation of BCA (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011). Consider a proposed project where it is known with certainty that, relative to the status quo, this project will have
$j$
impacts that will each affect
$n$
individuals. The effect of this proposed project on the utility of members of society with standing can be written as:
$$\begin{eqnarray}u^{1}-u^{0}=\mathop{\sum }_{i=1}^{n}f(p_{i1},p_{i2},\ldots ,p_{ij})\end{eqnarray}$$
where
$u^{1}$
and
$u^{0}$
represent utility levels under the proposed project and the status quo, respectively, and
$p_{ij}$
represents the effect of impact
$j$
of the proposed project on the change in an individual’s utility level. Now assume analyst only identifies a subset
$k$
of the
$j$
total impacts of the proposed project, where
$k<j$
. In this case, the measured change in societal utility is:
$$\begin{eqnarray}u_{m}^{1}-u_{m}^{0}=\mathop{\sum }_{i=1}^{n}f(p_{i1},p_{i2},\ldots ,p_{ik}).\end{eqnarray}$$
In cases where
$$\begin{eqnarray}\mathop{\sum }_{i=1}^{n}f(p_{i,j\notin k})\neq 0\end{eqnarray}$$
it follows that
$$\begin{eqnarray}u^{1}-u^{0}\neq u_{m}^{1}-u_{m}^{0}\end{eqnarray}$$
which states that the true change in societal utility that stems from the project does not equal the measured change in utility due to the omission of valued impacts from the analysis.
BCA is designed to estimate net social benefits in a money metric, rather than in utility, and uses the concepts of willingness to pay (WTP) or willingness to accept (WTA) as the basis for doing so – WTP is more commonly used in practice. In the context of the proposed policy discussed above, WTP is the income change required to maintain post-policy utility constant at the pre-policy (i.e., status quo) level. To the extent that measured impacts of a proposed policy diverge from the full set of impacts of that policy, true societal WTP for the proposed project will not equal measured WTP for that project, at least in cases where the analyst is unable to identify the full set of impacts of the proposed project and the omitted impacts have a nonzero effect on societal utility. When decisions are made on the basis of BCA – or at least influenced by them – it is possible that the omission of impacts may result in the approval of projects that have net social costs or, alternatively, the denial of projects that are socially beneficial. This puts a premium on analysts’ ability to identify the full set of policy impacts.
3 Existing guidance on identifying policy impacts
Given the clear importance of identifying all potential impacts of a policy for inclusion in a BCA, it would be reasonable to expect to find a significant literature providing guidance on this topic – a literature similar to that providing guidance on theoretical and empirical issues surrounding the measurement and estimation of identified policy impacts. However, a review of textbooks, government documents, and academic publications on the practice of BCA reveals that the field provides only minimal guidance on the topic.
Leading textbooks on BCA explicitly instruct practitioners to identify all policy impacts for inclusion in the analysis (e.g., Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011; Cellini & Kee, Reference Cellini and Kee2010) but offer little practical guidance for systematically doing so. Rather, they offer broad generalities such as “think broad and inclusively” (Cellini & Kee, Reference Cellini and Kee2010), or “identify a cause-and-effect relationship between some physical outcome of the project and the utility of human beings with standing” (Boardman et al., Reference Boardman, Greenberg, Vining and Weimer2011, p. 8). Other texts on the topic do not address issues of impact identification at all (e.g., Layard & Glaister, Reference Layard and Glaister1994; Campbell & Brown, Reference Campbell and Brown2003). Even the Center for Benefit-Cost Analysis’ report on principles and standards for the practice of BCA is written under the assumption that analysts will have little problem identifying all valued project impacts (Zerbe et al., Reference Zerbe, Davis, Garland and Scott2010).
BCA is often an important input to policy decisions, particularly in the regulatory realm. Unsurprisingly, then, several government documents provide agencies with guidance for conducting such analyses. Perhaps the most important of these documents is Circular A-4, which the Office of Management and Budget (OMB) published in September 2003 and provides guidance to federal agencies on the conduct of regulatory analysis – BCA is subsumed under the broader umbrella of regulatory analysis. This document discusses the importance of identifying all valued policy impacts – costs and benefits – and instructs agencies to explicitly discuss each of the impacts in the analysis, even if they cannot be valued monetarily. Instructions for how to go about ensuring that all impacts included in the analysis are limited to a suggestion to “seek out the opinions of those who will be affected by the regulation as well as the views of those individuals and organizations who may not be affected but have special knowledge or insight into the regulatory issues…Early consultation can be especially helpful” (U.S. Office of Management and Budget, 2003, p. 3). Consultation with knowledgeable stakeholders is undoubtedly an effective way to elicit potential policy impacts, but the approach – like interviews and focus groups more generally – can be difficult to design and implement in a systematic and replicable way.
A predecessor to OMB Circular A-4 – OMB Circular A-94 – also addresses issues of impact identification, stating that “both intangible and tangible benefits and costs should be recognized” and that “Analyses should include comprehensive estimates of the expected benefits and costs to society” (U.S. Office of Management and Budget, 1992). The document is silent, however, on outlining a process for ensuring that a comprehensive set of costs and benefits has been identified. Many agency-specific guidance documents – including those issued by the Departments of Justice, Housing and Urban Development, Transportation, and Health and Human Services, as well as the Environmental Protection Agency – are similarly structured, noting the importance of including all valued policy impacts in the regulatory analysis, but providing no meaningful guidance for doing so.
Another potential source of guidance for systematically identifying impacts for inclusion in a BCA is the academic literature, particularly peer-reviewed applications of the technique. To this end, we reviewed every article published in the Journal of Benefit-Cost Analysis since its inception in 2010, first determining whether the article qualified as an application of BCA. Of the 119 articles published since 2010, through a thorough reading of each abstract and methods section, we classified 24 as BCA applications. Among these 24 articles, 20 did not explicitly state the approach they used to identify policy impacts – only 4 articles engaged this issue in some form. These four articles stated that they identified policy impacts through reviews of the theoretical and empirical literatures on the topic and/or consultation with stakeholders (Danguy & van Pottelsberghe de la Potterie, Reference Danguy and van Pottelsberghe de la Potterie2011; Gillespie & Kragt, Reference Gillespie and Kragt2012; Liang & van Dijk, Reference Liang and van Dijk2012; Brännlund et al., Reference Brännlund, Carlén, Lundgren and Marklund2012).
Despite the lack of formal guidance for identifying policy impacts, perhaps practicing analysts – particularly those conducting BCAs for the federal government – clearly articulate their practices for identifying impacts for inclusion in a BCA. To gauge this possibility, we consulted OMB’s 2016 report to Congress on the benefits of costs of federal regulations (U.S. Office of Management and Budget 2016) and identified nine BCAs that are broadly representative of cabinet-level departments. We surveyed each of these nine BCAs, searching for a description of the approach used to identify policy impacts in the analysis. Of the nine BCAs reviewed, six made no mention of it, two focused on a single category of impacts, and one relied on a review of the currently available literature. These findings are reported below in Table 1. They make clear that, in both academic and public sector applications of BCA, a clear discussion of the approach and techniques used to identify policy impacts for inclusion in the analysis is the exception, not the rule.
Impact identification in major federal BCAs.
* The RIN, or Regulation Identifier Number is used on all relevant documents throughout the entire “lifecycle” of a rule to help members of the public find regulatory information at each stage of the process and promote informed participation.
In reaching this conclusion, we are not necessarily implying that these BCAs do not identify reasonable costs and benefits – or even the comprehensive set of them – for inclusion in the analysis. Rather, we are merely highlighting that the process analysts use to identify potential policy impacts receives little attention in most BCAs. However, examining the regulatory history of the BCAs in Table 1 makes clear that interest groups and other stakeholders routinely contend that the analyses ignore important policy impacts. Indeed, for nearly all the nine BCAs we reviewed, we found at least one instance in the regulatory history of a stakeholder group stating that the BCA supporting the proposed rule rests omits one or more important policy impacts, thus undermining the validity of the analysis. For example, United Pacific, which owns a chain of gas stations and convenience stories in the western U.S., commented on the BCA underlying the regulation requiring chain restaurants to post nutritional information on their menus. In their comment, United Pacific noted that the BCA accounts for neither the fact that calorie information would need to be posted in multiple places nor for the costs of replacing the information placards due to normal wear and tear. Similarly, in the case of the EPA’s rule regulating effluent water, the National Mining Association argued that the accompanying BCA fails to account for significant capital costs and that it does not recognize how the effluent water regulation interacts with other rules affecting the coal industry. Although these are just two examples, examination of the regulatory history of the other seven BCAs we reviewed revealed similar stakeholder concerns about the omission of relevant costs or benefits. Considered together, these regulatory comments suggest stakeholders are routinely concerned about BCAs failing to recognize the full set of policy impacts.
Critiques regarding the omission of policy impacts can be difficult for agencies to address. Given that the BCAs rarely contained a description of the approach used to identify policy impacts, it is difficult to defend the impacts selected for inclusion or exclusion when the method used to make those selections is not presented. More generally, Zerbe et al. (Reference Zerbe, Davis, Garland and Scott2010) identify transparency as one of five foundational principles of BCA, arguing that it facilitates a more comprehensive understanding of the analysis. In our context, a lack of transparency in identifying potential policy impacts renders it difficult for stakeholders to assess whether analysts are likely to have succeeded in identifying the full set of impacts. It also poses a difficulty for potential replication of the analysis. In the following section, we outline an approach to identifying potential policy impacts that is, in principle, systematic, transparent, and replicable.
4 Identifying potential policy impacts from Congressional hearings
We contend that descriptions of policy impacts articulated in formal legislative venues, which are designed, in part, to register public views on public policies, provide a basis for identifying impacts – as either benefits or costs – for inclusion in a BCA. Scholars of the public policy process have long noted that collective decision-making institutions like the U.S. Congress are explicitly designed to solicit and route information to decision-makers, most notably through the public hearing process (Simon, Reference Simon and Greenberger1971; Baumgartner & Jones, Reference Baumgartner and Jones1993). Indeed, the division of Congressional committee jurisdictions, often with considerable overlap, is served by committee hearings both as mechanisms for allocating legislative attention and as a substantive basis for informing legislators about societal concerns and options for addressing them.
4.1 Overview of Congressional hearings
In the U.S., the primary role of Congress is to develop and pass legislation to address public problems. As society has evolved, the problems facing Congress – and the agencies to whom Congress delegates authority – have become increasingly complex. Such complexity requires Congress and executive agencies to acquire significant knowledge and expertise to pass effective legislation (Baumgartner & Jones, Reference Baumgartner and Jones2015; Workman, Reference Workman2015). Over time, these institutions have developed several methods for acquiring and processing the requisite knowledge and expertise. Congressional hearings are arguably the most prominent and important of these methods for acquiring and processing policy-relevant information.
Although Congress has always had the authority to hold hearings to acquire information from those with policy expertise, it is only in the post-World War II era – a period coinciding with technological advances and increasing policy complexity – that hearings have been used as a primary means of information acquisition. Indeed, Congress has routinely held between 1000 and 2000 hearings annually over this period. Figure 1 presents a time series of the number of Congressional hearings held annually between 1946 and 2013. It demonstrates that the number of annual hearings rose from about 1000 in the early 1960s to around 2000 in the late 1970s and through much of the 1980s. The number of hearings has declined somewhat in recent years, but Congress regularly holds about 1500 annually.
Number of Congressional hearings, by year: 1946–2013.
Congressional committees – and their affiliated sub-committees – have significant discretion in the conduct of hearings. They have sole autonomy in determining the number of hearings the committee (or sub-committee) will hold, the topics of those hearings, the individuals who will testify at the hearings, and the format and order in which the witnesses testify. Despite this autonomy, there are important rules and norms that guide the hearings process, particularly with respect to witness selection. Congressional Research Service guides to the conduct of House and Senate hearings note that committees often select witnesses in a manner designed to ensure that a wide array of viewpoints are represented, although committees are free to choose witnesses that only represent selected points of view (Sachs, Reference Sachs2004; Carr, Reference Carr2006). Notably, though, both House and Senate rules permit the minority party to select witnesses for the hearing – this rule encourages at least some degree of diversity in witness viewpoints. Witnesses are generally drawn from diverse backgrounds and professions, and routinely include federal agency personnel, individuals from state and local governments, academia, the private sector, and advocacy organizations (Palmer, Reference Palmer2009). Think tanks are also well-represented, with research indicating that moderate organizations testify more frequently than their ideologically extreme peers (Lerner, Reference Lerner2018). Each witness selected to testify at a hearing typically submits a written statement in advance of the hearing, provides an abbreviated oral presentation of that statement at the hearing itself, and responds to questions from committee members.
Research indicates that hearings have become more partisan in recent years, with legislators increasingly exposed to one-sided information (Lewallen et al., Reference Lewallen, Theriault and Jones2016). Even at their peak in the early 2000s, though, hearings with one-sided information flows only accounted for about 30 percent of hearings. The remaining 70 percent contained information from a broad range of viewpoints. Moreover, other research makes clear that – even though hearings have become more partisan – the range of policy impacts discussed in those hearings has actually increased over time (Workman, Reference Workman2015; Baumgartner & Jones, Reference Baumgartner and Jones2015). Together, this suggests that the increased partisanship in recent times has affected how policy impacts are discussed, but not whether they are discussed. Also, for our purposes, we are not concerned with witnesses’ viewpoints on policy impacts, only with whether witnesses discuss a broad range of impacts in the hearings.
The potential utility of Congressional hearings for identifying policy impacts for inclusion in a BCA is clear. Hearings are explicitly intended to draw on the expertise of a broad set of stakeholders to elicit a wide range of information about a given policy area. Moreover, many policy issues serve as the subject of multiple hearings over time, providing an opportunity to compile an extensive public record – informed by a diverse set of experts, advocates, and stakeholders – on the policy topic in question. That said, there will undoubtedly be cases where a BCA addresses a topic that has not been the subject of many Congressional hearings. We suspect that such cases will be relatively few because if a topic is important enough to warrant the investments required of a BCA, then it is likely that it will have been the subject of multiple hearings. Empirically, this is evident in Table 1. For our sample of nine BCAs, the number of potentially relevant hearings, which we identified using broad topic codes from the Policy Agendas Project, ranged from 371 to 2144. The actual number of hearings analyzed will be much smaller though. In our example, described below, we began with over 2100 hearings but through a short and efficient expert reading of the hearing summaries, which are at most a few sentences long, ended with a sample of only 34. Our searches suggest that a significant, but manageable number of hearings will likely be available for major topics that have been the subject of BCAs in recent years.
4.2 A general process for identifying potential policy impacts from Congressional hearings
The record of hearings can be systematically mined to identify potential policy impacts for inclusion in a BCA. We propose a six-step process for identifying policy impacts from Congressional hearings. The first stage involves identifying the set of hearings potentially relevant to the policy under analysis. We recommend using the U.S. Congressional Hearings Dataset from the Policy Agendas Project for this purpose. This dataset is compiled from Abstracts of Congressional Publications and Legislative History Citations from the CIS and supplemented with data from the ProQuest Congressional database. It contains data on all Congressional hearings since 1946, with each hearing possessing several pieces of metadata, such as an ID number, the year of the hearing, and the Congressional chamber, among others.Footnote 4 Most important to identifying potentially relevant hearings are the classification codes that denote the subject of each hearing – each hearing contains both a major topic code as well as a more specific sub-topic code. The dataset also includes a qualitative summary description of each hearing that can inform analysts’ decisions regarding the potential relevance of the hearing. In the process of identifying potentially relevant hearings, we urge the analyst to err on the side of inclusion in selecting the topic codes that will correspond to the set of potentially relevant hearings.
After identifying all potentially relevant hearings, the second step in the process involves obtaining hearing transcripts and written statements from witnesses. This task can be accomplished by using the source code in the U.S. Congressional Hearings Dataset to locate each hearing in the ProQuest Congressional database. Analysts can then download a complete transcript of each relevant hearing along with the written statements that were submitted by witnesses in that hearing.
In the next step of the process, analysts use these transcripts and statements to develop an impact classification scheme. This process is likely to be an iterative one with the initial contents of the scheme informed by factors such as analysts’ topic knowledge, reviews of the literature, and stakeholder input. However, the scheme must be revised and expanded as the analyst begins to wade through hearing transcripts and witness statements. That is, the analyst must be cognizant of the fact that they are likely to encounter some impacts that were not included in their initial classification scheme. And when they encounter such an impact, the analyst must be ready to revise the scheme to incorporate the additional potential impact. Thus, the resulting process is both deductive, relying on researcher knowledge and theory, and inductive, allowing the data itself to identify relevant policy impacts. It is this second inductive stage, which will often co-occur with the actual classification stage, that helps maximize impact identification.
The fourth step of the process involves classifying the transcripts and witness statements using the classification scheme. Although we present the classification process as a separate step from the development of the classification scheme, in practice these two steps may move in parallel – the scheme will be refined throughout the classification process. With respect to classifying the transcripts and witness statements, our experience indicates that it is best to classify these documents at the sentence level. That is, the scheme should be applied to each sentence in the transcript and witness statement. Conducting this process at the sentence level may seem excessive, but we find that conducting it at any higher level – the statement or even the paragraph level – can result in the omission of impacts, as many witnesses address multiple impacts within a given statement, or even within a given paragraph. A sentence-level approach will undoubtedly result in classifying a large percentage of sentences as having “no substantive content,” but we believe such an approach is preferable to one that classifies impacts at a higher level and runs the risk of omitting references to potential policy impacts. However, under time and resource constraints, other levels of coding, such as the paragraph level, may still provide a broadly encompassing set of relevant values.
After finishing the classification process, the fifth step involves compiling the results into a dataset and describing the resulting categories. The dataset will be sentence level in nature – corresponding to the level at which the hearings were classified in the previous step – and will possess a nested structure, with each sentence assigned to both a witness and a hearing identifier. The primary analytic approach can consist of a simple summary of the proportion of sentences that address each impact in the classification scheme. However, the nested structure of the dataset – the connection of each sentence to a particular witness and hearing – allows for more insightful analysis of the data. For example, the analyst could describe the proportion of witnesses that address a given impact, or the proportion of hearings in which a given impact is mentioned. Moreover, it allows the analyst to assess the potential for clustering of impacts – whether the mention of one impact is related to addressing another among witnesses – which can provide an improved understanding of potential policy impacts that are often connected to one another. Similarly, including witness characteristics in the dataset allows the analyst to assess whether witnesses – or classes of witnesses – are more (or less) discuss a given impact or set of impacts. Again, this has the potential to provide the analyst with a more complete understanding of potential policy impacts, and particularly the interaction of those impacts. Although such data can provide analysts with numerous insights into potential policy impacts, at the end of the day analysts should simply focus on identifying all the impacts addressed in a nonzero proportion of sentences in the relevant hearings.
The final step requires the analyst to determine whether each impact in the dataset will, in fact, manifest as an impact in the specific policy proposal under analysis. Although all hearings analyzed in the steps above are relevant to the policy being analyzed, the policies and projects addressed in the hearings will undoubtedly vary in their similarity to the policy under analysis. It is for this reason that we describe the hearings as providing insight into potential policy impacts. It is incumbent upon the analyst to determine whether each potential policy impact identified from the hearing data will manifest as a valued impact in the specific policy they are analyzing. Some of the potential impacts identified from the hearing data will not translate into valued impacts. The details of the policy or program discussed in the hearing may differ significantly from the policy currently under consideration, or the context in which the topic was raised in the hearing is quite different from the context of the policy under analysis. But it is likely that many of the potential impacts identified from the hearings data will indeed manifest as valued impacts in the policy under consideration. In the following section, we apply the approach outlined here to identify potential policy impacts resulting from a proposal to alter operations of the GCD.
5 Identifying valued impacts from a proposal to modify operations of the GCD
5.1 Overview of GCD operations
Completed in 1963, the GCD is located on the Colorado River in northern Arizona and creates Lake Powell, which is the second largest reservoir in the United States. The Grand Canyon lies directly downstream of the GCD and is thus directly affected by its operations. The GCD was originally designed to achieve several goals, including flood control, water storage for irrigation and drinking supply, and the generation of hydroelectric power. In addition to achieving these goals, however, the GCD impacts many additional resources in the surrounding region, such as recreational amenities, wildlife habitat, and cultural sites.
Given the diverse goals and impacts of the GCD, it is perhaps unsurprising that the dam – and its operations – has long been a source of contention and controversy. The contention and controversy are generally between those who primarily value the electricity generation of the dam and those whose primary concerns lie with the environmental and recreational impacts of the dam. With respect to electricity generation, the GCD is responsible for approximately 70 percent of the hydroelectricity generated from the upper Colorado River basin (Jones et al., Reference Jones, Berrens, Jenkins-Smith, Silva, Carlson, Ripberger, Gupta and Carlson2016). This electricity, which is far less expensive than other sources of power in the region, is distributed to nearly 6 million customers by the Western Area Power Administration (WAPA). Notably, GCD hydropower is distributed to more than 50 Native American tribes and pueblos. On the environmental side, GCD operations shape the habitat for trout and other fish species in the river, as well as for birds and other animals that reside along its banks. Dam operations also affect river rafting conditions through the Grand Canyon, which is a popular tourist activity in the area. Considered together, the GCD clearly has a wide ranging and diverse set of impacts on its surrounding region.
In the first three decades after its completion, the GCD was operated in a manner designed to match electricity production with demand. That is, water was released through the dam – engaging the hydroelectric turbines – at greater rates during times of high electricity demand and at lower rates during periods corresponding to periods of lower demand. Critics of this approach to dam operations charged that it produced deleterious environmental effects – particularly with respect to fish habitats – and created uncertainty for recreational opportunities; rafting conditions were shaped by levels of electricity demand. These critics favored an operational approach that moderated or, ideally, eliminated the fluctuations in dam releases, which they contended would improve environmental conditions and better represent the river’s natural flow pattern.
In the early 1990s the U.S. Department of the Interior (DOI) – the agency responsible for GCD operations – initiated a reevaluation of dam operations and in the mid-1990s ultimately altered operations to moderate the flows. This decision was partially justified by an analysis concluding that the nonmarket benefits of altered operations in the environmental and recreational realms far exceeded the costs of reduced hydroelectricity generation. The GCD has effectively been operated under this paradigm in the years since – there have been small changes on the margin – but DOI has recently initiated efforts to develop and implement a “Long-Term Experimental and Management Plan” (LTEMP) that would further alter dam operations. Although DOI is considering a range of management activities in the LTEMP, the “preferred alternative” in the Environmental Impact Statement (EIS) accompanying the LTEMP would establish condition-dependent flow and nonflow actions that would be triggered by environmental resource conditions rather than demand for hydroelectricity. Like all policy changes, this change would generate a variety of impacts that must be considered in a comprehensive BCA. In the following section, we describe the approach we used to identify such impacts from Congressional hearings.
5.2 Identifying policy impacts of altering GCD operations from Congressional hearings
As outlined above, the first step in our search for potentially relevant impacts requires the identification of Congressional hearings that are pertinent to the topic at hand. We used the U.S. Congressional Hearings Dataset from the Policy Agendas Project to identify the population of relevant hearings. Given our interest in the GCD, we decided that one of two sub-topics may be relevant to our case – hearings about “Electricity” (sub-topic 802), which include issues about electricity, hydropower, and regulation of electrical utilities, and “Water Resources” (sub-topic 2104), which include issues related to water resources, water resource development and civil works, flood control, and research. From 1946 to 2013, there were 481 hearings about “Electricity” and 1632 hearings on “Water Resources.” After identifying this set of potentially relevant hearings, we used the qualitative descriptions to isolate the hearings about hydropower, dam operations, or the relationships between these and other considerations of natural resources, public lands, and affected communities. This resulted in a set of 34 relevant hearings that were held between 1995 and 2013. We started in 1995 to capture relatively recent impacts; we stopped in 2013 because that was the most recent year for which the Policy Agendas Project had collected data. Decisions at this stage can reduce potential time costs of the process either by focusing on more restrictive topic codes and key word searches or by constraining the time period of the analysis. Though the resulting analysis may be not be fully inclusive, it will likely still address a wider range of relevant impacts. Of primary importance in these decisions, however, is that topics/keywords and time ranges are documented explicitly to ensure the transparency and replicability of the BCA. For illustration, Table 2 provides some information about three of the hearings we included in our analysis. The table provides the qualitative description of the hearing, the topic and sub-topic assigned to the hearing, the number of witness statements (and sentences in those statements), and witness affiliations. The table illustrates that the witnesses come from a wide range of organizations, representing everything from federal agencies to local stakeholders and interest groups. The table also demonstrates that the selected hearings vary in their length, ranging from 14 to 23 witness statements each containing between 426 and 1274 sentences.
Summary information about 3 of 34 relevant hearings.
After isolating this set of 34 relevant hearings, we used the source code in the hearings dataset to locate and download each hearing from the ProQuest Congressional database, which provides a complete transcript of each hearing along with written statements that were submitted by witnesses. We then created a dataset of statements at each hearing by witness, affiliation, and sentence. In total, the dataset includes 409 statements, 344 unique witnesses,Footnote 5 and 25,916 sentences.
After assembling the dataset, we reviewed previous research and conducted stakeholder interviews to create a preliminary list of policy impacts that may result from altering dam operations (for details, see Lowry et al., Reference Lowry, Chermak, Brookshire, Shaneyfelt and Kobos2016 and Jenkins-Smith et al., Reference Jenkins-Smith, Carlson, Gupta, Jones, Ripberger, Wehde and Berrens2016). Using these methods, we developed a preliminary list of impacts to local culture, recreational opportunities, water resources, energy production/consumption, the economy, and the environment. We then read each transcript and witness statement for mention of impacts that were not included on the preliminary list. In doing so we found multiple references to the impact of dam operations on issues of governance. For instance, multiple witnesses argued that changing dam operations disrupts long-standing promises, agreements, and contracts between local communities. Others feared that changing dam operations would impose bureaucratic costs (i.e., regulatory requirements) on communities that are not equipped to handle such costs. Based on this discovery, we updated the list and then sorted each statement into one or more general categories. Impacts to culture were cited in 42 statements, recreation in 24, water in 214, energy in 91, the economy in 187, the environment in 140, and governance impacts came up in 140 statements.Footnote 6
Having identified the general impacts in each statement, we moved on to identify more specific impacts (sub-categories) within each general category. For example, impacts to recreation might affect tourism, fishing, or boating. Similarly, impacts to water might affect storage/supplies, flood control/safety, drinking water/water quality, water rights/ownership, irrigation, and/or water flows. To accomplish this, we re-read the statements within each general category and developed the list of sub-categories in Figure 2. We then categorized each sentence in the dataset
$(n=\text{25,916})$
by sub-category.Footnote
7
Approximately 80%
$(n=\text{20,741})$
of the classified sentences did not mention an impact and 3%
$(n=875)$
mentioned more than one specific impact. If the sentence did not mention an impact, we assigned it to a “No-impact” category; if it mentioned more than one specific impact, we assigned it to a general rather than specific impact category (i.e., recreation rather than tourism, fishing, or boating). Table 3 provides a list of specific impact categories and an example sentence from each sub-category. The first sentence in the table, for example, is from the Chairman of the Klamath Tribes who said, “Without the treaty resources like the endangered c’waam (Lost River Sucker), we simply do not have the ability to live as Klamath People in the way Creator intended” in his statement at a hearing on water resources in the Klamath River Basin. We classified this sentence as an impact to Native American Livelihood and Justice.
Specific impact categories and example sentences.
Figure 2 shows the frequency distribution of sub-category classifications as a proportion of the sentences that received a specific impact classification
$(n=4300)$
. The distribution indicates that three impacts – water storage/supplies, general economic costs/expenses, and species protection – are commonly discussed in hearings about hydropower and dam operations. Witnesses mention one of these impacts in roughly 30% of the sentences that received a specific impact classification. Flood control/safety, drinking water/water quality, and local economic benefits (i.e., jobs) are relatively common as well, appearing in almost 20% of the sentences with an impact classification. Below that, Native American livelihood and justice, habitat conservation, legal contracts/expectations, bureaucratic burdens, and collaboration are also relatively common. Though present, dialogue on the history/culture of hydropower, beauty/aesthetics, and recreation (tourism, fishing, and boating) is less common in Congressional hearings.
Proportion of sentences by general and specific impact category.
In addition to summarizing the data, this distribution of impact classifications provides information about the likelihood that each potential impact is relevant to a decision about GCD operations. It is reasonable to suspect that the impacts appearing in the largest proportion of classified sentences are those most likely to manifest as an impact of changing GCD operations. That said, we ultimately concluded that nearly all of these impacts should be valued in a BCA of changing GCD operations. Specifically, we concluded that changing GCD will likely impact Native American livelihood and justice, “ways of life” in farming and ranching communities, tourism, fishing, boating, water storage/supplies, water flows, air pollution/fossil fuel consumption, the reliability of energy production, species protection, habitat conservation, beauty/aesthetics, and legal contracts and expectations in the communities around the GCD. While a detailed justification and description of each impact is beyond the scope of this paper, it is important to note that many of these impacts are not accounted for in the analyses that currently inform decisions about GCD operations (Welsh et al., Reference Welsh, Bishop, Phillips and Baumgartner1995; Duffield et al., Reference Duffield, Neher and Patterson2016). However, in the context of GCD operations, Jones et al. (Reference Jones, Berrens, Jenkins-Smith, Silva, Ripberger, Carlson, Gupta and Wehde2018) demonstrate how substantive policy conclusions can be affected by electing to include only certain policy impacts in the analysis. In particular, Jones et al. (Reference Jones, Berrens, Jenkins-Smith, Silva, Ripberger, Carlson, Gupta and Wehde2018) demonstrate that an inclusive approach to identifying policy impacts, as opposed to an exclusive or restrictive one, generates an entirely different substantive policy recommendation – the inclusive approach to impact identification supports a policy of continuing current dam operations while the restrictive one leads to a recommendation of changing operations. To the extent that these analyses are influential factors in operational decisions, Jones et al. (Reference Jones, Berrens, Jenkins-Smith, Silva, Ripberger, Carlson, Gupta and Wehde2018) demonstrate that restrictive approaches to impact identification can result in biased analyses and suboptimal policy decisions.
6 Contextual considerations in using Congressional hearings to identify policy impacts
We recognize that our proposed methodology and process has limits of course, with perhaps the most notable being that the proposed process is quite resource-intensive. Thus, we suggest a set of criteria for BCA to meet in order to justify undertaking the process we developed. First, we believe our methodology is best applied to BCAs on well-established policy topics. Indeed, policy areas are often dynamic, with the set of interest groups engaged at one point in time significantly differing from the set of groups engaged at a different point in time. However, the fact that the set of groups engaged in the policy process may change over time does not necessarily imply a change in the set of policy impacts potentially relevant to a BCA on the topic. Indeed, it is the nature of politics that some groups – and their interests – can be favored at some points in time, but disfavored at other times, even if the impacts of the policy remain unchanged. As such, we view the ability of our proposed approach to incorporate information presented over long periods of time to be one of its most appealing features. It potentially allows us to identify policy impacts that, from a conceptual standpoint, should be included in a BCA, even if the interest groups serving as the voice for these impacts are not politically favored at the time the BCA is being executed.
This approach has the potential to give voice – to identify impacts – to stakeholders who may be overlooked by outreach efforts, but whose interests should nonetheless be accounted for in a BCA. However, we believe that using Congressional hearings as a source for identifying policy impacts will likely yield little extra information, relative to stakeholder consultation, in nascent policy areas. Second, policy areas that are highly contested will yield more information through the Congressional hearing arena than those with broad consensus. Following from this suggestion, we believe Congressional hearings provide a particularly valuable source of information about policy impacts when evidence itself is contested. In policy areas where opposing sides of the debate use narrow and different information sources, Congressional hearings will provide a forum for the dissemination of both sides of the debate.
Using Congressional hearings to identify policy impacts may also prove suboptimal for policies that are primarily administered at the state or local levels. In this case, Congressional hearings may only discuss the aspects of the policy that are relevant to the federal government, and largely ignore those aspects that are primarily determined by state or local governments. Such cases leave open the possibility that Congressional hearings will not allow analysts to identify the full set of policy impacts for inclusion in the BCA. However, policies with a substantial state or local component suggest the potential for other formal venues that may support identification of policy impacts using a comprehensive, systematic, and replicable approach. For example, state and local level deliberative institutions such as state legislatures or city councils may elicit relevant policy impacts in a manner similar to Congressional hearings at the federal level.
Returning to the federal level, another potential source of information for identifying policy impacts is comments on proposed regulations. Because agencies already analyze these comments during the regulatory process, the marginal cost of using them to identify policy impacts for inclusion in a BCA would be small, particularly relative to the hearing-based approach we outline above. Indeed, a comment-based approach to identifying policy impacts would, in principle, provide many of the same benefits that Congressional hearings do – comments are a formal, public venue for expressing policy preferences and positions. Further, comment-based approaches may avoid the potentially problematic selection inherent in using Congressional hearings – only a select set of stakeholders are invited to testify before Congress, but anyone can submit a regulatory comment. Moreover, from a legal perspective, comments may be preferable to hearings for identifying policy impacts because many members of the judiciary are hesitant to use legislative history as a basis for decisions. Such hesitation may be enhanced in our context, as our suggested approach could be interpreted as a proposal to incorporate legislative history into agencies’ rulemaking processes.
All of these considerations lead us to the conclusion that our approach of using Congressional hearings to identify policy impacts may be best suited for application in the academic realm, with the results of such applications made available to agency personnel for potential inclusion in BCAs conducted by the agency. Dividing labor in this manner would result in enhanced transparency surrounding the process used to identify the set of policy impacts included in a BCA without imposing additional regulatory requirements on agency personnel. We point to the policy analysis literature for a useful precedent for such a division of labor. In particular, we highlight Boardman et al.’s (Reference Boardman, Greenberg, Vining and Weimer1997) work that combs the academic literature to identify and catalog all available shadow prices – across a wide variety of policy domains – that practicing policy analysts can easily “plug in” to their analyses. We envision a similar evolution for our hearings-based approach to identifying policy impacts. Specifically, we believe our application of this approach in the context of dam operations may spark a series of similar applications in other policy domains, such as energy, the environment, education, health, housing, or any other policy area. As these applications accumulate, we intend to collect and catalog the results, and make them available to practitioners in a format that allows for straightforward integration into BCAs conducted by their organization.
7 Discussion and conclusion
BCAs are designed to provide a comprehensive accounting of each valued impact of the policy or project under analysis. Academics and practitioners alike have devoted significant time and effort to addressing the theoretical and empirical issues surrounding the valuation of policy impacts – this effort has produced widely accepted best practices. No comparable literature exists, however, to guide analysts on how to identify valued policy impacts in the first place. This paper represents an initial effort on this score. Specifically, we develop an approach for using Congressional hearings to identify potential policy impacts for inclusion in a BCA. This approach involves first identifying the set of hearings on the policy topic under analysis, classifying each sentence from each witness at these hearings using a scheme of potential policy impacts, and using the resulting data to inform the process of identifying valued impacts of the policy under analysis.
We believe that this approach has several advantages over current approaches to value identification. First, it allows for a degree of systematism, transparency, and replicability that other approaches often lack. By no means does our approach eliminate the need for analysts to make subjective judgments when choosing which impacts should be included in a BCA. However, it does place some structure around those judgments and, in doing so, may encourage the analyst to explicitly state the choices that were made (such as the keywords to identify hearing topics and the contents of the scheme used to classify witness statements). The Society for Benefit-Cost Analysis indicates transparency and replicability are foundational principles for the practice of BCA (Zerbe et al., Reference Zerbe, Davis, Garland and Scott2010), and we believe that our approach to identifying policy impacts has the potential to advance applications of the technique on this score.
Second, our approach to identifying policy impacts draws from the same process that the resulting BCA is intended to inform. BCAs are often intended to inform public decision-makers about the net social benefits. Our approach ensures that the inputs to these analyses – the benefits and costs – are identified from a context that these very same decision-makers use to explore policy details. In doing so, it allows the analyst to identify the impacts that are most important to those responsible for ultimately making the decision regarding the policy. Such knowledge has the potential to prove useful in the design and execution of a BCA.
Third, use of Congressional hearings to identify policy impacts has the potential to alert analysts to what we term “soft” impacts. Soft impacts are those that can be difficult to measure or quantify and are thus more likely to be absent from existing literature on the topic – studies tend to be biased toward the effects that can be measured or quantified with relative ease. Of course, the ease with which something is measured or quantified has no bearing on whether it should be included as a valued policy impact in a BCA. Hearings are disproportionately likely to allow analysts to identify such impacts because the format includes anecdotes and stories that witnesses use to convey information. In the GCD context, examples of “soft” impacts that routinely appeared in Congressional hearings but were absent from much of the literature – as well as other analyses of GCD operational changes – include concepts like culture, “ways of life,” and governance. These concepts are difficult to measure and quantify, but witness after witness highlighted how dam operations could affect these factors, making clear that they should be included as policy impacts in a BCA of changing dam operations. In doing so, they highlight the great promise Congressional hearings hold for assisting analysts in systematically and transparently identifying impacts of a policy that should be accounted for in a BCA.
In closing, we note that our approach is grounded in a viewpoint where all impacts – benefits and costs alike – affecting the welfare of individuals with standing should be included in a BCA (e.g., Zerbe, Reference Zerbe2004). In doing so, we effectively view the challenge of identifying all policy impacts as an empirical one. We acknowledge that such a view generates a few practical difficulties. At a minimum, there is the aforementioned challenge of identifying all of impacts for inclusion in a BCA, which can be overwhelming and time consuming from an analyst’s perspective. Moreover, it provides significant discretion to analysts, which could lead to selective identification of impacts in an effort to (dis)advantage any given policy. An alternative approach – one articulated by some theorists (e.g., Adler & Posner, Reference Adler and Posner2006) – could involve limiting the set of impacts that should be included in a BCA. Such an approach would have the upside of significantly clarifying the set of impacts necessary to include in a BCA. In doing so, though, it would circumscribe the set of factors allowed to enter an individual’s utility function, which violates the norm of “consumer sovereignty” underlying BCA. Our own view is that policy analysts need a reasoned middle ground between an unguided and opaque process for identifying impacts and a prescriptive screening-out of selected categories of impacts deemed inappropriate, despite being relevant to individual utilities. We believe the approach to identifying impacts for inclusion in BCAs developed in this paper can allow analysts to achieve this middle ground.
