1. MOTIVATION AND RESULTS

In introductory econometrics classes, students are sometimes warned of the perils of controlling for too many factors in multiple regression analysis. In particular, it may not make sense to hold fixed certain factors when estimating the partial effect of the variable of interest. Overcontrolling is often a result of using goodness-of-fit measures or statistical tests to decide whether to include certain covariates and forgetting about the ceteris paribus interpretation of regression coefficients.

The counterfactual setting of modern treatment effect analysis provides an ideal setting in which to illustrate how one can easily include too many factors. Plus, propensity-score-based methods of estimating treatment effects are becoming rather popular, and one sees a tendency to include many factors in estimating the propensity scores (the probabilities of the various treatment levels). This exercise provides a simple demonstration of how including certain controls—specifically, those that are themselves affected by treatment—generally violates the key ignorability assumption.

To formalize this conclusion, consider the counterfactual treatment setup where w is a discrete treatment outcome taking values in {0,1,2,…,J}, so that there are J + 1 treatment levels. The corresponding counterfactual outcomes are (y₀,y₁,…,y_J); that is, for a generic population unit, y_j is the outcome or response if the treatment level is w = j. (For the binary case, see, e.g., Rosenbaum and Rubin, 1983; Heckman, Ichimura, and Todd, 1997.) We do not observe each counterfactual outcome; rather, we observe only the outcome associated with the treatment level w. Write the observed outcome in terms of the treatment level and counterfactual outcomes as

where w_j ≡ 1[w = j] and 1[·] is the indicator function. Let x be a K-vector of observed covariates thought to be “related to” the treatment level and also the counterfactual outcomes. A key assumption in the treatment effect literature is ignorability of treatment: conditional on x, w and (y₀,y₁,…,y_J) are independent. A weaker version of the assumption—which often suffices for consistency of propensity-score or regression-adjustment methods—is conditional mean independence:

Typically, we think (2) is more plausible the richer is x; that is, the more factors that might predict treatment status, the better. Nevertheless, as in standard regression contexts, it is possible to include too many factors in x. In particular, one might inadvertently include in x variables (other than y, of course) that are themselves influenced by the treatment level. For example, in evaluating the effects of drug courts on recidivism, should we include in x a measure of postsentencing education levels or employment status? Almost certainly not. Differences in postsentencing education and employment, depending on whether the juvenile was sentenced in regular court or a drug court, are themselves responses to the policy treatment.

To obtain an unambiguous result, consider the extreme case where treatment is actually randomized with respect to (y₀,y₁,…,y_J). That is, assume that w and (y₀,y₁,…,y_J) are independent. Letting D(·) denote distribution, show that if D(x|w) ≠ D(x)—so that treatment is not randomized with respect to x—then the ignorability assumption in (2) cannot hold unless E(y_j|x) = E(y_j) for all j.