2.1. Information Criteria

There has been substantial progress in the area of IC over the last decades; see Rao and Wu (2001) for a recent survey. Progress has been made in extending results to possibly nonstationary time series processes; see Phillips (1996) and references therein for the extension of Rissanen's theorem to the case of time-series prediction. IC, although based on (quasi-)likelihood ratios (see Pötscher, 1991), treat models symmetrically, unlike hypothesis testing. This is the key motivation used by Granger, King, and White (1995) in advocating the use of IC in comparison of economic theories.

One limitation of IC lies however with their characteristic of estimators of the best model. No measure of strength of decision is usually available, unless a Bayesian paradigm is used. Bayesian analysis is however not the best suited method for communicating inferences when the elicitation of a priori distributions is subjective. Model averaging is not an option when inference is aimed at selecting among economic theories.

In general, one would like to complement the choice of a model with a likely (objective) assessment of whether this model is significantly distant from alternative ones. If one is to choose among different economic theories, this is particularly relevant.

Asymptotic probability statements on IC can be deduced from their connection to HT and MC procedures, which are discussed in Sections 2.2 and 2.3. This is because IC, HT, and MC are all based on the same (quasi-)likelihood theory. No investigation appears to exist in this direction in the literature, and any attempt along this line will probably be rewarded.

One other area where more theoretical contributions would be welcome is comparison of IC procedures. These comparisons should be aimed at selecting a single or at most a few best criteria. The many IC now in use often generate conflicting decisions, with no clue on how to resolve them.

At present, procedures are classified as efficient and/or consistent (see Rao and Wu, 2001). Among consistent criteria, no comparison is usually made on (higher order) asymptotics, as in the case of parameter estimation.

Here one needs to define (higher order) asymptotics in this context. As one possibility, consider two criterion functions h_jn(i), j = 1,2, based on a sample of size n, where i = 1,…,p indicates different models M₁,…,M_p. Let i₀ be the index of the correct model. Consistency of

is usually proved by showing that h_jn(i₀) = O_p(1) as n → ∞ whereas h_jn(i) = O(n^α_j) → ∞ for i ≠ i₀. The order n^α_j could depend also on i, but we assume for simplicity that it is uniform over incorrect models. Assume that both h_1n and h_2n are consistent. One could first compare α₁ and α₂, where the criterion that guarantees the fastest divergence should be favored.

Assume however that the two criteria h_1n and h_2n are both consistent and have the same divergence rate α₁ = α₂. One could then study the ratio q_n := h_1n /h_2n − 1 to ascertain if q_n is asymptotically positive or negative. More generally one could study the asymptotic distribution of q_n and prefer h_1n to h_2n if Pr(q_n ≥ 0) → η > ½. These comparisons could give a handle on how to select a best criterion, thus reducing the possibility of conflicting results.

2.2. Hypothesis Testing

Some selection procedures are based on HT. HT gives a probabilistic framework within which one can measure strength of decisions. Let Θ₁ ⊂ Θ₂ ⊂ ··· ⊂ Θ_p be nested parameter sets corresponding to models M₁,…,M_p. Some common misconceptions on HT in model selection are the following: (i) in nested models, a general to specific HT strategy is only associated with testing M_j−1 in M_j starting with j = p and proceeding to j = 0 (called “testing-down” strategy in the following discussion); (ii) HT may be applied only to nested models that contain the true data generating process (DGP).

Consider (i). In several instances it has proved useful to consider a general to specific strategy that tests M_i in M_p starting with i = 0 and proceeding up to p − 1. For later reference, call this procedure a “testing-up” sequence. This strategy is still general to specific, because each submodel is tested against the full model M_p.

The testing-up sequence has been applied, e.g., when testing for cointegration rank (see Johansen, 1992). Here M_j indicates a vector autoregressive (VAR) model with cointegration rank at most equal to j; let LR(i| j) be the likelihood ratio test comparing M_i and M_j. For the testing-up procedure, as sample size diverges, one has Pr(select M_r) → 1 − α and Pr(select M_i) → 0 for i < r, where r is the true cointegration rank and α is the size of each test in the sequence. These properties do not apply to the testing-down procedure, because the limit distribution of LR(j − 1| j) depends on which submodel M_j−h contains the DGP, h = 1,2,3,…,j − 1. Hence one sees that, in a nonstationary setting, the testing-up procedure is recommended, contrary to the common belief (i). Obviously, letting α decrease to zero as sample size increases generates a consistent selection criterion for cointegration rank.

Regarding issue (ii), it is important to note that hypothesis testing can be defined and used also for comparing misspecified models that do not contain the DGP, as shown by Vuong (1989) for independent and identically distributed (i.i.d.) observations. Extension of these results to dependent data and more than a pair of models appears to be a fruitful line of future research.

A final word of caution concerns the distribution of the estimators of the selected model, as shown in recent work by Leeb and Pötscher (2003).

2.3. Multiple Comparisons

Consider finally the case when model selection is the main goal of inference, i.e., one wishes to confront several economic theories with the same set of data. In this context MC procedures may find wider applications in econometrics; see Hsu (1996) for an introduction to MC.

As a simple example, consider the usual regression setup with two regressors,

Assume also that several competing theories exist on the explanation of y. Theory 1 predicts that β₁ > 0 and β₂ < 0, and theory 2 predicts that β₁ = β₂.

Of course one can compute the univariate t tests t_β1=0, t_β2=0, and t_β1=β2 or the corresponding confidence intervals. One can then combine these separate inferences, using probability inequalities such as Bonferroni's (this is called “deduced inference” in Hsu, 1996).

However, in principle, one would like to construct simultaneous confidence intervals for β₁, β₂, and β₁ − β₂ from the start, with overall coverage probability not inferior to 95%. This is called “direct inference” in Hsu (1996). Note that the direct method can exploit the fact that we wish to have three confidence intervals for a bivariate distribution of

, where a hat indicates the regression estimator.

One possible simple direct inference method is Scheffé's procedure, which states

where F_1−α is the 1 − α quantile of an F(k,n − k) distribution, and k is the number of regressors, in the example equal to 2. One can form simultaneous confidence intervals for any choice of

using (s²c′(X′X)⁻¹ckF_1−α)^1/2 as standard error. In our example we can choose c equal to c₁ := (1 : 0)′, c₂ := (0 : 1)′, c₃ := (1 : −1)′, obtaining three confidence intervals I₁, I₂, I₃. If the lower endpoint of I₁ is greater than 0 one can assert β₁ > 0; similarly if the upper endpoint of I₂ is negative, one can assert β₂ < 0. If finally I₃ contains 0, then one can assert β₁ = β₂. All these statements have a joint error rate bounded by 5%.

Note that, if some new economic theory comes along—theory 3, say, which predicts β₁ + 2β₂ = 1—one does not have to rework the confidence level of the confidence intervals I₁, I₂, I₃ from the start, but just needs to construct one more confidence interval I₄ for β₁ + 2β₂ − 1.

This feature is both a strength and a weakness of Scheffé's procedure. On one side it gives great flexibility to add any other confidence interval; on the other hand a price is paid in accommodating all these simultaneous confidence intervals. The price is that some possibly less comprehensive procedure gives tighter bounds for each confidence interval; see Hsu (1996).

MCs are widely used in analysis of variance (ANOVA) applications and in regression models; for the application of Scheffé-like procedures in nonlinear regression see, e.g., Johansen and Johnstone (1990). It is possible that many of the procedures within PcGets (see Granger and Hendry, 2005), could be (re)discussed or reorganized in this MC perspective.