2. ESTIMATION VIA OUTER PRODUCT OF GRADIENTS

Let G(x) = E(Y |X = x). It is easy to see from model (1) that

Therefore, the derivative or gradient of the regression function at any point has the same direction as θ₀. Based on this observation, Powell et al. (1989) and Härdle and Stoker (1989) proposed estimating θ₀ via E {∇G(X)}. They called this the average derivative estimation (ADE) method. Let f (x) be the probability density function of X. Integration by parts gives

Instead of estimating ∇G(x), Powell et al. (1989) and Härdle and Stoker (1989) proposed estimating l(x) = ∇ log f (x) by high-order kernel smoothers. Because the nonparametric estimator of a high-dimensional density function is used, the ADE procedure still suffers the “curse of dimensionality.” Another disadvantage for the method is that when E(g′(θ₀^{[top ]}X)) = 0, i.e.,

the method fails to estimate the index θ₀. To overcome this drawback, one can consider the outer product of the gradients, E {∇G(X)∇^{[top ]}G(X)}. Note that

has only one nonzero eigenvalue. Therefore, index θ₀ is the eigenvector corresponding to the largest eigenvalue of E {∇G(X)∇^{[top ]}G(X)}. This idea can be easily extended to the estimation of the multi-index model (2).

To implement the estimation, we first estimate the gradients by local polynomial smoothing. Specifically, we consider the local linear fitting in the form of the following minimization problem:

where X_ij = X_i − X_j and w_ij is a weight depending on the distance between X_i and X_j. Calculate

where

is the minimizer from (4). Then the first eigenvector,

, of

is an estimator of θ₀.

To allow the estimation to be adaptive to the structure of the dependency of Y on X, we may follow the idea of Xia et al. (2002) to refine the kernel weights by taking

and repeat the previous procedure, where K(.) is a kernel function, h is a bandwidth, and K_h(.) = K(./h)/h. We call this method refined outer product of gradients (rOPG) estimation. HJS also proposed a gradually adaptive procedure by taking

, where |.| denotes the Euclidean norm. To implement the estimation, we propose the following rOPG algorithm. Suppose θ is an initial estimate of θ₀ for which some discussion is given in Section 4.

where

. Denote the eigenvector again by θ.

In the algorithm, ρ_n(.) is a trimming function employed to handle the boundary points. In this paper, we take ρ_n(v) as a bounded function with bounded derivative on

such that ρ_n(v) = 1 if v > 2c₀ n^−ε; ρ_n(v) = 0 if v ≤ c₀ n^−ε for some constants ε > 0 and c₀ > 0. As an example, we can take

The choice of ε will be given subsequently.

The final eigenvector in the preceding algorithm is the rOPG estimator of θ₀, denoted by

. Compared with HJS, we here use single-indexing weight w_ij = K_h(θ^{[top ]}X_ij) directly. Another advantage over HJS is that we replace the average of derivatives in HJS by the outer product to avoid the problematic case of Eg′(θ₀^{[top ]}X) = 0 for HJS.

3. THE MINIMUM AVERAGE CONDITIONAL VARIANCE ESTIMATION

Because the link function is unknown, it needs to be estimated using a local linear smoother. Instead, θ₀ is a global parameter; it should be estimated via minimizing a global cost function. Based on this idea, many estimation methods have been proposed. Examples are Härdle et al. (1993) and Ichimura (1993).

The estimation of Härdle et al. (1993) can be stated as follows in the Nadaraya-Watson kernel smoothing framework. The estimator of θ₀ can be obtained by minimizing

with respect to h and θ, where

Note that the preceding minimization problem is very difficult to implement. Actually, Härdle et al. (1993) used the grid search algorithm for their calculation. The algorithm is infeasible in practice when the dimension of covariate is high.

Xia et al. (2002) simplified the procedure via the MAVE. The corresponding algorithm can be stated as follows. Suppose θ is an initial estimate of θ₀.

where

is defined in the rOPG algorithm.

The final vector θ := θ/|θ| is the rMAVE estimator of θ₀, denoted by

. Note that the steps are an explicit algorithm of Xia et al. (2002) for the single-index model. Similar to the other direct estimation methods, the preceding calculation is easy to implement. See Horowitz and Härdle (1996) for more discussion.

4. ASYMPTOTIC DISTRIBUTIONS AND COMPARISON

We impose the following conditions to obtain the asymptotic normalities of the estimators.

(C1) (Design). The density function f_θ(v) of θ^{[top ]}X and its derivatives up to third order are bounded on

for all θ : |θ − θ₀| < δ where δ > 0 is a constant, E|X|⁶ < ∞, and E|Y |³ < ∞.

(C2) (Link function). The conditional mean g_θ(v) = E(Y |θ^{[top ]}X = v), and its derivatives up to third order are bounded for all θ : |θ − θ₀| < δ where δ > 0.

(C3) (Kernel function). K(v) is a symmetric density function with finite moments of all orders and bounded derivative. Its Fourier transform is absolutely integrable.

(C4) (Bandwidth and trimming parameter). Bandwidth h ∝ n^−1/5 and trimming parameter

.

Assumption (C1) means that X may have discrete components providing that θ^{[top ]}X is continuous for θ in a small neighbor of θ₀; see also Ichimura (1993). The moment requirement on X is not strong. Härdle et al. (1993) obtained their estimator in a bounded area of

, which is equivalent to assuming that X is bounded; see also Härdle and Stoker (1989). We impose a slightly higher order moment requirement than second moment for Y to ensure that the optimal bandwidth in (C4) can be used in applying Lemma 6.6 in Section 6. The smoothness requirements on the link function in (C2) can be relaxed to the existence of a bounded second-order derivative at the cost of more complicated proofs and smaller bandwidth. Assumption (C3) includes the Gaussian kernel and the quadratic kernel. Assumption (C4) includes the commonly used optimal bandwidth in both the estimation of the link function and the estimation of the index θ₀. See Xia, Linton, and Härdle (2004) and Härdle et al. (1993).

Because our interest here is asymptotic efficiency, we assume in the following context that the initial value θ is in a small neighbor of θ₀: Θ_n = {θ : |θ − θ₀| ≤ C₀ n^−1/2+c₀} with

. Actually, Härdle et al. (1993) even assumed that the initial value is in a root-n neighbor of θ₀, {θ : |θ − θ₀| ≤ C₀ n^−1/2}. These assumptions are feasible because such an initial estimate is obtainable using the existing methods, such as Härdle and Stoker (1989), Powell et al. (1989), and Horowitz and Härdle (1996). Let μ_θ(x) = E(X|θ^{[top ]}X = θ^{[top ]}x), ν_θ(x) = μ_θ(x) − x, w_θ(x) = E(XX^{[top ]}|θ^{[top ]}X = θ^{[top ]}x), W₀(x) = ν_θ0(x)ν_θ0^{[top ]}(x), and W(x) = w_θ0(x) − μ_θ0(X)μ_θ0^{[top ]}(X). Let A⁺ denote the Moore–Penrose inverse of symmetric matrix A. We have the following asymptotic results for the estimators.

THEOREM 4.1 (rOPG). Under conditions (C1)–(C4), we have

where

THEOREM 4.2 (rMAVE). Under conditions (C1)–(C4) we have

where

By choosing a similar trimming function, the estimators in Härdle et al. (1993) and Ichimura (1993) have the same asymptotic covariance matrix as rMAVE. If we further assume that the conditional distribution of Y given X belongs to a canonical exponential family

for some known functions

, then Σ_rMAVE is the information lower bound in the semiparametric sense (Bickel, Klaassen, Ritov, and Wellner, 1993). See also the proofs in Carroll, Fan, Gijbels, and Wand (1997) and Härdle et al. (1993). In some other cases, we can also compare the efficiencies of the estimation methods as follows.

COROLLARY 4.3. If ε is independent of X, then

That is,

is more efficient than

. If, further, X is normal, then they are equivalent in terms of asymptotic efficiency, i.e.,

When the model has heteroskedasticity, we can follow the common approach by considering weighted estimations. Specifically, we consider the following model:

where ε is independent of X and s(.,.) is an unknown function. This model includes the most important binary choice model:

and the Tobit model (cf. Nishiyama and Robinson, 2005):

as its special cases. For model (6), we can consider weighted versions of the proposed estimation methods following the idea of Ichimura (1993). In calculation, we need only modify the algorithms by replacing

with

, where

Assuming that σ²(v) ≡ Es²(v,ε) is bounded away from 0 and has bounded third derivatives, Theorems 4.1 and 4.2 still hold with

Similar to the proof of Corollary 4.3 in Section 6, we have

That is, rMAVE is more efficient than rOPG for model (6). Again, if X is normal, then Σ_rMAVE = Σ_rOPG.

6. PROOFS OF THE THEOREMS

Recall that K is a symmetric density function. Thus, ∫K(v) dv = 1 and ∫vK(v) dv = 0. For ease of exposition, we further assume that μ₂ = ∫v²K(v) dv = 1. Otherwise, we can redefine K(v) := μ₂^1/2K(μ₂^1/2v). Let f_θ(v) be the density function of

where c ≥ 1. Suppose A_n is a matrix. By A_n = O(a_n) (or A_n = o(a_n)), we mean that all elements in A_n are O_a.s.(a_n) (or o_a.s.(a_n)) uniformly for

. Let δ_n = (nh/log n)^−1/2, τ_n = h² + δ_n, and δ_θ = |θ − θ₀|. For any vector V(v) of functions of v, we define (V(v))′ = dV(v)/dv.

LEMMA 6.1. (Basic results for kernel smoothers). Suppose E(Z|θ^{[top ]}X = θ^{[top ]}x) = m(θ^{[top ]}x) and its derivatives up to second order are bounded for all θ ∈ Θ_n and that E|Z|^r exists for some r > 3. Let (X_i,Z_i),i = 1,2,…, n, be an independent and identically distributed (i.i.d.) sample from (X,Z) and X_ix = X_i − x. If (C1), (C3), and (C4) hold, then

where μ_d = ∫K(v)v^d dv,d = 0,1,….

LEMMA 6.2 (Kernel smoothers in rOPG). Let

Under assumptions (C1)–(C4), we have

where w_θ(x) = w_θ(x) − μ_θ(x)μ_θ^{[top ]}(x).

LEMMA 6.3 (Kernel smoothers in rMAVE). Let

Under assumptions (C1)–(C4), we have

where

LEMMA 6.4. (Denominator in the rMAVE). Under assumptions (C1)–(C4), we have

where W_g0 = E {g′(θ₀^{[top ]}X)²ν_θ0(X)ν_θ0^{[top ]}(θ₀^{[top ]}X)} and F = E {(f_θ0(θ₀^{[top ]}X)ν_θ0(X))′/f_θ0(θ₀^{[top ]}X)}.

LEMMA 6.5 (Numerator in rMAVE). Under assumptions (C1)–(C4), if θ ∈ Θ_n we have

Proof of Theorem 4.1. By assumption (C1), we have

It follows from the Borel–Cantelli lemma that

Let

. Similarly, we have

Thus, we can exchange summations over

in the sense of almost sure consistency. By Lemma 6.1 and (C1), we have

Thus by the smoothness of ρ_n(.) with |ρ_n′(.)| < cn^ε for some c > 0 and (10), we have

We introduce operator E_j as follows. For any function G(X_i,Y_i,X_k,Y_k,X_{[ell ]},Y_{[ell ]}) with i ≠ j ≠ k, we define

Let C_ij = K_h(θ^{[top ]}X_ij)ρ_n(f_θ(θ^{[top ]}X_j))g′(θ₀^{[top ]}X_j){f_θ(θ^{[top ]}X_j)w_θ(X_j)}⁺. We have

Thus, by (C1) and Lemmas 6.6 and 6.7 in Section 6.2, we have

Suppose we start with θ^(k) = θ ∈ Θ_n. We have (hτ_n + hδ_θ + δ_θ²)n^ε = o(n^−1/2). It follows from (9)–(11) and Lemma 6.2 that

where

and

Recall that μ_θ0(x) = E(X|θ₀^{[top ]}X = θ₀^{[top ]}x) and W(x) = E(XX^{[top ]}|θ₀^{[top ]}X = θ₀^{[top ]}x) − μ_θ0(x)μ_θ0^{[top ]}(x). We have θ₀^{[top ]}μ_θ0(x) = θ₀^{[top ]}x, which implies θ₀^{[top ]}ν_θ0(x) = 0, and θ₀^{[top ]}W(x) = θ₀^{[top ]}xμ_θ0^{[top ]}(x) − θ₀^{[top ]}μ_θ0(x)μ_θ0^{[top ]}(x) = 0. It follows that

, and

We have

The estimator θ^(k+1) in the next iteration is the eigenvector of

. By Lemma 3.1 in Bai, Miao, and Rao (1991), we have

, i.e.,

Therefore, Theorem 4.1 follows from the preceding equation and the central limit theorem for

. █

The preceding proof indicates that starting from θ ∈ Θ_n, after one iteration the estimator is already root-n consistent. Fast convergence rate for the algorithm was also observed in HJS.

Proof of Theorem 4.2. After one iteration, we have that the new θ is

Because |θ| = 1 and |θ₀| = 1, we have

Recall that θ₀^{[top ]}W_g0 = 0 and θ₀^{[top ]}ν_θ0(x) = 0. By the definition of the Moore–Penrose inverse, we have W_g0⁺W_g0 = I − θ₀θ₀^{[top ]}. If we start with θ ∈ Θ_n, then (τ_n hn^ε + δ_θ)δ_θ /h² = o(n^−1/2). By these facts, Lemmas 6.4 and 6.5, and equations (9) and (10), we have

where

It is easy to check that

. Thus

Let θ^(k) be the value of θ after k iterations. We have

Recursing the preceding equation, we have

Thus as the iteration k → ∞, Theorem 4.2 follows immediately from the preceding equation and the central limit theorem. █

Proof of Corollary 4.3. If ε is independent of X, we have

Let A = {g′(θ₀^{[top ]}X)}⁻²W(X)⁺ and C = −g′(θ₀^{[top ]}X)⁻²(EA⁺)⁺. Then,

Thus, the first part of Corollary 4.3 follows. Without loss of generality, we assume that X is standard normal. We have

Thus,

We have completed the proof of the second part. █

6.1. Proofs of the Lemmas

Proof of Lemma 6.1. It follows from Lemma 6.6 and

where μ_d = ∫v^dK(v) dv. █

Proof of Lemma 6.2. By Lemma 6.6, we have

where

. If f_θ(θ^{[top ]}x) > n^−ε, then

where

. By the Taylor expansion of g(θ₀^{[top ]}X_i) at θ₀^{[top ]}x and θ₀ = θ + (θ₀ − θ), we have

where Δ_n(x,X_i,θ) = O(|θ₀^{[top ]}X_ix|³ + |θ^{[top ]}X_ix|·|X_ix|δ_θ + |X_ix|²δ_θ²). It is easy to check that

Note that μ₂ = μ₀ = 1. It follows from Lemma 6.1 that

By Lemma 6.6, it follows that

Thus,

For the noise term, we have from (12)

Combining (13)–(17), we complete the proof of Lemma 6.2. █

Proof of Lemma 6.3. It follows from Lemma 6.1 that

Thus, if f_θ(θ^{[top ]}x) > n^−ε then

Rewrite (13) as

It is easy to see that

Now, by Lemma 6.1 we have

Again, by Lemma 6.1 and (18), we have

Similar to (16), we have

For the noise term, we have from (18)

Combining (19)–(24), we complete the proof of Lemma 6.3. █

Proof of Lemma 6.4. Let (θ,B) be an orthogonal matrix. By Lemma 6.1, we have

where

. Note that

. Therefore,

where F_θ(x) = B^{[top ]}{f_θ(θ^{[top ]}x)ν_θ(x)}′/f_θ(θ^{[top ]}x). By (11), d_x^θ = g′(θ₀^{[top ]}x) + O(τ_n n^2ε/h)(1 + |x|⁴). Note that E {g′(θ₀^{[top ]}X_j)²C^θ₀(X_j)} = 2B₀^{[top ]}W_g0 B₀. Denote by D_n^θ the left-hand side of Lemma 6.4. Then

where F = E {g′(θ₀^{[top ]}X)²F_θ0(X)}. The law of large numbers (Chow and Teicher, 1978, Cor. 2, p. 122) is used for the summation of |X_j|⁶. Therefore by the matrix inversion formula in blocks (Schott, 1997), we have

where

. Note that B = B₀ + δ_θ. We have

. By the definition of the Moore–Penrose inverse, we have

Thus Lemma 6.4 follows from the facts that BB^{[top ]} = I − θθ^{[top ]} = I − θ₀θ₀^{[top ]} + O(δ_θ) and θ₀^{[top ]}W_g0 = 0. █

Proof of Lemma 6.5. By Lemma 6.3, we have

Therefore,

By Lemmas 6.1 and 6.3, we have for the third term on the right-hand side of the preceding equation

By Lemma 6.6 and Corollary 2 in Chow and Teicher (1978, p. 122), we have

Let

be the left-hand side in Lemma 6.5. By Lemma 6.3 and the above two equations in this proof, we have

where

By Lemma 6.1 and (9) and (10), we have

Let D_ij = ρ_n(f_θ(θ^{[top ]}X_j))g′(θ₀^{[top ]}X_j) f_θ⁻¹(θ^{[top ]}X_j)K_h(θ^{[top ]}X_ij)X_ij. We have E {E_j D_ij − g′(θ₀^{[top ]}X_i)ν_θ(X_i)}² = O{(h² + δ_θ)²}. Note that

. By Lemmas 6.6 and 6.7, we have

Let F_ij = K_h(θ^{[top ]}X_ij)X_ij. We have E_i(F_ij) = ν_θ(X_j) + O(h²)(1 + |X_j|). Note that Eν_θ(X_j) = 0. By Lemma 6.7 in Section 6.2

Thus Lemma 6.5 follows from (25)–(27). █

6.2. Two Supplementary Lemmas

LEMMA 6.6. Suppose m_n(χ,Z),n = 1,2,…, are measurable functions of Z with index

, where d is any integer number, such that (i) |m_n(χ,Z)| ≤ a_n M(Z) with E(M(Z)^r) < ∞ for some r > 2 and a_n increases with n such that a_n < c₀ n^1−2/r; (ii) E(m_n(χ,Z))² ≤ a_n m₀²(χ) with |m₀(χ) − m₀(χ′)| ≤ c|χ − χ′|^α₁, where α₁ > 0 and c > 0 are two constants (without loss of generality, we assume m₀(χ) ≥ 1); and (iii) |m_n(χ,Z) − m_n(χ′,Z)| ≤ |χ − χ′|^α₁n^α₂G(Z) with some α₂ > 0 and EG²(Z) exists. Suppose {Z_i,i = 1,…, n} is a random sample from Z. Then, for any positive α₀, we have

Proof of Lemma 6.6. The “continuity argument” approach is used here. See, e.g., Mack and Silverman (1982) and Härdle et al. (1993). For ease of exposition, let m_n(χ,Z) denote m_n(χ,Z)/m₀(χ). Note that

is bounded and its Borel measure is less than n^d+α₀. There are n^α₄ (α₄ > d + α₀) balls B_nk centered at χ_nk, 1 ≤ k ≤ n^α₄, with diameter less than cn^{−(2+α₂)/α₁}, such that

. Then

By part (ii) of Lemma 6.6, we have

By the strong law of large numbers, we have

almost surely. Let T_n = {n/(a_n log(n))}^1/2. Let m_n^o(χ,Z_i) = m_n(χ_nk,Z_i) I {|M(Z_i)| ≥ T_n} and m_n^I(χ_nk,Z_i) = m_n(χ_nk,Z_i) − m_n^o(χ_nk,Z_i). We have

where ξ_nk,i = m_n^I(χ_nk,Z_i) − E {m_n^I(χ_nk,Z_i)}. Note that

. Thus, for all sufficiently large n, with probability 1, |M(Z_i)| ≤ T_n for all i ≤ n. This implies that the first term on the right-hand side of (30) is eventually 0 with probability 1.

It follows from part (ii) of Lemma 6.6 that

By part (i) of Lemma 6.6, we have

Let N₃ = c₃{na_n log n}^1/2. By Bernstein's inequality, we have from (31) and (32) that

The last inequality holds if we choose c₃ sufficiently large. It follows that

By the Borel–Cantelli lemma, we have

Combining (30) and (34), we have

Therefore, Lemma 1 follows from (28), (29), and (35). █

LEMMA 6.7. Suppose φ(θ) and ξ(θ) are two measurable functions of (X,Y) such that

with

for some r > 2, and E {ξ(θ)|θ^{[top ]}X} = 0 for all

. Suppose (X_i,Y_i), φ_i(θ), and ξ_i(θ),i = 1,2,…, n are random copies of (X,Y), φ(θ), and ξ(θ), respectively. If (C1), (C3), and (C4) hold, then

Proof. Let Δ_n(θ) be the value in the absolute symbols. By the continuity of K(.), there are n₁ < cn^2q points θ_n,1,…, θ_n,n1 in Θ such that ∪_k=1^n₁{θ : |θ − θ_n,k| < h²δ_n²} ⊃ Θ and

The Fourier transform φ(s) = ∫exp(ιsv)K(v) dv will be used in the following expressions, where ι is the imaginary unit. Thus K(v) = ∫exp(−ιsv)φ(s) ds. We have

Following the same steps leading to (35), we have

almost surely, where c₅ and c₆ are constants that do not depend on s. Hence

Note that

Therefore, Lemma 6.7 follows from (36)–(38). █