2.1. Presentation of the Problem

Let S = (X,Y,Z) be an

-valued random vector. The data available consist of s₁,…,s_n realizations of S₁,…,S_n independent and identically distributed as f_S(.,.,.), an unknown density function with associated cumulative distribution function F_S(.,.,.). Let us assume that the third coordinate of S, Z, is square integrable, that is,

.

Define f (x,y) = ∫f_S(x,y,z) dz, φ(x,y) = ∫z f_S(x,y,z) dz, and the conditional expectation

on f (x,y) > 0.

Then the initial system to be studied is the following system, where the function m is defined on a subset of

and is at least continuous:

The first objective is to detail the properties of the solution (if it exists) of this system. Moreover, because it depends on an unknown parameter, it has to be estimated through a nonparametric method.

The estimator of the function m(x,y) is calculated using nonparametric kernel estimation. (Other nonparametric methods could also have been used, such as local polynomial modeling; see Fan and Gijbels, 1996.) Let K denote a kernel function of one dimension, centered and separable, and σ_n a bandwidth parameter. Consequently, the kernel estimator of m is defined on

as

where

are the respective estimators of f (x,y) and φ(x,y).

Remark 2.1. This expression could have been written with a kernel of dimension two, which corresponds to the two-dimensional random vector (X,Y). Nevertheless, to simplify the calculus and especially the proofs, we assume that the kernel function is centered and separable, to decompose it into two independent kernel functions of dimension one (see Wand and Jones, 1995, or Green and Silverman, 1994).

Remark 2.2. Moreover, a unique bandwidth parameter σ_n is considered for all the variables. A more general definition could have been used with two different smoothing parameters σ_n^X,σ_n^Y, which are approximately equal up to a multiplicative constant term. For the sake of simplicity, it is assumed that σ_n^X = σ_n^Y = σ_n.

Therefore, the associated estimated system (3), where

is the kernel estimator of

, is defined by

Before studying the convergence of two possible solutions, let us check whether or not there exists a unique solution for both systems and also if the solutions are stable under small perturbations of initial conditions. In other words, the objective of the next part is to study the well-posedness of a particular inverse problem.

2.2. Link with Inverse Problems Theory

The first equation of the system (2) can be rewritten the following way:

or equivalently

Denote by A an operator that depends on the arguments f, φ, and λ such that

The second equation may be more interesting to define the operator A because it is linear in the argument f. It may be in particular an easier way of writing when replacing the initial arguments f and φ by estimators

into the operator A and studying some asymptotic convergence properties.

Therefore, solving equation (2) is equivalent to solving

The objective is then to express the solution λ as a function of f and φ, that is, λ = Φ(f,φ). Such a problem obviously belongs to the inverse problem theory (Tikhonov & Arsenin, 1977; O'Sullivan, 1986; Florens, 2003). The properties of the solution λ will in particular depend on the regularity of the initial operator A.

A classical class of inverse problems deals with integral operators. Inversion of some integral operators has already been studied, such as the Fredholm integral (Groetsch, 1984) and the Abel-type equation (Garza, Hall, and Ruymgaart, 1999). These kinds of operators are usually linear and compact on Hilbert spaces

(Van Rooij and Ruymgaart, 1999). The differences between these examples and (4) are mainly twofold. First, without imposing restrictions on the form of the function m, the relation (4) defines a nonlinear equation to solve. Therefore, one important step of this analysis will be to try to linearize this relation to study its properties. Further, the operator to invert is a differential one and then, contrary to integral operators, does not have the good regularity properties expected for the previous ones. For example, A will not be continuous with respect to the uniform norm but rather for some Sobolev norm (detailed later).

Let us now study the well-posedness of our problem: if it admits a unique solution stable under perturbation of the function m (when m is replaced by a kernel estimator

).

2.3. Identification and Overidentification

The study of identification and overidentification is needed for both systems (2) and (3).

2.3.1. Study of the True Differential Equation.

Consider first the initial system (2). The proof for existence and uniqueness of a solution λ is given using a well-known analysis result on ordinary differential equations: the Cauchy–Lipschitz theorem (Schwartz, 1970).

Let I = [x₀ − a,x₀ + a] for a > 0 and C(I) be the Banach space of continuous functions x(t) on I with the sup norm ∥x∥_∞ = max_t∈I|x(t)|.

In what follows, without loss of generality, assume that x₀ = λ₀ = 0, so that I = [−a,a]. Define D = {(x,y);|x| ≤ a,|y| ≤ b} and C_b,0(I) = {u ∈ C(I);u(0) = 0,∥u∥_∞ ≤ b}. The following assumptions are made.

Assumption A. max_(x,y)∈D|m(x,y)| < b/a.

Assumption B. |m(x,y₁) − m(x,y₂)| ≤ k|y₁ − y₂|, ∀(x,y_i) ∈ D such that c = ka < 1.

LEMMA 2.1. Under Assumptions A and B, there exists a unique solution to (2), λ ∈ C_b,0(I).

This result comes from the Cauchy–Lipschitz theorem; the mathematical points are detailed in the Appendix.

Both Assumptions A and B lead to the following comments on the regularity imposed on m and f.

Remark 2.3. First, the important condition for proving existence and uniqueness of a solution is Assumption B. Indeed, Assumption A is just imposed by the local definition of our solution on D. Moreover, it is not very restrictive because the parameter a will usually be chosen small enough; in particular, if the function m is continuous, this assumption can be satisfied for a well-chosen a.

Remark 2.4. Assumption B imposes m to be continuous on D and to satisfy the Lipschitz condition uniformly on the second argument; in particular a sufficient condition on m to verify this assumption is to be continuously differentiable of order one on D (or that m is continuously differentiable of order two on D, as needed for the mean square convergence).

Remark 2.5. Thanks to the Cauchy–Lipschitz theorem, a unique solution λ can be defined inside a compact neighborhood of the initial condition (0,0). It could be interesting to find the optimal interval I and a maximal solution defined on this optimal interval, or, in other words, to check if the solution can be extended to a larger interval. Theoretically, under the same assumptions, it can be proved that a unique maximal solution exists. That means concretely to piece together local solutions if the intersection of their definition intervals is not empty.

Remark 2.6. Note that Assumptions A and B are in particular satisfied under standard regularity assumptions on f, such as continuity and continuous derivability, and it is not necessary to restrict the definition domain of f to compact support (contrary to standard assumptions in nonparametric estimation) or to worry about kernel estimation in the boundary of finitely supported random variables. In other words, in principle there is no need here for imposing compact supports on the variables X,Y,Z or for bounding f from below. The restriction to a compact neighborhood around (0,0) will be imposed in a second step when defining a unique solution to the differential equation. Then, construct the conditional expectation m(x,y) (and its associated estimator

) on every point where f (x,y) > 0 and just restrict it to a compact support D only to define the solution of the differential equation.

However, so as not to rule out simple cases such as uniform random variables, we do not define φ and f on

but consider that any compact subset Θ of

containing D is a good candidate for the definition domain of these functions.

Remark 2.7. Moreover, to be sure that the function m is defined on D, we implicitly assume that f is strictly positive on D. By continuity, under the assumption that f (0,0) > 0, it will always be possible to define a neighborhood around (0,0) where m is well defined.

To summarize, here are the assumptions imposed on φ and f.

Assumption A₁ (s₁).

(i) φ and f are continuous bounded functions on a compact subset

, such that D ⊂ int(Θ) (interior of Θ), and continuously differentiable of order s₁ on D;

(ii) f (0,0) > 0.

This assumption depends on one parameter s₁ that captures the regularity of the function f.

2.3.2. Study of the Estimated Differential Equation.

To calculate

, introduce some nonparametric kernel estimator with a kernel function K. As mentioned before, K is a one-dimensional kernel for sake of simplicity. Here are the assumptions required for K.

Assumption A₂ (r,s₂).

This assumption depends on two parameters, r and s₂, which correspond to the order and the regularity of the function K. When r = 2, K is of order two, which is a commonly used assumption. Further restrictions may be imposed on K, as in the next section, with the notion of kernel of order three (which is of course no longer symmetric). Such an additional assumption may be useful when solving the inverse problem, in particular when the operator to inverse regularizes the initial estimator

. This point will be explored later; let us just remark that solving this differential equation allows us to obtain a function λ whose regularity properties are increased compared to m. This smoothing property can obviously also be observed for the associated estimated functions

.

As noted in Remark 2.4, assume that m is continuously differentiable of order one on D; moreover thanks to Assumption A₂ (r,s₂ = 1), the estimator

is also at least continuously differentiable of order one on D. Therefore, the Cauchy–Lipschitz theorem proves that for all n there exists a unique local solution

to the system (3). In particular

Moreover, under the assumption of uniform convergence a.s. of

to (∂/∂y)m (the notation ∂/∂y (resp. ∂/∂x) represents the derivative with respect to the second (resp. first) variable), it is possible to define a global Lipschitz factor for the functions

, and m. This will be useful to check the stability of the inverse problem. Let us first assume the following properties to ensure the uniform convergence of

.

Assumption A₃.

(i) The derivative of K, K′, is bounded and Lipschitzian.

(ii) For n → +∞, σ_n → 0 and nσ_n³/log n → +∞.

LEMMA 2.2. Under Assumptions A₁ (s₁ = 1), A₂ (r,s₂ = 1) with r ≥ 2, and A₃, there exists almost surely a real k > 0 that defines a unique Lipschitz factor for all the functions

, and m on a compact neighborhood of (0,0).

2.3.3. Conclusion on Our Inverse Problem.

The last point to check is the stability of our solutions

, using the result of the previous lemma. Introduce another notation, the Picard operator associated with the equation (2):

such that

and define the estimated associated Picard operator:

As a consequence of the Cauchy–Lipschitz theorem, λ is a fixed point of

is a fixed point of

. Then, Lemma 2.2 allows us to derive the following result:

where d_∞ and d_Πk(I) are uniform norms defined, respectively, on the subsets I and C_b,0(I),

and c is a constant that does not depend on n. (See the Appendix for details.)

This inequality is quite interesting because it links known operators

to unknown functions

. Indeed,

represents the integral of the kernel regression, and it is easier to study the convergence of

. Therefore, this inequality gives a first result of convergence for the interest parameter λ, under the condition of uniform convergence of

. This is a first easy result to obtain without calculating the function λ explicitly. The next section will give results on the asymptotic rate of convergence of

in mean square and in law. So, this relation is a first step in exploiting the link between m and λ, but of course, this first analysis needs to be investigated.

Moreover, this inequality allows us to conclude that the inverse problem is stable, because the perturbation involved in the solution is controlled by the initial perturbation. Therefore, the system (2) defines a well-posed inverse problem, and there exists a unique solution that is stable.

The last result proves the uniform consistency in probability of our estimator. Indeed, following equation (5), we have

where C = a/(1 − c).

PROPOSITION 2.1. Assume the following properties:

. Then,

The next sections deal with numerical implementation, which is useful in practice, and results on the asymptotic convergence of the estimated solution

to the true one λ.

4.1. Linearization of the Problem

The following preliminary result tries to transform the nonlinear equation into a linear problem.

PROPOSITION 4.1. (i) Under the assumptions of Proposition 2.1, it can be proved that

where

.

where

.

This preliminary result is crucial for the rest of the paper. Indeed, our problem is split into two parts: a linear one that behaves as the integral of the density function and a residual term that converges to zero. The next step is to study the convergence of each term.

Remark 4.1. Both residual terms R_1,n(x) and R_2,n(x) depend on the argument x, in addition to the functions

. The results we demonstrate in what follows are pointwise and not uniform with respect to x. More precisely,

are calculated on the compact neighborhood D strictly included in the definition domain Θ. These norms do not depend on any argument x or y. Both residual terms can be expressed in the following way:

and both o_P(1) and O_P(1) are functions of one argument x, like the functions

.

Remark 4.2. To stick to the most general case, assume that (∂/∂x)m ≠ 0 and that (∂/∂y)m ≠ 0. Indeed, if (∂/∂y)m = 0, the solution would simply be

which will be much easier to study. Moreover, when (∂/∂x)m = 0, consider the particular autonomous equation, that is,

This is also a direct application of integration of nonparametric kernel estimators, with an

rate of convergence, the same result as expected for the case (∂/∂y)m = 0.

The following preliminary result allows us to get rid of the residual term in probability (see Serfling, 1980).

Assumption A₄. For

.

PROPOSITION 4.2. Assume the following properties:

. Then,

The next step is to concentrate on the asymptotic properties of the first term in the decomposition of Proposition 4.1, that is, mean square convergence and optimal choice of the bandwidth. The previous result will be useful at last to conclude for the asymptotic normality of our estimator, because the second part can be neglected in probability. Let us now concentrate on the linear part of our decomposition.

4.2. Asymptotic Mean Square Properties

The next result can be demonstrated with a second-order kernel. This is a usual context for studying kernel estimators, and it is sufficient to prove the asymptotic consistency of our estimated solution. Moreover, we assume from now on that m is a function continuously differentiable of order two. Thanks to Assumption A₂ (r,s₂ = 2), the associated estimator is also continuously differentiable of order two. We first derive a convergence in mean square for the linear part

:

THEOREM 4.1. Assume the following properties:

. Assume moreover that the partial derivatives of f are bounded. Then

where

is a version of Var(Z|X = t,Y = λ(t)).

This first theorem shows the consistency in mean square of

.

Some comparisons can already be derived with the initial nonparametric kernel estimator

. Indeed, recall the expansion of the asymptotic quadratic error for the kernel regression estimator. The asymptotic variance is of order 1/nσ_n², and the square bias is of order σ_n⁴ (Bosq, 1998, or Pagan and Ullah, 1999).

The difference is in the variance term, and it comes from the fact that there is a gain in dimension by applying an integral operator to the initial kernel regression. Indeed, the solution λ is in dimension one, whereas the function m is in dimension two.

4.3. Asymptotic Optimal Choice

The objective of this section is to study more carefully the asymptotic optimal choice of the bandwidth parameter by taking into account the particular smoothing properties of our inverse problem. As has been proved earlier, solving the differential equation allows us to transform a continuous differentiable function of order two, the function m, into a continuous differentiable function of order three, the function λ. Therefore, the minimax properties of our estimated interest parameter,

, should be studied within this second class of functions.

The next result simply derives the classical optimal bandwidth, obtained by minimizing the asymptotic mean square error with respect to σ_n for a fixed n. We will then extend this result and show that we can derive a smaller optimal bandwidth, more adapted to the C³ class of functions.

Let us first present the result of optimality with a second-order kernel.

COROLLARY 4.1. Under the previous assumptions, if

the value of σ_n that minimizes the asymptotic mean square error for a fixed n is

Our next issue is to take into account the properties of the solution λ to achieve a better rate of convergence, that is, the minimax rate of convergence for a given set of functions. Indeed, because m is assumed to be continuously differentiable of order two, λ is then continuously differentiable of order three. Therefore, the optimality will not be reached with a kernel of order two, but rather with one of order three.

The next results show that, without assuming that m is C³ for the two arguments, but only for the second one, a better rate of convergence can be achieved by taking into account the properties of the integral operator applied to the kernel regressor. Let us now present the result of convergence of the quadratic error with the third-order kernel.

THEOREM 4.2. Assume the following properties:

. Assume moreover that f is continuously differentiable of order three with respect to the second argument and that the partial derivatives of f are bounded. Then, the following expansion can be derived:

where

The three-order kernel reduces the bias term. This difference can be explained by a gain in regularity for the estimated solution, compared to the bias for the kernel regressor. Indeed, as mentioned before, the initial function m was continuously differentiable of order two, and the solution λ is then continuously differentiable of order three.

COROLLARY 4.2. Under the same assumptions,

the value of σ_n that minimizes the asymptotic mean square error for a fixed n is

Again, there is a difference with the result for the kernel regression because the optimal bandwidth was of order (1/n)^1/6.

The interest of this result is to take benefit from all the features of our inverse problem to obtain some better optimal results. Indeed, it has already been proved that the rate of convergence for the regression kernel is the best attainable in a minimax sense. Moreover, the optimal bandwidth σ_n* ∼ n^−1/7 leads to the minimax speed of convergence for the continuously differentiable function of order three in dimension one. The estimated solution then achieves the optimal convergence of order n^6/7.

4.4. Asymptotic Normality

Let us now derive an asymptotic normality theorem. This result is less new and has already been proved by Hausman and Newey (1995). In particular, the faster convergence rate for the solution is analogous to the results on averaging kernel estimators over some arguments that are given in Linton and Nielsen (1995) and Newey (1994); see also Robinson (1988).

Two theorems are presented, with the two different assumptions for the kernel order.

THEOREM 4.3. Assume the following properties:

. Assume moreover that the partial derivatives of f are bounded and that nσ_n⁵ → 0.

Therefore,

converges in law to a centered Gaussian distribution with an asymptotic variance term equal to

Remark 4.3. Under the assumptions of the previous theorem, the asymptotic distribution of

is centered to zero. An asymptotic bias equal to zero would have been obtained by undersmoothing the estimator and thus by assuming that σ_n converges faster to zero (than the optimal one). Under the assumption of optimal bandwidth choice, the asymptotic bias term is then equal to

THEOREM 4.4. Assume the following properties:

. Assume moreover that f is continuously differentiable of order three with respect to the second argument and that the partial derivatives of f are bounded. Under the additional assumption nσ_n⁷ → 0, it can be proved that

converges in law to a centered Gaussian distribution with an asymptotic variance term equal to

Remark 4.4. Under the assumption of optimal bandwidth choice, the asymptotic bias term is then equal to

The asymptotic variance given previously,

, very clearly depends on x through the second integral. In particular, when x is very close to zero, the variance is also close to zero. This is a logical property because the solution at point 0 is known and equal to zero. Therefore around this point there is less uncertainty about the estimated solution.

The asymptotic variance estimation has also been discussed by Hausman and Newey (1995) and Newey (1994). It is indeed useful for inference based on the asymptotic normality result and construction of confidence intervals. The asymptotic variance term can of course be estimated by replacing the unknown parameters by kernel estimators to study the consistence of this estimated variance (e.g., Newey, 1994). An alternative approach to variance estimation, or confidence intervals, is the bootstrap technique. The idea is then to sample the data randomly with replacement, estimate the solution

several times, and calculate the empirical variance of these solutions. Thus, the bootstrap provides a way to substitute computation for mathematical analysis if calculating the asymptotic distribution of an estimator is difficult (see, e.g., Horowitz, 2001b).

Remark 4.5. In the classical studies of density kernel estimation and conditional expectation kernel estimation, the covariance term can be neglected because its rate of convergence to zero is faster than the variance one. In this particular problem, let us analyze the covariance between

for x ≠ y:

Therefore, contrary to the usual results, as pointed out by Horowitz (2001a), the covariance term cannot be neglected in this case. An explanation of this result is that both arguments of the conditional expectation function m(t,λ(t)) depend on the same variable t. Under a more general hypothesis such as m(x,y), the covariance term would have been neglected compared to the variance one.