2. Function spaces

In this section, we will introduce some notations and preliminary results that will be used later on. First of all, let $\gamma$ be a nonnegative real number and $z=(x,y)=(x_1,\ldots,x_m,y_1,\ldots,y_k)\in \mathbb {R}^{m}\times \mathbb {R}^{k}$ with $m, k\geqslant 1$ and $m+k=N\geqslant 3$. In what follows, we designate by $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ the weighted Sobolev space

\[ \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{u\in L^{2}\big(\mathbb{R}^{N}\big):\frac{\partial u}{\partial x_i}, ~|x|^{\gamma}\frac{\partial u}{\partial y_j}\in L^{2}\big(\mathbb{R}^{N}\big), i=1,\ldots,m, j=1,\ldots,k\right\}. \]

For simplicity, if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we denote by $\nabla _{\gamma }$ the gradient operator defined by

\[ \nabla_{\gamma}u=\left(\nabla_{x}u, |x|^{\gamma}\nabla_{y} u \right)= \left(u_{x_1},\ldots,u_{x_m}, |x|^{\gamma}u_{y_1},\ldots,|x|^{\gamma}u_{y_k} \right), \]

and so,

\[ |\nabla_{\gamma}u|^{2}=|\nabla_{x}u|^{2}+|x|^{2\gamma}|\nabla_{y}u|^{2}. \]

We would like to mention that $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ is a Hilbert space, when endowed with the scalar product given by

\[ \langle u,v \rangle_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{N})} = \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u\nabla_{\gamma} v + uv \big) \,{\rm d}z, \]

whose the corresponding norm is

\[ ||u||_{\gamma}:=\sqrt{\langle u,u \rangle_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{N})}}=\left(\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{1}/{2}}. \]

By Sobolev inequality found in [Reference Monti33], we have

(2.1)\begin{equation} \left(\int_{\mathbb{R}^{N}}|u|^{2_{\gamma}^{*}} \, {\rm d}z\right)^{2/2_{\gamma}^{*}}\leqslant C(m,k,\gamma)\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big), \end{equation}

from where it follows that the embedding $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{2_{\gamma }^{*}}(\mathbb {R}^{N})$ is continuous, then there is $\mathcal {C}>0$ such that

\[ |u|_{2_{\gamma}^{*}}\leqslant \mathcal{C}||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big). \]

Since

\[ |u|_{2}\leqslant ||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big), \]

the interpolation inequality on the Lebesgue space ensures that the embedding $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is continuous for every $p\in [2, 2^{*}_{\gamma }]$. Thus, for each $p \in [2, 2^{*}_{\gamma }]$ there is $\mathcal {C}_p>0$ such that

(2.2)\begin{equation} |u|_{p}\leqslant \mathcal{C}_p||u||_{\gamma}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big). \end{equation}

Here, we would like point out that if $\Omega _1 \subset \mathbb {R}^{m}$ and $\Omega _2 \subset \mathbb {R}^{k}$ are smooth bounded domains, the embeddings (2.2) still hold by replacing $\mathbb {R}^{N}$ for any set of the form $\Omega =\mathbb {R}^{m} \times \Omega _2$, $\Omega =\Omega _1 \times \mathbb {R}^{k}$ or $\Omega =\Omega _1 \times \Omega _2$, that is,

(2.3)\begin{equation} |u|_{L^{p}(\Omega)}\leqslant \mathcal{C}_p||u||_{\mathcal{H}_{\gamma}^{1,2}(\Omega)}, \quad \forall u \in \mathcal{H}_{\gamma}^{1,2}(\Omega), \end{equation}

where

\[ \mathcal{H}_{\gamma}^{1,2}\big(\Omega\big)=\left\{u\in L^{2}\big(\Omega\big):\frac{\partial u}{\partial x_i}, \ |x|^{\gamma}\frac{\partial u}{\partial y_j}\in L^{2}\big(\Omega\big), i=1,\ldots,m, j=1,\ldots,k\right\} \]

and

\[ ||u||_{\mathcal{H}_{\gamma}^{1,2}(\Omega)}:=\left(\int_{\Omega}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{1}/{2}}. \]

The proof of (2.3) follows by showing the existence of an extension operator that can be done as in [Reference Brézis11, § 9.2]. Moreover, when $\Omega =\Omega _1 \times \Omega _2 \subset \mathbb {R}^{N}=\mathbb {R}^{m} \times \mathbb {R}^{k}$ the embedding $\mathcal {H}_{\gamma }^{1,2}(\Omega ) \hookrightarrow L^{p}(\Omega )$ is compact for every $p\in [1, 2^{*}_{\gamma })$, which is a consequence of the Strauss's compactness lemma [Reference Berestycki and Lions6, theorem A.I].

Setting the space

\[ \mathcal{H}_{\gamma}^{a}(\mathbb{R}^{N})=\left\{u\in\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):\int_{\mathbb{R}^{N}}a(z)|u|^{2} \, {\rm d}z <\infty\right\}, \]

it is easy to see that condition (A1) implies that $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is Hilbert space with scalar product and norm given by

\[ \langle u,v \rangle_{a} = \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u\nabla_{\gamma} v + a(z)uv \big) \,{\rm d}z \quad \hbox{and} \quad ||u||^{2}:=\sqrt{\langle u,u \rangle_{a}}. \]

Moreover, (A1) also yields $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is embedded continuously into $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$. Hence, by (2.2), $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ is embedded continuously in $L^{p}(\mathbb {R}^{N})$ for every $p\in [2, 2^{*}_{\gamma }]$.

Another important space that will consider in the present paper is $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$, which is the closure of $C_{0}^{\infty }(\mathbb {R}^{N})$ with relation to the norm

\[ \|\varphi \|_{\mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N})}=\left(\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z\right)^{{1}/{2}}, \quad \varphi \in C_{0}^{\infty}(\mathbb{R}^{N}). \]

From (2.1) we derive that the embedding $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{2^{*}_{\gamma }}(\mathbb {R}^{N})$ is continuous, that is, there is $\mathcal {S}>0$ such that

\[ |u|_{2^{*}_{\gamma}}\leqslant \mathcal{S}\|u\|_{\mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N})}, \quad \forall u \in \mathcal{D}_{\gamma}^{1,2}(\mathbb{R}^{N}). \]

2.1 Compactness results

In this section, we will prove some important compactness results that will be used in the proof of theorems 1.1, 1.2 and 1.3. Our first result is a version of a famous result due to [Reference Lions29, lemma I.1] for the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Lemma 2.1 Let $r>0, 2\leqslant q < 2^{*}_{\gamma }$ and $(u_n) \subset \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ be a bounded sequence with

\[ \sup_{y\in\mathbb{R}^{k}}\int_{\mathbb{R}^{m}\times B^{k}_r(y)}|u_n|^{q}\,{\rm d}z \longrightarrow 0 \quad \mbox{as} n\longrightarrow\infty, \]

where $B_r^{k}(y)=\{\xi \in \mathbb {R}^{k}\,:\,|\xi - y|< r\}.$ Then, $u_n\longrightarrow 0$ in $L^{p}(\mathbb {R}^{N})$ for $2< p<2^{*}_{\gamma }$.

Proof. As a first step we consider

\[ \mathbb{R}^{N}=\mathbb{R}^{m}\times\mathbb{R}^{k}\subset\bigcup_{y\in\mathbb{R}^{k}}\left(\mathbb{R}^{m}\times B_r(y)\right), \]

in such way that each point of $\mathbb {R}^{k}$ is contained in a finite quantity $l$ of balls of radius $r$. Now, let $q< s<2^{*}_{\gamma }$ and $\alpha :={(s-q)2^{*}_{\gamma }}/{(2^{*}_{\gamma }-q)s}$. Then, if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, the embedding (2.3) together with the interpolation inequality on the Lebesgue space says that

\begin{align*} |u|_{L^{s}(\mathbb{R}^{m}\times B_r(y))} & \leqslant |u|^{1-\alpha}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}|u|^{\alpha}_{L^{2^{*}_{\gamma}}(\mathbb{R}^{m}\times B_r(y))} \\ & \leqslant \mathcal{C}|u|^{1-\alpha}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}\left(\int_{\mathbb{R}^{m}\times B_r(y)}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z\right)^{{\alpha}/{2}}. \end{align*}

Choosing $\alpha =2/s$, we obtain the inequality

\[ \int_{\mathbb{R}^{m}\times B_r(y)}|u|^{s} \, {\rm d}z\leqslant \mathcal{C}^{s}|u|^{(1-\alpha)s}_{L^{q}(\mathbb{R}^{m}\times B_r(y))}\int_{\mathbb{R}^{m}\times B_r(y)}\big(|\nabla_{\gamma}u|^{2}+|u|^{2}\big) \, {\rm d}z, \]

which leads to

\[ \int_{\mathbb{R}^{N}}|u|^{s} \, {\rm d}z \leqslant \mathcal{C}^{s}l\sup_{y\in\mathbb{R}^{k}}\left(\int_{\mathbb{R}^{m}\times B_r(y)}|u|^{q} \,{\rm d}z\right)^{(1-\alpha)s/q}\int_{\mathbb{R}^{N}}\big(| \nabla_{\gamma}u|^{2}+|u|^{2}\big)\, {\rm d}z. \]

As $(u_n)$ is bounded sequence in $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ the last inequality allows us to conclude that $u_n\to 0$ in $L^{s}(\mathbb {R}^{N})$. Now, applying again the interpolation inequality on the Lebesgue spaces we conclude that $u_n \to 0$ in $L^{p}(\mathbb {R}^{N})$ for $2< p<2^{*}_{\gamma }$.

Next, we will show some compactness results that are crucial in our approach. Have this in mind, we will fix the following spaces:

\[ \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,|y|) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}, \]

\[ \mathcal{H}_{\gamma,rad_x}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,y) \ \hbox{for all} \ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}, \]

\begin{align*} \mathcal{H}_{\gamma,rad_x}^{1,2}(\mathbb{R}^{m} \times B_r(0))& =\left\{\mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{m} \times B_r(0)):u(x,y)\right.\nonumber\\ & \left.\quad =u(|x|,y)\ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times B_r(0)\right\}, \end{align*}

\[ \mathcal{H}_{\gamma, rad_x}^{a}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{H}_{\gamma}^{a}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,y) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\} \]

and

\[ \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big)=\left\{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big):u(x,y)=u(|x|,|y|) \ \hbox{for all}\ (x,y) \in \mathbb{R}^{m} \times \mathbb{R}^{k}\right\}. \]

Lemma 2.2 Let $p\in (2,2^{*}_{\gamma })$ and $r>0$. Then, the embedding of $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$ into $L^{p}(\mathbb {R}^{m}\times B_r(0))$ is compact.

Proof. It is enough to show that if $(u_n)$ is a sequence in $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$ such that $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma,rad_x}^{1,2}(\mathbb {R}^{m}\times B_r(0))$, then $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$. Let $\delta >0$ and $h \in C^{\infty }(\mathbb {R},[0,1])$ defined by

\[ h(t)=\left \{ \begin{array}{@{}ll} 0 & \mbox{if} \ |t|\leqslant \delta/2, \\ 1 & \mbox{if} \ |t|\geqslant \delta. \end{array} \right. \]

Notice that if $u\in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{m}\times B_r(0))$ and

\[ v(z):=h(|x|^{2})u(z), \quad z=(x,y)\in\mathbb{R}^{m}\times B_r(0), \]

then $|v|^{2}\leqslant |u|^{2}$,

\[ \left|\frac{\partial v}{\partial x_i}\right|^{2}\leqslant {C}\bigg(|u|^{2}+\left|\frac{\partial u}{\partial x_i}\right|^{2} \bigg), \quad i=1,\ldots,m \]

and

\[ \left|\frac{\partial v}{\partial y_j}\right|^{2} = h^{2}(|x|^{2})\left|\frac{\partial u}{\partial y_j}\right|^{2}, \quad j=1,\ldots,k. \]

Hence,

\begin{align*} \int_{\mathbb{R}^{m}\times B_r(0)}|\nabla_{y}v|^{2}\,{\rm d}z & = \int_{[|x|\geqslant\delta/2]\times B_r(0)}h^{2}(|x|^{2})|\nabla_{y}u|^{2}\,{\rm d}z \\ & \leqslant {C}\int_{[|x|\geqslant\delta/2]\times B_r(0)}|x|^{2\gamma}|\nabla_{y}u|^{2}\,{\rm d}z \\ & \leqslant {C}\int_{\mathbb{R}^{m}\times B_r(0)}|x|^{2\gamma}|\nabla_{y}u|^{2}\,{\rm d}z. \end{align*}

Therefore, $v\in H^{1}(\mathbb {R}^{m}\times B_r(0))$ and

\[ ||v||_{H^{1}(\mathbb{R}^{m}\times B_r(0))}\leqslant {C}||u||_{\mathcal{H}_{\gamma}^{1,2}(\mathbb{R}^{m}\times B_r(0))}. \]

Thereby, setting $v_n(x,y)=h(|x|^{2})u_n(x,y)$ we have $(v_n)$ is a bounded sequence in $H_{rad,x}^{1}(\mathbb {R}^{m}\times B_r(0))$ and $v_n \rightharpoonup 0$ in $H_{rad,x}^{1}(\mathbb {R}^{m}\times B_r(0))$. By [Reference Lions28, lemme III.2],

\[ v_n\to 0 \quad \hbox{in} \ L^{p}(\mathbb{R}^{m}\times B_r(0)),\quad 2< p<2^{*}. \]

Since $v_n(x,y)=u_n(x,y)$ for $(x,y) \in [|x| \geqslant \delta ] \times B_r(0)$, we derive that

(2.4)\begin{equation} u_n\to 0 \quad \hbox{in}\ L^{p}([|x| \geqslant \delta] \times B_r(0)),\quad 2< p<2^{*}. \end{equation}

Furthermore, recalling that if $\Omega =B_\delta (0) \times B_r(0) \subset \mathbb {R}^{N}$ the embedding $\mathcal {H}_{\gamma }^{1,2}(\Omega )$ into $L^{p}(\Omega )$ is compact for $1\leqslant p<2_{\gamma }^{*}$, then

(2.5)\begin{equation} u_n\to 0 \quad \hbox{in} L^{p}([|x|\leqslant\delta]\times B_r(0)). \end{equation}

Accordingly to (2.4) and (2.5), $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ with $p\in (2,2^{*}_{\gamma })$.

Lemma 2.3 Let $p\in (2,2^{*}_{\gamma })$. Then the embedding of $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact.

Proof. In what follows, we set $q \in (2,2^{*}_{\gamma })$ and

\[ k(y,r,O(k))=\{n \in \mathbb{N}\,:\, \exists g_1,g_2,\ldots,g_n \in O(k)\,:\,B^{k}_{r}(g_i(y))\cap B^{k}_r(g_j(y))=\emptyset\}. \]

From definition of $k(y,r,O(k))$,

\[ \lim_{|y| \to +\infty}k(y,r,O(k))={+}\infty. \]

A simple computation gives that for each $u \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$,

\[ \int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u|^{q}\,{\rm d}z \leqslant \frac{|u|_q^{q}}{k(y,r,O(k))}. \]

Arguing as in the last lemma, let $(u_n)$ be a sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. As $(u_n)$ is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, given $\epsilon >0$, the definition of $k(y,r,O(k))$ ensures that there is $R>0$ such that

\[ \sup_{|y| \geqslant R}\int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u_n|^{q}\,{\rm d}z< \frac{\epsilon}{2}, \quad \forall n \in\mathbb{N}. \]

On the other hand, lemma 2.2 yields

\[ \int_{\mathbb{R}^{m} \times B_{R+r}(0)}|u_n|^{q}\,{\rm d}z \to 0, \]

and so,

\[ \sup_{y \in \mathbb{R}^{k}}\int_{\mathbb{R}^{m} \times B^{k}_{r}(y)}|u_n|^{q}\,{\rm d}z \to 0. \]

Now the lemma follows by employing lemma 2.1.

Corollary 2.4 If (A1) holds, then the embedding $\mathcal {H}^{a}_{\gamma,rad_{x,y}}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact.

Proof. The corollary is an immediate consequence of the last lemma, because we have the continuous embedding of $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ into $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Lemma 2.5 Let $(u_n) \subset \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ be a bounded sequence with $u_n \rightharpoonup 0$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Then, for each $0< a< b<+\infty$ and $p \in (2,2^{*}_{\gamma })$,

\[ \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0. \]

Proof. In the sequel, let us consider $g \in C^{\infty }(\mathbb {R},\mathbb {R})$ satisfying

\[ g(t)=0 \quad \mbox{for} \ |t| \leqslant \frac{a}{2} \quad \mbox{or} \ |t| \geqslant b+1 \]

and

\[ g(t)=t \quad \mbox{for} \ a\leqslant |t| \leqslant b. \]

Using the function $g$, we set the sequence $v_n(x,y)=g(u_n(x,y))$. A direct computation shows that $v_n \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ and that it is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Thereby, by lemma 2.3, $v_n \to 0$ in $L^{p}(\mathbb {R}^{N})$, that is,

\[ \int_{\mathbb{R}^{N}}|v_n|^{p}\,{\rm d}z \to 0, \]

and so,

\[ \int_{\{a<|u_n|< b\}}|v_n|^{p}\,{\rm d}z \to 0. \]

Using the fact that $g(t)=t$ for $a \leqslant |t| \leqslant b$, it follows that

\[ \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0, \]

showing the lemma.

Lemma 2.6 Assume that (A1) and (A4) hold. Then the embedding of $\mathcal {H}_{\gamma,rad_x}^{a}(\mathbb {R}^{N}) \hookrightarrow L^{p}(\mathbb {R}^{N})$ is compact for all $p\in [2,2^{*}_{\gamma })$.

Proof. It suffices to show that for any sequence $(u_n) \subset \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$ with $u_n \rightharpoonup 0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, we must have $u_n\to 0$ in $L^{p}(\mathbb {R}^{N})$. By (A4), given $\varepsilon >0$, we can take $R_{\varepsilon }>0$ such that

\[ \frac{1}{a(x,y)}\leqslant\frac{\varepsilon}{2}, \quad \hbox{for all} (x,y)\in\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]. \]

From this,

\[ \int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} |u_n|^{2} \, {\rm d}z \leqslant\frac{\varepsilon}{2}\int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} a(x,y)|u_n|^{2} \, {\rm d}z < \varepsilon M, \]

where $M=\sup _{n}||u_n||^{2}_a$.

Notice that (A1) and (2.3) imply that the space $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{m}\times [|y|\geqslant R_{\varepsilon }])$ is continuously immersed in $L^{2^{*}_{\gamma }}(\mathbb {R}^{m}\times [|y|\geqslant R_{\varepsilon }])$. Then the last inequality together with the interpolation inequality ensures that for each $p\in (2, 2^{*}_{\gamma })$,

\[ \int_{\mathbb{R}^{m}\times [|y|\geqslant R_{\varepsilon}]} |u_n|^{p} \, {\rm d}z < \varepsilon^{{(1-\alpha)p}/{2}} \mathcal{C}_p, \]

where $\alpha ={(p-2)2^{*}_{\gamma }}/{(2^{*}_{\gamma }-2)p}$. Once again by (A1) and lemma 2.2, it is easy to conclude that the embedding of $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$ into $L^{p}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$ is compact for all $p\in [2,2^{*}_{\gamma })$, and so, $u_n\to 0$ in $L^{p}(\mathbb {R}^{m}\times B_{R_{\varepsilon }}(0))$. This proves the desired result.

3. Proof of theorem 1.1

From now on, since we intend to find nonnegative solutions, without loss of generality we assume that

\[ f(s)=0, \quad \forall s \leqslant 0. \]

The energy functional associated with problem $(P)_1$ is the functional $I:\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})\longrightarrow \mathbb {R}$ given by

\[ I(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2} +|u|^{2}\big) \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z. \]

The reader is invited to see that weak solutions of problem $(P)_1$ are critical points of $I$.

In order to overcome the loss of compactness involving the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we will restrict to functional $I$ to the space $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, and in this sense, an important tool in our approach is the Principle of Symmetric Criticality due to Palais (see [Reference Willem38, theorem 1.28]) that will be used to obtain solutions on the whole space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

Our first lemma establishes that $I$ satisfies the mountain pass geometry.

Proof. (i) From $(f_1)$ and $(f_2)$, there is a constant $C>0$ such that

\[ |F(t)|\leqslant \frac{1}{4}|t|^{2}+C|t|^{q}, \quad \hbox{for all} t\in\mathbb{R}. \]

Hence,

\[ I(u) = \frac{1}{2}||u||_{\gamma}^{2} -\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z \geqslant \frac{1}{2}||u||_{\gamma}^{2} - \frac{1}{4}\int_{\mathbb{R}^{N}}|u|^{2}\,{\rm d}z-C\int_{\mathbb{R}^{N}}|u|^{q}\,{\rm d}z. \]

Since the embedding $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \hookrightarrow L^{q}(\mathbb {R}^{N})$ is continuous,

\[ I(u)\geqslant \frac{1}{2}||u||_{\gamma}^{2}-\frac{1}{4}||u||_{\gamma}^{2}-C||u||_{\gamma}^{q}, \]

and so,

\[ I(u)\geqslant \frac{1}{4}||u||_{\gamma}^{2}-C||u||_{\gamma}^{q}. \]

Now, if $||u||_{\gamma }=\rho$ is small enough, it follows that

\[ I(u)\geqslant \frac{1}{4}\rho^{2}-C\rho^{q}:=\alpha>0. \]

(ii) By $(f_3)$ there is a nonnegative function $\varphi \in C_0^{\infty }(\mathbb {R}^{N}) \cap \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \setminus \{0\}$ satisfying

(3.1)\begin{equation} \displaystyle \int_{\mathbb{R}^{N}}(F(\varphi)-\frac{1}{2}|\varphi|^{2})\,{\rm d}z>0. \end{equation}

By using the function $\varphi$ above, for each $t>0$ we fix the function

\[ \varphi_t(z)=\varphi(z/t), \quad \forall z \in \mathbb{R}^{N}. \]

A direct computation leads to

\begin{align*} I(\varphi_t)& =\frac{t^{N-2}}{2}\int_{\mathbb{R}^{N}}|\nabla_{x}\varphi|^{2}\,{\rm d}z +\frac{t^{N+2\gamma-2}}{2}\int_{\mathbb{R}^{N}}|x|^{2\gamma}|\nabla_{y}\varphi|^{2}\,{\rm d}z\\ & \quad + t^{N}\left(\frac{1}{2}\int_{\mathbb{R}^{N}}|\varphi|^{2} \, {\rm d}z - \int_{\mathbb{R}^{N}} F(\varphi)\, {\rm d}z\right). \end{align*}

Now, the fact that $0\leqslant \gamma <1$ combines with (3.1) to give

\[ I(\varphi_t) \to -\infty \quad \mbox{as} \ t \to +\infty. \]

Therefore, the result follows by fixing $v=\varphi _t$ for $t$ large enough.

The last lemma permits to conclude that the mountain pass level given by

\[ d:=\inf_{g\in\Gamma}\max_{t\in [0,1]}I(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\xi \in C([0,1],\mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N})):\xi(0)=0 \hbox{and}\ \xi(1)=v \}, \]

is well defined.

Since we are not assuming in this section the Ambrosetti–Rabinowitz condition, we will adapt for our problem the same approach explored in [Reference Jeanjean23], which was also used in [Reference Hirata, Ikoma and Tanaka22].

Hereafter, let us consider an auxiliary functional $\tilde {I}:\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \to \mathbb {R}$ defined by

\[ \tilde{I}(s,u(z))=I(u({\rm e}^{{-}s}x,{\rm e}^{-(\gamma+1)s}y)), \quad z=(x,y) \in \mathbb{R}^{N}=\mathbb{R}^{m} \times \mathbb{R}^{k}, \]

that is,

\[ \tilde{I}(s,u)= \frac{{\rm e}^{(m+(\gamma+1)-2)s}}{2}\|u\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}+\frac{{\rm e}^{(m+(\gamma+1)k)s}}{2}|u|_{2}^{2}- {\rm e}^{(m+(\gamma+1)k)s}\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z. \]

It is easy to check that

\[ \tilde{I}(0,u)=I(u), \quad \forall u \in \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}). \]

In what follows, we equip the space $\mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ with the standard product norm $\|(s,u)\|_{\mathbb {R} \times \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})}=(|s|^{2}+\|u\|_{\gamma }^{2})^{{1}/{2}}$.

Arguing of the same way as in lemma 3.1, it is easy to see that $\tilde {I}$ also satisfies the mountain pass geometry, more precisely, we have the result below:

From now on, we denote by $\tilde {d}$, the mountain pass level associated with $\tilde {I}$ given by

\[ \tilde{d}:=\inf_{\tilde{g}\in\Gamma}\max_{t\in [0,1]}\tilde{I}(\tilde{g}(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\tilde{g} \in C([0,1],\mathbb{R} \times \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N})\,:\,\tilde{g}(0)=0 \hbox{and} \ \tilde{g}(1)=(0,v) \}. \]

From definition of $I$ and $\tilde {I}$, we derive the equality $d=\tilde {d}$, which permits to repeat the same arguments explored in the proof of [Reference Hirata, Ikoma and Tanaka22, propostion 4.2] (see also [Reference Jeanjean23, propostion 2.2]) to show the following proposition:

The next lemma establishes the boundedness of the sequence $(u_n) \subset \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ that was obtained in the last proposition.

Proof. From proposition 3.3,

\[ \left\{ \begin{array}{l} \dfrac{{\rm e}^{(m+(\gamma+1)-2)s_n}}{2}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)}^{2}+ \dfrac{{\rm e}^{(m+(\gamma+1)k)s_n}}{2}|u_n|_{2}^{2}-{\rm e}^{(m+(\gamma+1)k)s_n}\\ \quad\displaystyle \int_{\mathbb{R}^{N}}F(u_n)\,{\rm d}z=\tilde{d}+o_n(1)\\ \mbox{} \\ \dfrac{(m+(\gamma+1)-2)\,{\rm e}^{(m+(\gamma+1)-2)s_n}}{2}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}\\\quad +\dfrac{(m+(\gamma+1)k)\,{\rm e}^{(m+(\gamma+1)k)s_n}}{2} |u_n|_{2}^{2}\\ \mbox{}\\ \quad -(m+(\gamma+1)k)\,{\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle\int_{\mathbb{R}^{N}}F(u_n)\,{\rm d}z=o_n(1) \\ \mbox{}\\ {{\rm e}^{(m+(\gamma+1)-2)s_n}}\|u_n\|_{\mathcal{D}_{\gamma}^{1,2} \big(\mathbb{R}^{N}\big)}^{2}+{{\rm e}^{(m+(\gamma+1)k)s_n}}|u_n|_{2}^{2}- {\rm e}^{(m+(\gamma+1)k)s_n}\\ \quad\displaystyle \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=o_n(1). \end{array} \right. \]

The first and second equation of the previous system ensure that $(\|u_n\|_{\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})})$ is bounded. From $(f_1)-(f_2)$, there is $C>0$ such that

\[ |f(t)t| \leqslant \frac{1}{4}|t|^{2}+C|t|^{2^{*}}, \quad \forall t \in \mathbb{R}. \]

Using this information in the last equation of the above system, we get

\[ |u_n|_{2}^{2} \leqslant C\int_{\mathbb{R}^{N}}|u_n|^{2^{*}}\,{\rm d}z+o_n(1), \quad \forall n \in \mathbb{N}. \]

Now, employing (2.1) we arrive at

\[ |u_n|_{2}^{2} \leqslant C\|u_n\|_{\mathcal{D}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)}^{2}+o_n(1), \quad \forall n \in \mathbb{N}, \]

from where it follows that $(u_n)$ is a bounded sequence in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$.

3.1 Proof of theorem 1.1

By proposition 3.3 there is $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ satisfying

1. (a) $s_n \to 0;$

2. (b) $\tilde {I}(s_n,u_n) \to \tilde {d};$

3. (c) $({\partial }/{\partial u})\tilde {I}(s_n,u_n)\to 0$ strongly in $(\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) $({\partial }/{\partial s})\tilde {I}(s_n,u_n)\to 0$ in $\mathbb {R}$.

By lemma 3.4 the sequence $(u_n)$ is bounded in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, then for some subsequence, still denoted by itself, there is $u_0 \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

\[ u_n \rightharpoonup u_0 \quad \mbox{in} \ \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}). \]

The item $(d)$ implies that $({\partial }/{\partial u})\tilde {I}(s_n,u_n)w\to 0$ for all $w \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is,

\begin{align*} & {\rm e}^{(m+(\gamma+1)-2)s_n}\int_{\mathbb{R}^{N}}\nabla_{\gamma} u_n\nabla_{\gamma}w\,{\rm d}z\\ & \quad+{{\rm e}^{(m+(\gamma+1)k)s_n}}\int_{\mathbb{R}^{N}}u_nw\,{\rm d}z- {\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle \int_{\mathbb{R}^{N}}f(u_n)w\,{\rm d}z=o_n(1). \end{align*}

Since $s_n \to 0$, it is easy to check that the above equality leads to

\[ \int_{\mathbb{R}^{N}}\nabla_{\gamma} u_0\nabla_{\gamma} w\,{\rm d}z+\int_{\mathbb{R}^{N}}u_0w\,{\rm d}z-\displaystyle \int_{\mathbb{R}^{N}}f(u_0)w\,{\rm d}z=0, \quad \forall w \in \mathcal{H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb{R}^{N}), \]

showing that $I'(u_0)w=0$ for all $w \in \mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is, $u_0$ is a critical point of $I$. We claim that $u_0 \not = 0$, because if $u_0=0$ the lemma 2.3 gives that

(3.2)\begin{equation} \int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z \to 0, \quad \forall p \in (2,2^{*}_{\gamma}). \end{equation}

From $(f_1)-(f_2)$, given $\epsilon >0$, there is $C>0$ such that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon \int_{\mathbb{R}^{N}}|u_n|^{2}\,{\rm d}z+C\int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z. \]

As $(u_n)$ is bounded in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, it follows that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon M+C\int_{\mathbb{R}^{N}}|u_n|^{p}\,{\rm d}z, \]

where $M=\sup _{n \in \mathbb {N}} \int _{\mathbb {R}^{N}}|u_n|^{2}\,{\rm d}z$. Now, using the limit (3.2) we derive that

\[ \limsup_{n \to +\infty} \int_{\mathbb{R}^{N}}|f(u_n)u_n|\,{\rm d}z \leqslant \epsilon M. \]

As $\epsilon$ is arbitrary, the last limit yields

\[ \lim_{n \to +\infty} \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=0. \]

This limit combined together with the limit $({\partial }/{\partial u})\tilde {I}(s_n,u_n)u_n=o_n(1)$ allows to deduce that $u_n \to 0$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Hence,

\[ \tilde{I}(s_n,u_n) \to 0, \]

which is absurd, because $\tilde {I}(s_n,u_n) \to \tilde {d}>0$. Thus $u_0$ is a nontrivial critical point of $I$ in $\mathcal {H}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$.

Next, we will show that $u_0$ is in fact a critical point of $I$ in the space $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$. To see why, we will apply the Principle of Symmetric Criticality [Reference Palais35] (see also [Reference Willem38, theorem 1.28]). In what follows, let us denote by $G$ the following subgroup of $O(N)$:

\[ G=\left\{g\in O(N)\,:\,g(x,y)=(h(x),l(y))\ \hbox{with}\ h\in {O}(m) \ \mbox{and} \ l \in {O}(k) \right\} \]

and the action $G\times \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N}) \longrightarrow \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$ is given by

\[ (gu)(x,y)=u(h(x),l(y))\quad \hbox{for all} (x,y)\in\mathbb{R}^{m}\times\mathbb{R}^{k}, \]

which is isometric, that is, $\|gu\|_{\gamma }=\|u\|_{\gamma }$. Furthermore, the functional $I$ is invariant under the action of $G$, since for all $g\in G$ and $u \in \mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$,

\[ I(gu) = \frac{1}{2}||gu||^{2}_{\gamma} -\int_{\mathbb{R}^{N}}F(gu) \,{\rm d}z=\frac{1}{2}||u||^{2}_{\gamma} - \int_{\mathbb{R}^{N}}F(u) \,{\rm d}z = I(u). \]

From the above commentaries, it is easy to check that

\[ Fix(G) = \{ u\in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big)\,:\, gu=u, \forall g\in G\}=\mathcal{H}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big). \]

Hence, by the Principle of Symmetric Criticality of Palais, $u_0$ is a nontrivial critical point of $I$ in $\mathcal {H}_{\gamma }^{1,2}(\mathbb {R}^{N})$, and consequently $u_0$ is the required nontrivial solution of $(P)_1$. Finally, we can see that $u_0$ is nonnegative, because it satisfies

\[ \int_{\mathbb{R}^{N}}\big(\nabla_{\gamma} u_0\nabla_{\gamma}\varphi +u_0\varphi \big)\,{\rm d}z =\int_{\mathbb{R}^{N}} f(u_0)\varphi \,{\rm d}z, \quad \forall \varphi \in \mathcal{H}_{\gamma}^{1,2}\big(\mathbb{R}^{N}\big). \]

Then, choosing the test function $\varphi =u_0^{-}:=\min \{u,0\}$, we find that $||u_0^{-}||_{\gamma }=0$. This proves that $u_0$ is nonnegative.

4. Proof of theorem 1.2

In what follows, as in the previous section, without loss of generality we assume that

\[ f(s)=0, \quad \forall s \leqslant 0. \]

The energy functional associated with problem $(P)_0$ is the functional $J:\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})\longrightarrow \mathbb {R}$ given by

\[ J(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}|\nabla_{\gamma}u|^{2} \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z. \]

The reader is invited to see that weak solutions of problem $(P)_0$ are critical points of $I$.

In order to overcome the loss of compactness involving the space $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$, we will restrict to functional $J$ to the space $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. In this section, $\|\,\,\|$ denotes the norm $\|\,\,\|_{\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})}$.

Our first lemma shows that $J$ verifies the mountain pass geometry. However, we will omit its proof because the arguments are similar those used in the proof of lemma 3.1.

The last lemma permits to conclude that the mountain pass level given by

\[ d_0:=\inf_{g\in\Gamma}\max_{t\in [0,1]}J(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{g\in C([0,1],\mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big):g(0)=0 \ \hbox{and}\ g(1)=v \} \]

is well defined.

Arguing as in § 3, we will consider an auxiliary functional $\tilde {J}:\mathbb {R} \times \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}) \to \mathbb {R}$ given by

\[ \tilde{J}(s,u(z))=J(u({\rm e}^{{-}s}x,{\rm e}^{-(\gamma+1)s}y)), \quad z=(x,y) \in \mathbb{R}^{N}=\mathbb{R}^{m} \times \mathbb{R}^{k}, \]

that is,

\[ \tilde{J}(s,u)= \frac{{\rm e}^{(m+(\gamma+1)-2)s}}{2}\|u\|^{2}-{\rm e}^{(m+(\gamma+1)k)s}\int_{\mathbb{R}^{N}}F(u)\,{\rm d}z. \]

It is easy to prove that $\tilde {J}$ also satisfies the mountain pass geometry, more precisely we have the result below

From now on, we denote by $\tilde {d}_0$, the mountain pass level associated with $\tilde {J}$ given by

\[ \tilde{d}_0:=\inf_{\tilde{g}\in\Gamma}\max_{t\in [0,1]}\tilde{J}(\tilde{g}(t))\geqslant\alpha>0, \]

where

\[ \Gamma=\{\tilde{g} \in C([0,1],\mathbb{R} \times \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big):\tilde{g}(0)=0 \ \hbox{and}\ \tilde{g}(1)=(0,v) \}. \]

From definition of $J$ and $\tilde {J}$, we also derive that equality $d_0=\tilde {d}_0$, which permits to repeat the same arguments explored in the proof of [Reference Hirata, Ikoma and Tanaka22, propostion 4.2] (see [Reference Jeanjean23, propostion 2.2]) to get the following proposition

The next lemma establishes the boundedness of the sequence $(u_n) \subset \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ that was obtained in the last proposition. Since the proof follows the same spirit of lemma 3.4 we also omit its proof.

4.1 Proof of theorem 1.2

By proposition 4.3, there is $\{(s_n,u_n)\} \subset \mathbb {R} \times \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ satisfying

1. (a) $s_n \to 0;$

2. (b) $\tilde {J}(s_n,u_n) \to \tilde {d};$

3. (c) $({\partial }/{\partial u})\tilde {J}(s_n,u_n)\to 0$ strongly in $(\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N}))^{*}$;

4. (d) $({\partial }/{\partial s})\tilde {J}(s_n,u_n)\to 0$ in $\mathbb {R}$.

By lemma 3.4 the sequence $(u_n)$ is bounded in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, then for some subsequence, still denoted by itself, there is $u_0 \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$ such that

\[ u_n \rightharpoonup u_0 \quad \mbox{in} \ \mathcal{D}_{\gamma,rad_{x,y}}^{1,2}\big(\mathbb{R}^{N}\big). \]

The item (d) implies that $({\partial }/{\partial u})\tilde {J}(s_n,u_n)w\to 0$ for all $w \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is,

\[ {\rm e}^{(m+(\gamma+1)-2)s_n}\int_{\mathbb{R}^{N}}\nabla_{\gamma} u_n\nabla_{\gamma} w\,{\rm d}z-{\rm e}^{(m+(\gamma+1)k)s_n}\displaystyle \int_{\mathbb{R}^{N}}f(u_n)w\,{\rm d}z=o_n(1). \]

Since $s_n \to 0$, a simple computation guarantees that $u_0$ satisfies the equality below

\[ \int_{\mathbb{R}^{N}}\nabla_{\gamma} u_0\nabla_{\gamma} w\,{\rm d}z-\displaystyle \int_{\mathbb{R}^{N}}f(u_0)w\,{\rm d}z=0, \]

showing that $J'(u_0)w=0$ for all $w \in \mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, that is, $u_0$ is a critical point of $J$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. We claim that $u_0 \not = 0$, because if $u_0=0$, lemma 2.5 gives that

(4.1)\begin{equation} \int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z \to 0, \end{equation}

for all $0< a< b<+\infty$ and $p \in (2,2^{*}_{\gamma })$.

From $(f_4)-(f_5)$, given $\epsilon >0$ and $p \in (2,2^{*}_{\gamma })$, there are $C>0$ and $a< b<+\infty$ such that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon \int_{\mathbb{R}^{N}}|u_n|^{2^{*}_{\gamma}}\,{\rm d}z+C\int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z. \]

As $(u_n)$ is bounded in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$, it follows that

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n| \,{\rm d}z\leqslant \epsilon M+C\int_{\{a<|u_n|< b\}}|u_n|^{p}\,{\rm d}z, \]

where $M=\sup _{n \in \mathbb {N}}\int _{\mathbb {R}^{N}}|u_n|^{2^{*}_{\gamma }}\,{\rm d}z$. Now, using the limit (4.1) we see that

\[ \limsup_{n \to +\infty} \int_{\mathbb{R}^{N}}|f(u_n)u_n|\,{\rm d}z \leqslant \epsilon M. \]

Thereby, as $\epsilon$ is arbitrary, we can infer that

\[ \lim_{n \to +\infty} \int_{\mathbb{R}^{N}}f(u_n)u_n\,{\rm d}z=0. \]

This limit combined with the limit $({\partial }/{\partial u})\tilde {J}(s_n,u_n)u_n=o_n(1)$ ensures that $u_n \to 0$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. Hence,

\[ \tilde{J}(s_n,u_n) \to 0, \]

which is absurd, because $\tilde {J}(s_n,u_n) \to \tilde {d}_0>0$. Therefore, $u_0$ is a nontrivial critical point of $J$ in $\mathcal {D}_{\gamma,rad_{x,y}}^{1,2}(\mathbb {R}^{N})$. As in § 3, we can again use the Principle of Symmetric Criticality to prove that $u_0$ is a critical point of $J$ in $\mathcal {D}_{\gamma }^{1,2}(\mathbb {R}^{N})$.

5. Proof of theorem 1.3

In this section, the energy functional associated with $(P)_a$ is given by

\[ I(u)=\frac{1}{2}\int_{\mathbb{R}^{N}}\big(|\nabla_{\gamma}u|^{2} +a(z)|u|^{2}\big) \, {\rm d}z - \int_{\mathbb{R}^{N}} F(u)\, {\rm d}z, \quad \forall u \in \mathcal{H}_{\gamma}^{a}(\mathbb{R}^{N}). \]

Unfortunately the space $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$ does not have good compactness embedding to help us to find a nontrivial critical point $u$ for $I$. Following the same approach explored in the previous sections, we will restrict $I$ to the space $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$. Since $I$ satisfies the mountain pass geometry in $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$, there is $(u_n) \subset \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ such that

(5.1)\begin{equation} I(u_n)\longrightarrow d_a \quad \hbox{and}\quad I'(u_n)\longrightarrow 0, \end{equation}

where

\[ d_a:=\inf_{g\in\Gamma}\max_{t\in [0,1]}I(g(t))\geqslant\alpha>0, \]

where

\[ \Gamma_a=\{g\in C([0,1], \mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N})\,:\,g(0)=0 \ \hbox{and} g(1)=v \}. \]

Using well-known arguments, the condition $(f_7)$ helps us to prove that $(u_n)$ is bounded in $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$. Since $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ is reflexive, we can assume that there exists $u_0\in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ such that, up to subsequence,

1. • $u_n \rightharpoonup u_0$ in $\mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N}),$

2. • $u_n \rightarrow u_0$ in $L_{loc}^{p}(\mathbb {R}^{N})\ \text {for all}\ p\in (2,2^{*}_{\gamma })$,

3. • $u_n \rightarrow u_0\ a. e.\ \textit {in}\ \mathbb {R}^{N}$.

As $I'(u_n)v = o_n(1)$ for all $v \in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$, the above convergences yield

\[ I'(u_0)v = 0\quad \hbox{for all}\ v \in\mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N}), \]

from where it follows that $u_0$ is a critical point of $I$.

Lemma 5.1 Let $(u_n)\subset \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ be a sequence such that

\[ I(u_n)\longrightarrow d_a>0\quad \hbox{and}\quad I'(u_n)\longrightarrow 0. \]

Then, there exist $(y_n)\subset \mathbb {R}^{k}$ and $r, \beta >0$ such that

\[ \int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{q}\,{\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N} . \]

Proof. In what follows, we will argue by contradiction. If the lemma is not true, we can employ lemma 2.1 to get

\[ u_n\to 0\ \hbox{in}\ L^{q}(\mathbb{R}^{N})\quad \hbox{for}\ 2< q<2^{*}_{\gamma}. \]

From $(f_1)$ and $(f_2)$, for each $\varepsilon >0$, there is a constant $C_{\varepsilon }>0$ such that

\[ |f(u_n)u_n|\leqslant \varepsilon |u_n|^{2}+C_{\varepsilon}|u_n|^{q},\quad \hbox{for all}\ n\in\mathbb{N}. \]

Hence,

\[ \int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z\leqslant\varepsilon C+C_{\varepsilon}\int_{\mathbb{R}^{N}}|u_n|^{q}\, {\rm d}z, \]

and so,

\[ \limsup_{n}\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z\leqslant\varepsilon C, \]

leading to

\[ \lim_{n}\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z=0. \]

As $I'(u_n)u_n = o_n(1)$, we have

\[ ||u_n||_{\gamma}^{2}=I'(u_n)u_n+\int_{\mathbb{R}^{N}}|f(u_n)u_n|\, {\rm d}z=o_n(1), \]

that is,

\[ u_n\to 0\quad \hbox{in}\ \mathcal{H}^{a}_{\gamma,rad_{x}}(\mathbb{R}^{N}). \]

Therefore

\[ I(u_n)\to I(0)=0, \]

which is a contradiction, because $I(u_n)\longrightarrow d_a>0$.

5.1 Proof of theorem 1.3

In what follows, we will divide the proof into two cases:

Case 1: (A3) holds.

In this case, we can assume that the weak limit $u_0 \in \mathcal {H}^{a}_{\gamma,rad_{x}}(\mathbb {R}^{N})$ of the sequence given in (5.1) is nontrivial. Otherwise, by lemma 5.1, there exist $(y_n)\subset \mathbb {R}^{k}$ and $r, \beta >0$ such that

\[ \int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{2} \, {\rm d}z \geqslant\beta, \quad \forall n\in\mathbb{N}. \]

Increasing $r$ if necessary, we can suppose that $y_n \in \mathbb {Z}^{k}$. Now, setting the sequence

\[ v_n(z)=v_n(x,y)=u_n(x,y+y_n), \quad \forall z=(x,y)\in\mathbb{R}^{N}, \]

it follows that $(v_n)\subset \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. As we are supposing that (A3) occurs, a simple computation shows that

\[ ||v_n||_{a}=||u_n||_{a}. \]

Hence $(v_n)$ is bounded, and for a subsequence of $(v_n)$, still denoted by itself, there is $v\in \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$ such that $v_n \rightharpoonup v$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. By lemma 2.2 $v_n \rightarrow v$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ for all $p\in (2,2^{*}_{\gamma })$ and $v_n(z) \rightarrow v(z)$ almost everywhere in $\mathbb {R}^{N}$. Furthermore,

\[ I(v_n)=I(u_n)\quad \hbox{and}\quad I'(v_n)\longrightarrow 0, \]

from where it follows that $I'(v)w=0$ for all $w\in \mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, showing that $v$ is a critical point for $I$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. On the other hand, the equality

\[ \int_{\mathbb{R}^{m}\times B_r(0)}|v_n|^{q} \, {\rm d}z=\int_{\mathbb{R}^{m}\times B_r(y_n)}|u_n|^{q} \, {\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N}, \]

together with the limit $v_n \rightarrow v$ in $L^{p}(\mathbb {R}^{m}\times B_r(0))$ (see lemma 2.2) gives

\[ \int_{\mathbb{R}^{m}\times B_r(0)}|v|^{q} \, {\rm d}z\geqslant\beta, \quad \forall n\in\mathbb{N}, \]

showing that $v\neq 0$.

Case 1: (A4) holds.

If (A4) holds, lemma 2.6 implies that

\[ u_n \rightarrow u_0\quad in\ L^{p}(\mathbb{R}^{N})\quad \hbox{for all}\ p\in[2,2^{*}_{\gamma}). \]

This limit combined with lemma 2.6 ensures that

\[ \int_{\mathbb{R}^{N}}f(u_n)(u_n-u_0) \,{\rm d}z\longrightarrow 0. \]

Now, since $I'(u_n)\to 0$ and $u_n \rightharpoonup u_0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$, we find that

\begin{align*} ||u_n-u_0||_{a}^{2} & = I'(u_n)u_n + \int_{\mathbb{R}^{N}}f(u_n)u_n \,{\rm d}z - I'(u_n)u_n \\ & \quad- \int_{\mathbb{R}^{N}}f(u_n)u_0 \,{\rm d}z + o_n(1)=o_n(1), \end{align*}

which shows that $u_n \to u_0$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$. From this,

\[ I(u_0)=d_a>0, \]

and so, $u_0$ is a nontrivial critical point for $I$ in $\mathcal {H}_{\gamma, rad_x}^{a}(\mathbb {R}^{N})$.

Finally, in order to conclude the proof, we must show that $u$ is a critical point for $I$ in $\mathcal {H}_{\gamma }^{a}(\mathbb {R}^{N})$. As in§ 3 and 4, it is enough to apply the Principle of Symmetric Criticality. However, since the argument are the same we omit its proof.

6. Proof of theorem 1.4

Suppose that $u\in \mathcal {H}^{a}(\mathbb {R}^{N})$ is a solution of $(P)_a$. For any $R>0$, $0< r\leqslant R/2$, let $\eta \in C^{\infty }(\mathbb {R}^{N})$, $0\leqslant \eta \leqslant 1$ with

\[ \eta(z)=\left \{\begin{array}{@{}ll} 1 & \mbox{if} \ |z|\geqslant R, \\ 0 & \mbox{if} \ |z|\leqslant R-r, \end{array}\quad \hbox{and}\quad |\nabla\eta|\leqslant\frac{2}{r\sqrt{R_{\gamma}}} \right. \]

where $R_{\gamma }=\max \{1, R^{2\gamma }\}$. For each $L>0$, let us define

\[ u_{L}=\left \{\begin{array}{@{}ll} u & \mbox{if}\ u\leqslant L, \\ L & \mbox{if}\ u\geqslant L, \end{array} \right. \]

$v_L=\eta ^{2}uu_{L}^{2(\beta -1)}$ and $w_L=\eta uu_{L}^{\beta -1}$ with $\beta >1$ to be determined later on. Then

\[ \nabla_{\gamma}u\nabla_{\gamma}v_{L}=\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}+2(\beta-1)\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}+2\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \]

and choosing $v_L$ as test function, we obtain

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z+2(\beta-1)\int_{\mathbb{R}^{N}}\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}\, {\rm d}z \\ & \quad +\int_{\mathbb{R}^{N}}a(z)\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\!=\! \int_{\mathbb{R}^{N}} f(u)\eta^{2}uu_{L}^{2(\beta\!-\!1)} \, {\rm d}z\!-\!2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \, {\rm d}z. \end{align*}

As

\[ \int_{\mathbb{R}^{N}}\eta^{2}uu_{L}^{2\beta-3}\nabla_{\gamma}u\nabla_{\gamma}u_{L}\,{\rm d}z=\int_{[u\leqslant L]}\eta^{2}u^{2(\beta-1)}|\nabla_{\gamma}u|^{2} \,{\rm d}z\geqslant 0 \]

and $a(z)\geqslant a_0>0$ in $\mathbb {R}^{N}$ we derive that

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z+a_0\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\\ & \quad \leqslant \int_{\mathbb{R}^{N}} f(u)\eta^{2} uu_{L}^{2(\beta-1)} \,{\rm d}z-2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}\nabla_{\gamma} u\nabla_{\gamma}\eta \, {\rm d}z. \end{align*}

By conditions $(f_1)$ and $(f_2)$, $f(u)\leqslant ({a_0}/{2})u+Cu^{q-1}$, and so,

\begin{align*} & \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z\\ & \quad \leqslant C\int_{\mathbb{R}^{N}} \eta^{2}u^{q}u_{L}^{2(\beta-1)} \,{\rm d}z +2\int_{\mathbb{R}^{N}}\eta uu_{L}^{2(\beta-1)}|\nabla_{\gamma} u||\nabla_{\gamma}\eta| \, {\rm d}z. \end{align*}

Using Young's inequality with $\varepsilon >0$ sufficiently small, we get

(6.1)\begin{align} \int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)} \, {\rm d}z & \leqslant C\int_{\mathbb{R}^{N}} \eta^{2}u^{q}u_{L}^{2(\beta-1)} \,{\rm d}z \nonumber\\ & \quad +C_{\varepsilon}\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \, {\rm d}z. \end{align}

From (2.2),

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\left(\int_{\mathbb{R}^{N}}|\nabla_{\gamma}w_L|^{2}\,{\rm d}z+\int_{\mathbb{R}^{N}}|w_L|^{2}\,{\rm d}z\right), \]

which leads to

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\int_{\mathbb{R}^{N}}\left(\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}+ \eta^{2}(\beta-1)^{2}u^{2}u_{L}^{2(\beta-2)}|\nabla_{\gamma}u_{L}|^{2}\right.\\ & \left.\quad +u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2}\right) {\rm d}z \\ & \quad +\,\mathcal{C}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)}\, {\rm d}z, \end{align*}

and thus

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}}\eta^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}u|^{2}\,{\rm d}z +\frac{a_0}{2}\int_{\mathbb{R}^{N}}\eta^{2}u^{2}u_{L}^{2(\beta-1)}\,{\rm d}z\right.\nonumber\\ & \left.\quad +\, \int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \, {\rm d}z\right). \end{align*}

Then from (6.1)

(6.2)\begin{equation} |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{2}u^{q-2} \,{\rm d}z +\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \,{\rm d}z\right). \end{equation}

Using Hölder inequality with exponents ${{2^{*}_{\gamma }}}/({q-2})$ and ${\alpha ^{*}}/{2}:={{2^{*}_{\gamma }}}/{2^{*}_{\gamma }-(q-2)}$,

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{\alpha^{*}} \,{\rm d}z\right)^{2/\alpha^{*}}\left(\int_{\mathbb{R}^{N}} u^{2^{*}_{\gamma}} \, {\rm d}z \right)^{q-2/2^{*}_{\gamma}}\\ & \quad +\mathcal{C}\beta^{2}\int_{\mathbb{R}^{N}}u^{2}u_{L}^{2(\beta-1)}|\nabla_{\gamma}\eta|^{2} \,{\rm d}z. \end{align*}

Since $u\in L^{2^{*}_{\gamma }}(\mathbb {R}^{N})$, we choose $\beta ={{2^{*}_{\gamma }}}/{\alpha ^{*}}>1$ so that $u\in L^{\alpha ^{*}\beta }(\mathbb {R}^{N})\cap L^{2\beta }(\mathbb {R}^{N})$, and so,

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{\mathbb{R}^{N}} u^{\alpha^{*}\beta} \,{\rm d}z\right)^{2/\alpha^{*}} +\frac{\mathcal{C}\beta^{2}}{r^{2}}\int_{\mathbb{R}^{N}}u^{2\beta}\,{\rm d}z < \infty. \]

Hence

\[ \left(\int_{[|z|\geqslant R]} (uu_{L}^{\beta-1})^{2^{*}_{\gamma}} \,{\rm d}z\right)^{2/2_{\gamma}^{*}}\leqslant K <\infty. \]

By employing the Fatou's lemma in the variable $L$ we conclude that

(6.3)\begin{equation} \int_{[|z|\geqslant R]} u^{{(2^{*}_{\gamma})^{2}}/{\alpha^{*}}} \,{\rm d}z <\infty. \end{equation}

Once again from inequality (6.2),

\[ |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{q-2}u^{2\beta} \, {\rm d}z + \int_{[R\geqslant |z|\geqslant R-r]}u^{2\beta} \, {\rm d}z\right). \]

We now consider $t={(2^{*}_{\gamma })^{2}}/{\alpha ^{*}(q-2)}>1$ and $\beta ={2^{*}_{\gamma }(t-1)}/{2t}>1$. Then by Hölder inequality with exponents $t/(t-1)$ and $t$,

\begin{align*} |w_L|_{2^{*}_{\gamma}}^{2}& \leqslant \mathcal{C}\beta^{2}\left[\left(\int_{[|z|\geqslant R-r]} u^{(q-2)t} \, {\rm d}z\right)^{1/t}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t} \right.\\ & \quad \left.+\left(\int_{[R\geqslant |z|\geqslant R-r]}1 \, {\rm d}z\right)^{1/t} \left(\int_{[R\geqslant |z|\geqslant R-r]}u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}\right]. \end{align*}

It follows from (6.3) that

(6.4)\begin{equation} |w_L|_{2^{*}_{\gamma}}^{2}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}. \end{equation}

On the other hand,

\[ \left(\int_{[|z|\geqslant R]} u_{L}^{2^{*}_{\gamma}\beta} \, {\rm d}z\right)^{2/2^{*}_{\gamma}}\leqslant\left(\int_{\mathbb{R}^{N}} (\eta uu_{L}^{\beta-1})^{2_{\gamma}^{*}} \,{\rm d}z\right)^{2/2_{\gamma}^{*}}=|w_{L}|_{2^{*}_{\gamma}}^{2}. \]

This combined with (6.4) leads to

\[ |u_{L}|^{2\beta}_{L^{2^{*}_{\gamma}\beta}([|z|\geqslant R])}\leqslant\mathcal{C}\beta^{2}\left(\int_{[|z|\geqslant R-r]} u^{2\beta t/(t-1)} \, {\rm d}z\right)^{(t-1)/t}=\mathcal{C}\beta^{2}|u|^{2\beta}_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Hence, by Fatou's lemma

\[ |u|^{2\beta}_{L^{2^{*}_{\gamma}\beta}([|z|\geqslant R])}\leqslant\mathcal{C}\beta^{2}|u|^{2\beta}_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Setting $\chi :={2^{*}_{\gamma }(t-1)}/{2t}$ and $s:={2t}/({t-1})$, the last inequality allows to conclude that

\[ |u|_{L^{\chi^{m+1}s}([|z|\geqslant R])}\leqslant\mathcal{C}^{\sum_{i=1}^{m}\chi^{{-}i}}\chi^{\sum_{i=1}^{m}i\chi^{{-}i}}\leqslant|u|_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}. \]

Letting $m\to \infty$ we obtain

\[ |u|_{L^{\infty}([|z|\geqslant R])}\leqslant|u|_{L^{2^{*}_{\gamma}}([|z|\geqslant R-r])}, \]

showing that $\lim _{|z|\to \infty }u(z)=0$.

Finally, in order to show that $u\in L^{\infty }(\mathbb {R}^{N})$, we need only show that the above inequality is true on balls of $\mathbb {R}^{N}$, that is, there exist $R>0$ and $\mathcal {C}>0$ such that

\[ |u|_{L^{\infty}(B_{R}(z))}\leqslant|u|_{L^{2^{*}_{\gamma}}(B_{2R}(z))}, \quad \forall \ z\in\mathbb{R}^{N}. \]

The argument follows with some adjusts from the ideas found in [Reference Alves and Germano4, lemma 2.13].