2.1 Some general notation

We fix the notation that will be used throughout the paper. For a positive integer k, let $I_k$ be the $k\times k$ identity matrix. Let $B_k=T_k N_k\subset {\mathrm {GL}}_k$ be the standard upper triangular Borel subgroup, with $T_k$ the group of diagonal matrices and $N_k$ the upper triangular unipotent subgroup. Let $\overline {N}_k$ be the opposite of $N_k$ , i.e., $\overline {N}_k$ is the lower triangular unipotent subgroup of ${\mathrm {GL}}_k$ . For positive integers $m,n$ , let ${\mathrm {Mat}}_{m\times n}$ be the set of $m\times n$ matrices. We consider the following subgroups of ${\mathrm {GL}}_{m+n}$ given by

$$ \begin{align*}M_{m,n}=\left\{{\begin{pmatrix} g_1 &\\ &g_2 \end{pmatrix}, g_1\in {\mathrm{GL}}_m, g_2\in {\mathrm{GL}}_n}\right\}, N_{m,n}=\left\{{\begin{pmatrix} I_m & X\\ &I_n \end{pmatrix}, X\in {\mathrm {Mat}}_{m\times n}}\right\},\end{align*} $$

and $P_{m,n}=M_{m,n}N_{m,n}$ . Denote $w_{m,n}=\begin {pmatrix} &I_m\\ I_n&\end {pmatrix}$ .

In this section, let F be a global field and ${\mathbb {A}}$ be its ring of adeles. Let $\psi $ be a fixed nontrivial additive character of ${\mathbb {A}}$ , which is trivial on F, and we define a character on $N_k({\mathbb {A}})$ , still denoted by $\psi $ , by $\psi (u)=\psi (\sum _{i=1}^{k-1} u_{i,i+1})$ for $u\in N_k({\mathbb {A}})$ .

2.2 Eisenstein series

Notice that the modulus character of $P_{m,n}$ is given by

$$ \begin{align*}\delta_{P_{m,n}}({\mathrm {diag}}(a_1,a_2))=|\det(a_1)|^n |\det(a_2)|^{-m}, \quad a_1\in {\mathrm{GL}}_m, a_2\in {\mathrm{GL}}_n.\end{align*} $$

Let $\tau _1$ (resp. $\tau _2$ ) be an irreducible automorphic cuspidal representation of ${\mathrm {GL}}_{m}({\mathbb {A}})$ (resp. ${\mathrm {GL}}_n({\mathbb {A}})$ ), we write ${\boldsymbol \tau }=(\tau _1,\tau _2)$ . Given a pair of complex numbers $\mathbf { {s}}:=(s_1,s_2)$ , we consider the normalized induced representation

$$ \begin{align*}{\mathrm {I}}(\mathbf{{s}},{\boldsymbol \tau}):={\mathrm{Ind}}_{P_{m,n}({\mathbb {A}})}^{{\mathrm{GL}}_{m+n}({\mathbb {A}})}(\tau_1|\det|^{s_1-\frac{1}{2}}\otimes \tau_2|\det|^{-s_2+\frac{1}{2}}).\end{align*} $$

Concretely, we associate with each $u\in {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ the function $f_{\mathbf {{s}}}(h)=(u(h))(1), h\in {\mathrm {GL}}_{m+n}({\mathbb {A}})$ . Thus, the space ${\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ consists of all functions $f_{\mathbf {{s}}}: {\mathrm {GL}}_{m+n}({\mathbb {A}})\to {\mathbb {C}}$ satisfying

$$ \begin{align*}f_{\mathbf{{s}}}({\mathrm {diag}}(a,b)uh)=|\det(a)|^{s_1+\frac{n-1}{2}}|\det(b)|^{-s_2+\frac{1-m}{2}}\varphi_h(a,b),\end{align*} $$

where $a\in {\mathrm {GL}}_m({\mathbb {A}}),b\in {\mathrm {GL}}_n({\mathbb {A}}), u\in N_{m,n}({\mathbb {A}}),h\in {\mathrm {GL}}_{m+n}({\mathbb {A}})$ and, for a fixed h, the function $(a,b)\mapsto \varphi _h(a,b)$ is a cusp form in the space of $\tau =\tau _1\boxtimes \tau _2$ of the group $M_{m,n}({\mathbb {A}})={\mathrm {GL}}_m({\mathbb {A}})\times {\mathrm {GL}}_n({\mathbb {A}})$ .

Denote $\widehat {\mathbf {s}}:=(s_2,s_1), 1-{\widehat {\mathbf {s}}}:=(1-s_2,1-s_1)$ and $\widehat {\boldsymbol {\tau }}:=(\tau _2,\tau _1)$ . There is a standard intertwining operator

$$ \begin{align*}M_{w_{m,n}}:{\mathrm {I}}({\mathbf{s}},{\boldsymbol{\tau}})\to {\mathrm{I}}(1-{\widehat{\mathbf{s}}},\widehat{\boldsymbol{\tau}})\end{align*} $$

defined by the integral

$$ \begin{align*}M_{w_{m,n}}f_{\mathbf{{s}}}(g)=\int_{N_{n,m}({\mathbb {A}})}f_{\mathbf{{s}}}\left( w_{m,n} ug \right)du.\end{align*} $$

By [Reference Mœglin and WaldspurgerMW95, Theorem IV.1.8], the above integral is convergent in a cone of $\mathbf {{s}}$ and can be extended to a meromorphic function.

Notice that ${\mathrm {I}}(1-{\widehat {\mathbf {s}}},\widehat {\boldsymbol \tau })$ is the induced representation

$$ \begin{align*}{\mathrm{Ind}}_{P_{n,m}({\mathbb {A}})}^{{\mathrm{GL}}_{m+n}({\mathbb {A}})}(\tau_2|\det|^{(1-s_2)-\frac{1}{2}}\otimes \tau_1|\det|^{-(1-s_1)+\frac{1}{2}}),\end{align*} $$

which consists of all functions $f_{1-{\widehat {\mathbf {s}}}}$ satisfying

$$ \begin{align*}f_{1-{\widehat{\mathbf{s}}}}({\mathrm {diag}}(a,b)uh)=|\det(a)|^{1-s_2+\frac{m-1}{2}}|\det(b)|^{-(1-s_1)-\frac{n-1}{2}}\varphi_h(a,b).\end{align*} $$

In the above equation, ${\mathrm {diag}}(a,b)\in M_{n,m}({\mathbb {A}}),u\in N_{n,m}({\mathbb {A}}),h\in {\mathrm {GL}}_{m+n}({\mathbb {A}}),$ and for a fixed h, the function $(a,b)\mapsto \varphi _h(a,b)$ is a cusp form in the space of $\widehat \tau :=\tau _2\otimes \tau _1$ of the group $M_{n,m}({\mathbb {A}}).$

Given $f_{\mathbf {{s}}}\in {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ , we consider the Eisenstein series

(2.1) $$ \begin{align} E(h,f_{\mathbf{{s}}})=\sum_{\gamma \in P_{m,n}(F)\backslash {\mathrm{GL}}_{m+n}(F)}f_{\mathbf{ {s}}}(\gamma h).\end{align} $$

The above series converges to an analytic function of $\mathbf {{s}}$ in an open cone. When we assume that $\tau _1, \tau _2$ are unitary, the open cone can be taken as $\left \{{(s_1,s_2)\kern1.3pt{\in}\kern1.3pt {\mathbb {C}}^2\mid {\mathrm {Re}}(s_1\kern1.3pt{+}\kern1.3pt s_2)\kern1.3pt{>}\kern1.3pt \frac {m+n}{2}\kern1.3pt{+}\kern1.3pt1}\kern-1pt\right \}$ . Moreover, the above series $E(h,f_{\mathbf {{s}}})$ can be meromorphically continued to ${\mathbb {C}}^2$ . For the above statements, see [Reference Mœglin and WaldspurgerMW95, §II.1.5, IV.1.8]. Similarly, we can also consider the Eisenstein series

$$ \begin{align*}E(h,f_{1-{\widehat{\mathbf{s}}}})=\sum_{\gamma \in P_{n,m}(F)\backslash {\mathrm{GL}}_{m+n}(F)}f_{1-{\widehat{\mathbf{s}}}}(\gamma h),\end{align*} $$

for $f_{1-{\widehat {\mathbf {s}}}}\in {\mathrm {I}}(1-{\widehat {\mathbf {s}}},\widehat {\boldsymbol {\tau }})$ . The series $E(h,f_{1-{\widehat {\mathbf {s}}}})$ is again convergent in a cone and can be continued meromorphically to ${\mathbb {C}}^2.$

2.3 Global integrals

Fix a positive integer l. Let $m,n $ be non-negative integers such that $l>m+n$ . For a non-negative integer j with $0\le j \le l-m-n-1$ , we set $k=l-m-n-1-j\ge 0$ and consider the embedding

$$ \begin{align*}\iota_{j,m,n}:{\mathrm{GL}}_{m+n}\to {\mathrm{GL}}_l\end{align*} $$

$$ \begin{align*}\begin{pmatrix} a&b\\ c&d \end{pmatrix} \mapsto \begin{pmatrix} I_j &&& &\\ &a&&b&\\ &&1&&\\ &c&&d&\\ &&&&I_{k} \end{pmatrix}\end{align*} $$

for $a\in {\mathrm {Mat}}_{m\times m}, b\in {\mathrm {Mat}}_{m\times n}, c\in {\mathrm {Mat}}_{n\times m}, d\in {\mathrm {Mat}}_{n\times n}.$ We also consider the permutation matrix $s_{j,m,n}\in {\mathrm {GL}}_{l}$ defined by

(2.2) $$ \begin{align} s_{j,m,n}=\begin{pmatrix} 0&I_m &0&0&0\\ 0&0&0&I_n&0\\ I_{j}&0&0&0&0\\ 0&0&1&0&0\\ 0&0&0&0&I_{k} \end{pmatrix}. \end{align} $$

The embedding $\iota _{j,m,n}:{\mathrm {GL}}_{m+n}\to {\mathrm {GL}}_{l}$ can then be written as

$$ \begin{align*}\iota_{j,m,n}(h)=(s_{j,m,n})^{-1}\begin{pmatrix} h&\\ &I_{j+1+k}\end{pmatrix} s_{j,m,n}, \quad h\in {\mathrm{GL}}_{m+n}.\end{align*} $$

Next, we consider the subgroup $Y_{j,m,n}$ of ${\mathrm {GL}}_l$ defined by

$$ \begin{align*}Y_{j,m,n}=\left\{{\left(\begin{smallmatrix} v_1&x_1&x_2&x_3 &z\\ &I_{m}& &&y_3 \\ &&1&&y_2\\ &&&I_n&y_1\\ &&&&v_2 \end{smallmatrix}\right), v_1\in N_{j}, v_2\in N_{k}}\right\}.\end{align*} $$

To ease the notation, if $m,n$ are understood, we usually drop $m,n$ from the subscripts of the above notations. For example, we may write $Y_{j,m,n}$ as $Y_j$ , $\iota _{j,m,n}$ as $\iota _j$ , and $s_{j,m,n}$ as $s_j$ . We define a character $\psi _{j}$ on $Y_j(F)\backslash Y_j({\mathbb {A}})$ by

(2.3) $$ \begin{align}\psi_j \left( \left(\begin{smallmatrix} v_1&x_1&x_2&x_3 &z\\ &I_{m}& &&y_3 \\ &&1&&y_2\\ &&&I_n&y_1\\ &&&&v_2 \end{smallmatrix}\right) \right) =\psi( v_1) +\psi(v_2)+\psi(x_{2,j})+\psi(y_{2,1}), \end{align} $$

where we write $x_2=(x_{2,1}, \ldots , x_{2,j})^t, y_2=(y_{2,1}, \ldots , y_{2,j})$ .

Let $\pi $ be an irreducible cuspidal automorphic representation of ${\mathrm {GL}}_{l}({\mathbb {A}})$ , and for $\phi \in V_\pi $ , we consider the following Fourier coefficient of $\phi $ along $Y_j$ :

$$ \begin{align*}\phi_{Y_j,\psi_j}(h)=\int_{Y_j(F)\backslash Y_j({\mathbb {A}})}\phi(y\iota_j(h))\psi_j^{-1}(y)dy, \quad h\in {\mathrm{GL}}_{m+n}({\mathbb {A}}).\end{align*} $$

This is called a Bessel coefficient of $\phi $ . By Lemma 2.1, $\phi _{Y,\psi }$ is left ${\mathrm {GL}}_{m+n}(F)$ -invariant. Thus, for $f_{\mathbf {{s}}}\in {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ , we can consider the integral

(2.4) $$ \begin{align} I_j(\phi,f_{\mathbf{{s}}}):=\int_{{\mathrm{GL}}_{m+n}(F)\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\phi_{Y_j,\psi_j}(h)E(h,f_{\mathbf{ {s}}})dh. \end{align} $$

Similarly, we can also consider $I_j(\phi , M_{w_{m,n}}(f_{\mathbf {{s}}}))$ . Here we remark that if l is odd and $m=n$ , $I_{j}(\phi , f_{\mathbf {{s}}})$ is a Bessel period for $(\phi , E(\cdot ,f_{\mathbf {{s}}}))$ considered in [Reference Gan, Gross and PrasadGGP12, §13].

Proposition 2.2. The integral $I_j(\phi ,f_{\mathbf {{s}}})$ converges absolutely and uniformly in vertical strips in ${\mathbb {C}}$ for each variable $s_1,s_2$ in $\mathbf {{s}}=(s_1,s_2)$ , away from the poles of the Eisenstein series. Moreover, away from the poles of $E(h,f_{\mathbf { {s}}})$ and $E(h,M_{w_{m,n}}(f_{\mathbf {{s}}}))$ , we have

$$ \begin{align*}I_j(\phi,f_{\mathbf{{s}}})=I_j(\phi, M_{w_{m,n}}(f_{\mathbf{{s}}})).\end{align*} $$

Proof. The second statement follows from the functional equation of the Eisenstein series; see [Reference Mœglin and WaldspurgerMW95, Theorem IV.1.10]. For the first statement, it is sufficient to show that $\phi _{Y_j,\psi _j}$ is rapidly decreasing. The proof is similar to other situations that have appeared elsewhere; see [Reference Ben-Artzi and SoudryBAS09, Lemma 2.1] for one example. We provide some details below, following the same argument as in [Reference Ben-Artzi and SoudryBAS09, Lemma 2.1].

Let $\Omega $ be a compact subset of $B_{m+n}({\mathbb {A}})$ . Let c be a real number with $0<c<1$ , and we define a set $A_c$ as follows. We consider the embedding $\mathbb {R}_{>0}\hookrightarrow {\mathbb {A}}^\times $ given by $t\mapsto (t_v)_v$ , where $t_v=t$ if v is an archimedean place, and $t_v=1$ if v is a non-archimedean place. Denote the image of this embedding by $\mathbb {R}_{+, \Delta }$ . Then $A_c$ is the set of all ${\mathrm {diag}}(t_1, \dots , t_{m+n})$ , such that $t_i\in \mathbb {R}_{+, \Delta }$ and $t_1\ge ct_2\ge c^2t_3\ge \dots \ge c^{m+n-1}t_{m+n}$ . Then $\mathcal {S}=\Omega A_c K_{{\mathrm {GL}}_{m+n}({\mathbb {A}})}$ is a Siegel domain for ${\mathrm {GL}}_{m+n}({\mathbb {A}})$ . Similarly, let $\mathcal {S}^\prime =\Omega ^\prime A_c^\prime K_{{\mathrm {GL}}_{l}({\mathbb {A}})}$ be a Siegel domain for ${\mathrm {GL}}_{l}({\mathbb {A}})$ , where $\iota _{j}(\Omega )\subset \Omega ^\prime $ is a compact subset of $B_{l}({\mathbb {A}})$ and $A^\prime _c$ is defined similarly. We take c small enough and $\Omega , \Omega ^\prime $ large enough so that ${\mathrm {GL}}_{l}({\mathbb {A}})={\mathrm {GL}}_{l}(F)\mathcal {S}^\prime $ and ${\mathrm {GL}}_{m+n}({\mathbb {A}})={\mathrm {GL}}_{m+n}(F)\mathcal {S}$ . Now, let $h=\omega a k\in \mathcal {S}$ , where $\omega \in \Omega $ , $a= {\mathrm {diag}}(t_1, \dots , t_{m+n}) \in A_c$ and $k\in K_{{\mathrm {GL}}_{m+n}({\mathbb {A}})}$ . Associated with a, we define

$$ \begin{align*} b= {\mathrm {diag}}( c^{j}t_1, c^{j-1}t_1, \dots, ct_1, I_m, t_{m}, I_n, c^{-1}t_{m+n}, c^{-2}t_{m+n}, \dots, c^{-k}t_{m+n}). \end{align*} $$

Then $b\iota _j(a)\in A_c^\prime $ . Let $\Omega _b^\prime =\Omega ^\prime \cup \Omega ^\prime \cdot b^{-1}$ . For fixed $a\in A_c$ , $\Omega _b^\prime $ is a compact subset of $B_{l}({\mathbb {A}})$ which contains $\Omega ^\prime $ . Let $\mathcal {S}^\prime _b=\Omega ^\prime _b A_c^\prime K_{{\mathrm {GL}}_{l}({\mathbb {A}})}$ . This is a Siegel domain for ${\mathrm {GL}}_{l}({\mathbb {A}})$ , which contains $\mathcal {S}^\prime $ . Thus, $\iota _j( h)=(\iota _j(\omega ) b^{-1}) (b \iota _j(a)) \iota _j(k)\in \mathcal {S}_b^\prime $ . We fix a compact subset $Y_{j,0}\subset Y_j({\mathbb {A}})$ such that $Y_j({\mathbb {A}})=Y_j(F)Y_{j,0}$ . We may assume that $Y_{j,0}\subset \Omega ^\prime $ . Then we have

(2.5) $$ \begin{align} |\phi_{Y_j,\psi_j}(h)|\le \int_{Y_{j,0}} |\phi(y \iota_j(\omega) b^{-1} b \iota_j(ak)) |dy. \end{align} $$

Let $N>0$ be given. Since $\phi $ is rapidly decreasing in $\mathcal {S}^\prime $ , there exists a constant $c_0$ such that for all $\omega ^\prime \in \Omega ^\prime $ , $a^\prime \in A_c^\prime $ and $k^\prime \in K_{{\mathrm {GL}}_{l}({\mathbb {A}})}$ , we have

(2.6) $$ \begin{align} |\phi(\omega^\prime a^\prime k^\prime)|\le c_0 \| a^\prime\|^{-N}. \end{align} $$

Here, $\|\cdot \|$ is the norm on ${\mathrm {GL}}_{l}({\mathbb {A}})$ defined by

$$ \begin{align*} \|g\|=\prod_v \| g_v\|_v \end{align*} $$

where $g\in {\mathrm {GL}}_l({\mathbb {A}})$ , v runs over all places of F and $\| g_v\|_v$ is the local norm on ${\mathrm {GL}}_l(F_v)$ defined by

$$ \begin{align*} \|g_v\|_v=\max\{ |(g_v)_{i,j}|_v, |(g_v^{-1})_{i,j}|_v: 1\le i, j \le l\}. \end{align*} $$

When passing from the Siegel domain $\mathcal {S}^\prime $ to the Siegel domain $\mathcal {S}^\prime _b$ , the constant $c_0$ in (2.6) can be replaced by $c_0 \|b^{-1}\|^{N_0}=c_0 \|b\|^{N_0}$ , for some positive number $N_0$ , which does not depend on b (see [Reference Mœglin and WaldspurgerMW95, Sec. I.2.10, I.2.11]). Thus, in the integrand in (2.5), we have

$$ \begin{align*} |\phi(y \iota_j(\omega) b^{-1}b \iota_j(ak)) | \le c_0 \|b\|^{N_0} \|b\iota_j(a)\|^{-N}. \end{align*} $$

Notice that

$$ \begin{align*} \begin{aligned} \|b\| &= \max\{ c^{j}t_1, c^{j-1}t_1, \dots, ct_1, t_{m}, c^{-1}t_{m+n}, c^{-2}t_{m+n}, \dots, c^{-k}t_{m+n}, \ \\ & \quad\ c^{-j}t_1^{-1}, c^{-j+1}t_1^{-1}, \dots, c^{-1}t_1^{-1}, t_{m}^{-1}, c t_{m+n}^{-1}, c^{2}t_{m+n}^{-1}, \dots, c^{k}t_{m+n}^{-1} \}\\ &= \max\{ ct_1, c^{-j}t_1^{-1}, t_{m}, t_{m}^{-1} , c^{-k}t_{m+n}, ct_{m+n}^{-1} \}\\ &\le c^{\max\{1, -j, -k\}}\|a\| \end{aligned} \end{align*} $$

and

$$ \begin{align*} \begin{aligned} \|b \iota_j(a)\| &= \max\{ c^{j}t_1, c^{j-1}t_1, \dots, ct_1, t_1, t_2, \dots, t_{m+n}, c^{-1}t_{m+n}, c^{-2}t_{m+n}\dots, c^{-k}t_{m+n}, \ \\ & \quad\ c^{-j}t_1^{-1}, c^{-j+1}t_1^{-1}, \dots, c^{-1}t_1^{-1}, t_1^{-1}, \dots, t_{m+n}^{-1}, c t_{m+n}^{-1}, c^2 t_{m+n}^{-1}, \dots, c^{k}t_{m+n}^{-1}\} \\ &\ge \max\{ t_1, t_2, \dots, t_{m+n}, t_1^{-1}, t_2^{-1}, \dots, t_{m+n}^{-1} \}\\ &= \|a\|. \end{aligned} \end{align*} $$

We conclude that

(2.7) $$ \begin{align} |\phi(y \iota_j(\omega a k))| \le c_1 \|a\|^{N_0-N} \end{align} $$

where $c_1$ is a positive constant depending on c and $c_0$ . Since $Y_{j,0}$ is compact, we combine (2.6) and (2.7) to conclude that $\phi _{Y_j, \psi _j}$ is rapidly decreasing in $\mathcal {S}$ . This completes the proof.

2.4 Unfolding of the global integral $I_j(\phi ,f_{\mathbf {{s}}})$

For integers $m,n\ge 0$ , denote

$$ \begin{align*}Z_{m,n}=\left\{\begin{pmatrix} I_m &0&z\\ &1&0\\ &&I_n \end{pmatrix}: z\in {\mathrm {Mat}}_{m\times n} \right\}\subset {\mathrm{GL}}_{m+n+1}.\end{align*} $$

For a cusp form $\phi $ on ${\mathrm {GL}}_{m+n+1}(F)\backslash {\mathrm {GL}}_{m+n+1}({\mathbb {A}})$ , we define its constant term along $Z_{m,n}$ by

$$ \begin{align*}\phi_{Z_{m,n}}(g)=\int_{Z_{m,n}(F)\backslash Z_{m,n}({\mathbb {A}})}\phi\left(zg \right)dz.\end{align*} $$

We have the following expansion of $\phi _{Z_{m,n}}$ .

Lemma 2.3. For $\phi \in {\mathcal {A}}_0({\mathrm {GL}}_{m+n+1})$ , the space of cusp forms on ${\mathrm {GL}}_{m+n+1}(F)\backslash $ ${\mathrm {GL}}_{m+n+1}({\mathbb {A}})$ , we have

$$ \begin{align*} \phi_{Z_{m,n}}(g)=\sum_{\substack{\gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_m(F), \\ \gamma_2\in N_{n}(F)\backslash {\mathrm{GL}}_n(F)}}W_\phi^{\psi}\left(\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} g\right), \end{align*} $$

with absolute and uniform convergence on compact subsets of $g.$ Here $W_\phi ^{\psi }$ is the $\psi $ -Whittaker function of $\phi $ defined by

(2.8) $$ \begin{align} W_\phi^{\psi}(g)=\int_{N_{m+n+1}(F)\backslash N_{m+n+1}({\mathbb {A}})} \phi(ug)\psi^{-1}(u)du, \quad \psi(u)=\psi\left(\sum_{k=1}^{m+n} u_{k,k+1}\right). \end{align} $$

Note that when $n=0$ , the above expansion is just the usual Fourier expansion of cusp forms, due to Piatetski-Shapiro [Reference Piatetskii-ShapiroPS71] and Shalika [Reference ShalikaSha74]. On the other hand, the above version of the expansion is an easy consequence of the results of Piatetski-Shapiro and Shalika. We give a sketch of the proof below.

Proof. Let

$$ \begin{align*}Q_{m}=\left\{{\begin{pmatrix} g_1 &x\\ &1 \end{pmatrix}: g_1\in {\mathrm{GL}}_m, x\in {\mathrm {Mat}}_{m\times 1}}\right\}\end{align*} $$

be the usual mirabolic subgroup of ${\mathrm {GL}}_{m+1}$ . We consider the function $\phi _1=\phi _1^g$ on $Q_m(F)\backslash Q_m({\mathbb {A}})$ defined by

$$ \begin{align*}\phi_1\left(\begin{pmatrix} g_1 &x\\ &1 \end{pmatrix} \right)=\phi_{Z_{m,n}}\left(\begin{pmatrix} g_1&x &\\ &1&\\ &&I_n \end{pmatrix} g \right).\end{align*} $$

Then $\phi _1$ is a cuspidal automorphic form on $ Q_m(F)\backslash Q_m({\mathbb {A}})$ in the sense that for any parabolic subgroup $P=MU$ of $Q_m$ with unipotent radical U, we have

$$ \begin{align*}\int_{U(F)\backslash U({\mathbb {A}})}\phi_1(uq)du=0, \quad \forall q\in Q_m({\mathbb {A}}).\end{align*} $$

This can be checked easily using cuspidality of $\phi $ ; see [Reference CogdellCog07, Lemma 2.2] for a similar situation. Thus by the Fourier expansion for $\phi _1$ , see [Reference Piatetskii-ShapiroPS71, Reference ShalikaSha74], we get that

$$ \begin{align*}\phi_1(I_{m+1})=\sum_{\gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_{m}(F)}W_{\phi_1}^{\psi}\left(\begin{pmatrix} \gamma_1 &\\ &1 \end{pmatrix}\right),\end{align*} $$

with absolute and uniform convergence, where $W_{\phi _1}^\psi $ is the standard $\psi $ -Whittaker function of $\phi _1$ . Plugging in the definitions, we get that

$$ \begin{align*} \phi_{Z_{m,n}}(g)&=\sum_{ \gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_{m}(F)}\int \phi \!\left( \!\!\begin{pmatrix} u &x&z\\ &1&\\ &&I_n \end{pmatrix} \!\begin{pmatrix} \gamma_1&&\\&1&\\ &&I_n \end{pmatrix} g \!\right) \!\psi^{-1}(u)\psi^{-1}(x_m)dudxdz, \end{align*} $$

where $u=(u_{ij})\in N_{m}({\mathbb {A}})$ , $\psi ^{-1}(u)=\psi ^{-1}(\sum _i u_{i,i+1})$ and $x_m$ is the last component of x. Similarly, we consider the mirabolic subgroup $Q_n'$ of ${\mathrm {GL}}_{n+1}$ of the form

$$ \begin{align*}Q_n'=\left\{{\begin{pmatrix} 1&y\\ 0&g_2 \end{pmatrix}, y\in {\mathrm {Mat}}_{1\times n}, g_2\in {\mathrm{GL}}_n}\right\}.\end{align*} $$

For fixed $\gamma _1$ and g, we consider the function $\phi _2$ on $Q_n'(F)\backslash Q_n'({\mathbb {A}})$ defined by

$$ \begin{align*} \phi_2\left(\begin{pmatrix} 1&y\\ 0&g_2 \end{pmatrix} \right)&=\int \phi\left(\begin{pmatrix} u &x&z\\ &1&y\\ &&g_2 \end{pmatrix} \begin{pmatrix} \gamma_1&&\\&1&\\ &&I_n \end{pmatrix} g\right)\psi^{-1}(u)\psi^{-1}(x_m)dudxdz. \end{align*} $$

Again, $\phi _2$ is a cusp form on $Q_n'(F)\backslash Q_n'({\mathbb {A}})$ . By a slightly variant form of the Fourier expansion, see, for example, [Reference Cogdell and Piatetski-ShapiroCPS99, §1], Proposition, we have

$$ \begin{align*}\phi_2(I_{n+1})=\sum_{\gamma_2\in N_{n}(F)\backslash {\mathrm{GL}}_{n}(F)}W_{\phi_2}^{\psi}\left(\begin{pmatrix} 1&\\&\gamma_2 \end{pmatrix}\right).\end{align*} $$

The above series is also absolutely convergent. Note that

$$ \begin{align*}W_{\phi_2}^\psi\left(\begin{pmatrix} 1&\\&\gamma_2 \end{pmatrix} \right) &=\int \phi\left( \begin{pmatrix} u&x&z\\ &1&y\\ &&v \end{pmatrix} \begin{pmatrix} \gamma_1&&\\&1&\\ &&\gamma_2\end{pmatrix} g\right) \\ & \quad \times\psi^{-1}(u)\psi^{-1}(v)\psi^{-1}(x_m+y_1)dxdydudvdz\\ &=W_\phi^\psi\left( \begin{pmatrix} \gamma_1&&\\&1&\\ &&\gamma_2\end{pmatrix} g \right), \end{align*} $$

where $y_1$ in the first integral is the first component of y. The result follows.

Theorem 2.4. The integral $I_j(\phi ,f_{\mathbf {{s}}})$ is Eulerian. More precisely, when $\mathbf {{s}}$ is in the cone of absolute convergence of the Eisenstein series (2.1), we have

(2.9) $$ \begin{align}I_j(\phi,f_{\mathbf{{s}}})=\int_{N_{m+n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline{U}^{j,m,n}({\mathbb {A}})}W_{\phi}^\psi\left( \overline u \eta_{j} \iota_{j}(h)\right)\xi_{f_{\mathbf{ {s}}}}^{\psi^{-1}}(h)d\overline udh \end{align} $$

where

$$ \begin{align*} \overline{U}^{j,m,n}&=\left\{{\overline u(x,y)=\begin{pmatrix} I_m&&&&\\ x&I_{j}&&&\\ &&1&&\\ &&&I_{k}&\\ &&&y& I_n \end{pmatrix} : x\in {\mathrm {Mat}}_{j\times m}, y\in {\mathrm {Mat}}_{n\times k}}\right\},\\ \eta_j&=\eta_{j,m,n}=\begin{pmatrix} &I_m&&&\\ I_{j}&&&&\\ &&1&&\\ &&&&I_{k}\\ &&&I_n& \end{pmatrix},\\ \xi_{f_{\mathbf{{s}}}}^{\psi^{-1}}(h)&=\int_{N_{m}(F)\backslash N_{m}({\mathbb {A}})\times N_{n}(F)\backslash N_{n}({\mathbb {A}})} f_{\mathbf{{s}}}\left(\begin{pmatrix} u_1 & \\ & u_2 \end{pmatrix} h \right)\psi(u_1)\psi(u_2)du_1 du_2. \end{align*} $$

Proof. For notational convenience, we set $[G]:=G(F)\backslash G({\mathbb {A}})$ for an algebraic group G over F. In what follows, we assume that $\mathbf {{s}}$ is in the cone of absolute convergence, so that the Eisenstein series $I(h,f_{\mathbf {{s}}})$ is given by the convergent sum (2.1). By plugging (2.1) into the definition of $I_j(\phi , f_{\mathbf {{s}}})$ (see (2.4)), we get

(2.10) $$ \begin{align} I_j(\phi,f_s)&=\int_{P_{m,n}(F)\backslash{\mathrm{GL}}_{m+n}({\mathbb {A}})}\phi_{Y_j,\psi_j}(h)f_{\mathbf{{s}}}(h)dh \\ &=\int_{M_{m,n}(F)N_{m,n}({\mathbb {A}})\backslash{\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{[N_{m,n}]}\phi_{Y_j,\psi_j}(uh)duf_{\mathbf{ {s}}}(h)dh \nonumber\\ &=\int_{M_{m,n}(F)N_{m,n}({\mathbb {A}})\backslash{\mathrm{GL}}_{m+n}({\mathbb {A}})}\phi_{Y_j,\psi_j,N_{m,n}}(h)f_{\mathbf{ {s}}}(h)dh,\nonumber \end{align} $$

where

$$ \begin{align*} \phi_{Y_j,\psi_j,N_{m,n}}(h)&=\int_{[N_{m,n}]}\phi_{Y_j,\psi_j}(uh)du. \end{align*} $$

By the definition of $\phi _{Y_j,\psi _j}$ , together with the left ${\mathrm {GL}}_{l}(F)$ invariance of $\phi $ , we get

(2.11) $$ \begin{align} \begin{aligned} \phi_{Y_j,\psi_j,N_{m,n}}(h)&=\int_{[Y_j]\times [N_{m,n}]}\phi(y\iota_j(uh))\psi_j^{-1}(y)dudy\\ &=\int_{[Y_j]\times [\iota_j(N_{m,n})]}\phi(\eta_j y\iota_j(u)\eta_j^{-1} \eta_j \iota_j (h))\psi_j^{-1}(y)dudy. \end{aligned} \end{align} $$

Write

$$ \begin{align*} y=\left(\begin{smallmatrix} v_1&x_1&x_2&x_3 &z\\ &I_{m}& &&y_3 \\ &&1&&y_2\\ &&&I_n&y_1\\ &&&&v_2 \end{smallmatrix}\right) \in Y_j({\mathbb {A}}), \quad u=\begin{pmatrix} I_m & t\\ &I_n\end{pmatrix} \in N_{m,n}({\mathbb {A}}),\end{align*} $$

then we have

$$ \begin{align*}\eta_{j} y \iota_j(u) \eta_{j}^{-1}=\left(\begin{smallmatrix} I_m&0&0&y_3&t\\ x_1 &v_1&x_2&z&x_3\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right),\end{align*} $$

where $v_1\in N_{j}({\mathbb {A}}),v_2\in N_{k}({\mathbb {A}}), (x_1,x_2,x_3)\in {\mathrm {Mat}}_{j\times (m+n+1)}({\mathbb {A}}), z\in {\mathrm {Mat}}_{j\times k}({\mathbb {A}}), y_3\in {\mathrm {Mat}}_{m\times k}({\mathbb {A}}), y_2\in {\mathrm {Mat}}_{1\times k}({\mathbb {A}}), y_1\in {\mathrm {Mat}}_{n\times k}({\mathbb {A}})$ and $t\in {\mathrm {Mat}}_{m\times n}({\mathbb {A}})$ . Then (2.11) becomes

(2.12) $$ \begin{align}\phi_{Y_j,\psi_j,N_{m,n}}(h)=\int_{[Y_j]\times [\iota_j(N_{m,n})]} \phi\left(\left(\begin{smallmatrix} I_m&0&0&y_3&t\\ x_1 &v_1&x_2&z&x_3\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h) \right)\psi_j^{-1}(y)dydu.\end{align} $$

Write

$$ \begin{align*}Z=\begin{pmatrix} y_3 &t\\ z& x_3 \end{pmatrix} \in {\mathrm {Mat}}_{(m+j)\times (n+k)}({\mathbb {A}}).\end{align*} $$

The right side integral of (2.12) contains the following inner integral

$$ \begin{align*} \int_{[{\mathrm {Mat}}_{(m+j)\times (n+k)}]}\phi\left( \begin{pmatrix} I_{m+j} &&Z\\ &1&\\ &&I_{n+k} \end{pmatrix} g\right)dZ, \end{align*} $$

which is

(2.13) $$ \begin{align} \sum_{\substack{\gamma_1\in N_{m+j}(F)\backslash {\mathrm{GL}}_{m+j}(F) \\ \gamma_2\in N_{n+k}(F)\backslash {\mathrm{GL}}_{n+k}(F)}}W_\phi^\psi\left( \begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} g \right) \end{align} $$

by Lemma 2.3. Plugging (2.13) into (2.12), we get

(2.14) $$ \begin{align} \phi_{Y_j,\psi_j,N_{m,n}}(h)&=\sum_{\gamma_1,\gamma_2}\int W_\phi^{\psi}\left(\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} \left(\begin{smallmatrix} I_m&0&0&0&0\\ x_1 &v_1&x_2&0&0\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h) \right)\psi_j^{-1}(y)dy. \end{align} $$

To simplify the above integral (2.14), we consider its inner integral with respect to $x_2=[x^1,\dots , x^j]^t\in {\mathrm {Mat}}_{j\times 1}({\mathbb {A}})$ first, which is

$$ \begin{align*} \int_{(F\backslash {\mathbb {A}})^j}W_\phi^\psi\left(\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} \left(\begin{smallmatrix} I_m&0&0&0&0\\ 0 &I_j&x_2&0&0\\ 0 &0&1&0&0\\ 0 &0&0&I_k&0\\ 0&0&0&0& I_n \end{smallmatrix}\right) \left(\begin{smallmatrix} I_m&0&0&0&0\\ x_1 &v_1&0&0&0\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h) \right)\psi^{-1}(x^j)dx_2. \end{align*} $$

Write $\gamma _1=(\gamma _{pq})_{1\le p,q\le m+j}$ , then we have

$$ \begin{align*}\gamma_1\begin{pmatrix} 0\\ x_2 \end{pmatrix}=\begin{pmatrix} *\\ *\\ \vdots\\ \gamma_{m+j,m+1}x^1+\gamma_{m+j,m+2}x^2+\dots+\gamma_{m+j,m+j}x^j \end{pmatrix}.\end{align*} $$

After conjugation, we get

$$ \begin{align*} &W_\phi^\psi\left(\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} \left(\begin{smallmatrix} I_m&0&0&0&0\\ 0 &I_j&x_2&0&0\\ 0 &0&1&0&0\\ 0 &0&0&I_k&0\\ 0&0&0&0& I_n \end{smallmatrix}\right) g \right)\\ & \qquad =\psi(\gamma_{m+j,m+1}x^1+\dots+\gamma_{m+j,m+j}x^j ) \cdot W_\phi^{\psi}\left(\begin{pmatrix} \gamma_1&&\\ &1&\\ &&\gamma_2 \end{pmatrix} g\right), \end{align*} $$

with

$$ \begin{align*}g= \left(\begin{smallmatrix} I_m&0&0&0&0\\ x_1 &v_1&0&0&0\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h).\end{align*} $$

Thus the inner integral of (2.14) with respect to $x_2$ is

$$ \begin{align*} \int_{(F\backslash {\mathbb {A}})^j}\psi(\gamma_{m+j,m+1}x^1+\dots+(\gamma_{m+j,m+j}-1)x^j)dx^1\dots dx^j W_\phi^{\psi}\left(\left(\begin{smallmatrix} \gamma_1&&\\ &1&\\ &&\gamma_2 \end{smallmatrix}\right) g\right). \end{align*} $$

The above integral over $x^1,\dots ,x^j$ is $1$ if $\gamma _{m+j,m+1}=\dots =\gamma _{m+j,m+j-1}=0$ and ${\gamma _{m+j,m+j}=1}$ , and is zero otherwise. Note that if $\gamma _{m+j,m+1}=\dots =\gamma _{m+j,m+j-1}=0$ , as an element of the coset $N_{m+j}(F)\backslash {\mathrm {GL}}_{m+j}(F)$ , we can write

$$ \begin{align*}\gamma_1=\begin{pmatrix} \gamma_1' &\\ &1 \end{pmatrix} \begin{pmatrix} I_{m} &&\\ &I_{j-1} &\\ \xi && 1 \end{pmatrix},\end{align*} $$

with $\gamma _1'\in N_{m+j-1}(F)\backslash {\mathrm {GL}}_{m+j-1}(F), \xi \in {\mathrm {Mat}}_{1\times m}(F).$ By changing the summation notation, integral (2.14) becomes

(2.15) $$ \begin{align} &\phi_{Y_j,\psi_j,N_{m,n}}(h)=\sum_{\substack{\gamma_1\in N_{m+j-1}(F)\backslash {\mathrm{GL}}_{m+j-1}(F)\\ \gamma_2\in N_{n+k}(F)\backslash {\mathrm{GL}}_{n+k}(F)}}\sum_{\xi\in F^m}\\ & \qquad \int W_\phi^{\psi}\left(\left(\begin{smallmatrix} \gamma_1 &&\\ &I_2&\\ &&\gamma_2 \end{smallmatrix}\right) \left(\begin{smallmatrix} I_m &&&&\\ &I_{j-1} &&&\\ \xi &&1&&\\ &&&1&\\ &&&&I_{n+k} \end{smallmatrix}\right) \left(\begin{smallmatrix} I_m&0&0&0&0\\ x_1 &v_1&0&0&0\\ 0 &0&1&y_2&0\\ 0 &0&0&v_2&0\\ 0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h) \right) \nonumber\\ & \qquad \cdot \psi^{-1}(v_1)\psi^{-1}(v_2)\psi^{-1}(y^1)dx_1dy_1dy_2dv_1dv_2.\nonumber \end{align} $$

Here $y^1$ is the first component of the element $y_2\in {\mathrm {Mat}}_{1\times k}({\mathbb {A}}).$ In other words, $y_2=[y^1,\dots ,y^k]\in {\mathrm {Mat}}_{1\times k}({\mathbb {A}}).$ In (2.15), the summation over $\xi $ could be absorbed into the integral over the last row of $x_1$ . Thus, we get

(2.16) $$ \begin{align} \phi_{Y_j,\psi_j,N_{m,n}}(h)=&\sum_{\substack{\gamma_1\in N_{m+j-1}(F)\backslash {\mathrm{GL}}_{m+j-1}(F)\\ \gamma_2\in N_{n+k}(F)\backslash {\mathrm{GL}}_{n+k}(F)}}\int_{(F\backslash {\mathbb {A}})^*}\int_{{\mathbb {A}}^m}\\ &\int W_\phi^{\psi}\left(\begin{pmatrix} \gamma_1 &&\\ &I_2&\\ &&\gamma_2 \end{pmatrix} \left(\begin{smallmatrix} I_m&0&0&0&0&0\\ x_1' &v_1'&p&0&0&0\\ (x_{j1},\dots, x_{jm}) &0&1&0&0&0 \\ 0&0 &0&1&y_2&0\\ 0&0 &0&0&v_2&0\\ 0&0&0&0&y_1& I_n \end{smallmatrix}\right) \eta_{j} \iota_j(h) \right) \nonumber\\ &\quad \cdot \psi^{-1}(v_1)\psi^{-1}(v_2)\psi^{-1}(y^1)(\prod_{t=1}^m dx_{jt})dx_1'dy_1dy_2dv_1dv_2,\nonumber \end{align} $$

where we wrote $x_1=\begin {pmatrix} x_1'\\ (x_{j1},\dots , x_{jm}) \end {pmatrix}$ , $v_1=\begin {pmatrix} v_1'& p \\ &1 \end {pmatrix}$ with $p\in {\mathrm {Mat}}_{(j-1)\times 1}({\mathbb {A}})$ and $*$ in $(F\backslash {\mathbb {A}})^*$ denotes the number of variables other than the part in $(x_{j1},\dots ,x_{jm})$ . We next compute the inner integral over the p-part, which is similar to that above. Note that $\psi (v_1)=\psi (v_1')\psi (p^{j-1})$ , where $p=(p^1,\dots ,p^{j-1})^t$ . For $\gamma _1=(\gamma _{ab})_{1\le a,b\le m+j-1}\in {\mathrm {GL}}_{m+j-1}(F)$ , and $p=(p^1,\dots ,p^{j-1})^t$ we have

$$ \begin{align*}\gamma_1\begin{pmatrix} 0_{m\times 1} \\ p \end{pmatrix} =\begin{pmatrix} *\\ \vdots\\ *\\ \gamma_{m+j-1,m+1}p^1+\dots \gamma_{m+j-1,m+j-1}p^{j-1} \end{pmatrix}.\end{align*} $$

Thus the inner integral over p in (2.16) is

$$ \begin{align*} \int_{(F\backslash {\mathbb {A}})^{j-1}}\psi( \gamma_{m+j-1,m+1}p^1+\dots+(\gamma_{m+j-1,m+j-1}-1)p^{j-1})\prod_t dp^t W_\phi^{\psi} \!\left( \!\!\begin{pmatrix} \gamma_1&&\\ &1&\\ &&\gamma_2 \end{pmatrix} g \!\right)\! , \end{align*} $$

for certain appropriate g which should be self-evident from the context. The above integral is 1 if $\gamma _{m+j-1,m+1}=\dots =\gamma _{m+j-1,m+j-2}=0$ and $\gamma _{m+j-1,m+j-1}=1$ , and is zero otherwise. In this case, we can write that

$$ \begin{align*}\gamma_1=\begin{pmatrix} \gamma_1' &\\ &1 \end{pmatrix} \begin{pmatrix} I_m &&\\ &I_{j-2} &\\ \xi&&1 \end{pmatrix}\end{align*} $$

as an element in the coset $N_{m+j-1}(F)\backslash {\mathrm {GL}}_{m+j-1}(F)$ , where $\gamma _1'\in N_{m+j-2}(F)\backslash $ ${\mathrm {GL}}_{m+j-2}(F), \xi \in F^m$ . Similarly as above, by absorbing the summation over $\xi $ , we get that

$$ \begin{align*} &\phi_{Y_j,\psi_j,N_{m,n}}(h)=\sum_{\substack{\gamma_1\in N_{m+j-2}(F)\backslash {\mathrm{GL}}_{m+j-2}(F)\\ \gamma_2\in N_{n+k}(F)\backslash {\mathrm{GL}}_{n+k}(F)}}\int_{(F\backslash {\mathbb {A}})^*}\int_{{\mathbb {A}}^{2m}}\\ &\qquad \int W_\phi^{\psi}\left(\begin{pmatrix} \gamma_1 &&\\ &I_3&\\ &&\gamma_2 \end{pmatrix} \begin{pmatrix} I_m&0&0&0&0&0&0\\ x_1" &v_1"&p'&0&0&0&0\\ (x_{j-1,1},\dots,x_{j-1,m}) &0 &1 &0&0&0&0 \\ (x_{j1},\dots, x_{jm}) &0&0&1&0&0&0 \\ 0&0 &0&0&1&y_2&0\\ 0&0 &0&0&0&v_2&0\\ 0&0&0&0&0&y_1& I_n \end{pmatrix} \eta_{j} \iota_j(h) \right) \nonumber\\ &\qquad \quad \cdot \psi^{-1}(v_1')\psi^{-1}(v_2)\psi^{-1}(y^1)\left(\prod_{i=j-1}^j\prod_{t=1}^m dx_{it}\right)dx_1"dy_1dy_2dv_1'dv_2,\nonumber \end{align*} $$

where $v_1'=\begin {pmatrix} v_1"&p'\\ &1 \end {pmatrix}$ . An inductive argument shows that

$$ \begin{align*} \phi_{Y_j,\psi_j,N_{m,n}}(h)=&\sum_{\substack{\gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_{m}(F)\\ \gamma_2\in N_{n+k}(F)\backslash {\mathrm{GL}}_{n+k}(F)}}\int_{(F\backslash {\mathbb {A}})^*}\int_{{\mathrm {Mat}}_{j\times m}({\mathbb {A}})}\\ & W_\phi^\psi\left( \begin{pmatrix} \gamma_1&&\\ &I_{j+1}&\\ &&\gamma_2 \end{pmatrix} \begin{pmatrix} I_m &&&&&\\ x& I_{j} &&&&\\ &&1&&&\\ &&&1&y_2&\\ &&&&v_2&\\ &&&&y_1&I_n \end{pmatrix} \eta_{j} \iota_j(h) \right)\\ &\cdot \psi^{-1}(v_2)\psi^{-1}(y^1)dxdy_1dy_2 dv_2. \end{align*} $$

The integral over $y_1,y_2,v_2$ can be done similarly and we have

$$ \begin{align*} \phi_{Y_j,\psi_j,N_{m,n}}(h)=&\sum_{\substack{\gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_{m}(F)\\ \gamma_2\in N_n(F)\backslash {\mathrm{GL}}_n(F)}} \int_{{\mathrm {Mat}}_{j\times m}({\mathbb {A}})} \int_{{\mathrm {Mat}}_{n\times k}({\mathbb {A}})}\\ & \quad W_\phi^\psi\left( \begin{pmatrix} \gamma_1&&\\ &I_{j+k+1}&\\ &&\gamma_2 \end{pmatrix} \begin{pmatrix} I_m &&&&\\ x& I_{j} &&&\\ &&1&&\\ &&&I_{k}&\\ &&&y &I_n \end{pmatrix} \eta_{j} \iota_j(h) \right)dydx\\ =&\sum_{\substack{\gamma_1\in N_{m}(F)\backslash {\mathrm{GL}}_{m}(F)\\ \gamma_2\in N_n(F)\backslash {\mathrm{GL}}_n(F)}} \int_{{\mathrm {Mat}}_{j \times m}({\mathbb {A}})} \int_{{\mathrm {Mat}}_{n\times k}({\mathbb {A}})}\\ &\quad W_\phi^\psi\left( \begin{pmatrix} I_m &&&&\\ x& I_{j} &&&\\ &&1&&\\ &&&I_{k}&\\ &&&y &I_n \end{pmatrix} \eta_{j} \iota_j\left(\begin{pmatrix} \gamma_1&\\ &\gamma_2 \end{pmatrix} h \right) \right)dydx \end{align*} $$

We now plug the above formula into (2.10) to get

(2.17) $$ \begin{align} \begin{aligned} I_j(\phi,f_{\mathbf{{s}}})=&\int_{M_{m,n}(F)N_{m,n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\phi_{Y_j,\psi_j,N_{m,n}}(h)f_{\mathbf{{s}}}(h)dh\\ =&\int_{(N_m(F)\times N_n(F))N_{m,n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline U^{j,m,n}({\mathbb {A}})} W_\phi^\psi\left( \overline u \eta_{j} \iota_j( h ) \right)f_{\mathbf{{s}}}(h)d\overline udh. \end{aligned} \end{align} $$

In order to justify this step, we need to show that the double integral in the second line of (2.17) converges absolutely. This will be done in Subsection 2.6. From (2.17), we obtain

$$ \begin{align*} & I_j(\phi,f_{\mathbf{{s}}})\\ &\quad=\int_{N_{m+n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline U^{j,m,n}({\mathbb {A}})}\int_{N_m(F)\backslash N_m({\mathbb {A}})}\int_{N_n(F)\backslash N_n({\mathbb {A}})}W_\phi^\psi \!\left(\overline u \eta_{j}\iota_j \!\left( \!\begin{pmatrix} u_1 &\\ &u_2 \end{pmatrix}\! h \right)\!\right)\\ &\qquad\cdot f_{\mathbf{{s}}}\left(\!\begin{pmatrix} u_1 &\\[-3pt] &u_2 \end{pmatrix} h \right)du_2du_1dydxdh\\ &\quad=\int_{N_{m+n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline U^{j,m,n}({\mathbb {A}})}\int_{N_m(F)\backslash N_m({\mathbb {A}})}\int_{N_n(F)\backslash N_n({\mathbb {A}})}W_\phi^\psi\left(\overline u \eta_{j}\iota_j\left( h \right)\right)\\ &\qquad\cdot \psi^{-1}(u_1)\psi^{-1}(u_2) f_{\mathbf{{s}}}\left(\begin{pmatrix} u_1 &\\[-3pt] &u_2 \end{pmatrix} h \right)du_2du_1dydxdh\\ &\quad=\int_{N_{m+n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline{U}^{j,m,n}({\mathbb {A}})}W_{\phi}^\psi\left( \overline u \eta_{j} \iota_{j}(h)\right)\xi_{f_{\mathbf{{s}}}}^{\psi^{-1}}(h)d\overline udh. \end{align*} $$

The result follows.

Remark 2.5. Denote

(2.18) $$ \begin{align} \gamma_{m,n}:=\eta_{j,m,n}s_{j,m,n}^{-1}=\begin{pmatrix} I_m&&\\[-3pt] &&I_{l-m-n}\\[-3pt] &I_n& \end{pmatrix}. \end{align} $$

Then (2.9) can be rewritten as

$$ \begin{align*}I_j(\phi,f_{\mathbf{{s}}})=\int_{N_{m+n}({\mathbb {A}})\backslash {\mathrm{GL}}_{m+n}({\mathbb {A}})}\int_{\overline{U}^{j,m,n}({\mathbb {A}})}W_{\phi}^\psi\left( \overline u \gamma_{m,n}\begin{pmatrix} h&\\ & I_{l-m-n} \end{pmatrix} s_{j,m,n}\right)\xi_{f_{\mathbf{{s}}}}^{\psi^{-1}}(h)d\overline udh. \end{align*} $$

2.5 Unfolding of $I_j(\phi ,M_{w_{m,n}}(f_{\mathbf {{s}}}))$

Theorem 2.6. The integral $I_j(\phi ,M_{w_{m,n}}(f_{\mathbf {{s}}}))$ is Eulerian. More precisely, in the region of absolute convergence of the Eisenstein series $ E(h, M_{w_{m,n}}(f_{\mathbf {{s}}}))$ , we have

$$ \begin{align*} I_j(\phi,\widetilde f_{\mathbf{{s}}})=\int_{N_{n+m}({\mathbb {A}})\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})} \int_{\overline V^{j,m,n}} W_\phi^\psi\left(\overline u \gamma_{n,m}\begin{pmatrix} h&\\ &I_{l-m-n}\end{pmatrix} s_{j,m,n}\right) \xi_{\widetilde f_{\mathbf{{s}}}}^{\psi^{-1}}(h)d\overline udh, \end{align*} $$

where

$$ \begin{align*} \widetilde f_{\mathbf{{s}}}&=M_{w_{m,n}}(f_{\mathbf{{s}}}),\\[-3pt] \overline V^{j,m,n}&=\left\{{\begin{pmatrix} I_n&0&0&0&0\\ x&I_{j}&0&0&0\\ &&1&0&0\\ &&&I_{k}&0\\ &&&y&I_{m} \end{pmatrix} : x\in {\mathrm {Mat}}_{j\times n}, y\in {\mathrm {Mat}}_{m\times k}}\right\}=\overline{U}^{j,n,m},\\ \gamma_{n,m}&=\begin{pmatrix} I_n&&\\ &&I_{l-m-n}\\ &I_m& \end{pmatrix}, \\ \xi_{\widetilde f_{\mathbf{{s}}}}^{\psi^{-1}}(h)&=\int_{N_n(F)\backslash N_n({\mathbb {A}})\times N_m(F)\backslash N_m({\mathbb {A}})}\widetilde f_{\mathbf{{s}}}\left(\begin{pmatrix} u_1&\\ &u_2 \end{pmatrix} h \right)\psi(u_1)\psi(u_2)du_1du_2. \end{align*} $$

The proof is similar to the proof of Theorem 2.4. We give some details for completeness.

Proof. In the following, we assume that $m\ge n$ . If $n\le m$ , the matrix calculation performed below is a little bit different, but other parts of the proof go through, and the result is the same. Note that $\widetilde f_{\mathbf {{s}}}\in {\mathrm {I}}(1-{\widehat {\mathbf {s}}}, \widehat {\boldsymbol \tau })$ is left invariant under $N_{n,m}({\mathbb {A}})$ . When the Eisenstein series $E(h, \widetilde f_{\mathbf {{s}}}) $ is absolutely convergent, we can replace it by its infinite series definition to obtain

(2.19) $$ \begin{align} \begin{aligned} I_j(\phi,\widetilde f_{\mathbf{{s}}})&=\int_{P_{n,m}(F)\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\phi_{Y_j,\psi_j}(h)\widetilde f_{\mathbf{{s}}}(h)dh \\ &=\int_{M_{n,m}(F)N_{n,m}({\mathbb {A}})\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\int_{N_{n,m}(F)\backslash N_{n,m}({\mathbb {A}})}\phi_{Y_j,\psi_j}(uh)\widetilde {f}_{\mathbf{{s}}}(h)dudh \\ &=\int_{M_{n,m}(F)N_{n,m}({\mathbb {A}})\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\phi_{Y_j,\psi_j,N_{n,m}}(h)\widetilde {f}_{\mathbf{ {s}}}(h)dh, \end{aligned} \end{align} $$

where

$$ \begin{align*}\phi_{Y_j,\psi_j,N_{n,m}}(h):&=\int_{N_{n,m}(F)\backslash N_{n,m}({\mathbb {A}})}\phi_{Y_j,\psi_j}(uh) du\\ &=\int_{N_{n,m}(F)\backslash N_{n,m}({\mathbb {A}})}\int_{Y_j(F)\backslash Y_j({\mathbb {A}})}\phi(y\iota_j(u)\iota_j(h))\psi_j^{-1}(y)dydu. \end{align*} $$

Since $\phi $ is left ${\mathrm {GL}}_{l}(F)$ -invariant, we have

$$ \begin{align*}\phi(y\iota_j(u)\iota_j(h))=\phi\left(\gamma_{n,m}s_{j,m,n}y\iota_j(u)s_{j,m,n}^{-1} \gamma_{n,m}^{-1} \gamma_{n,m}\begin{pmatrix} h&\\ &I_{l-m-n} \end{pmatrix} s_{j,m,n} \right) .\end{align*} $$

Write

$$ \begin{align*} y=\begin{pmatrix} v_1&x_1&x_1'&x_2&x_3&z\\ &I_n&&&&y_3\\ &&I_{m-n}&&&y_3'\\ &&&1&&y_2\\ &&&&I_n&y^{\prime}_1\\ &&&&&v_2 \end{pmatrix} \in Y({\mathbb {A}}), \quad u=\begin{pmatrix} I_n&t_1&t_2\\ &I_{m-n}&\\ &&I_n \end{pmatrix} \in N_{n,m}({\mathbb {A}}), \end{align*} $$

with $v_1\in N_{j}({\mathbb {A}}), v_2\in N_{k}({\mathbb {A}})$ and other variables in appropriate matrices spaces. A matrix calculation shows that

$$ \begin{align*}\gamma_{n,m}s_{j,m,n}y\iota_j(u)s_{j,m,n}^{-1} \gamma_{n,m}^{-1}=\begin{pmatrix} I_n&0&0&y_3&t_1&t_2\\ x_1&v_1&x_2&z&x_1'+x_1t_1&x_3+x_1t_2\\ &&1&y_2&0&0\\ &&&v_2&0&0\\ &&&y_3'&I_{m-n}&0\\ &&&y^{\prime}_1&0&I_n \end{pmatrix}.\end{align*} $$

Thus we get

$$ \begin{align*} &\phi_{Y_j,\psi_j,N_{n,m}}(h)\\ &\qquad =\int_{[Y_j]\times [N_{n,m}]}\phi\left(\left(\begin{smallmatrix} I_n&0&0&y_3&t_1&t_2\\ x_1&v_1&x_2&z&x_1'&x_3\\ &&1&y_2&0&0\\ &&&v_2&0&0\\ &&&y_3'&I_{m-n}&0\\ &&&y^{\prime}_1&0&I_n \end{smallmatrix}\right) \gamma_{n,m}\begin{pmatrix} h&\\ &I_{l-m-n} \end{pmatrix} s_{j,m,n} \right)\psi_j^{-1}(y)dydu. \end{align*} $$

Denote

$$ \begin{align*}Z=\begin{pmatrix} y_3&t_1&t_2\\ z&x_1'&x_3 \end{pmatrix}\in [{\mathrm {Mat}}_{(n+j)\times (m+k)}].\end{align*} $$

Then inside the integral $ \phi _{Y_j,\psi _j,N_{n,m}}(h)$ , there is an inner integral

$$ \begin{align*}\int_{[{\mathrm {Mat}}_{(n+j)\times (m+k)}]}\phi\left( \begin{pmatrix} I_{n+j} &&Z\\ &1&\\ &&I_{m+k} \end{pmatrix} g\right)dZ,\end{align*} $$

which, by Lemma 2.3, is

$$ \begin{align*} \sum_{\substack{\gamma_1\in N_{n+j}(F)\backslash {\mathrm{GL}}_{n+j}(F)\\ \gamma_2\in N_{m+k}(F)\backslash {\mathrm{GL}}_{m+k}(F)}}W_\phi^\psi\left(\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} g \right). \end{align*} $$

Thus we get

$$ \begin{align*} & \phi_{Y_j,\psi_j,N_{n,m}}(h)\\ &\quad =\sum_{\substack{\gamma_1\in N_{n+j}(F)\backslash {\mathrm{GL}}_{n+j}(F)\\ \gamma_2\in N_{m+k}(F)\backslash {\mathrm{GL}}_{m+k}(F)}}\int W_\phi^\psi \!\left( \!\!\!\begin{pmatrix} \gamma_1 &&\\ &1&\\ &&\gamma_2 \end{pmatrix} \!\!\left( \!\begin{smallmatrix} I_n&0&0&0&0\\ x_1&v_1&x_2&0&0\\ &&1&y_2&0\\ &&&v_2&0\\ &&&y_1&I_{m}\\ \end{smallmatrix} \!\right)\! \gamma_{n,m}h s_{j,m,n} \!\right)\! \end{align*} $$

where $y_1=\begin {pmatrix} y_3' \\ y_1' \end {pmatrix} \in [{\mathrm {Mat}}_{m\times k}],$ and $h=\begin {pmatrix} h&\\ &I_{l-m-n} \end {pmatrix}$ . Note that the above formula is similar to (2.14). By the same method as in the proof of Theorem 2.4, we get that

$$ \begin{align*} \phi_{Y_j,\psi_j,N_{n,m}}(h)&=\sum_{\substack{\gamma_1\in N_n(F)\backslash {\mathrm{GL}}_n(F)\\ \gamma_2\in N_m(F)\backslash {\mathrm{GL}}_m(F)}}\int_{\overline V^{j,m,n}({\mathbb {A}})} W_\phi^\psi \left( \left(\begin{smallmatrix} \gamma_1&&\\ &I_{l-m-n}&\\ &&\gamma_2 \end{smallmatrix}\right) \overline v \gamma_{n,m}h s_{j,m,n} \right)d\overline v\\ &=\sum_{\substack{\gamma_1\in N_n(F)\backslash {\mathrm{GL}}_n(F)\\ \gamma_2\in N_m(F)\backslash {\mathrm{GL}}_m(F)}}\int_{\overline V^{j,m,n}({\mathbb {A}})} W_\phi^\psi \left( \overline v \gamma_{n,m}\left(\begin{smallmatrix}\gamma_1&&\\ &\gamma_2&\\ &&I_{l-m-n} \end{smallmatrix}\right) h s_{j,m,n} \right)d\overline v. \end{align*} $$

Plugging the above equation into (2.19), we get that

$$ \begin{align*} I(\phi,\widetilde f_{\mathbf{{s}}})&=\int_{\{(N_n(F)\times N_m(F))N_{n,m}({\mathbb {A}})\}\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\int_{\overline V^{j,m,n}({\mathbb {A}})}W_\phi^\psi(\bar v \gamma_{n,m} hs_{j,m,n})\widetilde f_{\mathbf{{s}}}(h)d\overline v dh\\ &=\int_{N_{n+m}({\mathbb {A}})\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\int_{\overline V^{j,m,n}({\mathbb {A}})}W_\phi^\psi(\overline v \gamma_{n,m} h s_{j,m,n})\\ &\quad\quad\cdot \int_{[N_n]\times [N_m]}\widetilde f_s\left(\begin{pmatrix} u_1 &\\ &u_2 \end{pmatrix} \right)\psi(u_1)\psi(u_2)du_1du_2 d\overline v dh\\ &=\int_{N_{n+m}({\mathbb {A}})\backslash {\mathrm{GL}}_{n+m}({\mathbb {A}})}\int_{\overline V^{j,m,n}({\mathbb {A}})}W_\phi^\psi(\overline v \gamma_{n,m} h s_{j,m,n})\xi^{\psi^{-1}}_{\widetilde f_{\mathbf{{s}}}}(h)d\overline v dh. \end{align*} $$

The result follows.

2.6 Convergence and justifications

In this subsection, we prove the convergence of the double integral in (2.17), for ${\mathrm {Re}}(s_1)\gg 0, {\mathrm {Re}}(s_2)\gg 0$ . This is standard and similar to many other situations like [Reference Ben-Artzi and SoudryBAS09]. In the following, we will only consider the number field case since the proof in the function field case is easier. Using the Iwasawa decomposition and the fact that $(N_m(F)\times N_n(F))\backslash (N_m({\mathbb {A}})\times N_n({\mathbb {A}}))$ is compact, the convergence of the double integral in (2.17), for ${\mathrm {Re}}(s_1)\gg 0, {\mathrm {Re}}(s_2)\gg 0$ , quickly reduces to the convergence of

$$ \begin{align*} \int_{T_{m+n}({\mathbb {A}})}\|t\|^{N_0} |\det(a)|^{{\mathrm{Re}}(s_1)+c_1}|\det(b)|^{-{\mathrm{Re}}(s_2)+c_2} \int_{\overline U^{j,m,n}({\mathbb {A}})} |W_\phi^\psi\left(\mathbf{{t}}_{m,n}(a, b) \overline u \right)| d\overline udt \end{align*} $$

where $t={\mathrm {diag}}(a, b)$ with $a\in T_m({\mathbb {A}}), b\in T_n({\mathbb {A}})$ and $\mathbf {{t}}_{m,n}(a,b)=\eta _j \iota _j( t) \eta _j^{-1}={\mathrm {diag}}(a,I_{l-m-n},b)$ . Here, $N_0$ , $c_1$ and $c_2$ are fixed given positive numbers. Note that the integration over $K_{{\mathrm {GL}}_l({\mathbb {A}})}$ is dropped, using reasoning similar to that in [Reference Ben-Artzi and SoudryBAS09, Remark 4.7] (in conjunction with Lemma 2.8 and (2.26)). We may assume that the Whittaker function $W_\phi ^\psi $ decomposes as $\prod _{v}W_v$ , where $W_v$ is a local $\psi _v$ -Whittaker function of $\pi _v$ , such that outside of a finite set S of places (including the archimedean ones), $\pi _v$ is unramified, and $W_v=W_v^0$ is the normalized unramified $\psi _v$ -Whittaker function of $\pi _v$ whose value at the identity is equal to 1. At a place $v\in S$ , we assume that $W_v(I_{l})=1$ . We assume that for v outside of S, $\psi _v$ is unramified. It suffices to prove, for ${\mathrm {Re}}(s_1)\gg 0, {\mathrm {Re}}(s_2)\gg 0$ , that we have

(2.20) $$ \begin{align} & \prod_{v} \int_{T_{m+n}(F_v)}\|t\|^{N_0}_v |\det(a)|_v^{{\mathrm{Re}}(s_1)+c_1}|\det(b)|_v^{-{\mathrm{Re}}(s_2)+c_2} \nonumber\\ &\quad \cdot \int_{\overline U^{j,m,n}(F_v)} |W_v\left(\mathbf{{t}}_{m,n}(a,b) \overline u \right)| d\overline udt<\infty. \end{align} $$

Lemma 2.7. Let v be a finite place of F. For fixed $W_v\in {\mathcal {W}}(\pi _v,\psi _v)$ , and $t={\mathrm {diag}}(a,b)$ with $a\in T_m(F_v), b\in T_n(F_v)$ , the function

$$ \begin{align*}\overline u\mapsto W_v(\mathbf{{t}}_{m,n}(a,b)\overline u ), \quad \overline u\in \overline U^{j,m,n}(F_v)\end{align*} $$

has compact support in $\overline U^{j,m,n}(F_v)$ , which is independent of t. If $W_v=W_v^0$ , then this support is in $\overline U^{j,m,n}({\mathcal {O}}_v)$ .

Suppose v is finite. By Lemma 2.7, to prove that the local integral of (2.20) at the place v converges, it suffices to show

(2.21) $$ \begin{align} \int_{T_{m+n}(F_v)}\|t\|^{N_0}_v |\det(a)|_v^{{\mathrm{Re}}(s_1)+c_1}|\det(b)|_v^{-{\mathrm{Re}}(s_2)+c_2} |W_v\left(\mathbf{ {t}}_{m,n}(a,b) \right)| dt<\infty \end{align} $$

for ${\mathrm {Re}}(s_i)\gg 0$ . Now we recall gauge estimates on Whittaker functions in [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS79a, Section 2]. A gauge on ${\mathrm {GL}}_l(F_v)$ is a function $\xi $ on ${\mathrm {GL}}_l(F_v)$ which is invariant on the left under $N_l(F_v)$ , on the right under ${\mathrm {GL}}_l({\mathcal {O}}_v)$ , and which on $T_l(F_v)$ has the form

(2.22) $$ \begin{align} \xi(t)=|t_1t_2\cdots t_{l-1}|_v^{-c}\Phi(t_1, t_2, \ldots, t_{l-1}) \end{align} $$

for

$$ \begin{align*} t={\mathrm {diag}}(t_1t_2\cdots t_l, t_2,\ldots, t_l, \ldots, t_{l-1}t_l, t_l)\in T_l(F_v), \end{align*} $$

where $c\ge 0$ is a real number and $\Phi \ge 0$ is a Schwartz-Bruhat function on $F_v^{l-1}$ . In particular, $\xi $ is invariant under the centre of ${\mathrm {GL}}_l(F_v)$ . Write $a\in T_m(F_v)$ and $b\in T_n(F_v)$ as

(2.23) $$ \begin{align} \begin{aligned} a&={\mathrm {diag}}(a_1\cdots a_m, a_2,\ldots, a_m, \dots, a_{m-1}a_m, a_m),\\ b&={\mathrm {diag}}(b_1^{-1}, b_1^{-1}b_2^{-1}, \dots , b_1^{-1}b_2^{-1}\cdots b_{n}^{-1}), \end{aligned} \end{align} $$

with $a_i\in F_v^\times , b_j\in F_v^\times $ . Then

$$ \begin{align*} \begin{aligned} |\det(a)|_v&=|a_1a_2^2\cdots a_{m-1}^{m-1}a_m^m|_v, \\ |\det(b)|_v&=|b_1^nb_2^{n-1}\cdots b_{n-1}^2 b_n|_v^{-1}, \end{aligned} \end{align*} $$

and

$$ \begin{align*} \mathbf{{t}}_{m,n}&({\mathrm {diag}}(a,b))\\ &={\mathrm {diag}}(a_1\cdots a_m, a_2\cdots a_m, \ldots, a_{m-1}a_m, a_m, 1, 1, \ldots, 1, b_1^{-1}, b_1^{-1}b_2^{-1}, \ldots , b_1^{-1}b_2^{-1}\cdots b_{n}^{-1}). \end{align*} $$

Then for a gauge $\xi $ on ${\mathrm {GL}}_l(F_v)$ , it follows from (2.22) that there is some real number $c\ge 0$ and a Schwartz-Bruhat function $\Phi $ on $F_v^{l-1}$ such that

(2.24) $$ \begin{align} \xi(\mathbf{{t}}_{m,n}({\mathrm {diag}}(a,b)))= |a_1\cdots a_m b_1\cdots b_n|_v^{-c} \Phi(a_1, a_2, \dots, a_m, 1, \dots, 1, b_1, b_2, \dots, b_n). \end{align} $$

Write $|\omega _\pi |_v=\alpha ^{c_0}$ , where $\alpha $ is a non-negative real-valued function on $F_v^\times $ and $c_0$ is a real number. By [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS79a, Proposition 2.3.6], for any Whittaker function $W_v\in \mathcal {W}(\pi _v,\psi _v)$ , there is a gauge $\xi $ such that

(2.25) $$ \begin{align} |W_v\otimes \alpha^{-c_0/l}|\le \xi. \end{align} $$

Then (2.21) follows from (2.24) and the estimate (2.25). This proves that the local integral from (2.20) over a finite place v is convergent.

Now we turn to the archimedean places. Let v be an archimedean place, so $F_v$ is either $\mathbb {R}$ or $\mathbb {C}$ . We recall the notion of gauge [Reference Jacquet and ShalikaJS90] in this setting, which is slightly different from the non-archimedean case. Let $\chi $ be a sum of positive characters of $T_l(F_v)$ trivial on the centre of ${\mathrm {GL}}_l(F_v)$ . A homogeneous gauge on ${\mathrm {GL}}_l(F_v)$ is a function $\xi $ on ${\mathrm {GL}}_l(F_v)$ of the form

$$ \begin{align*} \xi(ntk)=\chi(t)\Phi(t_1, t_2,\ldots, t_{l-1}), \end{align*} $$

where $n\in N_l(F_v)$ , $t={\mathrm {diag}}(t_1, \ldots , t_l)\in T_l(F_v)$ , k is in the maximal compact subgroup $K_l$ of ${\mathrm {GL}}_l(F_v)$ and $\Phi>0$ is a rapidly decreasing function in $l-1$ variables. Here, $\Phi $ being rapidly decreasing means that, for every set of integers $N_i$ , $1\le i \le l-1$ , there is a constant $C>0$ such that

$$ \begin{align*} \Phi(t_1, t_2, \ldots, t_{l-1})\le C\prod_i (1+ |t_i|_v^2)^{-N_i}. \end{align*} $$

We have the following estimate.

Lemma 2.8. Let v be an archimedean place. Let $\xi $ be a homogeneous gauge on ${\mathrm {GL}}_l(F_v)$ . Let $a\in T_m(F_v)$ and $b\in T_n(F_v)$ , with Iwasawa decompositions

$$ \begin{align*} \begin{aligned} a=n_1 {\mathrm {diag}}(t_1, \ldots, t_{m}) k_1,\quad b =n_2 {\mathrm {diag}}(t_{l-n+1}, \ldots, t_{l}) k_2, \end{aligned} \end{align*} $$

where $t_1, \ldots , t_{m}, t_{l-n+1}, \ldots , t_l$ are positive real numbers. Set $t_{m+1}=t_{m+2}=\cdots =t_{l-n}=1$ . Given positive integers $M_1, \ldots , M_j, N_1, \ldots , N_n$ , $L_1, \ldots , L_{l-1}$ , there exists a positive constant $C>0$ such that

$$ \begin{align*} \begin{aligned} & \xi \left( \begin{pmatrix} a&&&&\\ &I_j&&&\\&&1&&\\&&&I_k&\\&&&&b \end{pmatrix} \begin{pmatrix} I_m&&&&\\ x&I_j&&&\\&&1&&\\&&&I_k&\\&&&y&I_n \end{pmatrix}\right) \\ &\quad\le C \prod_{i=1}^j (1+||x_{i}||^2)^{-M_{i}} \prod_{i=1}^n (1+||y_{i}||^2)^{-N_{i}} \prod_{i=1}^{l-1} \left(1+\left| \frac{t_i}{t_{i+1}}\right|{}^2\right)^{-L_i}\chi(t_1, t_2, \ldots, t_l). \end{aligned} \end{align*} $$

Here, $\chi $ is a fixed sum of positive characters of $T_l(F_v)$ .

By [Reference Jacquet and ShalikaJS90, Proposition 2.1], for any Whittaker function $W_v\in \mathcal {W}(\pi _v, \psi _v)$ , there is a gauge $\xi $ on ${\mathrm {GL}}_l(F_v)$ such that

(2.26) $$ \begin{align} |W_v(g)|\le \xi(g), \quad g\in {\mathrm{GL}}_l(F_v). \end{align} $$

Combining (2.26) with Lemma 2.8, we conclude that the archimedean part of the product in (2.20) is convergent.

Now we are ready to prove (2.20). By Lemma 2.7 and due to the convergence of the local integral at each local place, it suffices to prove

(2.27) $$ \begin{align} \prod_{v\not\in S} \int_{T_{m+n}(F_v)}\|t\|^{N_0}_v |\det(a)|_v^{{\mathrm{Re}}(s_1)+c_1}|\det(b)|_v^{-{\mathrm{Re}}(s_2)+c_2} |W_v^0\left(\mathbf{{t}}_{m,n}(a,b) \right)| dt<\infty. \end{align} $$

Write $a, b$ as in (2.23). Similar to [Reference Ben-Artzi and SoudryBAS09, Lemma 4.3], in order for the function

$$ \begin{align*} \mathbf{{t}}_{m,n}(a,b) \mapsto W_v^0\left(\mathbf{{t}}_{m,n}(a,b)\right) \end{align*} $$

to be non-zero, we must have $a_1, \ldots , a_m, b_1, \ldots , b_n\in \mathcal {O}_{v}$ . Thus,

$$ \begin{align*} W_v^0\left(\mathbf{{t}}_{m,n}(a,b)\right) = \Phi^0(a_1, \ldots, a_m, b_1, \ldots, b_n) W_v^0\left(\mathbf{{t}}_{m,n}(a,b)\right), \end{align*} $$

where $\Phi ^0$ is the characteristic function of $\mathcal {O}_{v}^{m+n}$ . On the other hand, since $W_\phi ^\psi =\prod _{v\in S} W_v \cdot \prod _{v\not \in S} W_v^0$ is the Whittaker function of a cusp form $\phi $ , there are constants $B_0, C_0>0$ , such that, for all $h\in {\mathrm {GL}}_l({\mathbb {A}})$ ,

$$ \begin{align*} | W_\phi^\psi(h)|\le C_0 \| h\|^{B_0}. \end{align*} $$

By [Reference Mœglin and WaldspurgerMW95, §1.2.2 (ii)], there is a constant $C_1$ (which depends on $C_0$ ), such that, for $t={\mathrm {diag}}(a,b)\in \prod _{v\not \in S} T_{m+n}({\mathbb {A}}^S)$ with $a, b$ as in (2.23), we have

$$ \begin{align*} \left| \prod_{v\not\in S} W_v^0 \left(\mathbf{{t}}_{m,n}(a,b)\right) \right| \le C_1 \| a\|^{B_0} \| b\|^{B_0}. \end{align*} $$

Thus, the product of local integrals over $v\not \in S$ in (2.27) is bounded by

$$ \begin{align*} \begin{aligned} C_1 & \prod_{v\not\in S} \int_{F_v^{m+n}} \Phi^0(a_1, \ldots, a_m, b_1, \ldots, b_n) \|a\|^{N_0+B_0} \|b\|^{N_0+B_0} \\ & \cdot |a_1a_2^2\cdots a_{m-1}^{m-1}a_m^m|_v^{{\mathrm{Re}}(s_1)+c_1} |b_1^nb_2^{n-1}\cdots b_{n-1}^2 b_n|_v^{{\mathrm{Re}}(s_2)-c_2} \prod_{1\le i\le m} da_i \prod_{1\le j \le n} db_j. \end{aligned} \end{align*} $$

This product converges for ${\mathrm {Re}}(s_1)\gg 0, {\mathrm {Re}}(s_2)\gg 0$ as desired (see, for example, [Reference Ben-Artzi and SoudryBAS09, Lemma 4.5]). This finishes the proof of (2.27), and hence the convergence of the double integral in (2.17), for ${\mathrm {Re}}(s_1)\gg 0, {\mathrm {Re}}(s_2)\gg 0$ .

3.1 Definition of the local zeta integrals

Let $\pi $ be an irreducible generic representation of ${\mathrm {GL}}_l(F)$ and let ${\mathcal {W}}(\pi ,\psi )$ be its Whittaker model. Let $(\tau _1,V_{\tau _1})$ (resp. $(\tau _2,V_{\tau _2})$ ) be an irreducible generic representation of ${\mathrm {GL}}_m(F)$ (resp. ${\mathrm {GL}}_n(F)$ ). As in the last section, we write ${\boldsymbol \tau }=(\tau _1,\tau _2)$ and $\widehat {\boldsymbol \tau }=(\tau _2,\tau _1)$ . Let $\mathbf {{s}}=(s_1,s_2)\in {\mathbb {C}}^2$ be a pair of complex numbers. Then we can consider the induced representation

$$ \begin{align*}{\mathrm {I}}(\mathbf{{s}},{\boldsymbol \tau}):={\mathrm{Ind}}_{P_{m,n}(F)}^{{\mathrm{GL}}_{m+n}(F)}(\tau_1||^{s_1-\frac{1}{2}}\boxtimes \tau_2||^{-s_2+1/2}).\end{align*} $$

We fix $\psi ^{-1}$ -Whittaker functionals $\lambda _{i}$ of $\tau _{i}$ . Recall that a section $f_{\mathbf {{s}}}\in {\mathrm {I}}(\mathbf { {s}},{\boldsymbol \tau })$ is a function $f_{\mathbf {{s}}}: {\mathrm {GL}}_{m+n}(F)\to V_{\tau _1}\boxtimes V_{\tau _2}$ satisfying certain quasi-invariance properties. We consider the ${\mathbb {C}}$ -valued function

$$ \begin{align*}\xi_{f_{\mathbf{{s}}}}:{\mathrm{GL}}_{m+n}(F)\times {\mathrm{GL}}_m(F)\times {\mathrm{GL}}_n(F)\to {\mathbb {C}}\end{align*} $$

defined by

$$ \begin{align*}\xi_{f_{\mathbf{{s}}}}(h,a_1,a_2)=\lambda_{1}\boxtimes \lambda_{2}(\tau_{1}(a_1)\boxtimes \tau_{2}(a_2)(f_{\mathbf{ {s}}}(h))).\end{align*} $$

Set ${\mathcal {W}}(\mathbf {{s}},{\boldsymbol \tau },\psi ^{-1})=\left \{{\xi _{f_{\mathbf {{s}}}}: f_{\mathbf {{s}}}\in {\mathrm {I}}(\mathbf { {s}},{\boldsymbol \tau })}\right \}$ . Note that an element $\xi _{\mathbf {{s}}}$ satisfies

$$ \begin{align*} \xi_{\mathbf{{s}}}\left(\begin{pmatrix} b_1 &\\ &b_2 \end{pmatrix} u h, a_1,a_2 \right)=|\det(b_1)|^{s_1+\frac{n-1}{2}} |\det(b_2)|^{-s_2-\frac{m-1}{2}}\xi_{\mathbf{{s}}}(h,a_1b_1,a_2b_2), \end{align*} $$

for $a_1,b_1\in {\mathrm {GL}}_m(F), a_2,b_2\in {\mathrm {GL}}_n(F), u\in N_{m,n}(F), h\in {\mathrm {GL}}_{m+n}(F)$ . In particular

$$ \begin{align*} \xi_{\mathbf{{s}}}\left(\begin{pmatrix} u_1&\\ &u_2 \end{pmatrix} uh, I_m,I_n \right)&=\psi^{-1}(u_1)\psi^{-1}(u_2)\xi_s(h,I_m,I_n), \end{align*} $$

for $ u_1\in N_m(F), u_2\in N_n(F), u\in N_{m,n}(F), h\in {\mathrm {GL}}_{m+n}(F).$ We usually write $\xi _{\mathbf {{s}}}(h,I_m,I_n)$ as $\xi _{\mathbf {{s}}}(h)$ for simplicity.

Similarly, we can consider the space ${\mathcal {W}}(1-{\widehat {\mathbf {s}}}, \widehat {\boldsymbol \tau },\psi ^{-1})=\left \{{\xi _{f_{1-{\widehat {\mathbf {s}}}}}: f_{1-{\widehat {\mathbf {s}}}}\in {\mathrm {I}}(1-{\widehat {\mathbf {s}}},\widehat {\boldsymbol {\tau }})}\right \}$ . Let $M_{w_{m,n}}: {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })\to {\mathrm {I}}(1-{\widehat {\mathbf {s}}},\widehat {\boldsymbol \tau })$ be the standard intertwining operator defined by

$$ \begin{align*}M_{w_{m,n}}f_{\mathbf{{s}}}(h)=\int_{N_{n,m}(F)}f_{\mathbf{{s}}}(w_{m,n}uh)du.\end{align*} $$

This integral converges in a cone of $\mathbf {{s}}$ and can be extended meromorphically to ${\mathbb {C}}^2$ ; see [Reference Mœglin and WaldspurgerMW95]. Moreover, to obtain an element in $I(1-{\widehat {\mathbf {s}}}, \widehat {\boldsymbol \tau })$ , we need to identify $\tau _1\boxtimes \tau _2$ with $\tau _2\boxtimes \tau _1$ in the definition of $M_{w_{m,n}}$ . Equivalently, we need to compose the above integral with the map $\tau _1\boxtimes \tau _2\to \tau _2\boxtimes \tau _1$ defined by $v_1\boxtimes v_2\to v_2\boxtimes v_1$ , for $v_1\in \tau _1, v_2\in \tau _2$ . We adopt the convention that omits the above identification from the notation. The above intertwining operator gives a map

$$ \begin{align*}M_{w_{m,n}}: {\mathcal {W}}(\mathbf{{s}},{\boldsymbol \tau},\psi^{-1})\to {\mathcal {W}}(1-{\widehat{\mathbf{s}}},\widehat{\boldsymbol \tau},\psi^{-1})\end{align*} $$

defined by

$$ \begin{align*}M_{w_{m,n}}(\xi_{\mathbf{{s}}})(h,a_1,a_2)=\int_{N_{n,m}(F)}\xi_{\mathbf{{s}}} (w_{m,n}uh, a_2, a_1)du.\end{align*} $$

For $W\in {\mathcal {W}}(\pi ,\psi )$ , $\xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},{\boldsymbol \tau },\psi ^{-1})$ , and for j with $0\le j\le l-m-n-1$ , we consider the local zeta integrals

(3.1) $$ \begin{align} \Psi(W, \xi_{\mathbf{{s}}}; j):=\int_{N_{m+n}(F)\backslash {\mathrm{GL}}_{m+n}(F)}\int_{\overline U^{j,m,n}(F)}W\left(\overline u \gamma_{m,n}\begin{pmatrix} h&\\ &I_{l-m-n} \end{pmatrix} \right)\xi_{\mathbf{{s}}}(h)d\overline udh,\end{align} $$

where we recall that

$$ \begin{align*} \overline{U}^{j,m,n}&=\left\{{\overline u(x,y)=\begin{pmatrix} I_m&&&&\\ x&I_{j}&&&\\ &&1&&\\ &&&I_{k}&\\ &&&y& I_n \end{pmatrix}: \substack{ x\in {\mathrm {Mat}}_{j\times m}\\ y\in {\mathrm {Mat}}_{n\times k}}}\right\},\\ \gamma_{m,n}&=\eta_{j,m,n}s_{j,m,n}^{-1}=\begin{pmatrix} I_m&&\\ &&I_{l-m-n}\\ &I_n& \end{pmatrix}. \end{align*} $$

Here we remark that the natural numbers $m,n$ appearing in the local zeta integral (3.1) are determined by the section $\xi _{\mathbf {{s}}}$ , which is an element of ${\mathrm {Ind}}_{P_{m,n}(F)}^{{\mathrm {GL}}_{m+n}(F)}(\tau _1||^{s_1-1/2}\otimes \tau _2||^{-s_2+1/2})$ . In particular, if we take $\widetilde \xi _{1-{\widehat {\mathbf {s}}}}\in {\mathcal {W}}(1-{\widehat {\mathbf {s}}},\widehat {\boldsymbol \tau },\psi ^{-1})$ , we should have

(3.2) $$ \begin{align} \Psi(W,\widetilde\xi_{1-{\widehat{\mathbf{s}}}};j)=\int_{N_{m+n}(F)\backslash {\mathrm{GL}}_{m+n}(F)}\int_{\overline U^{j,n,m}(F)}W\left(\overline u \gamma_{n,m}\begin{pmatrix} h&\\ &I_{l-m-n} \end{pmatrix} \right)\widetilde\xi_{1-{\widehat{\mathbf{s}}}}(h)d\overline udh. \end{align} $$

Remark 3.1. In this remark, we assume that F is a global field. If $\phi =\otimes \phi _v$ is a cusp form on ${\mathrm {GL}}_l({\mathbb {A}})$ and $f_{\mathbf {{s}}}=\otimes f_{\mathbf {{s}},v}\in {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ is a pure tensor of a global section, then Theorem 2.4 and Theorem 2.6 imply that

$$ \begin{align*}I_j(\phi,f_{\mathbf{{s}}})=\prod_v \Psi(\rho(s_{j,m,n})W_v,\xi_{f_{\mathbf{{s}},v}};j), \quad I_j(\phi,\widetilde f_{\mathbf{ {s}}})=\prod_v \Psi(\rho(s_{j,m,n})W_v,\xi_{\widetilde f_{\mathbf{{s}},v}};j).\end{align*} $$

Here $\rho $ denotes the right translation.

Remark 3.2. In this remark, we consider the degenerate case when $m>0$ and $n=0$ . In this case, ${\boldsymbol \tau }=\tau _1$ is just a representation of ${\mathrm {GL}}_m(F)$ , and $\mathbf {{s}}=s$ is a single complex number. Moreover, an element $\xi _{\mathbf {{s}}}$ has the form $\xi _{\mathbf {{s}}}(h)=W'(h)|\det (h)|^{s-1/2}$ and we have $M_{w_{m,0}}(\xi _{\mathbf {{s}}})=\xi _{\mathbf {{s}}}.$ Thus

$$ \begin{align*} \Psi(W,\xi_{\mathbf{{s}}};j)=\int_{N_m(F)\backslash {\mathrm{GL}}_m(F)}\int_{ {\mathrm {Mat}}_{j\times m}(F)}&W\left(\begin{pmatrix} I_m &&\\ x&I_j&\\ &&I_{l-m-j} \end{pmatrix} \begin{pmatrix} h&\\ &I_{l-m} \end{pmatrix} \right)\\ & \cdot W'(h)|\det(h)|^{s-1/2}dxdh, \end{align*} $$

and

$$ \begin{align*} & \Psi(W,M_{w_{m,0}}(\xi_{\mathbf{{s}}});j)\\ &\quad =\int_{N_m(F)\backslash {\mathrm{GL}}_m(F)}\int_{ {\mathrm {Mat}}_{m\times k}(F)}&W\left(\begin{pmatrix} I_{j+1} &&\\ &I_k&\\ &y&I_{m} \end{pmatrix} \begin{pmatrix} &I_{l-m}\\ I_m& \end{pmatrix} \begin{pmatrix} h&\\ &I_{l-m} \end{pmatrix} \right)\\ &\qquad \cdot W'(h)|\det(h)|^{s-1/2}dydh. \end{align*} $$

Here we notice that $\gamma _{m,0}=I_l$ while $\gamma _{0,m}=\begin {pmatrix} &I_{l-m} \\ I_m & \end {pmatrix}.$ A simple change of variable shows that

$$ \begin{align*} &\Psi(W,\xi_{\mathbf{{s}}};j) \\ &\quad =\int_{N_m(F)\backslash {\mathrm{GL}}_m(F)}\int_{ {\mathrm {Mat}}_{j\times m}(F)}&W\left(\begin{pmatrix} h &&\\ x&I_j&\\ &&I_{l-m-j} \end{pmatrix} \right) W'(h)|\det(h)|^{s-1/2-j}dxdh. \end{align*} $$

One can compare the above integral with that defined by Jacquet–Piatetski-Shapiro–Shalika in [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83] and observe that

(3.3) $$ \begin{align} \Psi(W,\xi_{\mathbf{{s}}};j)=\Psi^{\mathrm{JPSS}}(s-j+\frac{l-m-1}{2}, W,W'; j), \end{align} $$

where $\Psi ^{\mathrm {JPSS}}$ denotes the integral defined in [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83, p.387]. On the other hand, for $W\in {\mathcal {W}}(\pi ,\psi )$ , we denote $\widetilde W(g)=W(J_l {}^t\!g^{-1})$ , which represents a Whittaker function of the contragredient representation $\widetilde \pi $ of $\pi $ . It is easy to check that

$$ \begin{align*} \Psi(W,M_{w_{m,0}}(\xi_{\mathbf{{s}}});j)=\int_{N_m(F)\backslash {\mathrm{GL}}_m(F)}\int_{ {\mathrm {Mat}}_{k\times m}(F)}&\widetilde W\left(\begin{pmatrix} h &&\\ y&I_k&\\ &&I_{j+1} \end{pmatrix} \begin{pmatrix} I_m &\\ & J_{l-m} \end{pmatrix} \right)\\ &\widetilde W'(h )|\det(h)|^{-s+1/2-k}dydh. \end{align*} $$

Thus we get

$$ \begin{align*} \Psi(W, M_{w_{m,0}}(\xi_{\mathbf{{s}}}); j)= \Psi^{\mathrm{JPSS}}\left(1-(s-j+\frac{l-m-1}{2}) ,\rho\left( \begin{pmatrix} I_m &\\ & J_{l-m}\end{pmatrix} \right)\widetilde W, \widetilde W'; l-m-1 \right). \end{align*} $$

Remark 3.3. Similarly, in the degenerate case where $m=0$ and $n>0$ , ${\boldsymbol \tau }=\tau _2$ is just a representation of ${\mathrm {GL}}_n(F)$ , $\mathbf {{s}}=s$ is a single complex number and an element $\xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},{\boldsymbol \tau },\psi ^{-1})$ has the form $\xi _{\mathbf {{s}}}(h)=W^{\prime \prime }(h)|\det (h)|^{-s+1/2}$ where $W^{\prime \prime }\in \mathcal {W}(\tau _2,\psi ^{-1})$ . In this case, we have

$$ \begin{align*} \Psi(W,\xi_{\mathbf{{s}}};j) = \Psi^{\mathrm{JPSS}}(s+j-\frac{l-n-1}{2}, \rho\left(\begin{pmatrix} I_n&\\ &J_{l-n} \end{pmatrix}\right) \widetilde W,\widetilde W^{\prime\prime}; l-n-j-1), \end{align*} $$

and

$$ \begin{align*} \Psi(W, M_{w_{0,n}}(\xi_{\mathbf{{s}}});j) = \Psi^{\mathrm{JPSS}}(1-(s+j-\frac{l-n-1}{2}), W, W^{\prime\prime}; l-n-j-1). \end{align*} $$

Proof. For $n=0$ , this was already proved in [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83] over non-archimedean local fields and in [Reference Jacquet and ShalikaJS90] over archimedean fields. Very similar statements can be found in many other places in the literature, for example, [Reference SoudrySou93], [Reference SoudrySou95], [Reference Ginzburg, Rallis and SoudryGRS98], [Reference Ben-Artzi and SoudryBAS09], and [Reference Cai, Friedberg and KaplanCFK22]. We provide some details here for completeness.

First, we consider the case where F is non-archimedean. By the Iwasawa decomposition and the fact that smooth vectors are finite under the maximal compact subgroup, we get that $\Psi (W,\xi _{\mathbf {{s}}}; j)$ is a finite sum of integrals of the form

$$ \begin{align*} & \int_{T_{m+n}(F)}\int_{\overline U^{m,n}(F)}W^\prime(\mathbf{{t}}_{m,n}(a,b)\overline u) d\overline u\\ &\quad \cdot W_{\tau_1}(a)W_{\tau_2}(b) |\det(a)|^{s_1+\frac{n-1}{2}-j}|\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1} dt \end{align*} $$

where $W^\prime \in \mathcal {W}(\pi ,\psi )$ , $W_{\tau _1}\in \mathcal {W}(\tau _1,\psi ^{-1})$ , $W_{\tau _2}\in \mathcal {W}(\tau _2, \psi ^{-1})$ , $t={\mathrm {diag}}(a, b)$ with $a\in T_m(F), b\in T_n(F)$ and $\mathbf {{t}}_{m,n}(a,b)={\mathrm {diag}}(a,I_{l-m-n},b)$ . Here the term $|\det (a)|^{-j} |\det (b)|^{k} $ comes from conjugating $\mathbf {{t}}_{m,n}(a,b)$ to the left of $\overline u$ and making a change of variables on $\overline u$ . By Lemma 2.7, the last integral is a finite sum of integrals of the form

(3.4) $$ \begin{align} \int_{T_{m+n}(F)} W^\prime(\mathbf{{t}}_{m,n}(a,b) ) W_{\tau_1}(a)W_{\tau_2}(b) |\det(a)|^{s_1+\frac{n-1}{2}-j}|\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1} dt. \end{align} $$

Now we recall the asymptotic expansion of Whittaker functions [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83, Section 2.5]. There is a finite set $X_l$ of functions on $T_l(F)$ such that for every $W\in \mathcal {W}(\pi ,\psi )$ we have

$$ \begin{align*} W(t)=\sum_{\chi\in X_l} \omega_{\pi}(t_l) \phi_\chi(t_1, t_2, \ldots, t_{l-1}) \chi(t) \end{align*} $$

where $t={\mathrm {diag}}(t_1t_2\cdots t_l, t_2\cdots t_l, \ldots , t_{l-1}t_l, t_l)\in T_l(F)$ and $\phi _\chi \in \mathcal {S}(F^{l-1})$ . Then for every $W\in \mathcal {W}(\pi ,\psi )$ , we have

(3.5) $$ \begin{align} |W(t)| \le \sum_{\eta\in Y_l} \phi_\eta(t_1, t_2, \ldots, t_{l-1}) \eta(t) \end{align} $$

where $\phi _\eta \in \mathcal {S}(F^{l-1})$ is non-negative and $\eta $ varies in another finite set $Y_l$ of finite functions on $T_l(F)$ . Applying the majorization (3.5) to $W^\prime $ (and the analogous ones for $W_{\tau _1}$ and $W_{\tau _2}$ ), we obtain the absolute convergence of the integral (3.4) for ${\mathrm {Re}}(s_i)\gg 0$ for $i=1,2$ . Hence $\Psi (W,\xi _{\mathbf {{s}}}; j)$ is absolutely convergent for ${\mathrm {Re}}(s_i)\gg 0$ for $i=1,2$ .

We continue to assume that F is non-archimedean. Since $N_{m+n}(F)T_{m+n}(F)\overline {N}_{m+n}(F)$ is an open dense subset of ${\mathrm {GL}}_{m+n}(F)$ whose complement has Haar measure zero, we may rewrite $\Psi (W,\xi _{\mathbf {{s}}}; j)$ as

(3.6) $$ \begin{align} \begin{aligned} \int_{T_{m+n}(F)} \int_{\overline{N}_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} &W\left(\overline u \gamma_{m,n}\begin{pmatrix} t \overline{v} &\\ &I_{l-m-n} \end{pmatrix} \right) \xi_{\mathbf{{s}}}( \overline{v}, a, b) \\ &|\det(a)|^{s_1+\frac{n-1}{2}} |\det(b)|^{-s_2-\frac{m-1}{2}} \delta_{B_{m+n}}(t)^{-1} d\overline u d\overline v dt, \end{aligned} \end{align} $$

where $t={\mathrm {diag}}(a, b)$ with $a\in T_m(F)$ , $b\in T_n(F)$ . Similar to [Reference SoudrySou93, Proposition 6.1], we choose $\xi _{\mathbf {{s}}}$ to have support in $B_{m+n}(F)\cdot \mathcal {V}_1$ , where $\mathcal {V}_1$ is a small open compact subgroup of ${\mathrm {GL}}_{m+n}(F)$ , and such that $\xi _{\mathbf {{s}}}(u, b_1, b_2)=W_{\tau _1}(b_1)W_{\tau _2}(b_2)$ for $u\in \mathcal {V}_1$ , $b_1\in T_{m}(F), b_2\in T_n(F)$ . Here, $W_{\tau _i}\in \mathcal {W}(\tau _i, \psi ^{-1})$ for $i=1, 2$ . We choose $\mathcal {V}_1$ to be so small that W is fixed by $\pi ( {\mathrm {diag}}(\overline {v}, I_{l-m-n}))$ for $\overline {v}\in \mathcal {V}_1$ . Thus, $\Psi (W,\xi _{\mathbf {{s}}}; j)$ is equal to

$$ \begin{align*} \begin{aligned} \mathrm{vol}(\mathcal{V}_1\cap \overline{N}_{m+n}(F)) \cdot \int_{T_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} &W\left(\overline u \gamma_{m,n}\begin{pmatrix} t &\\ &I_{l-m-n} \end{pmatrix} \right) W_{\tau_1}(a)W_{\tau_2}(b) \\ & |\det(a)|^{s_1+\frac{n-1}{2}} |\det(b)|^{-s_2-\frac{m-1}{2}} \delta_{B_{m+n}}(t)^{-1} d\overline u dt. \end{aligned} \end{align*} $$

We conjugate ${\mathrm {diag}}(t, I_{l-m-n})$ to the left of $\overline u$ and make a change of variable in $\overline u$ to get

$$ \begin{align*} \mathrm{vol}(\mathcal{V}_1\cap \overline{N}_{m+n}(F)) &\cdot \int_{T_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} \rho(\gamma_{m,n}) W\left( \begin{pmatrix} a & &\\ &I_{l-m-n}&\\ & & b \end{pmatrix} \overline u \right) W_{\tau_1}(a)W_{\tau_2}(b) \\ & |\det(a)|^{s_1+\frac{n-1}{2}-j} |\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1} d\overline u dt. \end{align*} $$

Now we choose W, $W_{\tau _1}$ and $W_{\tau _2}$ such that the function

$$ \begin{align*}(a,b,\overline u)\mapsto \rho(\gamma_{m,n})W\left( \begin{pmatrix} a & &\\ &I_{l-m-n}&\\ & & b \end{pmatrix} \overline u \right) W_{\tau_1}(a)W_{\tau_2}(b)\end{align*} $$

is the characteristic function of a small neighbourhood of $(I_m, I_n, I_l)$ . Thus, the integral can be made constant.

Now we assume F is archimedean. Similar to [Reference SoudrySou93, Lemma 5.2], there is a positive integer $A_0$ , such that for any $\xi _{\mathbf {{s}}}$ , there is a constant $c_{\mathbf {{s}}}>0$ , such that

$$ \begin{align*} |\xi_{\mathbf{{s}}}({\mathrm {diag}}(a, b)k) | \le c_{\mathbf{{s}}} |\det(a)|^{{\mathrm{Re}}(s_1)+\frac{n-1}{2}} |\det(b)|^{-{\mathrm{Re}}(s_2)-\frac{m-1}{2}} \|{\mathrm {diag}}(a, b)\|^{A_0}, \end{align*} $$

where $a\in T_m(F), b\in T_n(F)$ , and k is in the maximal compact subgroup $K_l$ of ${\mathrm {GL}}_l(F)$ . We then use the Iwasawa decomposition, (2.26) and Lemma 2.8 to conclude the absolute convergence of $\Psi (W,\xi _{\mathbf {{s}}}; j)$ .

Now we prove the non-vanishing of the integrals when F is archimedean. Write $\Psi (W,\xi _{\mathbf {{s}}}; j)$ in the form (3.6). Similar to [Reference Ginzburg, Rallis and SoudryGRS98, Proposition 6.7], we can choose $\xi _{\mathbf {{s}}}$ to have support in $P_{m,n}(F)\cdot \overline N_{m+n}(F)$ , and assume

$$ \begin{align*} \xi_{\mathbf{{s}}}\left(\begin{pmatrix} b_1 &\\ &b_2 \end{pmatrix} u \overline v, a_1,a_2 \right)=|\det(b_1)|^{s_1+\frac{n-1}{2}} |\det(b_2)|^{-s_2-\frac{m-1}{2}} \varphi_1(\overline v) W_{\tau_1}(a_1b_1)W_{\tau_2}(a_2b_2), \end{align*} $$

for $a_1,b_1\in {\mathrm {GL}}_m(F), a_2,b_2\in {\mathrm {GL}}_n(F), u\in N_{m,n}(F), \overline v\in \overline N_{m+n}(F)$ , $W_{\tau _i}\in \mathcal {W}(\tau _i,\psi ^{-1})$ for $i=1, 2$ , and $\varphi _1\in C_c^\infty (\overline N_{m+n}(F))$ . With this choice, $\Psi (W,\xi _{\mathbf {{s}}}; j)$ is equal to an integral of the form

(3.7) $$ \begin{align} \begin{aligned} \int_{T_{m+n}(F)} \int_{\overline{N}_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} &W\left(\overline u \gamma_{m,n}\begin{pmatrix} t \overline{v} &\\ &I_{l-m-n} \end{pmatrix} \right) \varphi_1(\overline{v})W_{\tau_1}(a) W_{\tau_2}(b) \\ &|\det(a)|^{s_1+\frac{n-1}{2}} |\det(b)|^{-s_2-\frac{m-1}{2}} \delta_{B_{m+n}}(t)^{-1} d\overline u d\overline v dt. \end{aligned} \end{align} $$

We consider the $d\overline v$ integration first. This is a convolution of $\varphi _1$ against $W\in \mathcal {W}(\pi ,\psi )$ . By the Dixmier-Malliavin Theorem [Reference Dixmier and MalliavinDM78], a linear combination of the $d\overline v$ integrals represents a general element of $\mathcal {W}(\pi ,\psi )$ (see [Reference Ginzburg, Rallis and SoudryGRS98, pp. 233] or [Reference SoudrySou95, pp. 219]). Thus, a suitable linear combination of integrals of the form (3.7) gives an integral of the form

$$ \begin{align*} \begin{aligned} \int_{T_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} &W\left(\overline u \gamma_{m,n}\begin{pmatrix} t &\\ &I_{l-m-n} \end{pmatrix} \right) W_{\tau_1}(a) W_{\tau_2}(b) \\ &|\det(a)|^{s_1+\frac{n-1}{2}} |\det(b)|^{-s_2-\frac{m-1}{2}} \delta_{B_{m+n}}(t)^{-1} d\overline u dt. \end{aligned} \end{align*} $$

We conjugate ${\mathrm {diag}}(t, I_{l-m-n})$ to the left of $\overline u$ to get

$$ \begin{align*} \begin{aligned} \int_{T_{m+n}(F)} \int_{\overline U^{j,m,n}(F)} & \rho(\gamma_{m,n})W\left( \begin{pmatrix} a & &\\ &I_{l-m-n}&\\ & & b \end{pmatrix} \overline u \right) W_{\tau_1}(a) W_{\tau_2}(b) \\ &|\det(a)|^{s_1+\frac{n-1}{2}-j} |\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1} d\overline u dt. \end{aligned} \end{align*} $$

Now we choose W so that $\rho (\gamma _{m,n})W(t\overline u)=\rho (\gamma _{m,n})W(t)\varphi _2(\overline u)$ for $t\in B_l(F)$ , $\overline u \in U^{j,m,n}(F)$ and $\varphi _2\in C_c^\infty (U^{j,m,n}(F))$ . Then the above integral becomes

$$ \begin{align*} \begin{aligned} \int_{\overline U^{j,m,n}(F)} \varphi_2(\overline u)d\overline u \cdot \int_{T_{m+n}(F)} & \rho(\gamma_{m,n})W\left( \begin{pmatrix} a & &\\ &I_{l-m-n}&\\ & & b \end{pmatrix} \right) W_{\tau_1}(a) W_{\tau_2}(b) \\ &|\det(a)|^{s_1+\frac{n-1}{2}-j} |\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1} dt. \end{aligned} \end{align*} $$

The $d\overline u$ integral is a nonzero constant for appropriate $\varphi _2$ . For appropriate $W, W_{\tau _1}, W_{\tau _2}$ , the $dt$ integral is holomorphic and nonzero in a neighbourhood of any given $\mathbf {{s}}$ . This proves that there is a linear combination of the local integrals $\Psi (W,\xi _{\mathbf {{s}}}; j)$ which is holomorphic and nonzero in a neighbourhood of any given $\mathbf {{s}}$ .

3.2 Local functional equations

Proposition 3.6. Let F be a non-archimedean local field of characteristic different from 2. There exists a meromorphic function $\Gamma (\mathbf { {s}},\pi \times {\boldsymbol \tau },\psi ;j)$ such that

$$ \begin{align*}\Psi(W,M_{w_{m,n}}(\xi_{\mathbf{{s}}});j) =\Gamma(\mathbf{{s}},\pi\times{\boldsymbol \tau},\psi;j)\Psi(W,\xi_{\mathbf{ {s}}};j),\end{align*} $$

for any $W\in {\mathcal {W}}(\pi ,\psi )$ and $\xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},{\boldsymbol \tau },\psi ^{-1})$ .

Proof. Recall that

$$ \begin{align*}Y_{j,m,n}=\left\{{\begin{pmatrix} u&*&*\\ &I_{m+n+1} &*\\ &&v \end{pmatrix}, u\in N_{j}, v\in N_{k}}\right\}\end{align*} $$

and we have defined a character $\psi _{j}$ on $Y_{j,m,n}(F)$ ; see (2.3). One can check that for any $y\in Y_{j,m,n}(F)$ , we have

(3.8) $$ \begin{align} \begin{aligned} \Psi(\rho(y)W,\xi_{\mathbf{{s}}};j)&=\psi_j(y)\Psi(W,\xi_{\mathbf{{s}}};j), \end{aligned} \end{align} $$

and for any $h=\begin {pmatrix} a& b\\ c&d\end {pmatrix} \in {\mathrm {GL}}_{m+n}(F)$ with $a\in {\mathrm {Mat}}_{m\times m}(F)$ ,

(3.9) $$ \begin{align} \Psi\left(\rho\left(\left(\begin{smallmatrix} I_j& & & & \\ &a&&b& \\ &&1&& \\ &c&&d& \\ &&&&I_k \end{smallmatrix}\right) \right)W,\rho(h)\xi_{\mathbf{{s}}};j\right)=\Psi(W,\xi_{\mathbf{{s}}};j). \end{align} $$

Let

$$ \begin{align*}H_{j,m,n}=\left\{{\left(\begin{smallmatrix} u&*&*&*&*\\ &a&&b&*\\ &&1&&*\\ &c&&d&*\\ &&&&v \end{smallmatrix}\right), u\in N_j, v\in N_k, \begin{pmatrix} a& b\\ c&d\end{pmatrix}\in {\mathrm{GL}}_{m+n} }\right\}={\mathrm{GL}}_{m+n}\ltimes Y_{j,m,n}.\end{align*} $$

One can define a representation $\nu _{\mathbf {{s}}}$ of $H_{j,m,n}(F)$ by $\nu _{\mathbf {{s}}}|_{{\mathrm {GL}}_{m+n}(F)}={\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ and $\nu _{\mathbf {{s}}}|_{Y_{j,m,n}(F)}=\psi _j$ . Then (3.8) and (3.9) imply that the bilinear form $(W,\xi _{\mathbf {{s}}})\mapsto \Psi (W,\xi _{\mathbf {{s}}};j)$ defines an element in

$$ \begin{align*}{\mathrm{Hom}}_{H_{j,m,n}(F)}(\pi\otimes \nu_{\mathbf{{s}}},1).\end{align*} $$

Similarly, the bilinear form $(W,\xi _{\mathbf {{s}}})\mapsto \Psi (W, M(\mathbf {{s}}, {\boldsymbol \tau })\xi _{\mathbf {{s}}}; j)$ satisfies the same properties (3.8) and (3.9) and hence $(W,\xi _{\mathbf {{s}}})\mapsto \Psi (W, M(\mathbf {{s}}, {\boldsymbol \tau })\xi _{\mathbf {{s}}}; j)$ also defines an element in

$$ \begin{align*}{\mathrm{Hom}}_{H_{j,m,n}(F)}(\pi\otimes \nu_{\mathbf{{s}}},1).\end{align*} $$

By the uniqueness of Bessel models (see [Reference Gan, Gross and PrasadGGP12] and [Reference ChanCha22, Corollary 5.11] when F is of characteristic zero, and [Reference MezerMez23, Theorem 1.9] when F is of positive characteristic different from 2), we have

(3.10) $$ \begin{align}\dim_{{\mathbb {C}}}{\mathrm{Hom}}_{H_{j,m,n}(F)}(\pi\otimes \nu_{\mathbf{{s}}},1)\le 1\end{align} $$

excluding a discrete set of $\mathbf {{s}}$ . This proves the existence of the gamma factor. By Proposition 3.5, there exist data $W, \xi _{\mathbf {{s}}}$ such that $\Psi (W, \xi _{\mathbf {{s}}}; j)$ is 1, which shows that $\Gamma (\mathbf {{s}},\pi \times {\boldsymbol \tau },\psi; j)$ is meromorphic.

Remark 3.7. If F is archimedean, the local integrals still define elements in

$$ \begin{align*}{\mathrm{Hom}}_{H_{j,m,n}(F)}(\pi\otimes\nu_{\mathbf{ {s}}},1).\end{align*} $$

It is known that this Hom space has dimension at most one when $j=0$ by [Reference Chen and SunCS15] and when $m=n$ by [Reference Gan, Gross and PrasadGGP12, Reference Jiang, Sun and ZhuJSZ10]. Thus, in these cases, we still have the local gamma factors $\Gamma (\mathbf {{s}}, \pi \times {\boldsymbol \tau },\psi; j)$ . As pointed out in [Reference ChanCha22] in the non-archimedean case, the general multiplicity one result (namely, when $m\ne n$ ) should also follow from the general framework of [Reference Gan, Gross and PrasadGGP12]. But it seems that this is not recorded anywhere.

Remark 3.8. By Remark 3.2 and Remark 3.3, we immediately obtain that

$$ \begin{align*} \begin{aligned} \Gamma((s_1,0), \pi\times(\tau_1, 0), \psi;j) =\omega_{\tau_1}(-1)^{l-1}\gamma( s_1-j+\frac{l-m-1}{2}, \pi\times\tau_1, \psi), \end{aligned} \end{align*} $$

and

$$ \begin{align*} \begin{aligned} \Gamma((0, s_2), \pi\times(0, \tau_2), \psi;j) =\omega_{\tau_2}(-1)^{l-1}\gamma(s_2+j-\frac{l-n-1}{2},\widetilde \pi\times \widetilde \tau_2,\psi). \end{aligned} \end{align*} $$

The gamma factor defined in Proposition 3.6 is indeed just a product of the JPSS local gamma factors defined in [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83]. More precisely, we have the following

Proposition 3.9. Let F be a local non-archimedean field of characteristic different from 2. Then we have

$$\begin{align*}\begin{aligned} \Gamma(\mathbf{{s}}, \pi\times(\tau_1, \tau_2), \psi;j) = \omega_{\tau_1}(-1)^{l-1}\omega_{\tau_2}(-1)^{l-1} \cdot \frac{\gamma(s_1+\frac{k-j}{2},\pi\times\tau_1,\psi) \gamma(s_2+\frac{j-k}{2},\widetilde \pi\times \widetilde \tau_2,\psi)}{\gamma(s_1+s_2,\tau_1\times \widetilde \tau_2,\psi)}. \end{aligned}\end{align*}$$

If $l=2r+1$ , $m=n$ and $j=r-m$ , the gamma factor is just the local gamma factor for ${\mathrm {U}}_{E/F}(2r+1)\times {\mathrm {Res}}_{E/F}({\mathrm {GL}}_m)$ at split places, and the above relation with JPSS local gamma factors was proved in [Reference Cheng and WangCW25]. To streamline the presentation and to avoid making the main body of the paper too long, we will defer the proof of Proposition 3.9 to Appendix A.

3.3 Unramified calculation

In this subsection, let F be a non-archimedean local field with ring of integers ${\mathcal {O}}$ . Let $\varpi \in {\mathcal {O}}$ be a fixed uniformizer, and $q=|{\mathcal {O}}/(\varpi )|$ . Our goal in this subsection is to compute the local zeta integral (3.1) when everything is unramified. In particular, we assume that $\pi $ is unramified with Satake parameters $\boldsymbol {\alpha }={\mathrm {diag}}(\alpha _1,\dots ,\alpha _{l})\in {\mathrm {GL}}_l({\mathbb {C}})$ and $\tau _1$ (resp. $\widetilde \tau _2$ ) is unramified with Satake parameters $\boldsymbol {\beta }^1={\mathrm {diag}}(\beta _1^1,\dots ,\beta _m^1)\in {\mathrm {GL}}_m({\mathbb {C}})$ (resp. $\boldsymbol {\beta }^2={\mathrm {diag}}(\beta _1^2,\dots ,\beta _n^2)\in {\mathrm {GL}}_n({\mathbb {C}})$ ). Moreover, we assume that $W\in {\mathcal {W}}(\pi ,\psi )$ is the spherical Whittaker function normalized by $W(I_l)=1$ , $\xi _{\mathbf {{s}}}$ is the spherical Whittaker function associated with the normalized spherical section $f_{\mathbf {{s}}}\in {\mathrm {I}}(\mathbf {{s}},{\boldsymbol \tau })$ . Let $W_{\tau _i}\in {\mathcal {W}}(\tau _i,\psi ^{-1})$ , $i=1,2$ , be the spherical Whittaker function of $\tau _i$ normalized such that $W_{\tau _1}(I_m)=1$ and $W_{\tau _2}(I_n)=1$ . From the definition of $\xi _{\mathbf {{s}}}$ , we get

$$ \begin{align*}\xi_{\mathbf{ {s}}}(t)=|\det(a)|^{s_1+\frac{n-1}{2}}|\det(b)|^{-s_2-\frac{m-1}{2}}W_{\tau_1}(a)W_{\tau_2}(b)\end{align*} $$

for $t={\mathrm {diag}}(a,b)\in T_{m+n}(F)$ , with $a\in T_m(F), b\in T_n(F)$ . By Iwasawa decomposition ${\mathrm {GL}}_{m+n}(F)=N_{m+n}(F)T_{m+n}(F)K_{m+n}$ , where $K_{m+n}={\mathrm {GL}}_{m+n}({\mathcal {O}})$ , we have

$$ \begin{align*}\Psi(W,\xi_{\mathbf{{s}}}; j)&=\int_{T_{m+n}(F)}\int_{\overline U^{j,m,n}(F)}W(\overline u \gamma_{m,n}{\mathrm {diag}}(t,I_{l-m-n}))\xi_{\mathbf{{s}}}(t) \delta_{B_{m+n}}(t)^{-1}d\overline u dt\\ &=\int_{T_{m+n}(F)}\int_{\overline U^{m,n}(F)}W(\mathbf{{t}}_{m,n}(a,b)\overline u)\xi_{\mathbf{{s}}}(t) |\det(a)|^{-j} |\det(b)|^{k}\delta_{B_{m+n}}(t)^{-1}d\overline u dt\\ &=\int_{T_{m+n}(F)}\int_{\overline U^{m,n}(F)}W(\mathbf{{t}}_{m,n}(a,b)\overline u)W_{\tau_1}(a)W_{\tau_2}(b)\\ &\quad \cdot |\det(a)|^{s_1+\frac{n-1}{2}-j}|\det(b)|^{-s_2-\frac{m-1}{2}+k} \delta_{B_{m+n}}(t)^{-1}d\overline u dt \end{align*} $$

where $t={\mathrm {diag}}(a,b)$ with $a\in T_m(F), b\in T_n(F)$ and $\mathbf {{t}}_{m,n}(a,b)={\mathrm {diag}}(a,I_{l-m-n},b)$ . Here the term $|\det (a)|^{-j} |\det (b)|^{k} $ comes from a modulus character when we change variables on $\overline u$ and the term $\delta _{B_{m+n}}(t)^{-1}$ comes from the corresponding Haar measure when we use the Iwasawa decomposition. By Lemma 2.7, we have

(3.11) $$ \begin{align} \Psi(W,\xi_{\mathbf{{s}}}; j)=&\int_{T_{m+n}(F)}W(\mathbf{{t}}_{m,n}(a,b))W_{\tau_1}(a)W_{\tau_{2}}(b)\\ & \quad \cdot |\det(a)|^{s_1+\frac{n-1}{2}-j}|\det(b)|^{-s_2-\frac{m-1}{2}+k}\delta_{B_{m+n}}^{-1}\left(\begin{pmatrix} a&\\ &b\end{pmatrix} \right)dadb \nonumber \\ =& \int_{T_{m+n}(F)}W(\mathbf{{t}}_{m,n}(a,b^*))W_{\tau_1}(a)W_{\tau_{2}}(b^*) \nonumber \\ & \quad \cdot |\det(a)|^{s_1+\frac{n-1}{2}-j}|\det(b)|^{s_2+\frac{m-1}{2}-k}\delta_{B_{m+n}}^{-1}\left(\begin{pmatrix} a&\\ &b^*\end{pmatrix} \right)dadb \nonumber \end{align} $$

where $b^*=J_n ^t\!b^{-1}J_n^{-1}$ , with $ J_n=\begin {pmatrix} &&1\\ &\unicode{x22F0} &\\ 1&& \end {pmatrix}$ . Note that the function $b\mapsto W_{\tau _2}(b^*)$ is just the normalized Whittaker function of $\widetilde \tau _2$ , namely, $W_{\tau _2}(b^*)=W_{\widetilde \tau _2}(b).$ We use the following notations following [Reference Jacquet and ShalikaJS81a]. For an m-tuple $\mathbf {x}=(x_1,\dots ,x_m)$ , we write $\varpi ^{\mathbf {{x}}}=(\varpi ^{x_1},\dots ,\varpi ^{x_m})$ . Then (3.11) can be written as

(3.12) $$ \begin{align} \Psi(W,\xi_{\mathbf{{s}}};j)=&\sum_{\mathbf{{x}},{\mathbf{y}}}W(\varpi^{(\mathbf{{x}},0,{\mathbf{y}}^*)})W_{\tau_1}(\varpi^{\mathbf{ {x}}})W_{\widetilde\tau_2}(\varpi^{{\mathbf{y}}})\delta_{B_{m+n}}^{-1}(\varpi^{(\mathbf{{x}},{\mathbf{y}}^*)})\\ &\cdot |\det(\varpi^{\mathbf{{x}}})|^{s_1+\frac{n-1}{2}-j} |\det(\varpi^{{\mathbf{y}}})|^{s_2+\frac{m-1}{2}-k}. \nonumber \end{align} $$

Here $\mathbf {{x}}$ (resp. ${\mathbf {y}}$ ) runs over all m-tuples (resp. n-tuples) of all integers, ${\mathbf {y}}^*=(-y_n,\dots ,-y_1)$ for ${\mathbf {y}}=(y_1,\dots ,y_n)$ and $(\mathbf {{x}},0,{\mathbf {y}}^*) $ denotes the l-tuple $(x_1,\dots ,x_m,0,\dots ,0,-y_n,\dots ,-y_1)$ with $l-m-n$ zeros in the middle. Denote by $T^+(m)$ the m-tuples of integers $\mathbf { {x}}=(x_1,\dots ,x_m)$ such that $x_1\ge x_2\ge \dots \ge x_m\ge 0$ . Similarly, we define $T^+(n)$ . By the Shintani-Casselman-Shalika formula [Reference ShintaniShi76, Reference Casselman and ShalikaCS80], we have $W(\varpi ^{(\mathbf {{x}},0,{\mathbf {y}}^*)})=0$ unless $\mathbf {{x}}\in T^+(m)$ and ${\mathbf {y}}\in T^+(n)$ . If $\mathbf {{x}}\in T^+(m)$ and ${\mathbf {y}}\in T^+(n)$ , we have $W(\varpi ^{(\mathbf {{x}},0,{\mathbf {y}}^*)})=\delta _{B_{l}}^{1/2}(\varpi ^{(\mathbf { {x}},0,{\mathbf {y}}^*)})S_{(\mathbf {{x}},0,{\mathbf {y}}^*)}(\alpha )$ . Here $S_{(\mathbf {{x}},0,{\mathbf {y}}^*)}$ denotes the Schur polynomial associated with $(\mathbf {{x}},0,{\mathbf {y}}^*)$ (see [Reference Fulton and HarrisFH91, Appendix A]), or more explicitly,

$$ \begin{align*}S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha)=\prod_{1\le i<j\le l}(\alpha_i-\alpha_j)^{-1} \cdot \det\begin{pmatrix} \alpha_1^{x_1+l-1} & \dots & \alpha_{l}^{x_1+l-1}\\ \vdots&&\vdots\\ \alpha_1^{-y_1}&\dots &\alpha_{l}^{-y_1} \end{pmatrix}.\end{align*} $$

Moreover, for $\mathbf {{x}}\in T^+(m), {\mathbf {y}}\in T^+(n)$ , we have

$$ \begin{align*}W_{\tau_1}(\varpi^{\mathbf{{x}}})=\delta_{B_{m}}^{1/2}(\varpi^{\mathbf{{x}}})S_{\mathbf{{x}}}(\beta^1), \quad W_{\widetilde \tau_2}(\varpi^{{\mathbf{y}}})=\delta_{B_n}^{1/2}(\varpi^{{\mathbf{y}}})S_{{\mathbf{y}}}(\beta^2).\end{align*} $$

We can check that

$$ \begin{align*} \delta_{B_m}(\varpi^{\mathbf{{x}}})&=|\varpi^{x_1}|^{m-1}|\varpi^{x_2}|^{m-3}\dots |\varpi^{x_m}|^{-m+1},\\ \delta_{B_n}(\varpi^{{\mathbf{y}}})&=|\varpi^{y_1}|^{n-1}|\varpi^{y_2}|^{n-3}\dots |\varpi^{y_n}|^{-n+1},\\ \delta_{B_{m+n}}(\varpi^{(\mathbf{{x}},\mathbf{ {y}}^*)})&=|\varpi^{x_1}|^{m-1}|\varpi^{x_2}|^{m-3}\dots|\varpi^{x_m}|^{-m+1}|\varpi^{y_n}|^{-n+1}\\ &\quad \dots |\varpi^{y_1}|^{n-1}|\det(\varpi^{\mathbf{{x}}})|^n|\det(\varpi^{{\mathbf{y}}})|^m,\\ \delta_{B_{l}}(\varpi^{(\mathbf{{x}},0,{\mathbf{y}}^*)})&=|\varpi^{x_1}|^{m-1}\dots |\varpi^{x_m}|^{1-m}|\varpi^{y_n}|^{1-n}\dots |\varpi^{y_1}|^{n-1}|\det(\varpi^{\mathbf{{x}}})|^{l-m}|\det(\varpi^{{\mathbf{y}}})|^{l-n}. \end{align*} $$

Combining the above formulas, (3.12) becomes

(3.13) $$ \begin{align} \Psi(W,\xi_{\mathbf{{s}}};j)&=\sum_{\substack{\mathbf{{x}}\in T^+(m)\\ {\mathbf{y}}\in T^+(n)}}S_{(\mathbf{{x}},0,\mathbf{ {y}}^*)}(\alpha)S_{\mathbf{{x}}}(\beta^1)S_{{\mathbf{y}}}(\beta^2)|\det(\varpi^{\mathbf{{x}}})|^{s_1+\frac{k-j}{2}}|\det(\varpi^{\mathbf{ {y}}})|^{s_2+\frac{j-k}{2}}. \end{align} $$

Proposition 3.10. We keep the notations as above. Then

$$ \begin{align*}\Psi(W,\xi_{\mathbf{{s}}}; j)=\frac{L(s_1+\frac{k-j}{2},\pi\times \tau_1)L(s_2-\frac{k-j}{2},\widetilde \pi\times \widetilde\tau_2)}{L(s_1+s_2,\tau_1\times \widetilde \tau_2)}.\end{align*} $$

Recall that $k=l-m-n-1-j$ and thus $k-j=l-m-n-1-2j.$

If $n=0$ , the above formula is the unramified calculation of the Jacquet–Piatetski-Shapiro–Shalika integral, see [Reference Jacquet and ShalikaJS81a, Proposition 2.4] and also [Reference CogdellCog04, Reference CogdellCog07]. If $l=2r+1, m=n$ and $j=r-m=k$ , the above unramified calculation is done in [Reference Gelbart and Piatetski-ShapiroGPS84] (when $r=1$ ), in [Reference TamirTam91] (for general r when $m=n=r$ ) with slightly different normalization, and in [Reference Ben-Artzi and SoudryBAS09] (when $m=n<r$ ), where this was the unramified calculation of L-functions for ${\mathrm {U}}_{2r+1,E/F}\times {\mathrm {Res}}_{E/F}({\mathrm {GL}}_r)$ at split places for a quadratic extension $E/F$ .

Proof. Without loss of generality, we assume that $m\ge n$ . Write $T_1=q^{-(s_1+\frac {k-j}{2})}, T_2=q^{-(s_2+\frac {j-k}{2})}$ . For an m-tuple $\mathbf {{x}}=(x_1,\dots ,x_m)$ , denote $|\mathbf {{x}}|=\sum _{i=1}^m x_i$ . An m-tuple $\mathbf {{x}}\in T^+(m)$ can be identified with a partition of $|\mathbf {{x}}|$ and can be represented by a Young diagram; see [Reference Fulton and HarrisFH91, §4] for example. We can then write (3.13) as

(3.14) $$ \begin{align} \Psi(W,\xi_{\mathbf{{s}}};j)=\sum_{\substack{\mathbf{{x}}\in T^+(m)\\ {\mathbf{y}}\in T^+(n)}}S_{(\mathbf{{x}},0,\mathbf{ {y}}^*)}(\alpha)S_{\mathbf{{x}}}(\beta^1)S_{{\mathbf{y}}}(\beta^2)T_1^{|\mathbf{{x}}|}T_2^{|{\mathbf{y}}|}. \end{align} $$

On the other hand, we have

$$ \begin{align*}L(s_1+s_2,\tau_1\times \widetilde \tau_2)=\det(I-\beta^1\otimes \beta^2 T_1T_2)^{-1}=\sum_{e\ge 0}{\mathrm{Tr}}({\mathrm{Sym}}^e(\beta^1\otimes \beta^2))(T_1T_2)^e.\end{align*} $$

Thus we get that

(3.15) $$ \begin{align} \begin{aligned} L(s_1+s_2,\tau_1\times \widetilde \tau_2)\Psi(W,\xi_{\mathbf{{s}}};j)=\sum_{\substack{\mathbf{{x}}\in T^+(m), {\mathbf{y}}\in T^+(n), e\ge 0} } &S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha)S_{\mathbf{{x}}}(\beta^1)S_{{\mathbf{y}}}(\beta^2)\\ &\cdot {\mathrm{Tr}}({\mathrm{Sym}}^e(\beta^1\otimes \beta^2))T_1^{|\mathbf{{x}}|+e}T_2^{|{\mathbf{y}}|+e}. \end{aligned} \end{align} $$

Since

$$ \begin{align*}L(s_1+\frac{k-j}{2},\pi\times \tau_1)=\sum_{c\ge 0}{\mathrm{Tr}}({\mathrm{Sym}}^c(\alpha\otimes \beta^1))T_1^c,\end{align*} $$

and

$$ \begin{align*}L(s_2+\frac{j-k}{2},\widetilde \pi\times \widetilde\tau_2)=\sum_{d\ge 0}{\mathrm{Tr}}({\mathrm{Sym}}^d(\widetilde\alpha\otimes \beta^2))T_2^d,\end{align*} $$

where $\widetilde \alpha ={\mathrm {diag}}(a_1^{-1},\dots ,a_l^{-1})$ is the Satake parameter for $\widetilde \pi ,$ we get that

(3.16) $$ \begin{align} & L(s_1+\frac{k-j}{2},\pi\times \tau_1)L(s_2+\frac{j-k}{2},\widetilde \pi\times \widetilde\tau_2) \nonumber \\ &\quad =\sum_{c\ge 0, d\ge 0}{\mathrm{Tr}}({\mathrm{Sym}}^c(\alpha\otimes \beta^1)){\mathrm{Tr}}({\mathrm{Sym}}^d(\widetilde\alpha\otimes \beta^2))T_1^c T_2^d. \end{align} $$

Comparing (3.15) and (3.16), in order to prove Proposition 3.10, it suffices to show

(3.17) $$ \begin{align} \begin{aligned} {\mathrm{Tr}}({\mathrm{Sym}}^c(\alpha\otimes \beta^1)){\mathrm{Tr}}({\mathrm{Sym}}^d(\widetilde\alpha\otimes \beta^2))=\sum_{e\ge 0}\sum_{\substack{\mathbf{{x}}\in T^+(m),{\mathbf{y}}\in T^+(n), e\ge 0\\ |\mathbf{{x}}|=c-e, |{\mathbf{y}}|=d-e}}&S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha)S_{\mathbf{{x}}}(\beta^1)S_{{\mathbf{y}}}(\beta^2)\\ &\cdot {\mathrm{Tr}}({\mathrm{Sym}}^e(\beta^1\otimes \beta^2)). \end{aligned} \end{align} $$

By [Reference Jacquet and ShalikaJS81a, Proposition 2.4], we have

$$ \begin{align*}{\mathrm{Tr}}({\mathrm{Sym}}^e(\beta^1\otimes \beta^2))=\sum_{{\mathbf{z}}\in T^+(n), |{\mathbf{z}}|=e}S_{(\mathbf{ {z}},0_{m-n})}(\beta^1)S_{{\mathbf{z}}}(\beta^2).\end{align*} $$

Here ${\mathbf {z}}=(z_1,\dots ,z_n)$ can be identified with a partition of $e=|{\mathbf {z}}|$ with at most n parts (since $m\ge n$ by our assumption) and $S_{{\mathbf {z}}}$ (resp. $S_{({\mathbf {z}},0_{m-n})}$ ) is the Schur polynomial defined by ${\mathbf {z}}$ with n (resp. m) variables. Similarly,

$$ \begin{align*}{\mathrm{Tr}}({\mathrm{Sym}}^c(\alpha\otimes \beta^1))&=\sum_{\mathbf{u}\in T^+(m), |{\mathbf{u}}|=c}S_{(\mathbf{ {u}},0_{l-m})}(\alpha)S_{{\mathbf{u}}}(\beta^1),\\ {\mathrm{Tr}}({\mathrm{Sym}}^d(\widetilde\alpha\otimes \beta^2))&=\sum_{\mathbf{v}\in T^+(n), |{\mathbf{v}}|=d}S_{(\mathbf{ {v}},0_{l-n})}(\widetilde \alpha)S_{{\mathbf{u}}}(\beta^2). \end{align*} $$

A simple matrix calculation shows that

$$ \begin{align*}S_{({\mathbf{v}},0_{l-n})}(\widetilde \alpha)=S_{(0_{l-n},{\mathbf{v}}^*)}(\alpha).\end{align*} $$

See also [Reference Fulton and HarrisFH91, Exercise 15.50] for a representation theoretic explanation of this formula. Thus the left hand side of (3.17) becomes

$$ \begin{align*} LHS=\sum_{{\mathbf{u}}\in T^+(m), |{\mathbf{u}}|=c}\sum_{{\mathbf{v}}\in T^+(n), |{\mathbf{v}}|=d}S_{({\mathbf{u}},0_{l-m})}(\alpha) S_{(0_{l-n},{\mathbf{v}}^*)}(\alpha)S_{{\mathbf{u}}}(\beta^1)S_{{\mathbf{v}}}(\beta^2), \end{align*} $$

while the right side of (3.17) becomes

$$ \begin{align*} RHS=\sum_{{\mathbf{z}}\in T^+(n)} \sum_{\substack{\mathbf{{x}}\in T^+(m),{\mathbf{y}}\in T^+(n), e\ge 0\\ |{\mathbf{x}}|=c-|{\mathbf{z}}|, |{\mathbf{y}}|=d-|{\mathbf{z}}|}}&S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha)S_{\mathbf{ {x}}}(\beta^1)S_{({\mathbf{z}},0)}(\beta^1)S_{{\mathbf{y}}}(\beta^2)S_{{\mathbf{z}}}(\beta^2) \end{align*} $$

By the Littlewood-Richardson rule, see [Reference Fulton and HarrisFH91, (A.8)] or [Reference MacdonaldMac95, §I.9], we have

$$ \begin{align*} S_{\mathbf{{x}}}(\beta^1)S_{{\mathbf{z}}}(\beta^1)=&\sum_{{\mathbf{u}}\in T^+(m), |{\mathbf{u}}|=c}c_{\mathbf{{x}}, {\mathbf{z}}}^{\mathbf{ {u}}} S_{{\mathbf{u}}}(\beta^1),\\ S_{{\mathbf{y}}}(\beta^2)S_{{\mathbf{z}}}(\beta^2)=&\sum_{{\mathbf{v}}\in T^+(n), |{\mathbf{v}}|=d} c_{{\mathbf{y}}, {\mathbf{z}}}^{\mathbf{ {v}}} S_{{\mathbf{v}}}(\beta^2), \end{align*} $$

where in the first equation, $({\mathbf {z}},0_{m-n})$ is identified with $|{\mathbf {z}}|$ as a partition of $e=|{\mathbf {z}}|$ with at most n parts, and $c_{\mathbf {{x}}, {\mathbf {z}}}^{{\mathbf {u}}}, c_{{\mathbf {y}}, {\mathbf {z}}}^{{\mathbf {u}}}$ are the Littlewood-Richardson coefficients as defined in [Reference Fulton and HarrisFH91, page 454] or [Reference MacdonaldMac95, §I.9.2]. Thus

$$ \begin{align*} RHS=\sum_{{\mathbf{u}}\in T^+(m), |{\mathbf{u}}|=c}\sum_{{\mathbf{v}}\in T^+(n),|{\mathbf{v}}|=d}\sum_{\substack{\mathbf{{x}}\in T^+(m), {\mathbf{y}},{\mathbf{z}}\in T^+(n)\\ |\mathbf{{x}}|+|{\mathbf{z}}|=c, |{\mathbf{y}}|+|{\mathbf{z}}|=d}}c_{\mathbf{{x}}, \mathbf{ {z}}}^ {\mathbf{u}} c_{{\mathbf{y}}, {\mathbf{z}}}^ {\mathbf{v}} S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha) S_{\mathbf{ {u}}}(\beta^1)S_{{\mathbf{v}}}(\beta^2). \end{align*} $$

Thus in order to prove (3.17) and hence Proposition 3.10, it suffices to prove that for any ${\mathbf {u}}\in T^+(m), {\mathbf {v}}\in T^+(n)$ , one has

(3.18) $$ \begin{align} S_{({\mathbf{u}},0_{l-m})}(\alpha) S_{(0_{l-n},{\mathbf{v}}^*)}(\alpha)=\sum_{\substack{\mathbf{{x}}\in T^+(m), {\mathbf{y}},\mathbf{ {z}}\in T^+(n)\\ |\mathbf{{x}}|+|{\mathbf{z}}|=c, |{\mathbf{y}}|+|{\mathbf{z}}|=d}}c_{\mathbf{{x}}, {\mathbf{z}}}^ {\mathbf{u}} c_{\mathbf{ {y}},{\mathbf{z}} }^{{\mathbf{v}}} S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}(\alpha). \end{align} $$

For ${\mathbf {v}}=(v_1,\dots ,v_n)\in T^+(n)$ , we write ${\widetilde {\mathbf {v}}}=(v_1,\dots ,v_1,v_1-v_n,\dots ,v_1-v_2,0)\in T^+(l)$ . Then $S_{(0_{l-n},{\mathbf {v}}^*)}(\alpha )=S_{{\mathbf {v}}}(\alpha ) D_{-v_1}(\alpha )$ , where $D_{-v_1}(\alpha )=\det ^{-v_1}(\alpha )$ following the notation of [Reference Fulton and HarrisFH91, §15.5]. Thus using the Littlewood-Richardson rule again, we have

$$ \begin{align*} S_{({\mathbf{u}},0_{l-m})}(\alpha) S_{(0_{l-n},{\mathbf{v}}^*)}(\alpha)&=D_{-v_1}(\alpha)\sum_{\lambda\in T^+(l), |\lambda|=|\widetilde v|+|{\mathbf{u}}|}c_{{\widetilde{\mathbf{v}}}, {\mathbf{u}}}^{ \lambda}S_\lambda(\alpha). \end{align*} $$

Write $\lambda =(\lambda _1,\dots ,\lambda _l)$ . By the definition of Littlewood-Richardson coefficients, if $c_{{\widetilde {\mathbf {v}}}, {\mathbf {u}}}^\lambda \ne 0$ , we must have $\lambda _{m+1}=\dots =\lambda _{l-n-1}=v_1$ , which means that $S_{\lambda }\cdot D_{-v_1}=S_{(\lambda _1-v_1,\dots ,\lambda _l-v_1)}$ must be of the form $S_{(\mathbf {{x}},0_{l-m-n},{\mathbf {y}}^*)}$ for $\mathbf {{x}}\in T^+(m)$ and ${\mathbf {y}}\in T^+(n)$ . Thus we get

$$ \begin{align*} S_{({\mathbf{u}},0_{l-m})}(\alpha) S_{(0_{l-n},{\mathbf{v}}^*)}(\alpha)=\sum_{\mathbf{{x}}\in T^+(m), {\mathbf{y}}\in T^+(n) } c_{{\widetilde{\mathbf{v}}}, {\mathbf{u}}}^{ \lambda}S_{(\mathbf{{x}},0,{\mathbf{y}}^*)}, \end{align*} $$

where $\lambda =(\lambda _1,\dots ,\lambda _l)=(\mathbf {{x}},0,{\mathbf {y}}^*)+(v_1,\dots ,v_1).$ Note that $|{\mathbf {u}}|-|\mathbf { {v}}|=|\mathbf {{x}}|-|{\mathbf {y}}|$ . Thus in order to prove (3.18), it suffices to show that for any fixed ${\mathbf {u}},\mathbf {{x}} \in T^+(m)$ and ${\mathbf {v}}, {\mathbf {y}}\in T^+(n)$ with $|{\mathbf {u}}|-|\mathbf {{x}}|=|{\mathbf {v}}|-|{\mathbf {y}}|$ ,

(3.19) $$ \begin{align} c_{{\widetilde{\mathbf{v}}},{\mathbf{u}}}^{\lambda}=\sum_{{\mathbf{z}}\in T^+(n)}c_{\mathbf{{x}}, {\mathbf{z}}}^{ {\mathbf{u}}} c_{\mathbf{ {y}}, {\mathbf{z}}}^{ {\mathbf{v}}}, \end{align} $$

where $ \lambda =(\lambda _1,\dots ,\lambda _l)=(\mathbf {{x}},0,{\mathbf {y}}^*)+(v_1,\dots ,v_1).$ The formula (3.19) was proved by Professor T. Tao in a MathOverflow answer [Reference TaoTao] using the hive model for Littlewood-Richardson coefficients introduced in [Reference Knutson and TaoKT99]. A proof of (3.19) based on Tao’s MathOverflow answer [Reference TaoTao] will be reproduced in §3.4 after we introduce some necessary notations and tools.

Remark 3.11. Here we give an example of (3.18). We take $l=4,m=2,n=1$ and ${\mathbf {u}}=(2,1),{\mathbf {v}}=(2)$ . One can check that there are 3 choices of ${\mathbf {z}}$ , which are ${\mathbf {z}}=(0),{\mathbf {z}}=(1), {\mathbf {z}}=(2)$ , and correspondingly, there are 3 choices of ${\mathbf {y}}$ given by ${\mathbf {y}}=(2),{\mathbf {y}}=(1),{\mathbf {y}}=(0)$ . When ${\mathbf {z}}=(0)$ , we must have $\mathbf {{x}}=(2,1)$ and when ${\mathbf {z}}=(2)$ , we must have $\mathbf {{x}}=(1)=(1,0)$ . But when ${\mathbf {z}}=(1)$ , there are two choices of $\mathbf {{x}}$ , which are $\mathbf {{x}}=(1,1)$ or $\mathbf {{x}}=(2)=(2,0)$ . One can check that in each case, $c_{\mathbf { {x}}, {\mathbf {z}}}^{ {\mathbf {u}}}c_{{\mathbf {y}}, {\mathbf {z}}}^{ {\mathbf {v}}}=1$ . Thus (3.18) becomes

$$ \begin{align*}S_{(2,1,0,0)}\cdot S_{(0,0,0,-2)}=S_{(2,1,0,-2)}+S_{(1,0,0,0)}+S_{(1,1,0,-1)}+S_{(2,0,0,-1)},\end{align*} $$

which could be checked directly using the Littlewood-Richardson rule by noting that $S_{(0,0,0,-2)}=S_{(2,2,2,0)}\cdot D_{-2}$ , where $D_{-2}=\det ^{-2}$ .

3.4 Proof of Tao’s formula (3.19)

An integral n-hive is an array of integers $a_{ij}$ for $0\le i, j, i+j\le n$ placed in the vertices of triangles of the following shape

Figure 1

hive.

which satisfies all of the following rhombus inequalities: for each rhombus of the following types

the sum of the two integers at the obtuse vertices must be greater than or equal to the sum of the two integers at the acute vertices.

Theorem 3.12 (Knutson-Tao, [Reference Knutson and TaoKT99]).

Let $\mathbf {{x}}=(x_1,\dots ,x_n), \mathbf { {y}}=(y_1,\dots ,y_n), {\mathbf {z}}=(z_1,\dots ,z_n)$ be partitions with $|{\mathbf {z}}|=|\mathbf {{x}}|+|{\mathbf {y}}|$ , then $c_{\mathbf { {x}},{\mathbf {y}}}^{{\mathbf {z}}}$ is the number of n-hives with boundary labels

Here the arrow and the number $x_i$ (resp. $y_j, z_k$ ) on the arrow indicate that the numbers increase by $x_i$ (resp. $y_j,z_k$ ) along the direction indicated by the arrow. One can normalize the above n-hive by assigning any integer to any fixed vertex.

We note that different normalization will give the same number of hives. The above theorem is proved in [Reference Knutson and TaoKT99]. See also the appendix of [Reference BuchBuc00] for a different proof given by W. Fulton.

Remark 3.13. We give a simple example which also appeared in [Reference BuchBuc00]. We have $c_{(2,1),(2,1)}^{(3,2,1)}=2$ , which can be computed in the following way. There are exactly two $3$ -hives with boundary conditions given below,

$$ \begin{align*}\begin{matrix} &&&3&&&\\ &&&&& \\ &&3&&5&&\\ &&&&& \\ &2&&x&&6&\\ &&&&&& \\0&&3&&5&&6 \end{matrix},\end{align*} $$

which are given by $x=4,5$ .

We temporarily call the following object an anti-n-hive: an array of integers placed in the vertices of triangles of the shape as in Figure 1 which satisfies the ‘reverse’ rhombus inequalities: for each rhombus below

the sum of the two integers at the obtuse vertices must be less than or equal to the sum of the two integers at the acute vertices.

For any n-hive, if we switch the sign of the number at each vertex, we will get an anti-n-hive. Note that this process will change the boundary conditions, which gives us the following direct corollary.

Corollary 3.14. Let $\mathbf {{x}}=(x_1,\dots ,x_n), {\mathbf {y}}=(y_1,\dots ,y_n), {\mathbf {z}}=(z_1,\dots ,z_n)$ be partitions with $|{\mathbf {z}}|=|\mathbf { {x}}|+|{\mathbf {y}}|$ , then $c_{\mathbf {{x}},{\mathbf {y}}}^{{\mathbf {z}}}$ is the number of anti-n-hives with boundary labels

Here the arrow and the number $x_i$ (resp. $y_j, z_k$ ) on the arrow indicate that the numbers increase by $x_i$ (resp. $y_j,z_k$ ) along the direction indicated by the arrow. One can normalize the above n-hive by assigning any integer to any fixed vertex.

Now we can prove Tao’s formula (3.19), which we restate below.

Proposition 3.15. Let $l,m,n$ be non-negative integers with $l\ge m+n+1$ and $m\ge n$ . Given $\mathbf {{x}}, {\mathbf {u}}\in T^+(m), {\mathbf {y}}, \mathbf { {v}}\in T^+(n)$ with $|{\mathbf {u}}|-|\mathbf {{x}}|=|{\mathbf {v}}|-|{\mathbf {y}}|\ge 0 $ , then

$$ \begin{align*}c_{{\widetilde{\mathbf{v}}}, {\mathbf{u}}}^{\lambda}=\sum_{{\mathbf{z}}\in T^+(n)}c_{\mathbf{{x}},{\mathbf{z}}}^{\mathbf{ {u}}} c_{{\mathbf{y}},{\mathbf{z}}}^{{\mathbf{v}}}.\end{align*} $$

Here ${\mathbf {u}}=(u_1,\dots ,u_m), \mathbf {{x}}=(x_1,\dots ,x_m), {\mathbf {y}}=(y_1,\dots ,y_n), {\mathbf {v}}=(v_1,\dots ,v_n)$ , $\mathbf { {y}}^*=(-y_n,\dots ,-y_2,-y_1)$ , ${\widetilde {\mathbf {v}}}=(0_{l-n},\mathbf { {v}}^*)+(v_1,\dots ,v_1)=(v_1,\dots ,v_1,v_1-v_n,\dots ,v_1-v_2,0)\in T^+(l)$ , and $\lambda =(\mathbf {{x}},0_{l-m-n},\mathbf { {y}}^*)+(v_1,\dots ,v_1)\in T^+(l)$ . Moreover, ${\mathbf {u}}$ in $c_{{\widetilde {\mathbf {v}}}, {\mathbf {u}}}^\lambda $ is viewed as an element in $T^+(l)$ in the obvious way, namely, ${\mathbf {u}}=({\mathbf {u}},0_{l-n})$ .

Proof. By Theorem 3.12 and Corollary 3.14, one can see that $c_{{\widetilde {\mathbf {v}}}, {\mathbf {u}}}^{\lambda }$ is the number of anti-l-hives with boundary conditions indicated below,

where $v_1$ in the left side boundary and bottom boundary means $(v_1,\dots ,v_1)\in T^+(l)$ . Here the two interior line segments are not important. For each hive above, we assume that its vertex integers are given by $(a_{ij})_{0\le i, j, i+j\le l}$ placed as in Figure 1. Then $(a_{ij}-(i+j)v_1)_{0\le i, j, i+j\le l}$ is also an anti-l-hive which has the boundary conditions as indicated in the following Figure 2. We also normalized the anti-l-hive so that the top vertex has value $0$ .

Figure 2

boundary conditions for anti-hives which represents $c_{{\widetilde {\mathbf {v}}}, {\mathbf {u}}}^\lambda $

Thus $ c_{{\widetilde {\mathbf {v}}}, {\mathbf {u}}}^\lambda $ is the number of anti-l-hives with boundary conditions as in Figure 2. Using the reverse rhombus inequality, we can check that an anti-l-hive as above must vanish completely in the quadrilateral $ABEF$ (including each side) in Figure 3. Moreover, inside the trapezoid $BCDE$ , the values of the hive on each horizontal line are the same. In particular, this means that there exists a ${\mathbf {z}}\in T^+(n)$ such that the boundary conditions on $CB$ and $DE$ are both given by ${\mathbf {z}}^*$ .

Figure 3

complete boundary conditions for anti-hives

Figure 3 Long description

A large triangle has vertices labeled 0 at the top, G at the bottom-left, and H at the bottom-right.

Along the perimeter:

- The left edge from G to 0 is labeled with a vector v star pointing upward and a segment label 0 sub l-n.

- The right edge from 0 to H is labeled with a segment label 0 sub l-m and a vector u pointing downward.

- The bottom edge from G to H is divided into segments G-C, C-D, and D-H. A vector y star points from G toward C. A segment label 0 sub l-m-n is below C-D. A vector x points from D toward H.

Internal structure:

- A horizontal line segment B-E is parallel to the base G-H. Point B is on the left edge and point E is on an internal diagonal.

- Point F lies on the right edge 0-H. A vector labeled 0 sub m-n points from E toward F.

- Two parallel upward-pointing vectors labeled z star connect the base to the internal horizontal line: one from C to B and one from D to E.

- The interior region near the top vertex 0 is labeled A.

Thus such a hive is uniquely determined by its values in the anti-hives $BGC$ and $FDH$ , with the indicated boundary conditions as in Figure 3. Conversely, given anti-hives $BGC$ and $FDH$ with boundary conditions as in Figure 3, we get an anti-hive with the boundary condition as in Figure 2 using a reverse process. Finally, note that the number of anti-hives $BGC$ is $c_{{\mathbf {y}},{\mathbf {z}}}^{{\mathbf {v}}}$ and the number of anti-hives $FDH$ is $c_{\mathbf { {x}},{\mathbf {z}}}^{{\mathbf {u}}}$ . Thus we get

$$ \begin{align*}c_{{\widetilde{\mathbf{v}}},{\mathbf{u}}}^{\lambda}=\sum_{{\mathbf{z}}\in T^+(n)} c_{\mathbf{{x}},{\mathbf{z}}}^{\mathbf{{u}}} c_{{\mathbf{y}},{\mathbf{z}}}^{\mathbf{v}}.\end{align*} $$

This concludes the proof.

4.1 On the gamma factors for ${\mathrm {GL}}_{2r}\times ({\mathrm {GL}}_r,{\mathrm {GL}}_r)$

Recall that if $m+n\le l-1$ , for a generic representation $\pi $ of ${\mathrm {GL}}_l(F)$ , $\tau _1$ (resp. $\tau _2$ ) of ${\mathrm {GL}}_m(F)$ (resp. ${\mathrm {GL}}_n(F)$ ), our local gamma factor $\Gamma (\mathbf {{s}},\pi \times (\tau _1,\tau _2),\psi )$ is defined by the local functional equation

$$ \begin{align*}\Psi(W,M_{w_{m,n}}(\xi_{\mathbf{{s}}});0)=\Gamma(\mathbf{{s}},\pi\times (\tau_1,\tau_2),\psi;0)\Psi(W,\xi_{\mathbf{ {s}}};0),\end{align*} $$

for all $W\in {\mathcal {W}}(\pi ,\psi )$ and $\xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},(\tau _1,\tau _2),\psi ^{-1}).$ See Proposition 3.6. From now on, we will drop the index $j=0$ in the local gamma factor and simply write $\Gamma (\mathbf {{s}},\pi \times (\tau _1,\tau _2),\psi ):=\Gamma (\mathbf {{s}},\pi \times (\tau _1,\tau _2),\psi ;0)$ . For $W\in {\mathcal {W}}(\pi ,\psi )$ , $\rho (\gamma _{m,n}^{-1})W$ is also an element in ${\mathcal {W}}(\pi ,\psi )$ . Thus, we have

(4.1) $$ \begin{align} \Psi(\rho(\gamma_{m,n}^{-1})W,M_{w_{m,n}}(\xi_{\mathbf{{s}}});0)=\Gamma(\mathbf{{s}},\pi\times (\tau_1,\tau_2),\psi)\Psi(\rho(\gamma_{m,n}^{-1})W,\xi_{\mathbf{{s}}};0), \end{align} $$

for all $\xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},(\tau _1,\tau _2),\psi ^{-1})$ . Here $\rho $ denotes the right translation, and $\gamma _{m,n}$ is the element in ${\mathrm {GL}}_l$ as defined after (3.1). The local functional equation (4.1) is the one we will use to prove our local converse theorem.

As explained in Remark 4.2, we also need the local gamma factors for $\Gamma (\mathbf {{s}}, \pi \times (\tau _1,\tau _2),\psi )$ when $l=2r$ and $m=n=r$ , which are not covered in our previous sections. This local gamma factor has been defined in [Reference Ben-Artzi and SoudryBAS09] and studied in [Reference MorimotoMor25]. We recall the definition now.

We first endowed $F^{2r}\oplus F^{2r}$ with a symplectic structure $\langle {~,~} \rangle $ defined by

$$ \begin{align*}\langle {(u_1,u_2),(v_1,v_2)} \rangle=2(u_1J_{2r}v_2^t-v_2J_{2r}u_2^t),\end{align*} $$

where $u_i, v_i\in F^{2r}$ are viewed as row vectors. For a nontrivial additive character $\psi $ of F and for a character $\mu $ of $F^\times $ , we can consider the Weil representation $\omega _{\psi ^{-1},\mu , \mu ^{-1}}$ of ${\mathrm {GL}}_{2r}(F)$ ; see [Reference MorimotoMor25, §2.2]. Note that we used a slightly different normalization. The Weil representation $\omega _{\psi ^{-1},\mu , \mu ^{-1}}$ can be realized on the space ${\mathcal {S}}(F^{r}\times F^{r})$ , the Bruhat-Schwartz functions on $F^{2r}$ . This is the Schrödinger model of the Weil representation. For example, we have the well-known formula

$$ \begin{align*}\left(\omega_{\psi^{-1}.\mu, \mu^{-1}}\left(\begin{pmatrix} I_r &X\\ &I_r \end{pmatrix} \right) \phi \right)(x,y)=\psi(x XJ_r y^t)\phi(x,y), X\in {\mathrm {Mat}}_{r\times r}(F).\end{align*} $$

In the following, we assume that $\mu $ is understood and omit it from the notation.

Now let $\pi $ be an irreducible generic representation of ${\mathrm {GL}}_{2r}(F)$ , $(\tau _1,\tau _2)$ be a pair of irreducible generic representations of ${\mathrm {GL}}_r(F)$ and $\mathbf {{s}}=(s_1,s_2)$ be a pair of complex numbers. For $W\in {\mathcal {W}}(\pi ,\psi ), \xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}},(\tau _1,\tau _2,),\psi ^{-1})$ , and $\phi \in {\mathcal {S}}(F^{2r})$ , we consider the local zeta integral

$$ \begin{align*}\Psi(W,\xi_{\mathbf{{s}}},\phi)=\int_{N_{2r}(F)\backslash {\mathrm{GL}}_{2r}(F)}W(g)\xi_{\mathbf{{s}}}(g) (\omega_{\psi^{-1}}(g)\phi)(e_r,e_r)dg,\end{align*} $$

where $e_r\in F^r$ is the vector $(0,0,\dots ,0,1)$ . There exists a meromorphic function $\Gamma (\mathbf {{s}}, \pi \times (\tau _1,\tau _2),\mu ,\psi )$ such that

(4.2) $$ \begin{align}\Psi(W, M_{w_{r,r}}\xi_{\mathbf{{s}}},\phi)=\Gamma(\mathbf{{s}}, \pi\times(\tau_1,\tau_2),\mu,\psi)\Psi(W,\xi_{\mathbf{{s}}},\phi)\end{align} $$

for any $W\in {\mathcal {W}}(\pi ,\psi ), \xi _{\mathbf {{s}}}\in {\mathcal {W}}(\mathbf {{s}}, (\tau _1,\tau _2),\psi ^{-1})$ and $\phi \in {\mathcal {S}}(F^r\times F^r).$ Note that in [Reference Ben-Artzi and SoudryBAS09] and [Reference MorimotoMor25], there is only a single complex variable involved in the local zeta integral and local gamma factor. Here we still use the two variable case.

4.2 Howe vectors

Our strategy for the proof of Theorem 4.1 is along the lines of that given in [Reference ZhangZha18] and [Reference ZhangZha19]. One basic tool for us is the partial Bessel functions associated with Howe vectors, as developed in [Reference BaruchBar95]. Here we recall the basic construction. Let $\psi $ be a fixed unramified additive character of F and we also view $\psi $ as a character of the maximal unipotent subgroup $N_l\subset {\mathrm {GL}}_l(F)$ in the usual way. For an integer $i\ge 0$ , we consider the open compact subgroup $K_{{\mathrm {GL}}_l}^i:=I_l+{\mathrm {Mat}}_{l\times l}({\mathfrak {p}}^i)$ of ${\mathrm {GL}}_l(F)$ . Consider the character $\theta _i$ of $K_{{\mathrm {GL}}_l}^i$ defined by

$$ \begin{align*}\theta_i(k)=\psi(\varpi^{-2i}(\sum_{s=1}^{l-1}k_{s,s+1})), \quad k=(k_{st})_{1\le s,t\le l}\in K_{{\mathrm{GL}}_l}^i.\end{align*} $$

One can check that $\theta _i$ is indeed a character of $K_{{\mathrm {GL}}_l}^i$ . Consider the element

$$ \begin{align*}d_i={\mathrm {diag}}(\varpi^{-i(l-1)},\varpi^{-i(l-3)},\dots,\varpi^{i(l-3)},\varpi^{i(l-1)}),\end{align*} $$

and $H_l^i=d_iK^id_i^{-1}$ , which is still an open compact subgroup of ${\mathrm {GL}}_l(F)$ . One sees that $H_l^i$ has the form

$$ \begin{align*}H_l^i=\begin{pmatrix} 1+{\mathfrak{p}}^i& {\mathfrak{p}}^{-i} &{\mathfrak{p}}^{-3i} &\dots\\ {\mathfrak{p}}^{3i} &1+{\mathfrak{p}}^i &{\mathfrak{p}}^{-i} &\dots\\ {\mathfrak{p}}^{5i} &{\mathfrak{p}}^{3i} & 1+{\mathfrak{p}}^i&\dots\\ \dots&\dots&\dots&\dots \end{pmatrix}.\end{align*} $$

We consider the character $\psi _i$ of $H_l^i$ defined by

$$ \begin{align*}\psi_i(h):=\theta_i(d_i^{-1}hd_i), \quad h\in H_l^i.\end{align*} $$

For a subgroup $U\subset {\mathrm {GL}}_l(F)$ , we denote $U^i:=U\cap H_l^i$ . For example, $N_l^i$ denotes $N_l\cap H_l^i$ . We also usually drop l from the notation if l is understood. It is easy to see that $\psi _i|_{N_l^i}=\psi |_{N_l^i}$ .

Let $(\pi ,V)$ be an irreducible generic representation of ${\mathrm {GL}}_l(F)$ and for $v\in V$ , we consider

$$ \begin{align*}v_i=\frac{1}{{\mathrm{vol}}(N_l^i)}\int_{N_l^i}\psi_i^{-1}(u)\pi(u)vdu.\end{align*} $$

If $W\in {\mathcal {W}}(\pi ,\psi )$ is the Whittaker function associated with v, then we denote $W_i=W_{v_i}$ . Note that

$$ \begin{align*}W_i(u_1gu_2)=\psi(u_1)\psi_i(u_2)W_i(g), \quad \forall g\in {\mathrm{GL}}_l(F), u_1\in N_l, u_2\in N_l^i.\end{align*} $$

Actually, there exists a positive integer $C(v)>0$ , such that $W_i$ satisfies the additional quasi-invariance property

(4.3) $$ \begin{align} W_i(ugh)=\psi(u)\psi_i(h)W_i(g)\end{align} $$

for all $u\in N_l,g\in {\mathrm {GL}}_l(F),h\in H_l^i$ if $i>C(v)$ ; see [Reference BaruchBar95, Lemma 3.2]. According to the proof [Reference BaruchBar95, Lemma 3.2], one can take $C(v)$ to be any integer such that v is fixed by $\pi (K^{C(v)}_l)$ .

Let $\omega $ be a character of $F^\times $ and we consider the space $C_c^\infty ({\mathrm {GL}}_l(F),\omega )$ consisting of smooth functions f on G such that f is compactly supported modulo $Z_l$ , the centre of ${\mathrm {GL}}_l(F)$ , and $f(zg)=\omega (z)f(g)$ . If $\pi $ is supercuspidal, let ${\mathcal {M}}(\pi )$ be the space of matrix coefficients of $\pi $ . Then ${\mathcal {M}}(\pi )\subset C_c^\infty ({\mathrm {GL}}_l(F),\omega _\pi )$ . For $f\in {\mathcal {M}}(\pi )$ , following [Reference Cogdell, Shahidi and TsaiCST17, page 2089], we consider the function

$$ \begin{align*}W^f(g)=\int_{N_l}\psi^{-1}(u)f(ug)du.\end{align*} $$

Note that the integral is convergent by assumption and defines an element in ${\mathcal {W}}(\pi ,\psi )$ . Moreover, for an appropriate choice of f, we can assume that $W^f(I_l)=1$ . See [Reference Cogdell, Shahidi and TsaiCST17, page 2089-2090]. Thus, we can consider $W^f_i$ . We also use the notation

$$ \begin{align*}{\mathcal {B}}_i(g,f)=W^f_i(g), \quad g\in {\mathrm{GL}}_l(F).\end{align*} $$

4.3 Weyl elements which support Bessel functions

Let $\Delta =\Delta ({\mathrm {GL}}_l)$ be the set of simple roots of ${\mathrm {GL}}_l(F)$ . Then $\Delta =\{\alpha _k: 1\le k\le l-1\}$ , where

$$ \begin{align*}\alpha_k({\mathrm {diag}}(t_1,\dots,t_l))=t_{k}/t_{k+1}, \quad {\mathrm {diag}}(t_1,\dots,t_l)\in T_l(F).\end{align*} $$

Let $\mathbf {{W}}=\mathbf {{W}}({\mathrm {GL}}_l)$ be the Weyl group of ${\mathrm {GL}}_l(F)$ . We sometimes identify $\mathbf {{W}}$ with the permutation matrix in ${\mathrm {GL}}_l(F)$ . Denote by e the identity element in $\mathbf {{W}}$ , which is represented by $I_l\in {\mathrm {GL}}_l(F)$ . For $w\in \mathbf {{W}}$ , denote $C(w)=BwB$ , where $B=B_l$ is the upper triangular subgroup of ${\mathrm {GL}}_{l}(F)$ . There is a Bruhat order on $\mathbf {{W}}$ , which is recalled as follows. Given $w_1,w_2\in \mathbf {{W}}$ , then $w_1\le w_2$ (or $w_2\ge w_1$ ) if and only if $C(w_1)\subset \overline {C(w_2)}$ . For $w\in \mathbf {{W}}$ , we denote $\Omega _w=\coprod _{w'\ge w}C(w')$ . Then $C(w)$ is closed in $\Omega _w$ and $\Omega _w$ is open in G.

Let ${\mathrm {B}}({\mathrm {GL}}_l)=\left \{{w\in \mathbf {{W}}({\mathrm {GL}}_l): \alpha \in \Delta , w\alpha>0\implies w\alpha \in \Delta }\right \}$ , which is the set of Weyl elements that can support partial Bessel functions.

Let $w_0=J_{l}\in {\mathrm {GL}}_l(F)$ , which represents the longest Weyl element of ${\mathrm {GL}}_l(F)$ . It is well-known that $w\in {\mathrm {B}}({\mathrm {GL}}_l)$ if and only if $w_0 w$ is the longest Weyl element of the Levi subgroup of a standard parabolic subgroup of ${\mathrm {GL}}_l(F)$ . For $w\in {\mathrm {B}}({\mathrm {GL}}_l)$ , let $P_w=M_wN_w$ be the corresponding parabolic subgroup such that $w_0 w=w_0^{M_w}$ , where $M_w$ is the Levi subgroup of $P_w$ and $w_0^{M_w}$ is the longest Weyl element of $M_w$ . Let $\theta _w$ be the subset of $\Delta $ which consists of all simple roots in $M_w$ . Then we have the relation

$$ \begin{align*}\theta_w=\left\{{\alpha\in \Delta| w\alpha>0}\right\}\subset \Delta.\end{align*} $$

The assignment $w\mapsto \theta _w$ is a bijection between ${\mathrm {B}}({\mathrm {GL}}_l)$ and subsets of $\Delta $ . Moreover, it is known that the assignment $w\mapsto \theta _w$ is order-reversing, i.e., $w'\le w$ if and only if $\theta _w\subset \theta _{w'}$ ; see [Reference Cogdell, Piatetski-Shapiro and ShahidiCPSS05, Proposition 2.11]. For example, we have $\theta _{w_0}=\emptyset $ and $\theta _{e}=\Delta $ .

Given a subset $\theta \subset \Delta $ , we will write the corresponding Weyl element in ${\mathrm {B}}({\mathrm {GL}}_l)$ by $w_\theta $ . For an integer k with $1\le k\le l-1$ , denote

$$ \begin{align*}\overline w_{k}=\begin{pmatrix} &I_{l-k}\\ I_k& \end{pmatrix} .\end{align*} $$

Proof. We have

$$ \begin{align*}w_{0}\overline w_k=\begin{pmatrix} J_k &\\ &J_{l-k} \end{pmatrix},\end{align*} $$

which is the longest Weyl element of the Levi subgroup

$$ \begin{align*}M_{\overline w_k}=\left\{{\begin{pmatrix} a&\\ &b \end{pmatrix}: a\in {\mathrm{GL}}_k(F), b\in {\mathrm{GL}}_{l-k}(F) }\right\}.\end{align*} $$

The set of simple roots in $M_{\overline w_k}$ is $\Delta -\left \{{\alpha _k}\right \}$ . Thus we have $\overline w_k\in {\mathrm {B}}({\mathrm {GL}}_l)$ and $\theta _{\overline w_k}=\Delta -\left \{{\alpha _k}\right \}$ .

Denote

$$ \begin{align*}\widetilde w_{n,m}=\begin{pmatrix} &&I_n\\ &I_{l-m-n}& \\ I_m&& \end{pmatrix}.\end{align*} $$

Proof. We have

$$ \begin{align*}w_0 \widetilde w_{n,m}=\begin{pmatrix}J_m &&\\ &J_{l-m-n}&\\&& J_n \end{pmatrix},\end{align*} $$

which is the longest Weyl element in the Levi subgroup

$$ \begin{align*}M_{\widetilde w_{n,m}}=\begin{pmatrix}a&&\\ &b&\\ &&c \end{pmatrix},a\in {\mathrm{GL}}_n, b\in {\mathrm{GL}}_{l-m-n}, c\in {\mathrm{GL}}_m.\end{align*} $$

Thus $\theta _{\widetilde w_{n,m}}=\Delta -\left \{{\alpha _m, \alpha _{l-n}}\right \}.$

Given $w,w'\in {\mathrm {B}}({\mathrm {GL}}_l)$ with $w>w'$ , define (following Jacquet [Reference JacquetJac16])

$$ \begin{align*}d_B(w,w')=\max\left\{{m\mid \textrm{ there exist }w^{\prime}_i\in {\mathrm {B}}({\mathrm{GL}}_l) \textrm{ with } w=w^{\prime}_m>w^{\prime}_{m-1}>\dots>w^{\prime}_0=w'}\right\}.\end{align*} $$

The number $d_B(w,w')$ is called the Bessel distance of $w,w'$ . By [Reference Cogdell, Piatetski-Shapiro and ShahidiCPSS05, Proposition 2.1] and Lemma 4.7, the set of elements in ${\mathrm {B}}({\mathrm {GL}}_l)$ which has Bessel distance 1 from the element $e\in {\mathrm {B}}({\mathrm {GL}}_l)$ are $\left \{{\overline w_k, 1\le k\le l-1}\right \}$ , i.e.,

(4.4) $$ \begin{align}\left\{{w| d_B(w,e)=1}\right\}=\left\{{\overline w_k| 1\le k\le l-1}\right\}.\end{align} $$

For $w,w'\in \mathbf {{W}}$ with $w<w'$ , we denote by $[w,w']$ the closed Bruhat interval $\left \{{w"\in \mathbf {{W}}({\mathrm {GL}}_l)| w\le w"\le w'}\right \}$ .

4.4 Cogdell-Shahidi-Tsai’s theory on partial Bessel functions

In this subsection, we review certain basic properties of partial Bessel functions developed by Cogdell-Shahidi-Tsai recently in [Reference Cogdell, Shahidi and TsaiCST17].

For $w\in {\mathrm {B}}({\mathrm {GL}}_l)$ , we denote

(4.5) $$ \begin{align}A_w=\left\{{a\in T_l(F)| \alpha(a)=1 \textrm{ for all }\alpha\in \theta_w}\right\}.\end{align} $$

The set $A_w$ is in fact the centre of $M_w$ .