2.1 An elementary description

We give a down-to-earth description of groups $\tilde{G}$ , $G$ , the involution $\unicode[STIX]{x1D703}$ and the vertex $\mathbf{x}$ . Let $\tilde{V}$ be an $n$ -dimensional hermitian space over $E$ , spanned by basis vectors $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ and equipped with the hermitian form given by $\langle \sum a_{i}\tilde{e}_{i},\sum b_{i}\tilde{e}_{i}\rangle _{\text{herm}}=\sum _{i=1}^{n}a_{n+1-i}b_{i}^{\ast }$ , where $a_{i},b_{i}\in E$ and $b_{i}^{\ast }$ is the conjugate of $b_{i}$ over $F$ . Then $\tilde{G}$ is such an algebraic group defined over $F$ for which $\tilde{G}(F)$ is isomorphic to the group of unitary operators on $\tilde{V}$ , i.e.  $E$ -linear operators on $\tilde{V}$ preserving the hermitian form.

Let $\unicode[STIX]{x1D6EC}=\text{span}_{{\mathcal{O}}_{E}}\{\tilde{e}_{1},\ldots ,\tilde{e}_{n}\}$ be a lattice in $\tilde{V}$ . Let $K$ be the subgroup of $\tilde{G}(F)$ consisting of unitary operators $g$ with $g(\unicode[STIX]{x1D6EC})=\unicode[STIX]{x1D6EC}$ . Then $K$ stabilizes a unique vertex on the Bruhat–Tits building of $\tilde{G}$ over $F$ , which (up to conjugation) is the vertex that we call $\mathbf{x}$ . We have the stabilizer group $\tilde{G}(F)_{\mathbf{x}}=K$ .

The hermitian form $\langle \cdot \,,\cdot \rangle _{\text{herm}}$ takes ${\mathcal{O}}_{E}$ values on $\unicode[STIX]{x1D6EC}$ . Its reduction mod $\unicode[STIX]{x1D70B}^{1/2}$ thus defines a quadratic form $\langle \cdot \,,\cdot \rangle$ on $V:=\unicode[STIX]{x1D6EC}/\unicode[STIX]{x1D70B}^{1/2}\unicode[STIX]{x1D6EC}$ . Write $e_{1},\ldots ,e_{n}$ to be the reduction of $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ , respectively. Then $\langle \cdot \,,\cdot \rangle$ on $V$ is defined by $\langle \sum a_{i}e_{i},\sum b_{i}e_{i}\rangle =\sum _{i=1}^{n}a_{n+1-i}b_{i}$ , where $a_{i},b_{i}\in k$ . The algebraic group $G$ then should be identified with the group of automorphisms of $V$ (not necessarily fixing $\langle \cdot \,,\cdot \rangle$ ); $G(k^{\prime })=\text{GL}(V\otimes _{k}k^{\prime })$ for any finite extension $k^{\prime }/k$ , and $\unicode[STIX]{x1D703}\curvearrowright G(k^{\prime })$ is the involution $\unicode[STIX]{x1D703}(g)=(g^{\text{t}})^{-1}$ where $g^{\text{t}}$ denotes the transpose of $g$ with respect to the quadratic form $\langle \cdot \,,\cdot \rangle$ . The differential of $\unicode[STIX]{x1D703}$ , still denoted by $\unicode[STIX]{x1D703}$ acts on $\mathfrak{g}=\text{Lie }G$ by $\unicode[STIX]{x1D703}(X)=-X^{\text{t}}$ .

The Lie algebra $\tilde{\mathfrak{g}}(F)$ is the space of anti-hermitian endomorphisms of $\tilde{V}$ . For any $d\in \frac{1}{2}\mathbb{Z}$ , one has the attached Moy–Prasad sublattice $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}=\{X\in \tilde{\mathfrak{g}}(F)\mid X(\unicode[STIX]{x1D6EC})\subset \unicode[STIX]{x1D70B}^{d}\unicode[STIX]{x1D6EC}\}$ .

Note that the isomorphism above depends on the choice of uniformizer $\unicode[STIX]{x1D70B}^{1/2}\in E$ . Lastly, the algebraic group $G$ has $\unicode[STIX]{x1D703}$ -stable Borel subgroups defined over $k$ . One such $B$ is given by that $B(k)$ consists of endomorphisms of $V$ that sends $e_{i}$ to a linear combination of $e_{1}$ , $e_{2},\ldots ,$ and $e_{i}$ . We also denote $B(0)=B\cap G(0)=(B^{\unicode[STIX]{x1D703}})^{o}$ . They are used in § 4. For $n\geqslant 3$ , all such $\unicode[STIX]{x1D703}$ -stable Borel subgroups are $G(0)$ -conjugate.

3.1 Odd case

In this subsection we have $n=2g+1$ and $G=\text{GL}_{2g+1}/_{k}=\text{GL}(V)$ . Recall that the vector space $V$ comes with a non-degenerate quadratic form $\langle \cdot \,,\cdot \rangle$ . We then have in the introduction an involution $\unicode[STIX]{x1D703}$ on $G$ which sends $g$ to $(g^{\text{t}})^{-1}$ , where $g^{\text{t}}$ is the adjoint of $g$ with respect to $\langle \cdot \,,\cdot \rangle$ . This induces an involution on $\mathfrak{g}$ , and we write $\mathfrak{g}(0)=\mathfrak{g}^{\unicode[STIX]{x1D703}=1}$ , $\mathfrak{g}(1)=\mathfrak{g}^{\unicode[STIX]{x1D703}=-1}$ . We have $\mathfrak{g}(1)\cong \operatorname{Sym}^{2}(V)$ as $G(0)$ -representations. As $\langle \cdot \,,\cdot \rangle$ provides a self-dual structure on $V$ , $\mathfrak{g}(1)\cong \text{End}^{\text{self}-\text{adj}}(V)$ is also the space of self-adjoint operators on $V$ .

The representation $G(0)\curvearrowright \mathfrak{g}(1)$ , or equivalently $\text{SO}(V)\curvearrowright \operatorname{Sym}^{2}(V)$ , was considered by Bhargava and Gross in [Reference Bhargava and GrossBG14]. An orbit in this representation is GIT-stable if and only if it is contained in $\mathfrak{g}(1)^{\text{rs}}:=\mathfrak{g}^{\text{rs}}\cap \mathfrak{g}(1)$ where $\mathfrak{g}^{\text{rs}}$ is the open subset of regular semisimple elements in the Lie algebra. We now fix an $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ .

Let $p_{T}(x)$ be the degree $2g+1$ monic characteristic polynomial of $T$ . Let $L=k[x]/p_{T}(x)$ be a degree $2g+1$ étale algebra over $k$ . Consider the Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . This is a commutative étale finite group scheme over $k$ of order $2^{2g+1}$ . It has a surjective norm map $Nm:\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ . Bhargava and Gross observed for $T\in \mathfrak{g}(1)^{\text{rs}}$ , we have canonical isomorphism $\text{Stab}_{G(0)}(T)\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . In fact, the map $T\mapsto p_{T}(x)$ is the GIT-quotient map $\mathfrak{g}(1)\mapsto \mathfrak{g}(1)/\!/G(0)$ ; we have $\mathfrak{g}(1)/\!/G(0)\cong \mathbb{A}^{2g+1}$ is the space of degree $n$ monic polynomials.

Let $C_{T}=(y^{2}=p_{T}(x))$ be a (smooth completion of) genus $g$ hyperelliptic curve. Let $J_{T}=\operatorname{Pic}^{0}(C_{T})$ . Since the $2$ -torsion $J_{T}[2]$ is generated by differences of Weierstrass points, one checks $J_{T}[2]\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . Consequently $J_{T}[2]\cong \text{Stab}_{G(0)}(T)$ .

If one fix such a $T$ , then the orbit of $T$ is $G(0)(k).T$ while the stable orbit of $T$ is $(G(0)(\bar{k}).T)\cap \mathfrak{g}(1)(\bar{k})$ . There could be more than one orbits inside a stable orbit, and relative to the choice of $T$ as a pinning they can be classified by $\ker (H^{1}(k,\text{Stab}_{G(0)}(T))\rightarrow H^{1}(k,G(0)))$ . When $k$ is a finite field, by Lang’s theorem, the latter pointed set is trivial, and thus we have $H^{1}(k,\text{Stab}_{G(0)}(T))\cong H^{1}(k,J_{T}[2])$ classifies orbits in the stable orbit of $T$ relative to the choice of a pinning.

The GIT-quotient map $\mathfrak{g}(1)\rightarrow \mathfrak{g}(1)/\!/G(0)$ has a Kostant section [Reference LevyLev09, Theorem 5.5]. Using the Kostant section as a pinning, a $G(0)(k)$ -orbit in $\mathfrak{g}(1)^{\text{rs}}(k)$ corresponds to a hyperelliptic curve $C_{T}$ together with a class in $H^{1}(k,J_{T}[2])$ . For $k$ a global field, Bhargava, Gross and others used this to study the average size of $2$ -Selmer groups of such hyperelliptic curves, see e.g. [Reference Bhargava and GrossBG13]. For this purpose, Wang developed the theory of pencil of quadrics [Reference WangWan13]. It turns out that his theory is very useful in describing the variety that we will encounter in orbital integrals.

On the vector space $V\oplus k$ we define two quadratic forms by $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{1}=\langle v_{1},v_{2}\rangle$ and $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{2}=\langle v_{1},Tv_{2}\rangle -c_{1}c_{2}$ . This defines a generic pencil of quadrics in the sense of Wang [Reference WangWan13, Introduction]. Recall that a subspace $W\subset V\oplus k$ is said to be isotropic with respect to a quadric (e.g.  $\langle \cdot \,,\cdot \rangle _{1}$ ) if the restriction of the quadratic form to $W$ is trivial. In his paper, Wang proved the following result.

In particular, $F_{T}$ is a torsor under $J_{T}$ and there is a doubling map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ . We review the group structure in the theorem. The group structure is determined by $(p)-[W]$ , i.e. how to subtract from $p\in C_{T}$ a subspace $[W]\in F_{T}$ . This is done as follows: recall that a ruling is a connected component of the variety parameterizing $(g+1)$ -dimensional subspace on which the quadratic form is trivial. A point $p=(x,y)$ on $C_{T}$ corresponds to a ruling of $\langle \cdot \,,\cdot \rangle _{2}-x\langle \cdot \,,\cdot \rangle _{1}$ (see [Reference WangWan13, p. 8]). There will be a unique $(g+1)$ -dimensional space $W^{\prime }$ in the ruling such that $W^{\prime }\supset W$ . Inside the space $W^{\prime }$ there will be, when counted with multiplicity, two $g$ -dimensional subspaces $W$ and $W^{\prime \prime }$ on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish. It is then defined $(p)-[W]:=[W^{\prime \prime }]$ . This uniquely characterizes the group structure on $G_{T}$ .

We give an example of $(p)-[W]$ when $p=\infty$ , the rational point on $C_{T}$ at infinity. Note that $\infty \in C_{T}$ correspond to the quadric $\langle \cdot \,,\cdot \rangle _{1}$ , which is degenerate and has only one ruling. Let $\unicode[STIX]{x1D70F}_{\infty }:V\oplus k\rightarrow V\oplus k$ be the map that sends $(v,c)$ to $(v,-c)$ . It is obvious that if $[W]\in F_{T}$ , then $[\unicode[STIX]{x1D70F}_{\infty }W]\in F_{T}$ . Let us fix such $W$ and write $W^{\prime \prime }=\unicode[STIX]{x1D70F}_{\infty }W$ . As $W\subset V\oplus k$ is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{2}$ , the second component is not contained in $W$ . Hence, $W^{\prime }:=W+k$ , where this $k$ is the second component of $V\oplus k$ , is $(g+1)$ -dimensional, and is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ as $W$ is. This says that $W^{\prime }$ and $W^{\prime \prime }$ satisfy the properties in the previous paragraph, and consequently $(\infty )-[W]=[W^{\prime \prime }]=[\unicode[STIX]{x1D70F}_{\infty }(W)]$ .

It is obvious that $G_{T}$ depends only on $T$ up to $G(0)(k)$ -conjugacy. As mentioned in the introduction the orbit of $T$ in its stable orbit may be characterized by a class $\unicode[STIX]{x1D6FC}_{T}\in H^{1}(k,J_{T}[2])$ . This class can be described as follows: by Theorem 3.1(i) and (ii) the map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ is étale and Galois with Galois group being $J_{T}[2]$ as a group scheme over $k$ . From the previous paragraph and [Reference WangWan13, Proposition 2.29], respectively, one has the following result.

We will later need an explicit description of the $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ . Let $p_{0},\ldots ,p_{2g},\infty$ be the Weierstrass points of $C_{T}$ . Then $J_{T}[2]$ is generated by $((p_{i})-(\infty ))$ , $0\leqslant i\leqslant 2g$ with the only relation $\sum _{i=0}^{2g}((p_{i})-(\infty ))=0$ . The action of $(p_{i})-(\infty )$ can be described as follows. Say $p_{i}=(x,0)$ . Then $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is a degenerate quadric with one-dimensional kernel $U$ . For $[W]\in (\times 2)^{-1}(\infty )$ , $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is trivial on the $(g+1)$ -dimensional space $W+U$ . There is exactly one $g$ -dimensional subspace $W^{\prime }\subset W+U$ , other than $W$ , on which $\langle \cdot \,,\cdot \rangle$ and $\langle \cdot ,T\cdot \rangle$ are also trivial. The $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ is then given by the following result.

For any $0\leqslant m\leqslant g$ , consider $j_{m}:\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by $j_{m}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-(m-1)(\infty )$ . Let $X_{T,m}$ be the image of $j_{m}$ , and let $\tilde{X}_{T,m}:=(\times 2)^{-1}(X_{T,m})$ be its preimage under the étale map $\times 2$ . For example $\tilde{X}_{T,0}=(\times 2)^{-1}(\infty )$ . We also put $\tilde{X}_{T,-1}=\emptyset$ . We shall relate $\tilde{X}_{T,m}$ with the following varieties $F_{T,m}$ , which could be thought as a generalized version of Hessenberg varieties considered by Goresky et al. [Reference Goresky, Kottwitz and MacPhersonGKM06].

For any finite extension $k^{\prime }/k$ , we call a flag of $k^{\prime }$ -subspaces $0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset (V\oplus k)\otimes _{k}k^{\prime }$ good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

3. (iii) $W^{g-1}\subset V\otimes _{k}k^{\prime }$ ;

4. (iv) $T(W^{i})\subset W^{i+2}$ for all $1\leqslant i\leqslant g-3$ ;

5. (v) $T(W^{g-2})\subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ .

Here $\unicode[STIX]{x1D70B}_{1}:(V\oplus k)\otimes _{k}k^{\prime }\rightarrow V\otimes _{k}k^{\prime }$ is the projection to the first factor. For $1\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset \unicode[STIX]{x1D70B}_{1}(W^{g-m+1})$ . Note this is simply $T(W^{g-m})\subset W^{g-m+1}$ when $m>1$ . Also a good flag is called $0$ -good if $W^{g}\subset V\otimes _{k}k^{\prime }$ . Next, for $0\leqslant m\leqslant g$ , an $m$ -good flag is called $m$ -excellent if it is also $n$ -good for $m<n\leqslant g$ . On the other hand, a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . Finally, a good flag is called $m$ -exact if it is $m$ -excellent and $m$ -general. Now let

The functor $F_{T,m}$ is easily seen from its very definition to be represented by a quasi-projective variety over $k$ which we will denote with the same notation. In fact, there is a projective variety $F_{T,\text{good}}$ that parameterize good flags, and $F_{T,m}\subset F_{T,\text{good}}$ is locally closed. There is a natural map $\tilde{j}:F_{T,\text{good}}\rightarrow F_{T}$ by sending a flag to $[W^{g}]$ . In the rest of this section, we work ‘geometrically,’ i.e. we replace $k$ by an algebraic closure $\bar{k}$ , so that we can omit the notation $\cdot \otimes _{k}k^{\prime }$ and so on. This section is mostly devoted to the proof of the following result.

We now begin our proof for the general case with two simple lemmas.

Let $F_{T,m}^{\text{ex}}\subset F_{T,\text{good}}$ be projective varieties parameterizing $m$ -excellent flags. The key is as follows.

We now come back to the proof of Lemma 3.8. Define $L$ to be the variety that parameterize $g$ -dimensional subspaces $W$ satisfying $U_{\lfloor (m+1)/2\rfloor }\subset W\subset U^{\lfloor (m+1)/2\rfloor }$ and that $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish on $W$ . In particular, by construction we have $U_{\lfloor (m+1)/2\rfloor }\subset W^{g-m+2\lfloor (m+1)/2\rfloor -1}\subset W^{g}$ and $W^{g}\subset (W^{g})^{\bot 2}\cap U^{0}\subset U^{\lfloor (m+1)/2\rfloor }$ , i.e.  $[W^{g}]\in L$ .

Define $\overline{V}$ to be the subquotient $\overline{V}\,:=\,U^{\lfloor (m+1)/2\rfloor }/U_{\lfloor (m+1)/2\rfloor }$ . Since $U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor -1})^{\bot 1}\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 1}$ and $U^{\lfloor (m+1)/2\rfloor }\subset (U_{\lfloor (m+1)/2\rfloor })^{\bot 2}\Rightarrow U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 2}$ , the two quadratic forms $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ restricts to be quadratic forms on  $\overline{V}$ . Denote still their restrictions by $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ , respectively. Let $L^{\prime }$ be the variety that parameterize $\lfloor m/2\rfloor$ -dimensional subspaces in $\overline{V}$ on which $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ are trivial. Then we have $\unicode[STIX]{x1D704}_{T}:L\xrightarrow[]{{\sim}}L^{\prime }$ which sends a $g$ -dimensional subspace of $V$ contained in $U^{\lfloor (m+1)/2\rfloor }$ to its image in $\overline{V}$ .

Consider the polynomials $p_{T}^{(i)}(x)=\text{disc}(\langle \cdot \,,\cdot \rangle _{1}-x\langle \cdot \,,\cdot \rangle _{2})|_{U^{i}/U_{i}}$ for $i=0,1,\ldots ,\lfloor (m+1)/2\rfloor$ . We claim $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ . To see this, observe that when we go from $U^{0}/U_{0}$ to $U^{1}/U_{1}$ , we quotient out $U_{1}/U_{0}$ , on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ is zero. Even more, $U_{1}/U_{0}$ is in the kernel of $\langle \cdot \,,\cdot \rangle _{1}$ while $\langle U_{1}/U_{0},U^{0}/U^{1}\rangle _{2}$ is non-trivial. This exactly says $p_{T}^{(0)}(x)=x^{2}p_{T}^{(1)}(x)$ . Repeating the argument gives the asserted result. We also note $\deg p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)=2m-2\lfloor (m+1)/2\rfloor +1=2\lfloor m/2\rfloor +1$ .

Let $\bar{C}_{T}$ be the hyperelliptic curve given by $(y^{2}=p_{T}^{(\lfloor (m+1)/2\rfloor )}(x))$ with completion smooth at infinity. Recall $T\in {\mathcal{V}}$ is such that our flag $\mathbb{F}$ is $m$ -excellent with respect to $T$ . One checks from definition that when $T$ runs over the subspace ${\mathcal{V}}$ , $p_{T}^{(0)}(x)$ runs over all monic polynomials of degree less than or equal to $2m+1$ that are divided by $x^{m+1}$ . Consequently, there exists a Zariski open subset $\unicode[STIX]{x1D6E5}\subset {\mathcal{V}}$ such that for $T\in \unicode[STIX]{x1D6E5}$ , $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is separable (i.e. having distinct roots) and $\bar{C}_{T}$ is smooth.

For such $T$ , the relation between $L^{\prime }$ and $\bar{C}_{T}$ is exactly that between $(\times 2)^{-1}(\infty )$ and $C_{T}$ in Lemma 3.2(i). In particular, we have a simply transitive $\operatorname{Pic}^{0}(\bar{C}_{T})[2]$ -action on $L^{\prime }$ described by Lemma 3.3. Let $\overline{\unicode[STIX]{x1D6FA}}_{0}\subset \overline{V}$ be the $\lfloor m/2\rfloor$ -dimensional subspace that corresponds to $W^{g}$ , i.e.  $\unicode[STIX]{x1D704}_{T}([W^{g}])=[\overline{\unicode[STIX]{x1D6FA}}_{0}]$ . Let $\bar{p}_{0},\ldots ,\bar{p}_{2\lfloor m/2\rfloor },\bar{\infty }$ be the Weierstrass points of $\bar{C}_{T}$ , the last one understood as the point at infinity. We define $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ by $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i-1})-(\bar{\infty })).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3, $\overline{\unicode[STIX]{x1D6FA}}_{i-1}$ and $\overline{\unicode[STIX]{x1D6FA}}_{i}$ intersect in codimension one. We note that the relation $\sum ((\bar{p}_{i})-(\bar{\infty }))=0$ implies $\overline{\unicode[STIX]{x1D6FA}}_{2\lfloor m/2\rfloor +1}=\overline{\unicode[STIX]{x1D6FA}}_{0}$ .

Now we pass this sequence from $L^{\prime }$ to $L$ . Let $\unicode[STIX]{x1D6FA}_{i}\subset V\oplus k$ be the preimage of $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for each $i$ . We also have $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ intersect in codimension one for each $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ . This will allow us to fetch the precise information we want about $[\unicode[STIX]{x1D6FA}_{i}]$ using the description of the group structure of $G_{T}$ following Theorem 3.1: the subspace $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ is $(g+1)$ -dimensional, and any quadratic form among $x_{1}\langle \cdot \,,\cdot \rangle _{1}+x_{2}\langle \cdot \,,\cdot \rangle _{2}$ factor through a linear pairing of $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i-1}$ and $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i}$ since the quadratic form is trivial on both $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ by assumption. Consequently some non-trivial combination of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ must vanish on $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . The space $\unicode[STIX]{x1D6FA}_{i-1}$ , $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ and $\unicode[STIX]{x1D6FA}_{i}$ then play the role of $W$ , $W^{\prime }$ and $W^{\prime \prime }$ in the paragraph following Theorem 3.1, respectively, which means there exists some $p_{i}\in C_{T}$ such that $(p_{i})-[\unicode[STIX]{x1D6FA}_{i-1}]=[\unicode[STIX]{x1D6FA}_{i}]$ .

Recall that $\unicode[STIX]{x1D6FA}_{2\lfloor m/2\rfloor +1}=\unicode[STIX]{x1D6FA}_{0}=W^{g}$ . Hence, we have

Put $p_{i}^{\ast }$ to be the hyperelliptic conjugate of $p_{i}$ for $i=2,4,\ldots ,2\lfloor m/2\rfloor$ and $p_{i}^{\ast }=p_{i}$ for odd $i$ , so that $(p_{i}^{\ast })-(\infty )=(-1)^{i+1}((p_{i})-(\infty ))\in J_{T}$ . Equation (3.1) then becomes

or, equivalently,

which says $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor +1}$ . This concludes the proof of Lemma 3.8 when $m$ is odd.

When $m$ is even, the fact that $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x^{m+1}$ implies that $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x$ . This means $(0,0)=\bar{p}_{i}$ is one of the Weierstrass point of  $\bar{C}_{T}$ . Recall $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i})-(\infty )).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3 we know $\langle \cdot \,,\cdot \rangle _{1}$ vanish on $\overline{\unicode[STIX]{x1D6FA}}_{i-1}+\overline{\unicode[STIX]{x1D6FA}}_{i}$ , and thus $\langle \cdot \,,\cdot \rangle _{1}$ also vanish on the preimage $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . But this says $p_{i}=\infty$ . Thus, (3.2) gives the better result $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor }=X_{T,m}$ and we are done.◻

3.2 Even case

In this subsection $n=2g+2$ ; $\tilde{G}=U_{2g+2}(E/F)$ , $G=\text{GL}_{2g+2}/_{k}=\text{GL}(V)$ , $G(0)=\text{SO}(V)$ and $\mathfrak{g}(1)=\operatorname{Sym}^{2}(V)$ where $V$ is a $(2g+2)$ -dimensional non-degenerate split quadratic space. The method in this subsection is almost identical to that of the previous one, and we only list the setting, definition and results here.

We have parallel result to Theorem 3.4. Again fix $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ . We also write $p_{T}(x)\in k[x]$ the monic characteristic polynomial of $T$ . Our hyperelliptic curve $C_{T}:=(y^{2}=p_{T}(x))$ now has two points above the infinity on $\mathbb{P}^{1}$ . We shall denote these two points by $\infty ^{(1)}$ and $\infty ^{(2)}$ . They are both defined over $k$ .

Consider $L=k[x]/p_{T}(x)$ . The Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ now has not only a surjective norm map $Nm\,:\,\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ but also a diagonal embedding $\unicode[STIX]{x1D6E5}\,:\,\unicode[STIX]{x1D707}_{2}\rightarrow \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . We have $\text{Stab}_{\text{O}(V)}(T)\cong \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ , also $\text{Stab}_{G(0)}(T)\cong \ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ , and lastly $J_{T}[2]\cong \text{Stab}_{G(0)}(T)/Z(G(0))\cong (\ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2}))/\unicode[STIX]{x1D6E5}(\unicode[STIX]{x1D707}_{2})$ .

We denote by $\langle \cdot \,,\cdot \rangle _{1}=\langle \cdot \,,\cdot \rangle$ the standard quadratic form on $V$ , i.e. the one which is invariant by $G(0)$ . Then $\infty ^{(1)}$ and $\infty ^{(2)}$ are just the two rulings of $\langle \cdot \,,\cdot \rangle _{1}$ . Define $\langle \cdot \,,\cdot \rangle _{2}$ on $V$ by $\langle v_{1},v_{2}\rangle _{2}=\langle v_{1},Tv_{2}\rangle _{1}$ . Then the theory of pencil of quadrics says the following.

We again write the doubling map $\times 2\,:\,F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ which is étale Galois with Galois group $J_{T}[2]$ . For $0\leqslant m\leqslant g$ with $m$ even, define $j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=1,2$ .

For $0<m\leqslant g$ with $m$ odd, we define $j_{m}^{(0)},j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=0,1,2$ .

Next, we introduce the notion of good flags. A flag of subspaces $0\subset W^{1}\subset \cdots \subset W^{g+1}\subset V$ is called good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ to $W^{g+1}$ is zero;

3. (iii) the restriction of $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

4. (iv) $T(W^{i})\subset W^{i+2}$ , for all $1\leqslant i\leqslant g-1$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset W^{g-m+1}$ . Here $W^{-1}=W^{0}=\{0\}$ , i.e. good flags are automatically $g$ -good. A flag is called $m$ -excellent if it is $n$ -good for $m\leqslant n\leqslant g$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . For any $0<m\leqslant g$ , we now define the notion of $m$ -exact flags (see also Remark 3.14). Let $\{W^{r}\}_{r=1}^{g+1}$ be any $m$ -excellent and $m$ -general flag. There always exists another $m$ -excellent flag $\{U^{r}\}_{r=1}^{g+1}$ satisfying $U^{g}=W^{g}$ but $U^{g+1}\not =W^{g+1}$ if $m$ is odd, or $U^{g-1}=W^{g-1}$ , $U^{g+1}=W^{g+1}$ but $U^{g}\not =W^{g}$ if $m$ is even. We say $\{W^{r}\}_{r=1}^{g+1}$ is $m$ -exact if $\{U^{r}\}_{r=1}^{g+1}$ is also $m$ -general. Lastly, a $0$ -excellent flag is said to be $0$ -exact.

Now let $F_{T,m}^{(1)}$ be the variety that parameterize $m$ -exact flags for which $W^{g+1}$ is in the ruling $\infty ^{(1)}$ , and $F_{T,m}^{(2)}$ be the variety that parameterizes those $m$ -exact flags for which $W^{g+1}$ is in the other ruling $\infty ^{(2)}$ . We have natural maps $\tilde{j}\,:\,F_{T,m}\,:=\,F_{T,m}^{(1)}\sqcup F_{T,m}^{(2)}\rightarrow F_{T}$ by sending $\{W^{r}\}_{r=1}^{g+1}$ to $W^{g}$ .

4.1 Geometric identification

The goal in this subsection is to prove Proposition 4.3. We begin with a lemma.

The main result in this subsection is to translate the following result from § 3.1.

It is now straightforward to verify that Proposition 4.3 follows from Lemma 4.6 and the following lemma.

4.2 Nilpotent orbital integrals

We now prove Proposition 4.4. Nilpotent orbital integrals, just like counting points on nilpotent Springer fibers, is usually purely combinatorial. Let $B^{\text{opp}}\subset G$ be the opposite Borel to $B$ with respect to $S$ . Our essential idea here is that after applying the formula of Ranga Rao [Reference Ranga RaoRR72, Theorem 1], we can do ‘reduction modulo $\unicode[STIX]{x1D70B}$ ’ and arrive at an integral over $G(0)(k)$ which is left invariant by $U(0)(k)$ and right invariant by $B^{\text{opp}}(0)(k)$ . This gives a combinatorial sum over $U(0)(k)\backslash G(0)(k)/B^{\text{opp}}(0)(k)$ , which is identified with the Weyl group of $G(0)$ .

To begin our proof, the formula of Ranga Rao in our case can be formulated as follows. There exists a maximal $F$ -split torus $\tilde{S}\subset \tilde{G}$ , whose corresponding apartment contains $\mathbf{x}$ and whose reduction at $\mathbf{x}$ is equal to $S(0)\subset G(0)$ . Moreover, after conjugation we may assume that the cocharacter $\unicode[STIX]{x1D70C}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{k}\rightarrow S(0)$ corresponds to $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{F}\rightarrow \tilde{S}$ and that $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ also acts on ${\tilde{N}}_{m}^{\prime }$ by weight $-2$ .

Fix such an $\tilde{S}$ and $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ . Denote by ${\mathcal{O}}$ in this subsection the orbit of ${\tilde{N}}_{m^{\prime }}$ . Write $\tilde{\mathfrak{g}}_{i}\subset \tilde{\mathfrak{g}}$ be the subspace on which $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ acts by weight $i$ . Then with suitably normalized measure, Ranga Rao’s formula says

where $\unicode[STIX]{x1D711}(\tilde{X})$ is an $\mathbb{R}$ -valued function on $\tilde{\mathfrak{g}}_{-2}(F)$ and the measure on the first integral is a Haar measure on $\tilde{\mathfrak{g}}_{{\leqslant}-2}$ . The space $\tilde{\mathfrak{g}}_{-2}$ can be interpreted as follows: let $\tilde{V}$ be the standard representation of $\tilde{G}/_{E}$ , i.e.  $\tilde{V}$ is a $(2g+1)$ -dimensional hermitian space over $E$ . Then $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ acts on $\tilde{V}$ with weights $(2g-m^{\prime }),(2g-m^{\prime }-2),\ldots ,m^{\prime },(m^{\prime }-1),\ldots ,1,0,-1,\ldots ,-m^{\prime },(-m^{\prime }-2),\ldots ,(-2g+m^{\prime })$ .

Denote by $\tilde{V}_{g},\ldots ,\tilde{V}_{-g}$ the one-dimensional $E$ -subspace with these weights, respectively. Let $\tilde{\mathfrak{g}}_{ij}$ , $-g\leqslant i,j\leqslant g$ be the one-dimensional $E$ -subspace of $\tilde{\mathfrak{g}}$ which maps $\tilde{V}_{j}$ to $\tilde{V}_{i}$ . One can then check

and

Note that $\tilde{\mathfrak{g}}_{ij}$ is not defined over $F$ unless $i+j=0$ , but $\tilde{\mathfrak{g}}_{ij}+\tilde{\mathfrak{g}}_{-j,-i}$ is always defined over  $F$ . Now we fix a ‘valuation-preserving’ identification of $\tilde{u} _{ij}\,:\,\tilde{\mathfrak{g}}_{ij}\rightarrow \mathbb{G}_{a}/_{E}$ so that $\tilde{u} _{ij}^{-1}(\unicode[STIX]{x1D70B}^{-1/2})$ is not in $\tilde{\mathfrak{g}}_{ij}\cap \tilde{\mathfrak{g}}(E)_{\mathbf{x},0}$ but $\tilde{u} _{ij}^{-1}(1)$ is. Let $|\cdot |\,:\,E\rightarrow \mathbb{R}$ be the extension of the standard norm on $F$ , i.e.  $|\unicode[STIX]{x1D70B}^{-1/2}|=q^{1/2}$ . One then computes

In (4.2), if $\tilde{X}\in \tilde{\mathfrak{g}}_{{\leqslant}-2}$ is such that $|\tilde{u} _{ij}(\tilde{X})|>q^{1/2}$ , then $\tilde{X}\not \in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ and $f_{m}(\text{Ad}(\tilde{h})(\tilde{X}))=0$ . Moreover, the value of $f_{m}(\text{Ad}(\tilde{h})(\tilde{X}))$ depends only on $\tilde{u} _{ij}(\tilde{X})$ modulo ${\mathcal{O}}_{E}$ since $f_{m}$ is locally constant by $\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}$ . Let us now denote by $Z_{m^{\prime }}$ the image of $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}\cap \tilde{\mathfrak{g}}_{{\leqslant}-2}(F)\cap {\mathcal{O}}$ in $\mathfrak{g}(1)(k)$ and by $(d\unicode[STIX]{x1D707})_{m^{\prime }}$ the push-forward of the measure $\unicode[STIX]{x1D711}(\tilde{X})\,d\tilde{X}|_{\tilde{\mathfrak{g}}_{{\leqslant}-2}(F)\cap \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}}$ to the finite set $Z_{m^{\prime }}$ . Also let $dh$ be the push-forward of the measure $d\tilde{h}$ from $\tilde{G}(F)_{\mathbf{x},0}$ to $G(0)(k)$ . Then we can rewrite (4.2) as

We can similarly define $V_{g},\ldots ,V_{-g}$ as one-dimensional $k$ -subspace on which $\unicode[STIX]{x1D70C}_{m^{\prime }}$ acts by strictly decreasing weight. In fact $V_{i}$ is just the line spanned by $v_{i}$ in the previous subsection (with $m$ replaced by $m^{\prime }$ ). We can then define $u_{ij}\,:\,\mathfrak{g}(k)\rightarrow k$ in the same way, scaled so that if we write the reduction maps $\text{red}_{1}\,:\,\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}\rightarrow \mathfrak{g}(1)$ and $\text{red}_{2}\,:\,\unicode[STIX]{x1D70B}^{-1/2}{\mathcal{O}}_{E}\rightarrow k$ , then $u_{ij}(\text{red}_{1}(\tilde{X}))=\text{red}_{2}(\tilde{u} _{ij}(\tilde{X}))$ for $\tilde{X}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ .

On the subset $Z_{m^{\prime }}^{o}\subset Z_{m^{\prime }}$ where the conclusion of the lemma holds, $(d\unicode[STIX]{x1D707})_{m^{\prime }}$ is nothing but a multiple of the counting measure. We will pretend it is exactly the counting measure and discuss the normalization constant later. The idea in the lemma can now be further applied to compute the integral; write

We have to compute $\sum _{h\in G(0)(k)}I_{m^{\prime }}(h)$ (up to a normalizing constant). Exactly as in the situation of the previous lemma, This function $I_{m^{\prime }}$ is invariant under left translation by $U(0)(k)$ and right translation by $B^{\text{opp}}(0)(k)$ , and we arrive at a sum over the Weyl group $S_{g}\ltimes \{\pm 1\}^{g}$ . To have $I_{m^{\prime }}(\unicode[STIX]{x1D70E}^{-1})\not =0$ , $\unicode[STIX]{x1D70E}$ needs to satisfy Condition 4.9.

Denote by $\unicode[STIX]{x1D6EF}_{m,m^{\prime }}\subset S_{g}$ the set of such $\unicode[STIX]{x1D70E}$ . For $\unicode[STIX]{x1D70E}\in \unicode[STIX]{x1D6EF}_{m,m^{\prime }}$ , the number of elements in the double coset $U(0)(k)\unicode[STIX]{x1D70E}^{-1}B^{\text{opp}}(0)(k)$ is given by $\#B(0)(k)\cdot q^{\unicode[STIX]{x1D6FF}_{1}(\unicode[STIX]{x1D70E})}$ . Also the sum $I_{m^{\prime }}(\unicode[STIX]{x1D70E}^{-1})=q^{\unicode[STIX]{x1D6FF}_{2}(\unicode[STIX]{x1D70E})-g^{2}}$ , where

We also have to figure out the normalization of measures. The single choice $X=N_{m^{\prime }}\in Z_{m^{\prime }}^{o}\subset \mathfrak{g}(1)(k)$ and $h=id\in G(0)(k)$ correspond to the lattice $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}/(\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}\cap \tilde{\mathfrak{g}}_{{\tilde{N}}_{m^{\prime }}})\subset \tilde{\mathfrak{g}}(F)/\tilde{\mathfrak{g}}_{{\tilde{N}}_{m^{\prime }}}(F)$ . A careful inspection of the normalization at the end of Appendix A shows that this lattice is to have measure $1$ .

In addition, as Ranga Rao’s method begins with Iwasawa decomposition $\tilde{G}=\tilde{B}\cdot \tilde{G}(F)_{\mathbf{x},0}$ , we also have to divide by how much they intersect. That is, we have to divide by the order of $(\tilde{G}(F)_{\mathbf{x},0}\cap \tilde{B})\tilde{G}(F)_{\mathbf{x},1/2}/\tilde{G}(F)_{\mathbf{x},1/2}$ , which is $\#B(0)(k)$ . In summary, all the way from (4.3) we have

Proposition 4.4 now follows from Proposition B.7. This completes the proof of Theorem 4.1.