2.1. Relative invariants and regularity

We begin by reviewing the basic setup about prehomogeneous vector spaces from [Reference Li21] (see also [Reference Li22, Chap. 6] or [Reference Kimura14], [Reference Sato27]). The following assumptions will remain in force throughout this article.

Then $X^+(\mathbb {R})$ is a union of finitely many $G(\mathbb {R})$ -orbits. The triplet $(G, \rho , X)$ forms a reductive prehomogeneous vector space over $\mathbb {R}$ . We say that a nonzero $f \in \mathbb {R}(X)$ is a relative invariant if there exists $\omega \in \mathbf {X}^*(G)$ such that $f(xg) = \omega (g) f(x)$ for all $(x,g) \in X \times G$ ; the character $\omega $ is unique, called the eigencharacter of f.

Relative invariants on an arbitrary prehomogeneous vector space are automatically homogeneous, according to [Reference Kimura14, Cor. 2.7].

If $f \in \mathbb {R}(X)$ is a relative invariant, then the logarithmic derivative $f^{-1} \mathop {}\!\mathrm {d} f$ defines a G-equivariant morphism $X^+ \to \check {X}$ . We say f is nondegenerate if $f^{-1} \mathop {}\!\mathrm {d} f$ is dominant (see [Reference Kimura14, Def. 2.14]).

The general theory of prehomogeneous vector spaces affords the basic relative invariants $f_1, \ldots , f_r \in \mathbb {R}[X]$ , say with eigencharacters $\omega _1, \ldots , \omega _r \in \mathbf {X}^*(G)$ under G-action, which define irreducible codimension-one components of $\partial X$ . Moreover,

$$ \begin{align*} \mathbf{X}^*_\rho(G) & := \left\{ \omega \in \mathbf{X}^*(G) : \text{eigencharacter of some relative invariant}\right\} \\ & = \bigoplus_{i=1}^r \mathbb{Z} \omega_i. \end{align*} $$

It is known that $\{\omega _1, \ldots , \omega _r\}$ is uniquely determined, whereas the $f_i$ corresponding to $\omega _i$ is unique up to $\mathbb {R}^\times $ . Call $\omega _1, \ldots , \omega _r$ the basic eigencharacters. Every relative invariant is proportional to $f_1^{a_1} \cdots f_r^{a_r}$ for a unique $(a_1, \ldots , a_r) \in \mathbb {Z}^r$ . The zero loci of basic relative invariants correspond to the irreducible components of $\partial X$ . Therefore, $f_1^{a_1} \cdots f_r^{a_r} \in \mathbb {R}[X]$ if and only if $a_1, \ldots , a_r \geq 0$ .

When G is split, the facts above have been reviewed in [Reference Li22, §6.2]. The general case follows by Galois descent from $\mathbb {C}$ to $\mathbb {R}$ ; specifically, $\operatorname {Gal}(\mathbb {C}|\mathbb {R})$ permutes the irreducible components of $\partial X_{\mathbb {C}}$ and

$$\begin{align*}\mathbf{X}^*_\rho(G) = \mathbf{X}^*_{\rho \otimes \mathbb{C}}(G_{\mathbb{C}})^{\operatorname{Gal}(\mathbb{C}|\mathbb{R})}. \end{align*}$$

Indeed, the only nontrivial part is that a priori, to each $\omega \in \mathbf {X}^*_{\rho \otimes \mathbb {C}}(G_{\mathbb {C}})^{\operatorname {Gal}(\mathbb {C}|\mathbb {R})} \subset \mathbf {X}^*(G)$ corresponds only a relative invariant $f \in \mathbb {C}(X)$ that is unique up to $\mathbb {C}^\times $ , but we may take $f \in \mathbb {R}(X)$ by the following technique: for every $\sigma \in \operatorname {Gal}(\mathbb {C}|\mathbb {R})$ , let $c_\sigma \in \mathbb {C}^\times $ be such that ${}^\sigma f = c_\sigma f$ , so that $\sigma \mapsto c_\sigma $ is a $1$ -cocycle; Hilbert’s Theorem 90 then implies that there exists $c \in \mathbb {C}^\times $ such that $cf$ is $\operatorname {Gal}(\mathbb {C}|\mathbb {R})$ -invariant as desired. Hence, $\omega \in \mathbf {X}^*_\rho (G)$ .

According to [Reference Kimura14, Th. 2.28] and our assumptions, $(G, \rho , X)$ is regular. Specifically,

• • $(\det \rho )^2 \in \mathbf {X}^*_\rho (G)$ and it corresponds to a nondegenerate relative invariant in $\mathbb {R}(X)$ ;

• • the dual triplet $(G, \check {\rho }, \check {X})$ is regular prehomogeneous as well;

• • $\mathbf {X}^*_{\rho }(G) = \mathbf {X}^*_{\check {\rho }}(G)$ ;

• • every nondegenerate relative invariant $f \in \mathbb {R}(X)$ induces an isomorphism $f^{-1} \mathop {}\!\mathrm {d} f: X^+ \stackrel {\sim }{\rightarrow } \check {X}^+$ of homogeneous G-spaces (see [Reference Kimura14, Th. 2.16]).

Again, for split G, these properties are reviewed in [Reference Li22, Th. 6.2.4], and the general case follows by Galois descent. We summarize below.

2.2. Density bundles

Below is a review of the formalism of densities, following [Reference Li22, §3.1].

Let Y be any real smooth manifold. Roughly speaking, the densities on Y are objects which can be integrated. More generally, for each $t \in \mathbb {R}$ , there is a real line bundle $\mathcal {L}^t$ of t-densities on Y, with $\mathcal {L} := \mathcal {L}^1$ . Denote by $C_c(Y, \mathcal {L}^t)$ the space of continuous sections of $\mathcal {L}^t$ over Y of compact support. Likewise, we have the space $C^\infty (Y, \mathcal {L}^t)$ of $C^\infty $ -sections of $\mathcal {L}^t$ .

The line bundles $\mathcal {L}^t$ come with:

• • canonical pairings

  $$\begin{align*}\mathcal{L}^s \otimes \mathcal{L}^t \to \mathcal{L}^{s+t}, \quad s, t \in \mathbb{R}, \end{align*}$$
  and a trivialization of $\mathcal {L}^0$ , subject to the unity, associativity, and commutativity constraints;

• • the integration as a linear functional

  $$\begin{align*}\int_Y: C_c(Y, \mathcal{L}) \to \mathbb{C}, \quad \xi \mapsto \int_Y \xi. \end{align*}$$

These data can be constructed by the following recipe. Denote by $\Omega _Y$ the line bundle of differential $1$ -forms on Y. To $\bigwedge ^{\mathrm {max}} \Omega _Y$ corresponds the $\mathbb {R}^\times $ -torsor $\mathcal {G}$ on Y, whose local sections are nonvanishing differential forms of top degree. Let $t \in \mathbb {R}$ . Using the group homomorphism $|\cdot |^t: \mathbb {R}^\times \to \mathbb {R}^\times _{> 0} \subset \mathbb {R}$ , we form the following real line bundle on Y:

$$\begin{align*}\mathcal{L}^t := \mathcal{G} \overset{|\cdot|^t}{\times} \mathbb{R}. \end{align*}$$

Specifically, let $U \subset Y$ be an open subset and let $\omega $ be a nonvanishing continuous section of $\bigwedge ^{\mathrm {max}} \Omega _Y$ over U. It yields a section $|\omega |^t$ of $\mathcal {G} \overset {|\cdot |^t}{\times } \mathbb {R}^\times _{> 0}$ , whence a section of $\mathcal {L}^t$ . In general, sections of $\bigwedge ^{\mathrm {max}} \Omega _Y$ can be locally expressed as $f \omega $ where f is a continuous function on Y; one defines unambiguously the section

$$\begin{align*}|f\omega|^t := |f| \cdot |\omega|^t \end{align*}$$

of $\mathcal {L}^t$ . We have $|\omega |^{s+t} = |\omega |^s |\omega |^t$ , and so forth. The integration of $\xi \in C_c(Y, \mathcal {L})$ is then performed via local charts and partition of unity, reducing everything to Lebesgue integrals. In particular, every continuous section of $\mathcal {L}$ gives rise to a Radon measure on Y.

Consequently, it makes sense to define the $L^p$ -space of sections of $\mathcal {L}^{1/p}$ as the completion of $C_c(Y, \mathcal {L}^{1/p})$ with respect to $\|\xi \|_{L^p} := \left ( \int _Y |\omega |^p \right )^{1/p}$ , where $1 \leq p \leq +\infty $ .

Below are some further properties of $\mathcal {L}^t$ .

• • Pullback: this is compatible with the pullback of differential forms. Given a morphism $\nu : Y \to Z$ and a section $\xi $ of $\mathcal {L}^t_Z$ , we write $\nu ^* \xi $ for the resulting section of $\mathcal {L}^t_Y$ .For $t=0$ , it is the pullback of functions, and for any differential form $\omega $ of top degree on Z, we have $\nu ^* |\omega |^t = |\nu ^* \omega |^t$ .

• • For any open subset $U \subset Y$ , we have $\mathcal {L}^t_Y |_U \simeq \mathcal {L}^t_U$ ; for any real analytic manifolds $Y_1, Y_2$ , we have $\mathcal {L}^t_{Y_1 \times Y_2} \simeq \mathcal {L}^t_{Y_1} \boxtimes \mathcal {L}^t_{Y_2}$ . Both isomorphisms are canonical.

• • Integration of densities satisfies the formula of change of variables

  $$\begin{align*}\int_Y \nu^* \xi = \int_X \xi, \quad \xi \in C_c(Y, \mathcal{L}). \end{align*}$$

• • If a Lie group H acts on the right of Y, then the bundles $\mathcal {L}^t$ have canonical H-equivariant structures.

Now, let $(G, \rho , X)$ as in Hypothesis 2.1. Every $\Omega \in \bigwedge ^{\mathrm {max}} \check {X}$ affords a translation-invariant t-density $|\Omega |^t$ . For every $g \in \operatorname {GL}(X)$ , we have $g^* |\Omega |^t = |g^* \Omega |^t = |\det g|^t |\Omega |^t$ . Most often, we will encounter the case $t = \frac {1}{2}$ , that is, the half-densities. If necessary, one can get rid of half-densities by the following observation.

We caution the reader that for homogeneous G-spaces in general, the density bundles are not necessarily equivariantly trivializable.

2.3. Fourier transform on Schwartz spaces

Given $(G, \rho , X)$ be as in Hypothesis 2.1, we follow the paradigm of §2.2 to define the following spaces.

• • $C^\infty (X^+) := C^\infty \left (X^+, \mathcal {L}^{1/2} \right )$ : the space of $C^\infty $ half-densities. It is a Fréchet space with respect to the standard topology as prescribed in [Reference Li22, §4.1] (see also [Reference Trèves32, §10, Exam. I] for the scalar-valued case).

  The seminorms in question involve a continuous metric on $\mathcal {L}^{1/2}$ , whose choice is immaterial since we consider only its supremum over compact subsets. In our case, one can even trivialize $\mathcal {L}^{1/2}$ to get a more canonical choice.

• • $L^2(X^+)$ : the Hilbert space of $L^2$ half-densities on $X^+(\mathbb {R})$ . It is the same as $L^2(X)$ .

• • $\mathcal {S}(X)$ : the Fréchet space of Schwartz–Bruhat sections in $L^2(X)$ . The relation of $\mathcal {S}(X)$ to the usual scalar-valued Schwartz–Bruhat space $\mathcal {S}_0(X)$ is straightforward: $\mathcal {S}(X) = \mathcal {S}_0(X) |\Omega |^{1/2}$ for any $\Omega \in \bigwedge ^{\mathrm {max}} \check {X} \smallsetminus \{0\}$ . In particular, $\mathcal {S}(X)$ is a nuclear Fréchet space.

  The topology on $\mathcal {S}_0(X)$ is described in [Reference Trèves32, §10, Exam. IV].

The spaces $C^\infty (X^+)$ and $\mathcal {S}(X)$ are smooth $G(\mathbb {R})$ -representations and $L^2(X^+)$ is a unitary $G(\mathbb {R})$ -representation.

Define the Fourier transform for half-densities $\mathcal {F}_\psi : \mathcal {S}(X) \stackrel {\sim }{\rightarrow } \mathcal {S}(\check {X})$ as in [Reference Li22, §6.1]: it is an isomorphism between Fréchet spaces, and extends to an isomorphism $L^2(X) \stackrel {\sim }{\rightarrow } L^2(\check {X})$ satisfying $\|\mathcal {F}\xi \| = \sqrt {A(\psi )} \|\xi \|$ for some constant $A(\psi )> 0$ (see Definition 2.8). Below is a recap of the formulas.

Let $\langle \cdot , \cdot \rangle : \check {X} \times X \to \mathbb {R}$ be the canonical pairing between $\check {X}, X$ , and similarly for $\langle \cdot , \cdot \rangle : \bigwedge ^{\mathrm {max}} \check {X} \times \bigwedge ^{\mathrm {max}} X \to \mathbb {R}$ . Given $\Omega \in \bigwedge ^{\mathrm {max}} \check {X} \smallsetminus \{0\}$ , we take the $\Psi \in \bigwedge ^{\mathrm {max}} X$ with $\langle \Omega , \Psi \rangle = 1$ and define the Fourier transforms

(2.1)

It is readily seen that $\mathcal {F}_\psi $ is independent of the choice of $\Omega $ . By working with half-densities, $\mathcal {F}$ becomes $G(\mathbb {R})$ -equivariant (see [Reference Li22, Th. 6.1.5]) and we do not have to choose Haar measures.

When there is no confusion about additive characters, we shall write $\mathcal {F}$ instead of $\mathcal {F}_\psi $ .

Every $f \in \mathbb {R}[X]$ induces a continuous endomorphism $\xi \mapsto f\xi $ on $\mathcal {S}(X)$ , namely by pointwise multiplication. On the other hand, every $\check {f} \in \mathbb {R}[\check {X}]$ can be viewed as a differential operator of constant coefficients on $X(\mathbb {R})$ , which can act on $\mathcal {S}(X)$ as follows: express $\xi \in \mathcal {S}(X)$ as $\xi = \xi _0 |\Omega |^{1/2}$ as before. Hence, $\check {f} \xi _0$ make sense, and we put

(2.2) $$ \begin{align}\begin{gathered} \check{f} \xi := \left( \check{f} \xi_0 \right) |\Omega|^{1/2} \; \in \mathcal{S}(X), \\ \text{and the same for } \xi \in C^\infty(X^+), \text{ and so on.} \end{gathered}\end{align} $$

This is clearly continuous in $\xi $ and independent of the choice of $\Omega $ .

A key observation is that the G-action on $\mathbb {R}[\check {X}]$ coincides with the G-action on differential operators: indeed, it suffices to compare these actions on $\mathbb {R}[\check {X}]^{\deg = 1}$ .

The same constructions also apply to the dual side. In particular, $\mathbb {R}[X]$ acts on $\mathcal {S}(\check {X})$ via differential operators of constant coefficients. In order to fix notations, we record the following common sense.

Lemma 2.7. There exists a constant $c(\psi ) \in \sqrt {-1} \cdot \mathbb {R}^\times $ , depending only on $\psi $ , such that for every homogeneous $f \in \mathbb {R}[X]$ and every $\xi \in \mathcal {S}(X)$ , we have

$$\begin{align*}\mathcal{F}(f\xi) = c(\psi)^{\deg f} \cdot f \mathcal{F}(\xi). \end{align*}$$

The next issue is the dependence on $\psi $ . For $a \in \mathbb {R}^\times $ , write $\psi _a(t) = \psi (at)$ and let $\nu _a: \check {X} \to \check {X}$ be the map $y \mapsto ay$ . We have $\nu _a^* |\Psi |^{1/2} = |\nu _a^* \Psi |^{1/2} = |a|^{\dim X /2} |\Psi |^{1/2}$ for all $\Psi \in \bigwedge ^{\mathrm {max}} X$ . One infers from (2.1) that

(2.3) $$ \begin{align}\begin{aligned} (\mathcal{F}_{\psi_a, |\Omega|} \xi_0)(\check{x}) & = (\mathcal{F}_{\psi, |\Omega|} \xi_0)(a\check{x}), \quad \check{x} \in \check{X}, \\ \mathcal{F}_{\psi_a} \xi & = |a|^{- \dim X /2} \nu_a^* \left( \mathcal{F}_\psi \xi \right). \end{aligned}\end{align} $$

To $\psi $ and $|\Omega |$ is associated the dual Haar measure $|\Psi |'$ on $\check {X}$ characterized by

$$\begin{align*}\int_{\check{X}} \left| \mathcal{F}_{\psi, |\Omega|} \xi_0 \right|{}^2 |\Psi|' = \int_X \left| \xi_0 \right|{}^2 |\Omega|. \end{align*}$$

• • If $|\Omega |$ is replaced by $t|\Omega |$ where $t \in \mathbb {R}_{> 0}$ , then $|\Psi |'$ gets multiplied by $t^{-1}$ .

• • If $\psi $ is replaced by $\psi _a$ , then $|\Psi |'$ gets multiplied by $|a|^{\dim X}$ .

Definition 2.8. Let $\Omega \in \bigwedge ^{\mathrm {max}} \check {X} \smallsetminus \{0\}$ ; take the $\Psi \in \bigwedge ^{\mathrm {max}} X$ such that $\langle \Psi , \Omega \rangle = 1$ . Define the $|\Psi |'$ as before, with respect to $|\Omega |$ and $\psi $ . Set

$$\begin{align*}A(\psi) := \frac{|\Psi|}{|\Psi|'} \; \in \mathbb{R}_{> 0}. \end{align*}$$

By the foregoing discussions, $A(\psi )$ depends only on $\psi $ and X. Furthermore,

$$\begin{align*}A(\psi_a) = |a|^{-\dim X} A(\psi), \quad a \in \mathbb{R}^\times. \end{align*}$$

Using the dual Haar measures $|\Omega |$ and $|\Psi |'$ , the Fourier inversion formula reads

$$ \begin{align*} \mathcal{F}_{-\psi, |\Psi|'} \mathcal{F}_{\psi, |\Omega|} = \mathrm{id}_{\mathcal{S}_0(X)}. \end{align*} $$

Proof. Take $\Omega $ , $\Psi $ , $|\Psi |'$ as above. Let $\xi = \xi _0 |\Omega |^{1/2} \in \mathcal {S}(X)$ . Apply (2.1) twice to see

$$ \begin{align*} & \mathcal{F}_{-\psi} \mathcal{F}_\psi \xi = \mathcal{F}_{-\psi} \left( \mathcal{F}_{\psi, |\Omega|} \xi_0 \cdot |\Psi|^{1/2} \right) = \left( \mathcal{F}_{-\psi, |\Psi|} \mathcal{F}_{\psi, |\Omega|} \xi_0 \right) \cdot |\Omega|^{1/2} \\ &\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad = \frac{|\Psi|}{|\Psi|'} \cdot \left( \mathcal{F}_{-\psi, |\Psi|'} \mathcal{F}_{\psi, |\Omega|} \xi_0 \right) \cdot |\Omega|^{1/2} = A(\psi) \xi , \end{align*} $$

as asserted.

Remark 2.11. These properties motivate us to define the self-dual version of $\mathcal {F}_\psi $ , namely,

$$\begin{align*}\mathcal{F}^{\mathrm{sd}}_\psi := A(\psi)^{-1/2} \mathcal{F}_\psi. \end{align*}$$

It satisfies $\mathcal {F}^{\mathrm {sd}}_{-\psi } \mathcal {F}^{\mathrm {sd}}_\psi = \mathrm {id}_{\mathcal {S}(X)}$ and extends to a $G(\mathbb {R})$ -equivariant isometry $L^2(X) \stackrel {\sim }{\rightarrow } L^2(\check {X})$ .

3.1. Coefficients of representations

Let $\pi $ be an SAF representation in the sense of [Reference Bernstein and Krötz4], also known as Casselman–Wallach representation; note that $V_\pi $ is nuclear. Following [Reference Li22, §4.1], we set

$$\begin{align*}\mathcal{N}_\pi(X^+) := \operatorname{Hom}_{G(\mathbb{R})}(\pi, C^\infty(X^+)), \end{align*}$$

where the $\operatorname {Hom}_{G(\mathbb {R})}$ is the continuous and $G(\mathbb {R})$ -equivariant $\operatorname {Hom}$ -space between continuous representations.

For $\eta \in \mathcal {N}_\pi (X^+)$ and $v \in V_\pi $ , we call $\eta (v) \in C^\infty (X^+)$ a generalize matrix coefficient of $\pi $ on $X^+(\mathbb {R})$ , with values in half-densities.

Remark 3.1. Let $C^\infty (X^+; \mathbb {C})$ denote the usual topological vector space of $C^\infty $ -functions on $X^+(\mathbb {R})$ . It is more common to consider scalar-valued generalized matrix coefficients arising from $\operatorname {Hom}_{G(\mathbb {R})}(\pi , C^\infty (X^+; \mathbb {C}))$ , yet there is little difference: Lemma 2.6 furnishes the isomorphism

$$ \begin{align*}\begin{aligned} \operatorname{Hom}_{G(\mathbb{R})}(\pi, C^\infty(X^+)) & \stackrel{\sim}{\rightarrow} \operatorname{Hom}_{G(\mathbb{R})}(\pi, C^\infty(X^+; \mathbb{C})) \\ \eta & \mapsto \eta_0 := \eta \cdot |\phi|^{1/4} |\Omega|^{-1/2} \end{aligned}\end{align*} $$

with $\phi $ , $\Omega $ as in Lemma 2.6.

Recall the following.

Let $C^\infty _c(X^+) := C^\infty _c(X^+, \mathcal {L}^{1/2})$ . As explained in [Reference Li22, §4.1] or [Reference Trèves32, §13], $C^\infty _c(X^+)$ carries a natural topology through

$$\begin{align*}C^\infty_c(X^+) = \varinjlim_{\substack{\Omega \subset X^+(\mathbb{R}) \\ \text{compact}}} C^\infty_\Omega(X^+), \end{align*}$$

where $C^\infty _\Omega (X^+) := \left \{ u \in C^\infty _c(X^+): \operatorname {Supp}(u) \subset \Omega \right \}$ carries the seminorms given by suprema of derivatives, using any continuous metric on $\mathcal {L}^{1/2}$ . It then becomes a smooth $G(\mathbb {R})$ -representation on an LF-space (= strict inductive limit of Fréchet spaces). The inclusion $C^\infty _c(X^+) \hookrightarrow \mathcal {S}(X)$ is equivariant and continuous. Upon choosing a volume form, these facts reduce to the well-known setting of scalar-valued functions. Elements of $C_c^\infty (X^+)^\vee $ are nothing but distributions on the open subset $X^+(\mathbb {R})$ of $X(\mathbb {R})$ , which fits into the classical picture if we choose volume forms.

Notice that $G(\mathbb {R})$ acts linearly on $C_c^\infty (X^+)^\vee $ , and the inclusion map $C^\infty (X^+) \hookrightarrow C^\infty _c(X^+)^\vee $ is $G(\mathbb {R})$ -equivariant.

Since $\dim _{\mathbb {C}} \mathcal {N}_\pi (X^+)$ is finite, one can talk about holomorphic or meromorphic families inside $\mathcal {N}_\pi (X^+)$ unambiguously. Fix a maximal compact subgroup $K \subset G(\mathbb {R})$ .

Proof. Clearly, (i) implies (ii). Now, assume (ii) and write $Z_\lambda (\eta , v, \xi ) := \int _{X^+(\mathbb {R})} \eta (v)|f|^\lambda \xi $ (temporarily, but see Proposition 6.5). Observe that if $\eta \in \mathcal {N}_\pi (X^+)$ satisfies $Z_\lambda (\eta , v, \xi ) = 0$ for all $(v, \xi ) \in V_\pi ^{K\text {-fini}} \times C^\infty _c(X^+)$ , then $\eta (v) = 0$ for all $v \in V_\pi ^{K\text {-fini}}$ ; hence, $\eta = 0$ by its continuity.

Since $\dim _{\mathbb {C}} \mathcal {N}_\pi (X^+)$ is finite, these linear functionals generate $\mathcal {N}_\pi (X^+)^\vee $ and there is a finite subset $F \subset V_\pi ^{K\text {-fini}} \times C^\infty _c(X^+)$ such that

$$ \begin{align*} \mathcal{N}_\pi(X^+) & \hookrightarrow \mathbb{C}^F \\ \eta & \mapsto \left( Z_\lambda(\eta, v, \xi) \right)_{(v, \xi) \in F}. \end{align*} $$

As $\omega \mapsto Z_\lambda (\eta _\omega , v, \xi )$ is holomorphic for each $(v, \xi )$ , the property (i) follows at once.

Using the embedding $X^+ \hookrightarrow X$ , we let $\mathcal {D}(X^+)$ act on the left of $C^\infty (X^+)$ as follows.

Definition 3.6. Let $u = u_0 |\Omega |^{1/2} \in C^\infty (X^+)$ where $u_0 \in C^\infty (X^+; \mathbb {C})$ and $\Omega \in \bigwedge ^{\mathrm {max}} \check {X} \smallsetminus \{0\}$ . For $D \in \mathcal {D}(X^+)$ , set

$$\begin{align*}Du := (Du_0) \cdot |\Omega|^{1/2}. \end{align*}$$

This makes $C^\infty (X^+)$ into a left $\mathcal {D}(X^+)$ -module, independently of the choice of $\Omega $ . The recipe is compatible with (2.2).

Write $D \mapsto {}^g D = gDg^{-1}$ for the left action of $g \in G$ on differential operators, and similarly for functions, volume forms, and so forth. It is routine to see that ${}^g u = {}^g u_0 \cdot |{}^g \Omega |^{1/2}$ and

$$ \begin{align*} {}^g (Du) & = {}^g (Du_0) \cdot |{}^g \Omega|^{1/2} = ({}^g D) ({}^g u_0) \cdot |{}^g \Omega|^{1/2} \\ & = ({}^g D) ({}^g u). \end{align*} $$

The case of real-analytic differential operators is completely analogous.

All the foregoing constructions apply to the dual triplet $(G, \check {\rho }, \check {X})$ as well. By Proposition 2.2 and the canonicity of density bundles, for any given $\pi $ , we can take a nondegenerate relative invariant f to obtain an isomorphism

where $(f^{-1} \mathop {}\!\mathrm {d} f)_*: C^\infty (X^+) \stackrel {\sim }{\rightarrow } C^\infty (\check {X}^+)$ is the transport of structure applied to half-densities.

3.2. Statement of the main theorems

The following constructions and statements are extracted from [Reference Li21], [Reference Li22].

Definition 3.8. Choose basic relative invariants $f_1, \ldots , f_r \in \mathbb {R}[X]$ as in §2.1, with eigencharacters $\omega _1, \ldots , \omega _r$ . For every $\lambda = \sum _{i=1}^r \omega _i \otimes \lambda _i \in \Lambda _{\mathbb {C}}$ , we write

$$\begin{align*}|f|^\lambda := \prod_{i=1}^r |f_i|^{\lambda_i}, \quad |\omega|^\lambda := \prod_{i=1}^r |\omega_i|^{\lambda}, \end{align*}$$

so that $|f|^\lambda : X(\mathbb {R}) \to \mathbb {R}_{\geq 0}$ has $G(\mathbb {R})$ -eigencharacter $|\omega |^\lambda $ .

For $\lambda = \sum _{i=1}^r \omega _i \otimes \lambda _i \in \Lambda _{\mathbb {C}}$ and $\kappa = \sum _{i=1}^r \omega _i \otimes \kappa _i \in \Lambda _{\mathbb {R}}$ , the notation $\operatorname {Re}(\lambda ) \underset {X}{\geq } \kappa $ signifies that $\operatorname {Re}(\lambda _i) \geq \kappa _i$ for all i; the notation $\operatorname {Re}(\lambda ) \underset {X}{\gg } 0$ signifies that $\operatorname {Re}(\lambda _i) \gg 0$ for all i.

Definition 3.9 (Generalized zeta integral)

Let $\pi $ be an SAF representation of $G(\mathbb {R})$ . For all $\eta \in \mathcal {N}_\pi (X^+)$ , $v \in V_\pi $ , $\xi \in \mathcal {S}(X)$ , and $\lambda \in \Lambda _{\mathbb {C}}$ with $\operatorname {Re}(\lambda ) \underset {X}{\gg } 0$ (see the discussion below), set

$$\begin{align*}Z_\lambda(\eta, v, \xi) := \int_{X^+(\mathbb{R})} \eta(v) |f|^\lambda \xi. \end{align*}$$

The integrand is a density on $X^+(\mathbb {R})$ ; hence, the integral makes sense. If we write $\xi = \xi _0 |\Omega |^{1/2}$ and $\eta (v) = \eta _0(v) |\phi |^{-1/4} |\Omega |^{1/2}$ (as in Remark 3.1), arrange that $|\phi |^{1/4} = |f|^{\lambda _0}$ for some $\lambda _0 \in \frac {1}{4} \Lambda _{\mathbb {Z}}$ , and consider the invariant measure $\mathop {}\!\mathrm {d} \mu := |\phi |^{-1/2} |\Omega |$ on $X^+(\mathbb {R})$ , then

$$ \begin{align*} Z_\lambda(\eta, v, \xi) & = \int_{X(\mathbb{R})} \eta_0(v) |f|^{\lambda - \lambda_0} \xi_0 |\Omega| \\ & = \int_{X^+(\mathbb{R})} \eta_0(v) |f|^{\lambda + \lambda_0} \xi_0 \mathop{}\!\mathrm{d} \mu. \end{align*} $$

We will also view $Z_\lambda (\eta , \cdot , \cdot )$ as a family of bilinear forms in $(v, \xi )$ . Implicit in the definition above is the convergence of $Z_\lambda (\eta , v, \xi )$ for $\operatorname {Re}(\lambda ) \underset {X}{\gg } 0$ . This is made precise in the following main result.

For the meaning of holomorphy for $\left (V_\pi \hat {\otimes } \mathcal {S}(X) \right )^\vee $ -valued functions, see §1.5.

Note that we do not assert that $L(\eta , \lambda )$ is the greatest common divisor of the zeta integrals.

In view of the results above, $Z_\lambda (\eta , v, \xi )$ may now be viewed as a meromorphic family of trilinear forms in $(\eta , v, \xi )$ on the whole $\Lambda _{\mathbb {C}}$ , which are jointly continuous (recall that $\dim _{\mathbb {C}} \mathcal {N}_\pi (X^+) < +\infty $ ).

For the dual triplet $(G, \check {\rho }, \check {X})$ , we also form the space $\mathcal {N}_\pi (\check {X}^+)$ . Since $\mathbf {X}^*_\rho (G) = \mathbf {X}^*_{\check {\rho }}(G)$ , the same $\Lambda _{\mathbb {C}}$ parameterizes both zeta integrals $Z_\lambda $ (on X) and $\check {Z}_\lambda $ (on $\check {X}$ ). Fix basic relative invariants $\check {f}_1, \ldots , \check {f}_r$ for $(G, \check {\rho }, \check {X})$ and write $|\check {f}|^\lambda := \prod _{i=1}^r |\check {f}_i|^{\lambda _i}$ , by which we define the zeta integrals $\check {Z}_\lambda $ . Observe that $|\omega |^\lambda : G(\mathbb {R}) \to \mathbb {R}^\times _{>0}$ does not depend on the eigencharacters $\omega _1, \ldots , \omega _r$ : it is determined by $\lambda \in \Lambda _{\mathbb {C}} \subset \mathbf {X}^*(G) \otimes \mathbb {C}$ , and thus the notation works uniformly for both $(G, \rho , X)$ and $(G, \check {\rho }, \check {X})$ .

Now comes the local functional equation. For $\pi $ as above and $\eta \in \mathcal {N}_\pi (X^+)$ , put

(3.1) $$ \begin{align} \eta_\lambda: v \mapsto \eta(v) |f|^\lambda, \end{align} $$

which belongs to $\mathcal {N}_{\pi \otimes |\omega |^\lambda }(X^+)$ for all $\lambda \in \Lambda _{\mathbb {C}}$ . The same definition applies to $\check {X}^+$ as well.

Theorem 3.13. Assume that $\pi $ is an SAF representation with central character. There is a meromorphic family of linear maps $\gamma (\pi , \lambda ): \mathcal {N}_\pi (\check {X}^+) \to \mathcal {N}_\pi (X^+)$ (i.e., its matrix entries are meromorphic in $\lambda $ ), depending on $\psi $ , such that

$$\begin{align*}\check{Z}_\lambda\left( \check{\eta}, v, \mathcal{F}\xi \right) = Z_\lambda\left( \gamma(\lambda, \pi)(\check{\eta}), v, \xi \right), \end{align*}$$

for all $\check {\eta } \in \mathcal {N}_\pi (\check {X}^+)$ , $v \in V_\pi $ , $\xi \in \mathcal {S}(X)$ , and all $\lambda \in \Lambda _{\mathbb {C}}$ off the poles of $Z_\lambda $ and $\check {Z}_\lambda $ .

Moreover:

1. (i) $\gamma (\pi , \lambda )$ is uniquely characterized by this equality;

2. (ii) if $L(\check {\eta }, \lambda )$ is as in Theorem 3.12, then $L(\check {\eta }, \lambda ) \gamma (\pi , \lambda )$ is holomorphic in $\lambda $ ;

3. (iii) $\gamma (\pi , \lambda + \mu )(\check {\eta })_\lambda = \gamma (\pi \otimes |\omega |^\lambda , \mu )(\check {\eta }_\lambda )$ for all $\mu , \lambda , \check {\eta }$ (see (3.1)).

Theorems 3.10, 3.12, and 3.13 are stated as axioms in [Reference Li22, Chap. 4] in an abstract setting; as to the special case of prehomogeneous vector spaces, see also [Reference Li22, Chap. 6]. Theorems 3.10 and 3.12 have also appeared in [Reference Li21] when $X^+$ is an essentially symmetric homogeneous G-space. It is routine to see that these assertions are unaffected by the choice of basic relative invariants $f_1, \ldots , f_r$ (see [Reference Li22, Lem. 4.5.4] for a precise statement).

The proofs of the theorems above will occupy the rest of this article.

We close this subsection by addressing the dependence of $\gamma $ -factors on the additive character $\psi $ of $\mathbb {R}$ . Let $a \in \mathbb {R}^\times $ and define:

• • $d(\lambda ) = \sum _{i=1}^r \lambda _i \cdot \deg \check {f}_i$ for all $\lambda = \sum _{i=1}^r \omega _i \otimes \lambda _i \in \Lambda _{\mathbb {C}}$ ;

• • $m_a: C^\infty (\check {X}^+) \to C^\infty (\check {X}^+)$ is the pullback along the automorphism $\nu _{a^{-1}}: y \mapsto a^{-1} y$ of $\check {X}^+(\mathbb {R})$ , so that $m_a$ is $G(\mathbb {R})$ -equivariant.

We write $\gamma (\pi , \lambda ) = \gamma (\pi , \lambda; \psi )$ , $\mathcal {F} = \mathcal {F}_\psi $ , and so forth. Denote the $\gamma $ -factor defined relative to $(G, \check {\rho }, \check {X})$ and $\psi $ by $\check {\gamma }(\pi , \lambda; \psi )$ .

Proof. Both assertions rely on the uniqueness of $\gamma $ -factors in Theorem 3.13. Assertion (i) results from Proposition 2.10. As to (ii), we apply (2.3) to see that when $\operatorname {Re}(\lambda ) \underset {\check {X}}{\gg } 0$ ,

$$ \begin{align*} &\check{Z}_\lambda\left( \check{\eta}, v, \mathcal{F}_{\psi_a} \xi \right) = |a|^{-\dim X /2} \int_{\check{X}^+(\mathbb{R})} \check{\eta}(v) |\check{f}|^\lambda \nu_a^* (\mathcal{F}_\psi \xi) \\ &\qquad\qquad = |a|^{-\dim X /2} \int_{\check{X}^+(\mathbb{R})} \nu_{a^{-1}}^* \left( \check{\eta}(v) | \check{f} |^\lambda \right) \mathcal{F}_\psi \xi \quad (\because\ \text{change of variables}) \\ &\qquad\qquad\qquad\qquad\qquad = |a|^{-d(\lambda) - (\dim X /2)} \int_{\check{X}^+(\mathbb{R})} m_a (\check{\eta}(v)) \cdot |\check{f}|^\lambda \cdot \mathcal{F}_\psi \xi \\ &\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad = Z_\lambda \left( |a|^{-d(\lambda) - (\dim X /2)} \gamma(\lambda, \pi; \psi) \circ m_a (\check{\eta}), v, \xi \right). \end{align*} $$

The equality extends by meromorphic continuation, and (ii) follows.

3.3. Examples

In all the examples below, we fix $\Omega \in \bigwedge ^{\mathrm {max}} \check {X}$ , $\Psi \in \bigwedge ^{\mathrm {max}} X$ with $\langle \Omega , \Psi \rangle = 1$ .

Example 3.16 (Sato–Shintani)

Let $(G, \rho , X)$ be as in Hypothesis 2.1, but take $\pi = \mathbf {1}$ , the trivial representation. Decompose $X^+(\mathbb {R})$ into $G(\mathbb {R})$ -orbits $\bigsqcup _{i=1}^k O_i$ . For $1 \leq i \leq k$ , let $c_i$ be the function on $X^+(\mathbb {R})$ , which is $1$ on $O_i$ and zero elsewhere. Take an invariant half-density $|\phi |^{-1/4} |\Omega |^{1/2}$ as in Lemma 2.6, with $|\phi |^{1/4} = |f|^{\lambda _0}$ where $\lambda _0 \in \frac {1}{4} \Lambda _{\mathbb {Z}}$ (see Definition 3.9). Then

By choosing a nondegenerate relative invariant to obtain $X^+ \stackrel {\sim }{\rightarrow } \check {X}^+$ , the $G(\mathbb {R})$ -orbits in $X^+(\mathbb {R})$ and $\check {X}^+(\mathbb {R})$ are in bijection, both labeled by $\{1, \ldots , k\}$ . In particular, we can define $\check {\eta }_1, \ldots , \check {\eta }_k$ .

For $\phi $ , $\lambda _0$ as above, $1 \leq i \leq k$ and $\operatorname {Re}(\lambda ) \underset {X}{\gg } 0$ , we obtain

$$ \begin{align*} Z_\lambda\left( \eta_i, 1, \xi \right) = \int_{O_i} |f|^{\lambda - \lambda_0} \xi_0 |\Omega|. \end{align*} $$

By recalling the definition of $\lambda _0$ and identifying $\Lambda _{\mathbb {C}}$ with $\mathbb {C}^r$ by the basis $\omega _1, \ldots , \omega _r$ , we see that $Z_\lambda \left ( \eta _i, 1, \xi \right )$ equals the archimedean local zeta integral $Z_i(\lambda + \lambda _0, \xi _0)$ defined in [Reference Sato27, §1.4].

For $\check {X}$ , we have $\check {Z}_i(\cdots )$ as well. Observe that:

• • the avatar of $\lambda _0$ for $\check {X}$ is $-\lambda _0$ ;

• • $\mathbf {X}^*_\rho (G) \otimes \mathbb {C} = \mathbf {X}^*_{\check {\rho }}(G) \otimes \mathbb {C}$ , but their isomorphisms to $\mathbb {C}^r$ induced by basic eigencharacters differ by $-1$ , by Proposition 2.3.

Use the standard additive character $\psi $ in Example 2.9 to perform Fourier transform. Together with the observations above, the theorem $\mathbf {R}$ in [Reference Sato27, p. 471] gives meromorphic functions $\left ( \Gamma _{ij} \right )_{1 \leq i,j \leq k}$ such that for all $\xi = \xi _0 |\Psi |^{1/2} \in \mathcal {S}(\check {X})$ ,

$$\begin{align*}Z_i \left( \lambda, \mathcal{F} \xi_0 \right) = \sum_{j=1}^k \Gamma_{ij}(\lambda - 2\lambda_0) \check{Z}_j \left( \lambda - 2\lambda_0, \xi_0 \right). \end{align*}$$

This can be rewritten as

$$\begin{align*}Z_{\lambda - \lambda_0} \left( \eta_i, 1, \mathcal{F} \xi \right) = \sum_{j=1}^k \Gamma_{ij}(\lambda - 2\lambda_0) \check{Z}_{\lambda - \lambda_0} \left( \check{\eta}_j, 1, \xi \right). \end{align*}$$

In our framework, $\check {\gamma }(\lambda , \mathbf {1}; \psi )$ is thus represented by the matrix $\left ( \Gamma _{ij}(\lambda - \lambda _0) \right )_{1 \leq i, j \leq k}$ of meromorphic functions, with respect to the bases $\{\eta _i\}_i$ and $\{\check {\eta }_j\}_j$ . Proposition 3.14 implies that $\gamma (\lambda , \mathbf {1}; \psi ) = \gamma (\lambda , \mathbf {1}, \psi _{-1}) \circ m_{-1}$ is represented by the matrix $\left ( \Gamma _{ij}(\lambda - \lambda _0) \right )_{1 \leq i, j \leq k}^{-1} \cdot P_{-1}$ , where $P_{-1}$ is the matrix corresponding to the permutation of the $G(\mathbb {R})$ -orbits in $\check {X}^+(\mathbb {R})$ induced by $y \mapsto -y$ .

Example 3.17 (Godement–Jacquet)

Let D be a central simple $\mathbb {R}$ -algebra of dimension $n^2$ , and let $G = D^\times \times D^\times $ act on $X := D$ , by

This is a regular prehomogeneous vector space with $X^+ = D^\times $ , which is spherical (in fact, the group case of a symmetric space). The relative invariants are generated by the reduced norm $\mathrm {Nrd}$ , up to $\mathbb {R}^\times $ . Accordingly, $\mathbf {X}^*_\rho (G)$ is generated by $(g, h) \mapsto \mathrm {Nrd}(h)^{-1} \mathrm {Nrd}(g)$ .

We may identify X with $\check {X}$ via the perfect pairing $(x, y) \mapsto \mathrm {Trd}(xy)$ on $X \times X$ , where $\mathrm {Trd}$ is the reduced trace. As discussed in [Reference Li22, Lem. 6.4.1], $(G, \check {\rho }, \check {X})$ then becomes X with the flipped action , and $(G, \rho , X)$ is regular. In fact, $\mathrm {Nrd}$ is nondegenerate, and the induced equivariant isomorphism $X^+ \stackrel {\sim }{\rightarrow } \check {X}^+$ is $x \mapsto x^{-1}$ (cf. [Reference Li22, Prop. 6.4.2]). We still write $X^+$ and $\check {X}^+$ in order to distinguish the G-actions. Note that $\mathrm {Nrd}$ is a basic relative invariant for both X and $\check {X}$ , but with opposite eigencharacters.

The irreducible SAF representations $\pi $ with $\mathcal {N}_\pi (X^+) \neq \{0\}$ take the form $\sigma \boxtimes \check {\sigma }$ , where $\sigma $ is an irreducible SAF representation of $D^\times (\mathbb {R})$ . Ditto for $\mathcal {N}_\pi (\check {X}^+)$ . Note that $|\det |^{-n} |\Omega |$ defines a Haar measure on $D^\times (\mathbb {R})$ . The spaces $\mathcal {N}_{\sigma \boxtimes \check {\sigma }}(X^+)$ and $\mathcal {N}_{\sigma \boxtimes \check {\sigma }}(\check {X}^+)$ are spanned, respectively, by matrix coefficient maps

$$ \begin{align*} \eta_{\Omega}: v \otimes \check{v} & \mapsto \langle \check{v}, \pi(\cdot) v \rangle |\det|^{-n/2} |\Omega|^{1/2}, \\ \check{\eta}_{\Psi}: v \otimes \check{v} & \mapsto \langle \check{\pi}(\cdot) \check{v}, v \rangle |\det|^{-n/2} |\Psi|^{1/2}. \end{align*} $$

With these choices, we write $\xi = \xi _0 |\Omega |^{1/2} \in \mathcal {S}(X)$ , $\check {\xi } = \check {\xi }_0 |\Psi |^{1/2} \in \mathcal {S}(\check {X})$ , and let $Z^{\mathrm {GJ}}(\cdots )$ (resp. $\gamma ^{\mathrm {GJ}}(\cdots )$ ) stand for the usual Godement–Jacquet zeta integrals in [Reference Goldfeld and Hundley11, (15.4.3)] (resp. the usual Godement–Jacquet $\gamma $ -factors). Here, we choose the standard additive character $\psi $ as in Example 2.9. It turns out that

$$ \begin{align*} Z_\lambda \left( \eta_{\Omega}, v \otimes \check{v}, \xi \right) & = Z^{\mathrm{GJ}}\left( \lambda + \frac{1}{2}, \langle \check{v}, \pi(\cdot) v \rangle, \xi_0 \right), \\ \check{Z}_\lambda \left( \check{\eta}_{\Psi}, v \otimes \check{v}, \xi \right) & = Z^{\mathrm{GJ}}\left( -\lambda + \frac{1}{2}, \langle \check{\pi}(\cdot) \check{v}, v \rangle, \xi_0 \right), \\ \gamma(\sigma \boxtimes \check{\sigma}, \lambda)\left( \check{\eta}_{\Psi} \right) & = \gamma^{\mathrm{GJ}}\left( \lambda + \frac{1}{2}, \sigma \right) (\eta_\Omega) , \end{align*} $$

so that functional equation in Theorem 3.13 reduces to the usual Godement–Jacquet functional equation. We refer to [Reference Li22, §6.4] for detailed explanations when D is split; the general case is analogous.

3.4. The complex case

Hypothesis 2.1 and the main theorems in §3.2 can all be formulated over $\mathbb {C}$ . The complex case can be reduced to the previous case over $\mathbb {R}$ as follows.

Let us write $\operatorname {Res} := \operatorname {Res}_{\mathbb {C} | \mathbb {R}}$ for the functor of restriction of scalars à la Weil along $\mathbb {C} | \mathbb {R}$ , applied to $\mathbb {C}$ -varieties, and so forth. If a triplet $(G, \rho , X)$ over $\mathbb {C}$ satisfies Hypothesis 2.1, so does $(\operatorname {Res} G, \operatorname {Res} \rho , \operatorname {Res} X)$ ; recall that $(\operatorname {Res} X)(\mathbb {R}) = X(\mathbb {C})$ , $(\operatorname {Res} X^+)(\mathbb {R}) = X^+(\mathbb {C})$ , and $(\operatorname {Res} G)(\mathbb {R}) = G(\mathbb {C})$ . There are canonical isomorphisms

If $f \in \mathbb {C}(X)$ is a relative invariant of eigencharacter $\omega \in \mathbf {X}^*_\rho (G)$ , then its norm $f \cdot \overline {f} \in \mathbb {R}(\operatorname {Res} X)$ has the eigencharacter in $\mathbf {X}^*_{\operatorname {Res} \rho }(\operatorname {Res} G)$ corresponding to $\omega $ .

Taking contragredient commutes with $\operatorname {Res}$ , since the pairing $\operatorname {tr}_{\mathbb {C}|\mathbb {R}} \circ \langle \cdot , \cdot \rangle : X(\mathbb {C}) \times \check {X}(\mathbb {C}) \to \mathbb {R}$ is perfect. Fix an additive character $\psi $ of $\mathbb {R}$ and take the additive character $\psi _{\mathbb {C}} := \psi \circ \operatorname {tr}_{\mathbb {C}|\mathbb {R}}$ of $\mathbb {C}$ . The Schwartz spaces for $X(\mathbb {C})$ and $(\operatorname {Res} X)(\mathbb {R})$ can be identified, and so do the Fourier transforms.

Finally, an SAF representation $\pi $ of $G(\mathbb {C})$ is the same as an SAF representation of $(\operatorname {Res} G)(\mathbb {R})$ on the same Fréchet space. The recipes above identifies $\mathcal {N}_\pi (X^+)$ (over $\mathbb {C}$ ) with $\mathcal {N}_\pi (\operatorname {Res} X^+)$ (over $\mathbb {R}$ ). Hence, the generalized matrix coefficients on $X^+(\mathbb {C})$ are the same as those on $(\operatorname {Res} X^+)(\mathbb {R})$ . All in all, the generalized zeta integral (Definition 3.9) for $(G, \rho , X)$ reduces immediately to the case for $(\operatorname {Res} G, \operatorname {Res} \rho , \operatorname {Res} X)$ , and the theorems in §3.2 carry over verbatim.

4.1. Proof of convergence

Fix $\eta \in \mathcal {N}_\pi (X^+)$ and write $\eta = \eta _0 |\phi |^{-1/4} |\Omega |^{1/2}$ for some chosen $\Omega \in \bigwedge ^{\mathrm {max}} \check {X} \smallsetminus \{0\}$ .

The argument for Theorem 3.10 is the same as that of [Reference Li21, Th. 6.6.4]. It is based on the following result. Some terminologies from real algebraic geometry such as Nash functions will be needed; we refer to [Reference Bochnak, Coste and Roy5, Chap. 8] for details.

Proposition 4.1. There exist a continuous seminorm $q: V_\pi \to \mathbb {R}_{\geq 0}$ and a Nash function $p: X^+(\mathbb {R}) \to \mathbb {R}_{\geq 0}$ , such that

$$ \begin{align*} |\eta_0(v)(x)| \leq q(v) p(x), \quad v \in V_\pi, \; x \in X^+(\mathbb{R}). \end{align*} $$

Proof. First, by [Reference Li23, Th. 10.5], there exists a weight function $w: X^+(\mathbb {R}) \to \mathbb {R}_{\geq 1}$ together with a continuous seminorm $q: V_\pi \to \mathbb {R}_{\geq 0}$ such that:

• • w is continuous and subanalytic,

• • $\{x \in X^+(\mathbb {R}): w(x) \leq B \}$ is compact for all $B> 0$ ,

• • $|\eta _0(v)(x)| \leq w(x) q(v)$ for all $v \in V_\pi $ and $x \in X^+(\mathbb {R})$ .

In fact, this can be deduced from the moderate growth of SAF representations. It is also deducible from the finer results in [Reference Knop, Krötz and Schlichtkrull17].

Next, we embed X as an open dense subset of a smooth projective $\mathbb {R}$ -variety $\overline {X}$ and let $\partial \overline {X} := \overline {X} \smallsetminus X^+$ . By [Reference Li23, Prop. 7.5], $1/w$ extends uniquely to a subanalytic continuous function $\overline {X}(\mathbb {R}) \to \mathbb {R}_{\geq 0}$ , still denoted as $1/w$ , whose zero locus is exactly $\partial \overline {X}(\mathbb {R})$ .

Note that $\overline {X}(\mathbb {R})$ is affine in the real sense (see [Reference Bochnak, Coste and Roy5, Th. 3.4.4]). Therefore, $\partial \overline {X}(\mathbb {R})$ is also the zero locus of some polynomial function $p_0: \overline {X}(\mathbb {R}) \to \mathbb {R}_{\geq 0}$ (see [Reference Bochnak, Coste and Roy5, Prop. 2.1.3]). We claim that there exist constants $a \in \mathbb {Z}_{\geq 1}$ and $C \in \mathbb {R}_{> 0}$ such that

$$\begin{align*}p_0 \leq C \cdot (1/w)^a \quad \text{over}\; \overline{X}(\mathbb{R}). \end{align*}$$

Indeed, this is due to the compactness of $\overline {X}(\mathbb {R})$ and Łojasiewicz’s inequality for subanalytic functions (see [Reference Li23, Th. 6.4]).

All in all, we have

$$\begin{align*}|\eta_0(v)(x)| \leq w(x) q(v) \leq \left( C p_0^{-1} \right)^{1/a} q(v), \quad x \in X^+(\mathbb{R}). \end{align*}$$

Notice that $C p_0^{-1}$ and its ath root are positive Nash functions on $X^+(\mathbb {R})$ . This completes the proof.

The notations below are the same as those in Theorem 3.10.

Proof of Theorem 3.10. As in Definition 3.9, we write

$$\begin{align*}Z_\lambda(\eta, v, \xi) = \int_{X(\mathbb{R})} \eta_0(v) |f|^{\lambda - \lambda_0} \xi_0 |\Omega|, \end{align*}$$

where $\xi _0 \in \mathcal {S}_0(X)$ .

By [Reference Li21, Lem. 6.6.5], whose proof applies under our Hypothesis 2.1, there exist $\mu \in \Lambda _{\mathbb {R}}$ and a Nash function $p_1$ on $X(\mathbb {R})$ such that $|f|^\mu p \leq p_1$ . Hence,

$$\begin{align*}|f|^{\operatorname{Re}(\lambda) - \lambda_0} p \leq |f|^{\operatorname{Re}(\lambda) - \lambda_0 - \mu} p_1. \end{align*}$$

Therefore, Proposition 4.1 implies

(4.1) $$ \begin{align} \left| \eta_0(v) |f|^{\lambda - \lambda_0} \xi_0 \right| \leq q(v) |f|^{\operatorname{Re}(\lambda) - \lambda_0 - \mu} p_1 |\xi_0|. \end{align} $$

We claim that, when $\operatorname {Re}(\lambda )$ is constrained in some compact subset $C \subset \Lambda _{\mathbb {R}}$ satisfying $\theta \underset {X}{\geq } \lambda _0 + \mu $ for all $\theta \in C$ , the function $|f|^{\operatorname {Re}(\lambda ) - \lambda _0 - \mu }$ is uniformly bounded by a polynomial function. To see this, take $\lambda ^* = \sum _{i=1}^r 2\lambda ^*_i \otimes \omega _i \in 2\Lambda _{\mathbb {Z}}$ such that $\lambda ^* \underset {X}{\geq } \theta - \lambda _0 - \mu $ for all $\theta \in C$ . Then

$$\begin{align*}|f|^{\operatorname{Re}(\lambda) - \lambda_0 - \mu} \leq \prod_{i=1}^r \left( 1 + |f_i|^{2\lambda^*_i} \right). \end{align*}$$

On the other hand, $p_1$ is Nash over $X(\mathbb {R})$ , and by [Reference Bochnak, Coste and Roy5, Prop. 2.6.2], every Nash function on $X(\mathbb {R})$ is bounded by some polynomial. It follows that the right-hand side of (4.1) is integrable over $X(\mathbb {R})$ after multiplied by $|\Omega |$ , when $\operatorname {Re}(\lambda ) \underset {X}{\gg } 0$ . The implied lower bound $\kappa $ of $\operatorname {Re}(\lambda )$ depends only on $\pi $ and $\eta $ . As $\dim _{\mathbb {C}} \mathcal {N}_\pi (X^+)$ is finite, $\kappa $ can even be made uniform in $\eta $ .

When $(v, \xi )$ is fixed, the holomorphy in $\lambda $ and the boundedness in vertical strips in the range of convergence follow from (4.1) and the bound on $|f|^{\operatorname {Re}(\lambda ) - \lambda _0 - \mu }$ just obtained.

In view of the topology on $\mathcal {S}(X)$ , the continuity of $Z_\lambda (\eta , v, \xi )$ in $\xi $ follows easily from (4.1). The continuity in v follows from that of $q(\cdot )$ . Since $V_\pi $ and $\mathcal {S}(X)$ are Fréchet, joint continuity in $(v, \xi )$ follows.

4.2. Proof of meromorphic continuation

Consider the sheaves $\mathscr {D}_{X^+}$ on $X^+$ and $\mathscr {D}_{X^+_{\mathbb {C}}}$ on $X^+_{\mathbb {C}}$ (recall §1.5). Any distribution u on $X^+(\mathbb {R})$ generates a $\mathscr {D}_{X^+}$ -module $\mathscr {D}_{X^+} \cdot u$ , which complexifies into $\mathscr {D}_{X^+_{\mathbb {C}}} \cdot u$ on $X^+_{\mathbb {C}}$ . We refer to [Reference Li21], [Reference Li23] for more a more detailed review of algebraic $\mathscr {D}$ -modules, including especially the notion of holonomicity.

Fix a maximal compact subgroup $K \subset G(\mathbb {R})$ .

The notations below are the same as those in Theorem 3.12.

Proof of Theorem 3.12. The argument for meromorphic continuation is exactly the same as [Reference Li21, Ths. 6.8.2 and 6.8.4]. It proceeds in two stages.

First, for $v \in V_\pi ^{K\text {-fini}}$ , one employs the method of Bernstein–Sato b-functions as explained in [Reference Brylinski and Delorme8, Appendice]. The sole input here is the holonomicity established in Proposition 4.2. This step corresponds to [Reference Li21, Th. 6.8.2]; it also produces the holomorphic function $L(\eta , \lambda )$ .

Second, let $\mathcal {S}(G)$ be the algebra of Schwartz measures on $G(\mathbb {R})$ , which acts on $\mathcal {S}(X)$ and also on any SAF representation of $G(\mathbb {R})$ . One uses $V_\pi = \pi (\mathcal {S}(G)) V_\pi ^{K\text {-fini}}$ to treat general $v \in V_\pi $ . This corresponds to [Reference Li21, Th. 6.8.4], the main analytic device being the Gelfand–Shilov principle of [Reference Li21, Prop. 6.8.3]. Specifically, the recipe is

$$\begin{align*}v = \sum_{i=1}^m \pi(\Xi_i) v_i \implies LZ_\lambda \left(\eta, v, \xi \right) := \sum_{i=1}^m LZ_\lambda\left( \eta, v_i, \check{\Xi}_i \xi \right), \end{align*}$$

where $v_i \in V_\pi ^{K\text {-fini}}$ , $\Xi _i \in \mathcal {S}(G)$ , and $\check {\Xi }_i(g) = \Xi _i(g^{-1})$ . In [Reference Li21], this is shown to be well defined and compatible with the original definition in the range of convergence.

The joint continuity and $G(\mathbb {R})$ -invariance of the bilinear form $LZ_\lambda (\eta , \cdot , \cdot )$ are also established in [Reference Li21].

4.3. Order of tempered distributions

The results below will be applied to prove the functional equation.

Choose a basis for the $\mathbb {R}$ -vector space X. This gives rise to the dual basis of $\check {X}$ , the standard volume form $\Omega $ , and the standard norm $\|\cdot \|$ on $X(\mathbb {R}) \simeq \mathbb {R}^n$ . The elements of $\mathcal {S}(X)^\vee \simeq \mathcal {S}_0(X)^\vee $ can be viewed as tempered distributions on $X(\mathbb {R})$ .

Recall that the topology on $\mathcal {S}_0(X)$ is determined by the seminorms

$$\begin{align*}\|\xi_0\|_{a,b} := \sup_{\substack{|\underline{\alpha}| \leq a \\ |\underline{\beta}| \leq b}} \; \sup_{x \in X(\mathbb{R})} \left| x^{\underline{\beta}} \cdot \partial^{\underline{\alpha}} \xi(x) \right| , \quad a,b \in \mathbb{Z}_{\geq 0}, \end{align*}$$

where $\underline {\alpha } = (\alpha _1, \ldots , \alpha _n)$ , $\underline {\beta } = (\beta _1, \ldots , \beta _n)$ , $|\alpha | := \sum _i \alpha _i$ , and $\partial ^{\underline {\alpha }} = \partial _1^{\alpha _1} \cdots \partial _n^{\alpha _n}$ , $x^{\underline {\beta }} = x_1^{\beta _1} \cdots x_n^{\beta _n}$ are the standard terminologies of multi-indices.

Some basic facts:

• • Every tempered distribution has order $\leq (a, b)$ for sufficiently large $a, b$ .

• • If $a' \geq a$ and $b' \geq b$ , then having order $\leq (a, b)$ implies having order $\leq (a', b')$ .

The following is also well known.

Next, consider the tempered distributions $Z_\lambda (\eta , v, \cdot )$ for $\lambda $ in the range of convergence.

Proof. This stems from the estimate (4.1) in the proof of Theorem 3.10 and the subsequent discussions: with the notations therein, it suffices to take an even b such that

$$\begin{align*}|f(x)|^{\operatorname{Re}(\lambda) -\lambda_0 - \mu} p_1(x) \leq C (1 + \|x\|^2)^{b/2} \end{align*}$$

for all $x \in X(\mathbb {R})$ , where C is some constant.

For the next proposition, take a holomorphic function $L(\eta , \lambda )$ and define $LZ_\lambda (\eta , \cdot , \cdot )$ as in Theorem 3.12. We shall also fix a maximal compact subgroup $K \subset G(\mathbb {R})$ .

5.1. On certain Capelli operators

In this subsection, we take G to be a connected reductive $\mathbb {C}$ -group, and the algebraic varieties are taken over $\mathbb {C}$ . For any smooth G-variety Z, there is a natural homomorphism $\mathcal {U}(\mathfrak {g}) \to \mathcal {D}(Z)$ which restricts to $\mathcal {Z}(\mathfrak {g}) \to \mathcal {D}(Z)^G$ .

Now, consider a finite-dimensional $\mathbb {C}$ -vector space X with a right G-action, given by a homomorphism $\rho : G \to \operatorname {GL}(X)$ between algebraic groups.

Being multiplicity-free implies the existence of an open dense G-orbit $X^+ \subset X$ ; hence, $(G, \rho , X)$ is a prehomogeneous vector space. In contrast with Hypothesis 2.1, here $X^+$ is not necessarily affine and $(G, \rho , X)$ is not necessarily regular.

Fix a Borel subgroup $B \subset G$ and set $T := B/B_{\mathrm {der}}$ . Let X be a multiplicity-free G-space. There are decompositions

(5.1) $$ \begin{align} \begin{aligned} \mathbb{C}[X] & = \bigoplus_{\lambda \in \mathbf{X}^*(T)^+} \mathcal{P}_\lambda, \\ \mathbb{C}[\check{X}] & = \bigoplus_{\lambda \in \mathbf{X}^*(T)^+} \mathcal{D}_\lambda, \end{aligned}\end{align} $$

where

• • $\mathbf {X}^*(T)^+$ denotes the set of dominant weights in $\mathbf {X}^*(T)$ ,

• • $\mathcal {P}_\lambda $ is the simple G-submodule with lowest weight $-\lambda $ , occurring with multiplicity $\leq 1$ ,

and the decomposition of $\mathbb {C}[\check {X}]$ is obtained from that of $\mathbb {C}[X]$ by duality; in particular, $\mathcal {D}_\lambda $ is the simple G-submodule with highest weight $\lambda $ , with multiplicity $\leq 1$ . Thus, $\check {X}$ is also a multiplicity-free G-space.

Note that in [Reference Knop16], G acts on the left of X. One switches between left and right actions on X by $g^{-1} x = xg$ , and the left G-module $\mathbb {C}[X]$ remains unaffected. Ditto for $\mathbb {C}[\check {X}]$ .

Theorem 5.3 (See [Reference Howe and Umeda12] or [Reference Knop16])

For a multiplicity-free G-space X, we have an isomorphism of $\mathbb {C}$ -algebras

where we regard $p_\lambda \in \mathbb {C}[X]$ and $q_\lambda \in \mathbb {C}[\check {X}]$ as algebraic differential operators on X.

Notice that if G, X descend to a subfield of $\mathbb {C}$ , so does C.

We will mainly use the terms with $\dim \mathcal {P}_\lambda = 1 = \dim \mathcal {D}_\lambda $ . In other words, we consider relative invariants $p \in \mathbb {C}[X]$ and $q \in \mathbb {C}[\check {X}]$ with opposite eigencharacters $-\lambda $ and $\lambda $ . Then $p \otimes q$ are automatically G-invariant and $C(p \otimes q)$ is an instance of the Capelli operators introduced in [Reference Howe and Umeda12].

5.2. Knop’s Harish-Chandra isomorphism

For any smooth G-variety X, Knop [Reference Knop15, Th. 6.5] has defined a Harish-Chandra isomorphism which realizes $\mathfrak {Z}(X)$ as the coordinate algebra of some explicitly defined variety. Below we review the simpler case of multiplicity-free spaces, following [Reference Knop16].

Fix a multiplicity-free G-space X (Definition 5.1) over $\mathbb {C}$ , with open G-orbit $X^+$ . Fix a Borel subgroup $B \subset G$ , and let $T := B/B_{\mathrm {der}}$ . Let W be the corresponding abstract Weyl group acting on T (see [Reference Chriss and Ginzburg10, p. 137]). Define:

• • $\Lambda (X)^+ := \left \{ \lambda \in \subset \mathbf {X}^*(T): \mathcal {P}_\lambda \neq 0 \right \}$ where $\mathcal {P}_\lambda $ is as in (5.1), and let $\Lambda (X) \subset \mathbf {X}^*(T)$ be the subgroup generated by $\Lambda (X)^+$ ;

• • $\mathfrak {a}^*_X := \Lambda (X) \otimes \mathbb {R}$ , which is a subspace of $\mathfrak {a} := \mathbf {X}^*(T) \otimes \mathbb {R}$ ;

• • $\rho := \frac {1}{2} \sum \alpha \; \in \mathfrak {a}^*$ where $\alpha $ ranges over the positive roots;

• • $W_X$ : the little Weyl group of $X^+$ , which is a reflection group acting on $\mathfrak {a}_X^*$ and embeds into the normalizer $N_W\left ( \rho + \mathfrak {a}_X^* \right )$ .

By [Reference Knop16, Sect 3.2], the submonoid $\Lambda (X)^+$ in $\mathbf {X}^*(T)$ is generated by linearly independent elements $\chi _1, \ldots , \chi _m$ in $\mathbf {X}^*(T)$ , and its $\mathbb {Z}$ -span $\Lambda (X)$ is the group of all weights of B-eigenfunctions in $\mathbb {C}(\check {X})$ . We refer to [Reference Knop16] for further details.

For all $D \in \mathcal {D}(X)^G$ and $\lambda \in \Lambda (X)$ , the action of D on $\mathcal {P}_\lambda $ must be a scalar, say $c_D(\lambda )$ , due to multiplicity-freeness. We obtain a map $\mathcal {D}(X)^G \to \mathrm {Maps}(\Lambda (X)^+, \mathbb {C})$ , mapping D to $c_D$ .

Theorem 5.4 (Knop [Reference Knop16, Sect 4.8])

For every $D \in \mathcal {D}(X)^G$ , the function $c_D$ extends uniquely to a polynomial $c_D \in \mathbb {C}[\mathfrak {a}_X^*]$ , and the map

is an injective homomorphism of $\mathbb {C}$ -algebras, with image equal to $\mathbb {C}[\rho + \mathfrak {a}_X^*]^{W_X}$ or equivalently $\mathbb {C}[\left ( \rho + \mathfrak {a}_X^* \right ) /\!/ W_X]$ , where $/\!/$ denotes the categorical quotient.

Remark 5.5. Whenever $\mathcal {D}(X)^G$ acts on some $\mathbb {C}$ -vector space V and $v \in V$ is a joint generalized eigenvector therein, we may attach an infinitesimal character $\chi _v$ to v; it is an element of $\left ( \rho + \mathfrak {a}_X^* \right ) /\!/ W_X$ . Specifically, for all $D \in \mathcal {D}(X)^G$ , we have

$$\begin{align*}N \gg 0 \implies \left(D - \mathrm{HC}(D)(\chi_v) \cdot \mathrm{id}_V \right)^N v = 0. \end{align*}$$

As in §2.1, define $\mathbf {X}^*_\rho (G) \subset \mathbf {X}^*(G)$ to be the lattice of eigencharacters of relative invariants. In fact, $\mathbf {X}^*_\rho (G) \subset \Lambda (X)^{W_X}$ . Every relative invariant can be viewed as an algebraic differential operator on $X^+$ of order zero.