2.1. Hyperbolic geometry

Let us recall some basic facts about hyperbolic surfaces, referring the reader to Borthwick’s book [Reference Borthwick1] for a comprehensive discussion. One of the standard models for the hyperbolic plane is the Poincaré half-plane

\begin{equation*} {\mathbb{H}}^2=\{ x+iy\in {\mathbb{C}}\ :\ y\gt 0\} \end{equation*}

endowed with its standard metric of constant curvature $-1$ ,

\begin{equation*} ds^2=\frac {dx^2+dy^2}{y^2}. \end{equation*}

The group of orientation-preserving isometries of $({\mathbb{H}}^2, ds)$ is isomorphic to $\mathrm{PSL}_2({\mathbb{R}})$ . It acts on the extended complex plane $\overline {{\mathbb{C}}} = {\mathbb{C}} \cup \{ \infty \}$ (and hence also on ${\mathbb{H}}^2$ ) by Möbius transformations

\begin{equation*} \gamma =\begin{pmatrix} a&b\\c&d\end{pmatrix}\in \mathrm{PSL}_2({\mathbb{R}}),\quad z\in \overline {{\mathbb{C}}} \Longrightarrow \gamma (z) = \frac {az+b}{cz+d}. \end{equation*}

A non-identity element $\gamma \in \mathrm{PSL}_2({\mathbb{R}})$ is either

• • hyperbolic if $\vert \mathrm{tr}(\gamma )\vert \gt 2$ , which implies that $\gamma$ has two distinct fixed points on the boundary $\partial {\mathbb{H}}^2$ ,

• • parabolic if $\vert \mathrm{tr}(\gamma )\vert =2$ , which implies that $\gamma$ has precisely one fixed point on $\partial {\mathbb{H}}^2$ , or

• • elliptic if $\vert \mathrm{tr}(\gamma )\vert \lt 2$ , which implies that $\gamma$ has precisely one fixed point in the hyperbolic plane ${\mathbb{H}}^2$ .

2.2. Hyperbolic surfaces and Fuchsian groups

Every hyperbolic surface $X$ is isometric to a quotient $\Gamma \backslash {\mathbb{H}}^2$ , where $\Gamma$ is a Fuchsian group, that is, a discrete subgroup $\Gamma \subset \mathrm{PSL}_2({\mathbb{R}})$ . A Fuchsian group $\Gamma$ is called

• • torsion-free if it contains no elliptic elements,

• • non-cofinite if the quotient $\Gamma \backslash {\mathbb{H}}^2$ has infinite-area,

• • non-elementary if it is generated by more than one element, and

• • geometrically finite if it is finitely generated, which is equivalent with $\Gamma \backslash {\mathbb{H}}^2$ being geometrically and topologically finite.

The Fuchsian groups $\Gamma$ considered in this paper satisfy all the above conditions. The limit set $\Lambda$ of $X$ is a fractal subset of the boundary $\partial {\mathbb{H}}^2 \cong \mathbb{R}\cup \{ \infty \}$ , consisting of accumulation points of all orbits of the action of $\Gamma$ on ${\mathbb{H}}^2$ . It has a Cantor-like structure and its Hausdorff dimension, denoted by $\delta$ , lies strictly between $0$ and $1$ .

Furthermore, $\Gamma$ is called convex cocompact if it is finitely generated and if it contains neither parabolic nor elliptic elements. This is equivalent with the convex core of $X = \Gamma \backslash {\mathbb{H}}^2$ being compact. By a result of Button [Reference Button6], every infinite-area, convex cocompact hyperbolic surface $X$ can be realized as the quotient of ${\mathbb{H}}^2$ by a so-called Schottky group $\Gamma$ , which we will define in Section 2.6 below, see also [Reference Borthwick1, Theorem 15.3].

Since we work exclusively with torsion-free Fuchsian groups, the distinction between $\mathrm{PSL}_2({\mathbb{R}})$ and $\mathrm{SL}_2({\mathbb{R}})$ is immaterial. Hence, we will use $\mathrm{SL}_2({\mathbb{R}})$ throughout this paper.

2.3. Spectral theory of infinite-area hyperbolic surfaces

Let us review some aspects of the spectral theory of infinite-area hyperbolic surfaces. Again, we refer the reader to [Reference Borthwick1] for an in-depth account of the material given here. The $ L^{2}$ -spectrum of the Laplace–Beltrami operator $\Delta _X$ on an infinite-area hyperbolic surface $X$ was described by Lax–Phillips [Reference Lax and Phillips26] and Patterson [Reference Patterson32] as follows:

• • The absolutely continuous spectrum is equal to $ [1/4, \infty )$ .

• • The pure point spectrum is finite and contained in the interval $(0,1/4)$ . In particular, there are no eigenvalues embedded in the continuous spectrum.

• • If $\delta \leqslant 1/2$ , then the pure point spectrum is empty. If $\delta \gt 1/2$ , then $\lambda _0(X) = \delta (1-\delta )$ is the smallest eigenvalue.

In light of these facts, the resolvent operator

\begin{equation*} R_X(s) \stackrel {\text{def}}{=} \big ( \Delta _X - s(1-s)\big )^{-1}\colon L^2(X) \to L^2(X) \end{equation*}

is defined for all $s\in {\mathbb{C}}$ with $ \mathrm{Re}(s)\gt 1/2$ and $s(1-s)$ not being an $L^2$ -eigenvalue of $\Delta _X$ . From Guillopé–Zworski [Reference Guillopé and Zworski14], we know that the resolvent extends to a meromorphic family

(15) \begin{equation} R_X(s) \colon C_{c}^{\infty }(X) \to C^{\infty }(X) \end{equation}

on $\mathbb{C}$ with poles of finite rank. The poles of $R_X(s)$ are called the resonances of $ X$ and the multiplicity of a resonance $\zeta$ is the rank of the residue operator of $R_X(s)$ at $s=\zeta$ . We denote by $\mathcal{R}(X)$ the set multiset of resonances of $X$ repeated according to multiplicities. Resonances are contained in the half-plane $\mathrm{Re}(s)\leqslant \delta$ . There is a simple resonance at $s=\delta$ ; there are no other resonances on the vertical line $\mathrm{Re}(s)=\delta$ . Any resonance $s$ on the half-plane $\mathrm{Re}(s)\gt \frac {1}{2}$ corresponds to an eigenvalue $\lambda = s(1-s)$ of $\Delta _X$ , see Figure 2. In other words, the set $\Omega (X)$ defined in the introduction can be re-expressed as

\begin{equation*} \Omega (X) \stackrel {\text{m}}{=} \mathcal{R}(X) \cap \left \{ \mathrm{Re}(s) \gt \frac {1}{2} \right \}. \end{equation*}

In particular, $\Omega (X)$ is empty for $\delta \leqslant \frac {1}{2}$ .

Figure 2.

The $s$ -plane: distribution of resonances for infinite-area $ \Gamma \backslash {\mathbb{H}}^2$ in the case $ \delta \gt \frac {1}{2}$ .

2.4. Twisted Selberg zeta function

Let $\Gamma \subset \mathrm{PSL}_2({\mathbb{R}})$ be a finitely generated Fuchsian group. The set of prime periodic geodesics on $X=\Gamma \backslash {\mathbb{H}}^2$ is bijective to the set $[\Gamma ]_{\mathsf{prim}}$ of $\Gamma$ -conjugacy classes of primitive hyperbolic elements in $\Gamma$ . We denote by $\ell (\gamma )$ the length of the geodesic corresponding to the conjugacy class $[\gamma ] \in [\Gamma ]_{\mathsf{prim}}$ .

The Selberg zeta function is defined for $\mathrm{Re}(s) \gt \delta$ by the infinite product

(16) \begin{equation} Z_{\Gamma }(s)\stackrel {\text{def}}{=}\prod _{k=0}^\infty \prod _{[\gamma ] \in [\Gamma ]_{\mathsf{prim}}}\left ( 1 - e^{-(s+k)\ell (\gamma )}\right ), \end{equation}

and it has a meromorphic continuation to $s\in \mathbb{C}.$

There is a beautiful correspondence between resonances of $X$ and the zeros of $Z_\Gamma (s)$ established in [Reference Borthwick, Judge and Perry2, Reference Patterson and Perry33]. Concretely, the zero set of $Z_\Gamma (s)$ consists of the following:

• • “Topological” zeros at the non-positive integers. The topological zero at $s=-k$ with $k\in \mathbb{N}_0$ has multiplicity $-(2k+1)\chi (X).$

• • The set $\mathcal{R}(X)$ of resonances of $X$ .

Moreover, if $\Gamma$ has no parabolic elements, which is the case of interest to us, then $Z_\Gamma (s)$ has no poles. We note that the resonances and the topological zeros could potentially overlap at $-\mathbb{N}_0$ . However, outside this set, the multiplicity of a zero $s$ of $Z_\Gamma (s)$ matches its multiplicity as a resonance of $X$ . In particular, the first zero of $Z_\Gamma (s)$ (furthest to the right) is the simple resonance at $s=\delta$ . Therefore, any problem about resonances and eigenvalues can be rephrased as a question about the distribution of the zeros of the Selberg zeta function.

Let $(\rho ,V)$ be a finite-dimensional unitary representation of $\Gamma$ . Consider the twisted Selberg zeta function

(17) \begin{equation} Z_{\Gamma }(s,\rho )\stackrel {\text{def}}{=}\prod _{k=0}^\infty \prod _{[\gamma ] \in [\Gamma ]_{\mathsf{prim}}} \det \nolimits _V\left ( I_V - \rho (\gamma )e^{-(s+k)\ell (\gamma )}\right ). \end{equation}

Clearly, if $\rho = \textbf {1}_{\mathbb{C}}$ is the trivial, one-dimensional representation of $\Gamma$ , then (17) reduces to classical Selberg zeta function (16). Furthermore, the product definition (17) implies that

(18) \begin{equation} Z_\Gamma (s,\rho _1 \oplus \rho _2) = Z_\Gamma (s,\rho _1)Z_\Gamma (s,\rho _2), \end{equation}

where $\rho _1 \oplus \rho _2$ denotes the orthogonal direct sum of $\rho _1$ and $\rho _2$ .

2.5. Venkov–Zograf induction formula

One of the main properties of twisted Selberg zeta functions is the Venkov–Zograf induction formula [Reference Venkov41, Reference Venkov and Zograf42]. It states that for any finite-index subgroup $\Gamma '\subset \Gamma$ the following relationship holds true:

(19) \begin{equation} Z_{\Gamma '}(s) = Z_\Gamma (s,\lambda _{\Gamma /\Gamma '}), \end{equation}

where

(20) \begin{equation} \lambda _{\Gamma /\Gamma '} \stackrel {\text{def}}{=} \mathrm{Ind}_{\Gamma '}^{\Gamma }(\textbf {1}_{\Gamma '}) \end{equation}

is the induced representation of the trivial one-dimensional representation $\textbf {1}_{\Gamma '}$ of $\Gamma '$ to the larger group $\Gamma .$ We also refer to the more recent paper [Reference Fedosova and Pohl11] for a proof of this formula based on the Frobenius character formula.

Let $g_1, \ldots , g_n$ be a full set of representatives in $\Gamma$ of the left cosets in $\Gamma /\Gamma '$ , where $n=[\Gamma :\Gamma ']$ is the index of $\Gamma '$ in $\Gamma$ . The induced representation can be thought of as acting on the space

(21) \begin{equation} V_{\Gamma /\Gamma '} \stackrel {\text{def}}{=} \mathrm{span}_{\mathbb{C}} \{ g_1, \ldots , g_n \} = \left \{ \sum _{i=1}^n \alpha _i g_i : \alpha _1, \ldots , \alpha _n\in \mathbb{C} \right \}. \end{equation}

By definition, for each $\gamma \in \Gamma$ and for each $i\in [n]$ , there exists $\sigma (i)\in [n]$ and $\tilde {\gamma }\in \Gamma '$ such that $ \gamma g_i = g_{\sigma (i)} \tilde {\gamma }.$ The action of $\lambda _{\Gamma /\Gamma '}$ is then given by

\begin{equation*} \lambda _{\Gamma /\Gamma '}(\gamma )\left ( \sum _{i=1}^n \alpha _i g_i \right ) = \sum _{i=1}^n \alpha _i g_{\sigma (i)}. \end{equation*}

In fact, $\sigma \in S_n$ is a permutation of $[n]$ and, with respect to the basis $\{ g_1, \ldots , g_n\}$ , $\lambda _{\Gamma /\Gamma '}(\gamma )$ acts on $V_{\Gamma /\Gamma '}$ by the permutation matrix associated to $\sigma .$ Furthermore, the induced representation splits as an orthogonal direct sum

\begin{equation*} \lambda _{\Gamma /\Gamma '} = \textbf {1}_\Gamma \oplus \lambda _{\Gamma /\Gamma '}^0, \end{equation*}

where $\lambda _{\Gamma /\Gamma '}^0$ is an $(n-1)$ -dimensional representation acting on the subspace

(22) \begin{equation} V_{\Gamma /\Gamma '}^0 \stackrel {\text{def}}{=} \left \{ \sum _{i=1}^n \alpha _i g_i \in V_{\Gamma /\Gamma '} : \sum _{i=1}^n \alpha _i = 0 \right \}. \end{equation}

Thanks to (18), we can factorize

(23) \begin{equation} Z_{\Gamma '}(s) = Z_\Gamma (s) Z_\Gamma (s,\lambda _{\Gamma /\Gamma '}^0). \end{equation}

We conclude that any “new” resonance for $X' = \Gamma '\backslash {\mathbb{H}}^2$ (that is, a resonance which has greater multiplicity in $X'$ than in $X$ ) must be a zero of $Z_\Gamma (s,\lambda _{\Gamma /\Gamma '}^0).$ In particular, if $\lambda$ is a “new” eigenvalue for $X'$ , then we have $\lambda = s(1-s)$ for some $s\in [\frac {1}{2},\delta ]$ satisfying $Z_\Gamma (s,\lambda _{\Gamma /\Gamma '}^0)=0.$

2.6. Schottky groups

Let us now recall the definition of Schottky groups.

• • Define the alphabet $ \mathcal{A} = \{ 1, \ldots , 2m\}$ and for each $ a\in \mathcal{A}$ define

  \begin{equation*} \overline {a} \stackrel {\text{def}}{=} \begin{cases} a+m &\text{ if } a \in \{ 1,\ldots , m\} \\ a-m &\text{ if } a \in \{ m+1,\ldots , 2m\} \end{cases} \end{equation*}

• • Fix open disks $D_1, \ldots , D_{2m} \subset \mathbb{C}$ centered on the real line with mutually disjoint closures.

• • Fix isometries $ \gamma _1, \ldots \gamma _{2m}\in \mathrm{SL}_2(\mathbb{R})$ such that for all $ a\in \mathcal{A}$

  \begin{equation*} \text{$\gamma _a ( \overline {\mathbb{C}}\smallsetminus D_{\overline {a}} ) = D_a $ and $\gamma _{\overline {a}} = \gamma _a^{-1}.$} \end{equation*}
  (In the notation of [Reference Borthwick1, §15] we have $m=r$ and $\gamma _a = S_a^{-1}$ .)

• • Let $ \Gamma \subset \mathrm{SL}_2({\mathbb{R}})$ be the group generated by the elements $ \gamma _1, \ldots \gamma _{2m}$ . This is a free group on $ m$ generators, see for instance [Reference Borthwick1, Lemma 15.2].

Figure 3.

A configuration of Schottky disks and isometries with $ m=3$ .

An example configuration of Schottky disks and corresponding isometries is depicted in Figure 3.

Throughout the rest of this paper, $\Gamma$ is a non-elementary Schottky group with Schottky data $D_1, \ldots , D_{2m}$ and $\gamma _1, \ldots , \gamma _{2m}$ as above. This assumption will not be repeated in the sequel.

2.7. Combinatorial notation for words

Let $\Gamma$ be a Schottky group as in Section 2.6. We will follow the combinatorial notation of Dyatlov–Zworski [Reference Dyatlov and Zworski9] for indexing elements in the free group $\Gamma$ .

• • A word $\textbf {a}$ in the alphabet $\mathcal{A} =\{ 1,\ldots ,2m \}$ is a finite string $\textbf {a}=a_1 \ldots a_n$ with $a_1, \ldots , a_n \in \mathcal{A}$ . For technical reasons, we also introduce the empty word $\emptyset$ , a string of length zero.

• • A word $\textbf {a}=a_1 \ldots a_n$ is said to be “reduced” if $a_{j} \neq \overline {a_{j+1}}$ for all $j=1,\ldots , n-1$ . For all $n\in \mathbb{N}$ denote by $\mathcal{W}_n$ the set of (reduced) words of length $ n$ :

  \begin{equation*} \mathcal{W}_n = \left \{ a_1\cdots a_n : \text{$a_1, \ldots , a_n \in \mathcal{A}$ s.t. $a_{j} \neq \overline {a_{j+1}} $ for all $j=1,\ldots , n-1$} \right \}. \end{equation*}
  Moreover, put $ \mathcal{W}_0 = \{ \emptyset \}$ where $ \emptyset$ is the empty word.

• • Let $ \mathcal{W} = \bigsqcup _{n\geqslant 0} \mathcal{W}_n$ be the set of all reduced words and write $ \vert \textbf {a}\vert = n$ if $ \textbf {a}\in \mathcal{W}_n$ . In other words, $\vert \textbf {a}\vert$ is the reduced word length of $\textbf {a}$ . Given $m\in \mathbb{N}$ let $\mathcal{W}_{\geqslant m} = \bigsqcup _{n\geqslant m} \mathcal{W}_n$ be the set of all reduced words whose length is at least $m$ , and let $ \mathcal{W}^\circ = \mathcal{W}_{\geqslant 1}$ be the set of all non-empty reduced words.

• • Given a word $ \textbf {a} = a_1\cdots a_n \in \mathcal{W}^\circ$ write $ \textbf {a}' = a_1\cdots a_{n-1} \in \mathcal{W}$ . Note that $ \mathcal{W}$ is a tree with root $ \emptyset$ and $ \textbf {a}'$ is the parent of $ \textbf {a}.$

• • For $ \textbf {a} = a_1\cdots a_n \in \mathcal{W}$ and $ \textbf {b} = b_1\cdots b_m \in \mathcal{W}$ , write $ \textbf {a}\to \textbf {b}$ if either $ \textbf {a}= \emptyset$ , or $ \textbf {b}= \emptyset$ , or $ a_n \neq \overline {b_1}$ . Note that in this case, $ \textbf {a}\textbf {b} \in \mathcal{W}$ , that is, if $ \textbf {a}\to \textbf {b}$ , then the concatenation $\textbf {a}\textbf {b}$ is also a reduced word.

• • Given a word $\textbf {a} = a_1\cdots a_n$ , let $\overline {\textbf {a}} = \overline {a_n}\cdots \overline {a_1}$ be its “mirror” word.

• • Write $ \textbf {a}\prec \textbf {b}$ if $ \textbf {a}$ is a prefix $ \textbf {b}$ , that is, if $ \textbf {b} = \textbf {ac}$ for some $ \textbf {c}\in \mathcal{W}$ .

• • There is a one-to-one correspondence between reduced words $\textbf {a} = a_1\cdots a_n \in \mathcal{W}$ and elements $\gamma _{\textbf {a}} = \gamma _{a_1}\cdots \gamma _{a_n} \in \Gamma .$ Moreover, we have $ \gamma _{\textbf {ab}} = \gamma _{\textbf {a}} \gamma _{\textbf {b}}$ , $ \gamma _{\textbf {a}}^{-1} = \gamma _{\overline {\textbf {a}}}$ , and $\gamma _{\textbf {a}} = I$ if and only if $ \textbf {a} = \emptyset .$

• • For $ \textbf {a} = a_1 \cdots a_n \in \mathcal{W}^\circ$ , we define the disk

  \begin{equation*} D_{\textbf {a}}\,:\!=\,\gamma _{\textbf {a}'}(D_{a_n}). \end{equation*}
  If $ \textbf {a}\prec \textbf {b}$ , then $ D_{\textbf {b}}\subset D_{\textbf {a}}$ . On the other hand, if $ \textbf {a}\nprec \textbf {b}$ and $ \textbf {b}\nprec \textbf {a}$ , then $ D_{\textbf {a}}\cap D_{\textbf {b}} = \emptyset$ . We also define the real interval
  (24) \begin{equation} I_{\textbf {a}} \,:\!=\, D_{\textbf {a}}\cap \mathbb{R} \end{equation}
  and we denote by $ \vert I_{\textbf {a}}\vert$ its length which is equal to the diameter of $ D_{\textbf {a}}.$ The following uniform contraction property holds true: there are constants $C\gt 0$ and $0 \lt \alpha \lt 1$ such that for all $\textbf {a}\in \mathcal{W}^\circ$
  (25) \begin{equation} \vert I_{\textbf {a}}\vert \leqslant C \alpha ^{\vert \textbf {a}\vert }, \end{equation}
  see also Lemma 2.4 below.

• • Denote

  \begin{equation*} \text{$D = \bigsqcup _{a\in \mathcal{A}} D_a $ and $ I = \bigsqcup _{a\in \mathcal{A}} I_a$.} \end{equation*}

• • In the above notation, the limit set of $\Gamma$ may be re-expressed as follows:

  \begin{equation*} \Lambda = \bigcap _{n\geqslant 1} \bigsqcup _{\textbf {a}\in \mathcal{W}_n} I_{\textbf {a}} \subset \mathbb{R}. \end{equation*}

2.8. Twisted transfer operators

In what follows, let $V$ be a finite-dimensional complex vector space with hermitian inner product $\langle \cdot ,\cdot \rangle _V$ and induced norm $\Vert v\Vert _V = \sqrt {\langle v,v\rangle _V}.$ Let $\rho \colon \Gamma \to \mathrm{U}(V)$ be a unitary representation. Recall that a representation is unitary if for any element $\gamma \in \Gamma$ and all vectors $ v,w\in V$ , one has $\langle \rho (\gamma )v,\rho (\gamma )w\rangle _V = \langle v,w\rangle _V$ and in particular $\Vert \rho (\gamma )v\Vert _V = \Vert v\Vert _V$ .

Let $H^2(D,V)$ be the Hilbert space of $V$ -valued, square-integrable, holomorphic functions on $D = \bigsqcup _{a\in \mathcal{A}} D_a$ :

(26) \begin{equation} H^2(D,V) \stackrel {\text{def}}{=} \left \{ \text{$f\colon D\to V$ holomorphic} \left \vert \; \left \Vert f \right \Vert \lt \infty \right .\right \}, \end{equation}

with $L^2$ -norm given by

\begin{equation*} \Vert f\Vert ^2 \stackrel {\text{def}}{=} \int _D \Vert f(z)\Vert _V^2 \textrm {dvol}(z). \end{equation*}

Here, “ $\textrm {vol}$ ” denotes the Lebesgue measure on the complex plane. On this space, we define for all $s\in \mathbb{C}$ the twisted transfer operator

(27) \begin{equation} \mathcal{L}_{s,\rho }\colon H^2(D,V) \to H^2(D,V) \end{equation}

by the formula

(28) \begin{equation} \mathcal{L}_{s,\rho }f(z) \stackrel {\text{def}}{=} \sum _{\substack { a \in \mathcal{A} \\ a\to b }} \gamma _a'(z)^s \rho (\gamma _a)^{-1} f(\gamma _a (z))\ \mathrm{if}\ z\in D_b. \end{equation}

Since the derivative on the right satisfies $\gamma _a'(z)\gt 0$ for all $z\in I_b = D_b\cap {\mathbb{R}}$ , the complex power $\gamma _a'(z)^s$ is uniquely defined and holomorphic for $z\in D_b$ and $s\in \mathbb{C}.$ More concretely, we define

\begin{equation*} \gamma _a'(z)^s \stackrel {\text{def}}{=} \exp ( s \mathbb{L}(\gamma _a'(z)) ), \end{equation*}

with

(29) \begin{equation} \mathbb{L}(z) = \log \vert z\vert + \mathrm{arg}(z), \end{equation}

where $\mathrm{arg}\colon \mathbb{C}\smallsetminus (\!-\infty , 0] \to (\!-\pi ,\pi )$ is the principal value of the argument.

We note that if $(\rho ,V) = (\textbf {1}_\Gamma ,\mathbb{C})$ is the trivial, one-dimensional representation, $H^2(\Omega ,V)$ reduces to the classical Bergman space $H^2(D)$ . Moreover, the operator in (27) reduces to the standard transfer operator $\mathcal{L}_s$ , which can be found for instance in Borthwick’s book [Reference Borthwick1, Chapter 15]. This operator is trace class for every $s\in \mathbb{C}$ and its Fredholm determinant is equal to the twisted Selberg zeta function of the Schottky group $\Gamma$ :

(30) \begin{equation} Z_{\Gamma }(s,\rho ) = \det (1-\mathcal{L}_{s,\rho }). \end{equation}

A proof of this identity for general twists can be found in [Reference Jakobson, Naud and Soares21]. Since $\mathcal{L}_{s,\rho }$ depends holomorphically on the variable $s\in \mathbb{C}$ , this identity shows that $Z_{\Gamma }(s,\rho )$ extends to an entire function.

2.9. Partitions and refined transfer operators

We now define the so-called refined transfer operators which were introduced by Dyatlov–Zworski [Reference Dyatlov and Zworski9]. These are generalizations of the standard transfer operator $\mathcal{L}_{s}$ . Given a finite subset $Z\subset \mathcal{W}$ , let

• • $Z'=\{ \textbf {a}' : \textbf {a}\in Z \}$ and

• • $\overline {Z} =\{ \overline {\textbf {a}} : \textbf {a}\in Z \}.$

For any $s\in \mathbb{C}$ and any finite-dimensional unitary representation $\rho \colon \Gamma \to \mathrm{U}(V)$ , we define the operator

(31) \begin{equation} \mathcal{L}_{Z,s,\rho }\colon H^2(D,V) \to H^2(D,V) \end{equation}

by the formula

(32) \begin{equation} \mathcal{L}_{Z,s,\rho }f(z) \stackrel {\text{def}}{=} \sum _{\substack { \textbf {a}\in (\overline {Z})' \\ \textbf {a}\to b}} \gamma _{ \textbf {a}}'(z)^s \rho ( \gamma _{ \textbf {a}})^{-1} f( \gamma _{ \textbf {a}} (z))\ \mathrm{if}\ z\in D_b. \end{equation}

Note that $\mathcal{L}_{Z,s,\rho }=\mathcal{L}_{s,\rho }$ for $Z=\mathcal{W}_2$ , the set of reduced words of length two.

In terms of the limit set, a finite set $Z\subset \mathcal{W}^\circ$ is a partition if we have the disjoint union

\begin{equation*} \Lambda = \bigsqcup _{\textbf {b}\in Z} ( I_{\textbf {b}} \cap \Lambda ). \end{equation*}

Recall that $\mathcal{W}_n$ denotes the set of reduced words of length $n$ . The sets $\mathcal{W}_n$ for $n\geqslant 2$ are trivial examples for partitions and, in this case, we have $\mathcal{L}_{\mathcal{W}_n,s} = \mathcal{L}_s^{n-1}.$

The fundamental fact about partitions is the following result:

Lemma 2.2. Let $Z \subset \mathcal{W}_{\geqslant 2}$ be a partition. There exists a holomorphic family of trace class operators $T_{Z,s,\rho }$ such that

(33) \begin{equation} 1-\mathcal{L}_{Z,s,\rho } = (1+T_{Z,s,\rho })(1-\mathcal{L}_{s,\rho }). \end{equation}

In particular, we have

(34) \begin{equation} \det (1-\mathcal{L}_{Z,s,\rho } )= \det (1+T_{Z,s,\rho }) Z_\Gamma (s,\rho ). \end{equation}

Proof. First note that the identity in (34) follows from taking determinants on both sides of (33) and then applying (30). Thus, we only need to prove (33). We argue by induction on $\sum _{\textbf {b}\in Z'} \vert \textbf {b}\vert$ . If $Z = \mathcal{W}_2$ , then $\mathcal{L}_{Z,s,\rho } = \mathcal{L}_{s,\rho }$ , so (33) holds true with $T_{\mathcal{W}_2,s,\rho } = 0$ . If $Z \neq \mathcal{W}_2$ , choose a word $\textbf {d}c\in Z$ of maximal length, where $\textbf {d}\in \mathcal{W}^\circ$ and $c\in \mathcal{A}$ .

Then $Z$ has the form (see Figure 4)

(35) \begin{equation} Z = (Z_0 \smallsetminus \{ \textbf {d} \}) \sqcup \{ \textbf {d}a \,:\, a\in \mathcal{A}, \textbf {d}\to a \}, \end{equation}

where $Z_0$ is a partition containing $\textbf {d}$ . By the inductive hypothesis, we have

(36) \begin{equation} 1-\mathcal{L}_{Z_0,s,\rho } = (1+T_{Z_0,s,\rho })(1-\mathcal{L}_{s,\rho }). \end{equation}

Observe that we can write

\begin{equation*} \mathcal{L}_{Z,s,\rho } = \sum _{\textbf {a}\in (\overline {Z})'} \nu _{\textbf {a},s,\rho }, \end{equation*}

where $\nu _{\textbf {a},s,\rho }\colon H^2(D,V)\to H^2(D,V)$ is the trace class operator defined by

\begin{equation*} \nu _{\textbf {a},s,\rho }(f)(z) = \begin{cases} \rho (\gamma _{\textbf {a}})^{-1}\gamma _{\textbf {a}}'(z)^s f(\gamma _{\textbf {a}} (z)) &\text{if $\textbf {a}\to z$}\\ 0 &\text{else}. \end{cases}. \end{equation*}

It follows from (35) that

(37) \begin{equation} \mathcal{L}_{Z,s,\rho } = \mathcal{L}_{Z_0 \smallsetminus \{ \textbf {d} \},s,\rho } + \sum _{a\in \mathcal{A}} \nu _{\overline {\textbf {d}a}', s,\rho }. \end{equation}

Note that for any $a\in \mathcal{A}$ such that $\textbf {d}\to a$ , we have the rule $\overline {\textbf {d}a}' = \overline {a} \overline {\textbf {d}}'$ . Applying the chain rule yields

\begin{equation*} \nu _{\overline {\textbf {d}a}',s, \rho } = \nu _{\overline {\textbf {d}}',s,\rho } \nu _{a,s,\rho }. \end{equation*}

Inserting this into (37), we get

\begin{align*} \mathcal{L}_{Z,s,\rho } &= \mathcal{L}_{Z_0 \smallsetminus \{ \textbf {d} \},s,\rho } + \sum _{a\in \mathcal{A}} \nu _{\overline {\textbf {d}}', s,\rho } \nu _{a,s,\rho }\\ &= \mathcal{L}_{Z_0,s,\rho } - \nu _{\overline {\textbf {d}}', s,\rho } + \nu _{\overline {\textbf {d}}', s,\rho } \mathcal{L}_{s,\rho } \\ &= \mathcal{L}_{Z_0,s,\rho } - \nu _{\overline {\textbf {d}}', s,\rho } (1-\mathcal{L}_{s,\rho }). \end{align*}

Together with (36), we now obtain

\begin{equation*} 1-\mathcal{L}_{Z,s,\rho } = 1- \mathcal{L}_{Z_0,s,\rho } + \nu _{\overline {\textbf {d}}', s,\rho } (1-\mathcal{L}_{s,\rho }) = (1+T_{Z_0,s,\rho } + \nu _{\overline {\textbf {d}}', s,\rho })(1-\mathcal{L}_{s,\rho }). \end{equation*}

This proves (33) with

\begin{equation*} T_{Z,s,\rho } \stackrel {\text{def}}{=} T_{Z_0,s,\rho } + \nu _{\overline {\textbf {d}}', s,\rho }. \end{equation*}

Recall that the sum of trace class operators remains trace class. In particular, $T_{Z,s,\rho }$ is trace class. That $T_{Z,s,\rho }$ is a holomorphic family in $s\in {\mathbb{C}}$ follows by a similar reasoning. The proof is finished.

Figure 4.

An example of a partition $Z$ , with elements of $Z$ shaded gray in the tree of words $\mathcal{W}$ in the alphabet $\mathcal{A} = \{ 1,2,3,4 \}$ . The solid black word is one possible choice of $\textbf {d}$ in the proof of Lemma 2.2.

Let us now introduce the partitions relevant in this paper, again following [Reference Dyatlov and Zworski9]. For any sufficiently small $\tau \gt 0$ , called the resolution parameter, we define

(38) \begin{equation} Z(\tau ) \stackrel {\text{def}}{=} \left \{ \mathbf{a} \in \mathcal{W}^{\circ } : |I_{\mathbf{a}}| \leqslant \tau \lt |I_{\mathbf{a}'}| \right \}. \end{equation}

The set $Z(\tau )$ is a partition by virtue of (25). We now define the $\tau$ -refined transfer operator by

(39) \begin{equation} \mathcal{L}_{\tau ,s,\rho } \stackrel {\text{def}}{=} \mathcal{L}_{Z(\tau ),s,\rho }. \end{equation}

By Equation (32), the operator $\mathcal{L}_{\tau ,s,\rho }$ is explicitly defined as follows: for each $f\in H^2(D,V)$ and each fixed $b\in \mathcal{A}$ , we put for all $z\in D_b$ ,

(40) \begin{equation} \mathcal{L}_{\tau ,s,\rho } f(z) = \sum _{\substack { \textbf {a}\in Y(\tau ) \\ \textbf {a}\to b}} \gamma _{ \textbf {a}}'(z)^s \rho (\gamma _{ \textbf {a}})^{-1} f(\gamma _{ \textbf {a}}(z)) \end{equation}

where

(41) \begin{equation} Y(\tau ) \stackrel {\text{def}}{=} \overline {Z(\tau )}'. \end{equation}

Note that the operator (39) is well-defined as long as $Y(\tau )\subset W^\circ$ , or equivalently, $Z(\tau )\subset W_{\geqslant 2}$ . This condition holds true once $\tau \gt 0$ is small enough, see Lemma 2.7 below.

We use this specific family of operators because it allows us to control both the size of $Y(\tau )$ as well as the size of the derivatives $\gamma _{\textbf {a}}'$ when $\textbf {a}\in Y(\tau )$ , see Lemma 2.7 below.

2.10. Some useful bounds for Schottky groups

We now record some useful estimates for Schottky groups when acting on the hyperbolic plane. The following basic estimates are due to Naud [Reference Naud29]:

We now introduce additional notations, following Magee–Naud [Reference Naud and Magee30]. For every $a\in \mathcal{A}$ , choose a point $o_a \in I_a$ . From now on, we view these points as fixed. For any $\textbf {a}\in \mathcal{W}^\circ$ , we define

\begin{equation*} o_{\textbf {a}} \stackrel {\text{def}}{=} o_b \end{equation*}

for some $b\in \mathcal{A}$ such that $\textbf {a}\to b$ , and we let $o_\emptyset$ be some arbitrary point in $D$ . Next, for any $\textbf {a}\in \mathcal{W}$ we define

\begin{equation*} \Upsilon _{\textbf {a}} \stackrel {\text{def}}{=} \gamma _{\textbf {a}}'(o_{\textbf {a}}). \end{equation*}

As it turns out, the derivatives $\gamma _{\textbf {a}}'(z)$ for any $z\in D_b$ , where $b\in \mathcal{A}$ is such that $\textbf {a}\to b$ , are all comparable to $\Upsilon _{\textbf {a}}$ in a uniform way. We have the following estimates from [Reference Naud and Magee30]:

Combining Lemma 2.4 and Lemma 2.5 yields

Lemma 2.6 (Complex powers of derivatives). For all $b\in \mathcal{A}$ , $\textbf {a}\in \mathcal{W}^\circ$ with $\textbf {a}\to b$ , $z\in D_b$ , and $s=\sigma +it$ with $\sigma \gt 0$ , we have

\begin{equation*} \vert \gamma _{\textbf {a}}'(z)^s\vert \ll (C \Upsilon _{\textbf {a}})^\sigma e^{C\vert t\vert }, \end{equation*}

where $C\gt 0$ and the implied constant depend solely on $\Gamma$ .

Recall the definitions of the sets $Z(\tau )$ and $Y(\tau )$ in (38) and (41), respectively. Also crucial are the following estimates:

2.11. Hilbert–Schmidt norm of refined transfer operators

Given a trace class operator $A\colon H\to H$ on a separable Hilbert space $H$ , the Hilbert–Schmidt norm is defined by

(42) \begin{equation} \Vert A\Vert _{\mathrm{HS}}^2 \stackrel {\text{def}}{=} \mathrm{tr} \left ( A A^{\ast } \right ), \end{equation}

where $A^{\ast }$ denotes the adjoint of $A$ . The goal of this subsection is to prove

Lemma 2.8 (HS-norm). For any finite-dimensional unitary representation $\rho \colon \Gamma \to \mathrm{U}(V)$ , the Hilbert–Schmidt norm of the operator $\mathcal{L}_{\tau ,s,\rho }$ is given by the formula

(43) \begin{equation} \Vert \mathcal{L}_{\tau ,s,\rho } \Vert _{\mathrm{HS}}^2 = \sum _{b\in \mathcal{A}}\sum _{\substack { \textbf {a},\textbf {b}\in Y(\tau ) \\ \textbf {a},\textbf {b}\to b }} \mathrm{tr} \left ( \rho (\gamma _{\textbf {a}}^{-1} \gamma _{\textbf {b}}) \right ) \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)}, \end{equation}

where

(44) \begin{equation} \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)} = \int _{D_b} \gamma '_{\textbf {a}}(z)^s \overline {\gamma '_{\textbf {b}}(z)^s } B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z))\, \textrm {dvol}(z). \end{equation}

Here, $B_{D}(z,w)$ is the reproducing kernel of the Bergman space $H^2(D).$ Moreover, for all $\sigma_0 \gt 0$ , for all $s = \sigma + it$ with $\sigma \geqslant \sigma_0$ , for all $b\in \mathcal{A}$ and for all $\textbf {a},\textbf {b}\in Y(\tau )$ with $\textbf {a},\textbf {b}\to b$ , we have the bound

(45) \begin{equation} \vert \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)}\vert \ll _\Gamma (C\tau )^{2\sigma_0} e^{C \vert t\vert }, \end{equation}

for some constant $C=C(\Gamma )\gt 0$ .

Proof. Formulas for the HS-norm of similar operators are given in [Reference Naud and Magee30, Lemma 4.7] and in [Reference Pohl and Soares34, Proposition 5.5]. We will provide an alternative but essentially equivalent argument. We denote by $B_D(z,w)$ the reproducing kernel of the classical Bergman space $H^2(D)$ over $D = \bigsqcup _{b\in \mathcal{A}} D_b$ . This kernel satisfies $\overline {B_{D}(\cdot ,w)}\in H^2(D)$ for all $w\in D$ and

\begin{equation*} \int _{D} B_{D}(z,w) f(w) \textrm {dvol}(w) = f(z) \end{equation*}

for all $f\in H^2(D)$ and all $z\in D$ , and is uniquely defined by these two properties. Hence, using the formula in (40), $\mathcal{L}_{\tau ,s,\rho }$ can be rewritten as the integral operator

\begin{equation*} \mathcal{L}_{\tau ,s,\rho } f(z) = \int _D K_{\tau ,s,\rho }(z,w) f(w)\textrm {dvol}(w), \end{equation*}

where the kernel is given for all $z,w\in D$ by

\begin{equation*} K_{\tau ,s,\rho }(z,w) = \sum _{\substack { \textbf {a}\in Y(\tau ) \\ \textbf {a}\to b }} \gamma '_{\textbf {a}}(z)^s \rho (\gamma _{\textbf {a}})^{-1} B_D(\gamma _{\textbf {a}}(z),w), \quad \text{if $z\in D_b$.} \end{equation*}

Note that if the points $z,w\in D$ are fixed, then $K_{\tau ,s,\rho }(z,w)$ is an element of the endomorphism ring $\mathrm{End}(V)$ of $V$ . The Hilbert–Schmidt norm on $\mathrm{End}(V)$ is defined by

\begin{equation*} \Vert A\Vert _2 = \sqrt {\mathrm{tr}_V(A A^\ast )}, \quad A\in \mathrm{End}(V), \end{equation*}

where $\mathrm{tr}_V$ is the trace of $V$ . We will drop the subscript $V$ from the notation, writing only $\mathrm{tr}_V = \mathrm{tr}$ . Hence, the Hilbert–Schmidt norm of $K_{\tau ,s,\rho }(z,w)$ (when viewed as an element of $\mathrm{End}(V)$ ) is given by

\begin{align*} \Vert K_{\tau ,s,\rho }(z,w)\Vert _2^2 &= \textrm {tr}\left ( K_{\tau ,s,\rho }(z,w) K_{\tau ,s,\rho }(z,w)^\ast \right )\\ &= \sum _{\substack { \textbf {a},\textbf {b}\in Y(\tau ) \\ \textbf {a},\textbf {b}\to b }} \mathrm{tr} \left ( \rho (\gamma _{\textbf {a}}^{-1} \gamma _{\textbf {b}}) \right ) \gamma '_{\textbf {a}}(z)^s \overline {\gamma '_{\textbf {b}}(z)^s } B_D(\gamma _{\textbf {a}}(z),w) \overline {B_D(\gamma _{\textbf {b}}(z),w)}. \end{align*}

Note that for the second equality, we used the linearity of the trace and the rule $ \rho (\gamma _{\textbf {a}})^\ast \rho ( \gamma _{\textbf {b}}) = \rho (\gamma _{\textbf {a}}^{-1} \gamma _{\textbf {b}}),$ which follows from the unitarity of the representation $\rho$ . The Hilbert–Schmidt norm of $\mathcal{L}_{\tau ,s,\rho }$ can now be computed as

\begin{align*} \Vert \mathcal{L}_{\tau ,s,\rho } \Vert _{\mathrm{HS}}^2 &= \int _D \int _D \Vert K_{\tau ,s,\rho }(z,w)\Vert _2^2 \textrm {dvol}(w)\textrm {dvol}(z)\\ &= \sum _{b\in \mathcal{A}} \int _{D_b} \int _D \Vert K_{\tau ,s,\rho }(z,w)\Vert _2^2 \textrm {dvol}(w)\textrm {dvol}(z)\\ &= \sum _{b\in \mathcal{A}} \sum _{\substack { \textbf {a},\textbf {b}\in Y(\tau ) \\ \textbf {a},\textbf {b}\to b }} \mathrm{tr} \left ( \rho (\gamma _{\textbf {a}}^{-1} \gamma _{\textbf {b}}) \right ) \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)}, \end{align*}

where, in the last line, we put

(46) \begin{equation} \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)} \stackrel {\text{def}}{=} \int _{D_b} \gamma '_{\textbf {a}}(z)^s \overline {\gamma '_{\textbf {b}}(z)^s } \left ( \int _D B_D(\gamma _{\textbf {a}}(z),w) \overline {B_D(\gamma _{\textbf {b}}(z),w)} \textrm {dvol}(w) \right )\textrm {dvol}(z). \end{equation}

By the defining property of the Bergman kernel, we have

\begin{align*} \int _D B_D(\gamma _{\textbf {a}}(z),w) \overline {B_D(\gamma _{\textbf {b}}(z),w)}\, \textrm {dvol}(w) &= \int _D B_D(\gamma _{\textbf {a}}(z),w) B_D(w,\gamma _{\textbf {b}}(z)) \,\textrm {dvol}(w) \\ &= B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z)), \end{align*}

which when inserted into (46) yields

\begin{equation*} \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)} = \int _{D_b} \gamma '_{\textbf {a}}(z)^s \overline {\gamma '_{\textbf {b}}(z)^s } B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z))\, \textrm {dvol}(z), \end{equation*}

as claimed in (44).

Let us now prove the bound in (45). Fix $b\in \mathcal{A}$ and $\textbf {a},\textbf {b}\in Y(\tau )$ so that $\textbf {a},\textbf {b}\to b$ . Combining Lemmas 2.6 and 2.7 implies that for all $z\in D_b,$

\begin{equation*} \vert \gamma _{\textbf {a}}'(z)^{s}\vert \ll (C \tau)^{\sigma } e^{C \vert t\vert} \leqslant (C \tau)^{\sigma_0} e^{C \vert t\vert}\end{equation*}

for some constant $C=C(\Gamma )\gt 0$ , where for the second estimate we assume that $\tau \gt0$ is small enough so that $C\tau \leqslant 1$ . Thus, applying the triangle inequality, it follows that

\begin{align*} \vert \mathcal{I}_{\textbf {a},\textbf {b}}^{(b)}\vert &\leqslant \int _{D_b} \vert \gamma _{\textbf {a}}'(z)^{s}\vert \vert \gamma _{\textbf {b}}'(z)^{s} \vert \vert B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z)) \vert \textrm {dvol}(z)\\ &\ll (C\tau )^{2\sigma_0 } e^{2C \vert t\vert } \sup _{z\in D_b} \vert B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z)) \vert . \end{align*}

It remains to prove that

(47) \begin{equation} \sup _{z\in D_b} \vert B_D(\gamma _{\textbf {a}}(z), \gamma _{\textbf {b}}(z)) \vert \ll _\Gamma 1. \end{equation}

Note that $B_D(z,w)$ equals zero unless the points $z$ and $w$ belong to the same Schottky disk $D_b$ , in which case $B_D(z,w) = B_{D_b}(z,w)$ . Let $r_b\gt 0$ and $c_b\in \mathbb{R}$ be the radius and the center of $D_b$ , respectively. We then have the following formula for the Bergman kernel over $D_b$ , see for instance [Reference Duren and Schuster8, Chapter 1]:

\begin{equation*} B_{D_b}(z,w) = \frac {r_b^2}{\pi ^2 \left ( r_b^2 - (z-c_b)(\overline {w}-c_b) \right )^2}. \end{equation*}

Applying the elementary inequality $\vert R^2 - u v\vert ^2\geqslant R^2 (R-\vert u\vert ) (R-\vert v\vert )$ , valid for all $R\gt 0$ and all points $u,v\in {\mathbb{C}}$ with $\vert u\vert , \vert v\vert \leqslant R$ , we find

\begin{align*} \vert r_b^2 - (z-c_b)(\overline {w}-c_b) \vert ^2 &\geqslant r_b^2 ( r_b-\vert z-c_b\vert ) (r_b-\vert w-c_b\vert )\\ &= r_b^2\, \textrm {dist}( z,\partial D_b )\, \textrm {dist}\,( w,\partial D_b ). \end{align*}

Here, $\textrm {dist}( \cdot ,\partial D_b )$ denotes the minimal euclidean distance to the boundary $\partial D_b$ . We deduce that

(48) \begin{equation} \vert B_{D_b}(z,w)\vert \ll \frac {1}{\textrm {dist}( z,\partial D_b ) \,\textrm {dist}( w,\partial D_b )}. \end{equation}

By the uniform contraction property from Lemma 2.4, there exists some $c = c(\Gamma )\gt 0$ such that for all $\textbf {a}\in \mathcal{W}^\circ$ with $\textbf {a}\to b$ , we have

\begin{equation*} \textrm {dist}( \gamma _{\textbf {a}}(z),\partial D )\geqslant c. \end{equation*}

Inserting this into (48) yields the desired bound (47), completing the proof.

4.1. Reducing the main theorem to counting zeros of zeta functions

We first reformulate the counting of new eigenvalues for Hecke congruence covers in terms of zeros of twisted Selberg zeta functions. Fix a finitely generated group $\Gamma \subseteq \mathrm{SL}_2({\mathbb{Z}})$ and put $X=\Gamma \backslash {\mathbb{H}}^2$ as well as $X_0(q)=\Gamma _0(q)\backslash {\mathbb{H}}^2.$ Recall that $\lambda \in {\mathbb{R}}$ is said to be a “new” eigenvalue for the Laplacian on $X_0(q)$ if it occurs with greater multiplicity than in $X$ . We denote by $N_\Gamma ^{\mathrm{new}}(q,\sigma )$ the number of new eigenvalues of the form $\lambda = s(1-s)$ with $s\in [ \sigma ,\delta ]$ , repeated according to multiplicity.

Let $\lambda _q$ be the induced representation of the trivial representation of the subgroup $\Gamma _0(q)\subset \Gamma$ to $\Gamma$ ,

\begin{equation*} \lambda _q = \mathrm{Ind}_{\Gamma _0(q)}^\Gamma (\textbf {1}_{\Gamma _0(q)}) \end{equation*}

and let $\lambda _q^0 = \lambda _q \ominus \textbf {1}_\Gamma .$ Recall from Section 2.5 that the Selberg zeta function of $\Gamma _0(q)$ can be written as

(59) \begin{equation} Z_{\Gamma _0(q)}(s) = Z_{\Gamma }(s, \lambda _q) = Z_{\Gamma }(s) Z_{\Gamma }(s, \lambda _q^0) , \end{equation}

In particular, this implies that $N_\Gamma ^{\mathrm{new}}(q,\sigma )$ is equal to the number of zeros of $Z_{\Gamma }(s, \lambda _q^0)$ inside $[\sigma ,\delta ].$

Lemma 4.1 (Tensor product). For any square-free $q\in \mathbb{N}$ , it holds that

\begin{equation*} \lambda _q \cong \bigotimes _{p\vert q} \lambda _p, \end{equation*}

where the tensor product is taken over all prime divisors of $q$ .

Proof. For the rest of the proof, define $G_p \stackrel {\text{def}}{=} \pi _p(\Gamma )$ , where $\pi _p \colon \Gamma \to \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}})$  is the reduction modulo $p$ map $\gamma \mapsto \gamma \mod p$ . Clearly, $G_p$ is a subgroup of $ \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}})$ . Observe that $\Gamma _0(p)$ is equal to the pre-image $\pi _p^{-1}(B_p)$ , where $B_p$ is the subgroup of $G_p$ consisting of upper triangular matrices:

\begin{equation*} B_p \stackrel {\text{def}}{=} \left \{ \begin{pmatrix} \ast &\ast \\0&\ast \end{pmatrix} \right \} \leqslant G_p. \end{equation*}

We have $X_p \stackrel {\text{def}}{=} B_p \backslash G_p \cong \Gamma _0(p)\backslash \Gamma$ , so $\lambda _p = \mathrm{ind}_{\Gamma _0(p)}^{\Gamma }(\textbf {1}_{\Gamma _0(p)})$ is equivalent to the regular representation on the space $\ell ^2(X_p)$ of functions $f\colon X_p \to \mathbb{C}$ endowed with the standard inner product. This representation, which by abuse of notation we also denote by $\lambda _p$ , is defined for all $f\in \ell ^2(X_p)$ , $ \gamma \in G_p$ and $x\in X_p$ as follows:

\begin{equation*} \lambda _p(\gamma )f(x) = f(x\gamma ). \end{equation*}

Since $G_q \cong \prod _{p\vert q} G_p,$ the claim can now be checked easily.

Lemma 4.1 implies that for any square-free $q\in \mathbb{N}$ , we have

(60) \begin{equation} \lambda _q \cong \bigotimes _{p\vert q} \lambda _p \cong \bigotimes _{p\vert q} (\textbf {1}_\Gamma \oplus \lambda _p^0) \cong \textbf {1}_\Gamma \oplus \bigoplus _{\substack { d\vert q}} \left ( \bigotimes _{p\vert d} \lambda _p^0 \right ), \end{equation}

where the notation $d\vert q$ means that the sum is taken over all divisors of $q$ . Now, we define $\nu _1 \stackrel {\text{def}}{=} \textbf {1}_\Gamma$ , and for any square-free positive integer $q\in \mathbb{N}$ ,

(61) \begin{equation} \nu _q \stackrel {\text{def}}{=} \bigotimes _{p\vert q} \lambda _{p}^0, \end{equation}

where the product is over all prime divisors of $q$ . In particular, for every prime $p$ we have

\begin{equation*} \nu _p = \lambda _{p}^0. \end{equation*}

Thus, Equation (60) becomes

\begin{equation*} \lambda _q = \bigoplus _{\substack { d\vert q}} \nu _d, \end{equation*}

where the sum is taken over all divisors of $q$ . In light of (18), this yields the factorization

\begin{equation*} Z_{\Gamma }(s, \lambda _q) = \prod _{\substack { d\vert q}} Z_\Gamma (s, \nu _d). \end{equation*}

Dividing both sides by $Z_\Gamma (s)$ this is equivalent to

(62) \begin{equation} Z_{\Gamma }(s, \lambda _q^0) = \prod _{\substack { d\vert q \\ d\neq 1}} Z_\Gamma (s, \nu _d). \end{equation}

Denote by $N_\Gamma (\nu , \sigma )$ the number of zeros of $Z_{\Gamma }(s,\nu )$ in the interval $[\sigma,\delta ]$ , counted with multiplicity. From (62) we now deduce the key relation

(63) \begin{equation} N_\Gamma ^{\mathrm{new}}(q,\sigma ) = \sum _{\substack { d\vert q \\ d\neq 1}}N_\Gamma (\nu _d, \sigma ). \end{equation}

Now, consider the following dyadic sums:

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{prime}}(x,\sigma ) \stackrel {\text{def}}{=} \sum _{\substack { p \sim x \\ \text{$p$ prime} }} N_\Gamma (\lambda _p^0,\sigma ) = \sum _{\substack { p \sim x \\ \text{$p$ prime} }} N_\Gamma (\nu _p,\sigma ). \end{equation*}

and

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma ) \stackrel {\text{def}}{=} \sum _{\substack { q \sim x \\ \text{$q$ sqf} }} N_\Gamma (\nu _q,\sigma ). \end{equation*}

The notation $p,q\sim x$ stands for $x/2 \lt p,q \leqslant x$ . The first sum is over primes, while the second sum is over square-free integers.

Our main Theorem 1.5 will follow from

Let us derive Theorem 1.5 from this proposition.

Proof of Theorem 1.5 assuming Proposition 4.2 . Recall that Theorem 1.5 makes the following assertions (conditional on the Lindelöf Hypothesis and GRH, respectively): if $\delta \gt \frac {1}{2},$ then for any $\epsilon \gt 0,$ we have

(64) \begin{equation} \sum _{\substack {2\leqslant q \leqslant x \\ \text{$q$ sqf} }} N_\Gamma ^{\mathrm{new}}\left ( q, \sigma \right ) \ll _{\epsilon ,\Gamma } \max \left \{ x, x^{1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + \epsilon } \right \}, \end{equation}

where

\begin{equation*} A = \max \left \{ \frac {1+\delta }{3}, 1-\frac {2}{3}\delta \right \}. \end{equation*}

Moreover, if $\delta \gt \frac {3}{4}$ , then for any $\epsilon \gt 0,$ we have

(65) \begin{equation} \sum _{\substack { 2\leqslant p \leqslant x \\ \text{$p$ prime} }} N_\Gamma ^{\mathrm{new}}(p, \sigma ) \ll _{\epsilon ,\Gamma } x^{1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + \epsilon }, \end{equation}

where

\begin{equation*} A = \frac {2\delta }{3}. \end{equation*}

Let us begin with the deducing the estimate in (65) from Proposition 4.2. Using a dyadic decomposition, we obtain

\begin{equation*} \sum _{\substack { p\leqslant x \\ \text{$p$ prime}}} N_\Gamma ^{\mathrm{new}}(p,\sigma )\leqslant \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil }\sum _{\substack { 2^{k-1} \lt p\leqslant 2^k \\ \text{$p$ prime}}} N_\Gamma ^{\mathrm{new}}(p,\sigma )= \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } \mathcal{S}_\Gamma ^{\mathrm{prime}}(2^k,\sigma ). \end{equation*}

It is now clear that the second estimate from Proposition 4.2,

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma ) \ll _{\epsilon , \Gamma } x^{1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + \epsilon }, \end{equation*}

implies

\begin{equation*} \sum _{\substack { p\leqslant x \\ \text{$p$ prime}}} N_\Gamma ^{\mathrm{new}}(p,\sigma ) \ll _{\epsilon ,\Gamma } \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } 2^{k \left ( 1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right )+\epsilon \right ) } \ll x^{ 1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right )+\epsilon }. \end{equation*}

In the same vein, we obtain the estimate in (64), although the derivation is a bit more complicated. Using the relation (63), rearranging sums, and noticing that every divisor $ d\vert q$ of a square-free integer $q$ is also square-free, we obtain

(66) \begin{align} \sum _{\substack { q\leqslant x \\ \text{$q$ sqf}}} N_\Gamma ^{\mathrm{new}}(q,\sigma ) &= \sum _{\substack { q\leqslant x \\ \text{$q$ sqf}}} \sum _{\substack { d\vert q \\ d\neq 1}}N_\Gamma (\nu _d, \sigma ) \end{align}

(67) \begin{align} &\leqslant \sum _{\substack { d\leqslant x \\ \text{$d$ sqf}}} N_\Gamma (\nu _d, \sigma ) \left \lfloor \frac {x}{d} \right \rfloor \end{align}

(68) \begin{align} &\leqslant x \left ( \sum _{\substack { d\leqslant x \\ \text{$d$ sqf}}} \frac {N_\Gamma (\nu _d, \sigma )}{d} \right ). \end{align}

Breaking up the sum in the last line into dyadic pieces gives

(69) \begin{equation} \sum _{\substack { q\leqslant x \\ \text{$q$ sqf}}} N_\Gamma ^{\mathrm{new}}(q,\sigma ) \ll x \left ( \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } \frac {1}{2^k} \sum _{\substack { 2^{k-1} \lt d\leqslant 2^{k} \\ \text{$d$ sqf}}} N_\Gamma (\nu _d, \sigma )\right ) = x \left ( \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } \frac {\mathcal{S}_\Gamma ^{\mathrm{sqf}}(2^k,\sigma )}{2^k}\right ). \end{equation}

Inserting the first estimate from Proposition 4.2,

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{prime}}(x,\sigma ) \ll _{\epsilon ,\Gamma } x^{1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + \epsilon }, \end{equation*}

we then get

\begin{equation*} \sum _{\substack { q\leqslant x \\ \text{$q$ sqf}}} N_\Gamma ^{\mathrm{new}}(q,\sigma ) \ll _{\epsilon ,\Gamma } x \left ( \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } 2^{k \left ( - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right )+\epsilon \right ) } \right ). \end{equation*}

If $\sigma \geqslant \frac {\delta + A}{2} + A\epsilon$ , then $\frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) \geqslant 2\epsilon ,$ and so we can use a geometric series summation to estimate

\begin{equation*} \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } 2^{k \left ( - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right )+\epsilon \right ) } \leqslant \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } 2^{- \epsilon k} \ll _\epsilon 1. \end{equation*}

On the other hand, if $\sigma \lt \frac {\delta + A}{2} + A\epsilon$ , we can estimate

\begin{equation*} \sum _{0 \leqslant k \leqslant \lceil \log _2 x\rceil } 2^{k \left ( - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right )+\epsilon \right ) } \ll _\epsilon x^{-\frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + C \epsilon }. \end{equation*}

for some absolute constant $C\gt 0$ . Adding these two bounds and returning to (69), we see that for all $\sigma \in [\frac {1}{2},\delta ]$ ,

\begin{equation*} \sum _{\substack { q\leqslant x \\ \text{$q$ sqf}}} N_\Gamma ^{\mathrm{new}}(q,\sigma ) \ll _\epsilon x \left ( 1 + x^{-\frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + C \epsilon }\right ) \ll \max \left \{ x, x^{1 - \frac {2}{A} \left ( \sigma - \frac {\delta + A}{2} \right ) + C\epsilon } \right \}. \end{equation*}

Relabeling $C\epsilon$ as $\epsilon$ concludes the proof. We have therefore proved that Proposition 4.2 implies our main Theorem 1.5.

4.2. Reducing Proposition 4.2 to estimating sums of HS-norms

Next, we apply Proposition 3.1 to reduce the main Proposition 4.2 on dyadic sums to

Proof of Proposition 4.2 assuming Proposition 4.3 . Applying Proposition 3.1 to the family of representations $(\nu _q)_{\substack { q \sim x \\ \text{ $q$ sqf } }}$ gives

\begin{align*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma ) &= \sum _{\substack { q \sim x \\ \text{ $q$ sqf } }} N_\Gamma (\nu _q, \sigma )\\ &\leqslant C K^2 \max _{\substack { \mathrm{Re}(s)\geqslant \sigma -\frac {\alpha }{K} \\ \vert \mathrm{Im}(s)\vert \leqslant \beta K }} \left ( \sum _{\substack { q \sim x \\ \text{ $q$ sqf } }} \Vert \mathcal{L}_{\tau , s, \nu _q} \Vert _{\mathrm{HS}}^2 \right ) + \left ( \sum _{\substack { q \sim x \\ \text{ $q$ sqf } }} \dim (\nu _q) \right ) K^2 (C\tau )^{2K} \end{align*}

By Lemma 4.9 below, the dimension of $\nu _q$ is bounded by $\dim (\nu _q) \ll _\Gamma q$ , whence

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma ) \ll C K^2 \max _{\substack { \mathrm{Re}(s)\geqslant \sigma -O(\frac {1}{K}) \\ \vert \mathrm{Im}(s)\vert \leqslant O(K) }} \left ( \sum _{\substack { q\sim x \\ \text{$q$ sqf}}} \Vert \mathcal{L}_{\tau , s, \nu _q} \Vert _{\mathrm{HS}}^2 \right ) + x^2 K^2 (C\tau )^{2K}. \end{equation*}

Applying Proposition 4.3 and setting $\tau = x^{-1/A},$ we find

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma )\ll _\epsilon e^{O(K)} x^{1-\frac {2}{A}( \sigma - \frac {\delta + A}{2} )+ \epsilon + O(\frac {1}{K})} + x^2 K^2 (C x^{-1/A})^{2K}. \end{equation*}

As a final step, we take $K =(\log x)^{1/2}$ . Observing that $e^{O(K)}, x^{O(\frac {1}{K})} \ll _\epsilon x^{\epsilon }$ for any $\epsilon \gt 0$ , and noting that the second term on the right-hand side gets absorbed by the first, we conclude that

\begin{equation*} \mathcal{S}_\Gamma ^{\mathrm{sqf}}(x,\sigma ) \ll _{\epsilon ,\Gamma } x^{1-\frac {2}{A}( \sigma - \frac {\delta + A}{2} )+ \epsilon }, \end{equation*}

proving the first estimate in Proposition 4.2. Deducing the second estimate in Proposition 4.2 from Proposition 4.3 is analogous.

The rest of this paper is devoted to the proof of Proposition 4.3.

4.3. The main number-theoretic ingredient

The estimation of Hilbert–Schmidt norms appearing above reduces to bounding certain arithmetic sums involving traces of induced representations, which are closely connected to character sums involving the Kronecker symbol. The aim of this subsection is to establish the main number-theoretic bound of this paper. First, we need some definitions.

Moreover, we endow the space of $2\times 2$ real matrices with the Frobenius norm

\begin{equation*} \left \Vert \begin{pmatrix} a&\,\,b\\c&\,\,d\end{pmatrix} \right \Vert = \sqrt {a^2 + b^2 + c^2 + d^2}, \end{equation*}

and we write $I=\begin{pmatrix} 1&0\\0&1\end{pmatrix}$ for the identity.

We can now state the key result:

Proposition 4.5 (Main number-theoretic ingredient). Let $\Gamma$ be a finitely generated subgroup of $\mathrm{SL}_2({\mathbb{Z}})$ . Let $\psi \colon (0,\infty )\to {\mathbb{C}}$ be a $C^\infty$ -function with compact support.

If we assume the Lindelöf Hypothesis for quadratic L-functions, then for any hyperbolic element $\gamma \in \Gamma$ , for all $x \gg 1$ , and for any $\epsilon \gt 0$ , the following inequality holds:

(70) \begin{equation} \sum _{\text{ $q$ sqf }} \mathrm{tr}(\nu _q(\gamma )) \psi ( \frac {q}{x} ) \ll _{\epsilon ,\psi ,\Gamma } x^{\frac {1}{2}} \mathcal{Q}(\gamma )^{\frac {1}{2}+\epsilon } \mathcal{D}(\gamma )^{\epsilon }. \end{equation}

Moreover, if we assume GRH for quadratic $L$ -functions and if $\gamma \in \Gamma$ is a hyperbolic element with $\Vert \gamma \Vert \lt \frac {1}{12} x^2$ , then for all $x$ sufficiently large in terms of $\Gamma$ and for any $\epsilon \gt 0$ , the following inequality holds:

(71) \begin{equation} \sum _{\substack { p\sim x \\ \text{$p$ prime} }} \log (p) \mathrm{tr}(\nu _p(\gamma )) \ll _{\epsilon ,\Gamma } x^{\frac {1}{2}+\epsilon }. \end{equation}

Recall that the Legendre symbol is defined for all integers $a$ and all odd primes $p$ by

\begin{equation*} \left ( \frac {a}{p}\right ) = \begin{cases} 1 &\text{ if $a=x^2 \mod p$ for some $x\in \mathbb{F}_p \smallsetminus \{ 0 \} $}\\ 0 &\text{ if $a=0 \mod p$}\\ -1 &\text{ else.} \end{cases} \end{equation*}

There is a standard way of extending the Legendre symbol to a Dirichlet character in the bottom argument. For $p=2$ we define

\begin{equation*} \left ( \frac {a}{2}\right ) = \begin{cases} 0 &\text{ if $a$ is even}\\ 1 &\text{ if $a = \pm 1 \mod 8$}\\ -1 &\text{ if $a = \pm 3 \mod 8$}.\\ \end{cases} \end{equation*}

The Kronecker symbol is defined for all $n\in \mathbb{N}$ by

\begin{equation*} \left ( \frac {a}{n}\right ) = \left ( \frac {a}{p_1}\right )^{r_1} \cdots \left ( \frac {a}{p_m}\right )^{r_m}, \end{equation*}

where $n=p_1^{r_1} \cdots p_m^{r_m}$ is the prime factorization of $n$ . Clearly, if $n=p$ is an odd prime, then the Kronecker symbol is just the Legendre symbol. When either the top or the bottom argument is held constant, the Kronecker symbol behaves like a completely multiplicative function with respect to the other argument. In fact, it is well known that if $D \equiv 0,1$ or $2 \mod 4$ , then $\chi _D(n) = \left ( \frac {D}{n}\right )$ is a non-principal Dirichlet character of conductor at most $4\vert D\vert$ . We refer to [Reference Montgomery and Vaughan28, Chapter 9] for more background on quadratic characters and Kronecker symbols.

The reason we are specifically interested in the Kronecker symbol will become clear in the next lemma.

Lemma 4.6 (Trace formula). Denote by

\begin{equation*} \pi _p \colon \Gamma \to \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}}), \quad \gamma \mapsto \gamma \mod p. \end{equation*}

the reduction modulo $p$ map. Let $\Gamma$ be a subgroup of $ \mathrm{SL}_2(\mathbb{Z})$ and let $p$ be a prime such that $\pi _p \colon \Gamma \to \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}})$ is onto. Then, we have

(72) \begin{equation} \mathrm{tr}(\nu _p(\gamma )) = \mathrm{tr}(\lambda _p^0(\gamma )) = \begin{cases} p &\text{ if $\gamma =\pm I \mod p $ }\\ \left ( \frac {\mathcal{D}(\gamma )}{p} \right ) &\text{ else}, \end{cases} \end{equation}

and in particular, $\dim (\lambda _p^0 ) = p$ .

Proof. Fix a prime $p$ such that $\pi _p \colon \Gamma \to \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}})$ is onto. Let $G_p = \pi _p(\Gamma )$ and let $B_p\subseteq G_p$ be the subgroup of upper triangular matrices. Put $X_p = B_p \backslash G_p$ . Recall from the proof of Lemma 4.1 that $\lambda _p$ is equivalent to the regular representation on $\ell ^2(X_p)$ , that is, for all $f\in \ell ^2(X_p)$ , $ \gamma \in G_p$ and $x\in X_p$ , we have

\begin{equation*} \lambda _p(\gamma )f(x) = f(x\gamma ). \end{equation*}

Recall also that $\lambda _p = \lambda _p^0 \oplus \textbf {1}_\Gamma$ , so the trace satisfies

\begin{equation*} \mathrm{tr}( \lambda _p^0(\gamma ) ) = \mathrm{tr}( \lambda _p(\gamma ) ) -1. \end{equation*}

Since $\pi _p \colon \Gamma \to \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}})$ is assumed to be onto, we have $G_p\cong \mathrm{SL}_2({\mathbb{Z}}/p{\mathbb{Z}}).$ The proof rests on the following two claims:

1. (i) The map sending a line $\ell$ in $\mathbb{F}_p^2$ to $\mathrm{Stab}_{G_p}(\ell )$ is a bijection from $\mathbb{P}(\mathbb{F}_p^2)$ to the set of conjugates of $B_p$ in $G_p$ ;

2. (ii) The normalizer of $B_p$ in $G_p$ equals $B_p$ , so we can identify to the set of conjugates of $B_p$ in $G_p$ with $X_p=B_p \backslash G_p$ .

We proceed in two steps: first, we show that for all $\gamma \in G_p$ , the trace $\mathrm{tr}( \lambda _p(\gamma ) )$ is equal to the number $L(\gamma )$ of lines $\ell \subseteq \mathbb{F}_p^2$ such that $\gamma \ell = \ell$ , and then we compute $L(\gamma )$ .

Step 1: Let $\delta _x$ be the function on $X_p$ taking $1$ at $x$ and $0$ elsewhere. Note that $\{ \delta _x : x\in X_p\}$ is an orthonormal basis for $\ell ^2(X_p)$ . Thus, $\mathrm{tr}( \lambda _p(\gamma ) )$ is equal to the number of fixed points of the action of $\gamma$ on $X_p$ . Note that $B_p g \gamma = B_p g$ if and only if $\gamma$ belongs to $g^{-1}B_p g$ . By Claim (i), $X_p$ corresponds to the set of conjugates of $B_p$ in $G_p$ , so $\mathrm{tr}( \lambda _p(\gamma ) )$ is equal to the number of conjugates of $B_p$ containing $\gamma$ . By Claim (i), this is equal to $L(\gamma )$ .

Step 2: Recall that $\mathcal{D}(\gamma )=\mathrm{tr}(\gamma )^2-4$ is the discriminant of the characteristic polynomial of $\gamma$ . If $\mathcal{D}(\gamma )=0$ , then $\gamma$ has a double eigenvalue $\lambda$ , namely $\lambda =\pm 1.$ The eigenspace $E_\lambda$ has dimension either $2$ or $1$ . If $\dim (E_\lambda )=2$ , then $\gamma = \pm I$ , so $\gamma$ fixes every line in $\mathbb{P}(\mathbb{F}_p^2)$ . Thus, in this case, we have $L(\gamma ) = \# \mathbb{P}(\mathbb{F}_p^2) = \frac {p^2-1}{p-1}=p+1$ , so $\mathrm{tr}( \lambda _p^0(\gamma ) )=L(\gamma )-1 = p.$ If $\dim (E_\lambda )=1$ , then $\gamma$ only fixes the line $E_\lambda$ , whence $\mathrm{tr}( \lambda _p^0(\gamma ) )=L(\gamma )-1 = 0 = \left ( \frac {\mathcal{D}(\gamma )}{p} \right ).$

Now, suppose alternatively that $\mathcal{D}(\gamma )\neq 0$ . If $\mathcal{D}(\gamma )$ is a square in $\mathbb{F}_p^\times$ , then it has two distinct eigenvalues. In this case, $L(\gamma )=2$ as $\gamma$ fixes exactly its two eigenspaces (which are different lines). If $\mathcal{D}(\gamma )$ is not a square in $\mathbb{F}_p^\times$ , then $\gamma$ has no eigenvalue in $\mathbb{F}_p$ , in which case $L(\gamma )=0$ . In both cases, we have

\begin{equation*} L(\gamma )-1 = \left ( \frac {\mathcal{D}(\gamma )}{p} \right ). \end{equation*}

This completes the proof.

Remark 4.7. There is a more pedestrian proof of Lemma 4.6 based on the Frobenius induction formula (also known as the Mackey formula). Let $s, t \in G_p$ denote the elements

\begin{equation*} s = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}, \quad t = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}. \end{equation*}

One can verify by direct computation that the $p+1$ elements

(73) \begin{equation} I = t^0,\, t,\, t^2,\, \ldots ,\, t^{p-1},\, s \end{equation}

form an explicit set of representatives for the (left or right) cosets of $B$ in $G_p$ . If $\pi _p \colon \Gamma \to G_p$ is surjective, then $\lambda _p$ is equivalent to the induced representation $\mathrm{Ind}_{B_p}^{G_p}(\mathbf{1}_{B_p})$ . Using the Frobenius induction formula, the trace of $\lambda _p$ can be expressed in terms of these representatives as

(74) \begin{equation} \mathrm{tr}(\lambda _p(\gamma )) = \mathbf{1}_{B_p}(s^{-1} \gamma s) + \sum _{j=0}^{p-1} \mathbf{1}_{B_p}(t^{-j} \gamma t^j), \end{equation}

where $\mathbf{1}_{B_p}$ denotes the indicator function of $B_p$ . A somewhat tedious but straightforward calculation then yields the same formula claimed in Lemma 4.6 .

Also helpful is the following result:

Proof. The first statement is an observation due to Sarnak–Xue [Reference Sarnak and Xue37]. Write

\begin{equation*} \gamma = \begin{pmatrix} a&b\\c&d\end{pmatrix} \in \mathrm{SL}_2(\mathbb{Z}). \end{equation*}

If $\gamma \equiv \pm I \mod q$ , then there are $ a', b', c', d'\in {\mathbb{Z}}$ such that

\begin{equation*} \text{$ a = a'q \pm 1 $, $ b = b'q $, $ c = c'q $, and $ d = d'q \pm 1 $.} \end{equation*}

Furthermore, the relation $ad-bc = 1$ gives

(75) \begin{equation} 1 = (a'd' -b'c') q^{2} \pm (a'+d')q + 1, \end{equation}

which forces

\begin{equation*} a'+d' \equiv 0 \mod q. \end{equation*}

But this implies that

(76) \begin{equation} \mathrm{tr}(\gamma ) = a+d = (a'+d' )q \pm 2 \equiv \pm 2 \mod q^2, \end{equation}

as claimed. Now, let us prove the second claim. Since $\gamma$ is assumed to be hyperbolic, we have $\vert \mathrm{tr}(\gamma )\vert \gt 2$ . Combining this with the congruence in (76), we deduce that for all $q\geqslant 2$ ,

(77) \begin{equation} \vert a + d\vert = \vert \mathrm{tr}(\gamma )\vert \geqslant q^{2}-2 \geqslant q^2/2. \end{equation}

It follows that for all $q\geqslant 2$ ,

\begin{align*} \Vert \gamma \Vert ^2 &= a^2 + b^2 + c^2 + d^2\\ &= (a+d)^2 + (b-c)^2 - 2 \\ &\geqslant (a+d)^2 - 2\\ &\geqslant q^{4}/4 - 2\\ &\geqslant q^4/8. \end{align*}

Thus, $\Vert \gamma \Vert \geqslant \sqrt {q^4/8} \gt q^2 /3$ , as claimed.

Lemma 4.9. Let $\Gamma \subset \mathrm{SL}_2({\mathbb{Z}})$ be a finitely generated subgroup. Then, for every prime $p$ sufficiently large in terms of $\Gamma$ , the reduction modulo $p$ map $\pi _p\colon \Gamma \to \mathrm{SL}_2(\mathbb{Z}/p \mathbb{Z})$ is onto. Consequently, for any square-free positive integer $q\in \mathbb{N}$ , it holds that

\begin{equation*} \dim (\nu _q) \ll _\Gamma q. \end{equation*}

Proof. The first part is well known from Gamburd [Reference Gamburd12]: there exists $p_0 = p_0(\Gamma )$ such that for all primes $p\geqslant p_0$ the map $\pi _p\colon \Gamma \to \mathrm{SL}_2(\mathbb{Z}/p \mathbb{Z})$ is onto. To establish the dimension bound, we apply Lemmas 4.1 and 4.6, which together imply that

\begin{equation*} \dim (\nu _q) = \prod _{p\vert q} \dim (\nu _p) \ll _\Gamma \prod _{\substack {p\vert q \\ p\gt p_0}} \dim (\nu _p) = \prod _{\substack {p\vert q \\ p\gt p_0}} p \leqslant q, \end{equation*}

as claimed.

We are now prepared to finish the

Proof of Proposition 4.5. For the rest of this proof, fix a finitely generated subgroup $\Gamma \subset \mathrm{SL}_2({\mathbb{Z}})$ and some hyperbolic element $\gamma \in \Gamma .$

We start with the estimate in (70). Consider the following Dirichlet series:

\begin{equation*} F(s;\,\gamma )\stackrel {\text{def}}{=} \sum _{\text{ $q$ sqf }} \frac {\mathrm{tr}(\nu _q(\gamma ))}{q^s}. \end{equation*}

This series is absolutely convergent for $\mathrm{Re}(s)\gt 2$ , since $\vert \mathrm{tr}(\nu _q(\gamma ))\vert \leqslant \dim (\nu _q) \ll _\Gamma q$ by Lemma 4.9. Standard Mellin inversion tells us that for any $C^\infty$ -function $\psi \colon (0,\infty )\to {\mathbb{C}}$ , we have

(78) \begin{equation} \sum _{\text{$q$ sqf}} \mathrm{tr}(\nu _q(\gamma )) \psi ( \frac {q}{x} ) = \frac {1}{2\pi i} \int _{(3)} F(s;\,\gamma ) x^s \widehat {\psi }(s) ds, \end{equation}

where the integral on the right is integration along the vertical line $\mathrm{Re}(s)=3$ and $\widehat {\psi }$ stands for the Mellin transform

\begin{equation*} \widehat {\psi }(s) = \int _0^\infty \psi (t)t^{s-1}dt. \end{equation*}

We need to understand the analytical properties of $F(s;\,\gamma ).$ From the definition of $\nu _q$ in (61), it follows that if $q$ is a square-free integer, then $\mathrm{tr}(\nu _q(\gamma )) = \prod _{p\vert q} \mathrm{tr}(\nu _p(\gamma ))$ , where the product is taken over all prime divisors of $q$ . We can, therefore, express $F(s;\,\gamma )$ as an Euler product over primes:

\begin{equation*} F(s;\,\gamma ) = \prod _{p \in P} \left ( 1+\frac {\mathrm{tr}(\nu _p(\gamma ))}{p^s} \right ). \end{equation*}

Now, define $\mathcal{B}_1$ as the set of primes $p$ for which the map $\pi _p\colon \Gamma \to \mathrm{SL}_2(\mathbb{Z}/p\mathbb{Z})$ is not surjective. Further, given an element $\gamma \in \Gamma$ , define $\mathcal{B}_2(\gamma )$ as the set of primes $p$ for which $\gamma \equiv \pm I \mod p$ . Clearly, $\mathcal{B}_2(\gamma )$ is a finite set, since $\gamma \neq \pm I$ . In fact, by Definition 4.4, we have

\begin{equation*} \prod _{p\in \mathcal{B}_2(\gamma )} p = \mathcal{Q}(\gamma ), \end{equation*}

which implies that

(79) \begin{equation} \# \mathcal{B}_2(\gamma ) = o( \log \mathcal{Q}(\gamma )). \end{equation}

Moreover, by Lemma 4.9, we have $\# \mathcal{B}_1(\gamma ) \ll _\Gamma 1$ . Let $\mathcal{B}(\gamma )$ denote the union of $ \mathcal{B}_1$ and $ \mathcal{B}_2(\gamma )$ . By Lemma 4.6, for all primes $p$ outside the set $\mathcal{B}(\gamma )$ , the trace $\mathrm{tr}(\nu _p(\gamma ))$ equals $ \chi _{\mathcal{D}(\gamma )}(p)$ , where $\chi _{D}(n) = \left ( \frac {D}{n} \right )$ denotes the Kronecker symbol. Formally,

(80) \begin{equation} p\notin \mathcal{B}(\gamma ) \Longrightarrow \mathrm{tr}(\nu _p(\gamma )) = \chi _{\mathcal{D}(\gamma )}(p). \end{equation}

From now on, let $D$ be some positive, non-square, integer such that $D$ is $0$ or $1$ mod $4$ . (We are interested only in the case $D=\mathcal{D}(\gamma )$ but the results can be stated more generally.) In this case, the Kronecker symbol $ \chi _D(\cdot )$ is a real-valued, non-principal Dirichlet character of conductor (that is, smallest modulus) $m\leqslant 4D$ . The associated Dirichlet L-function is the entire function

\begin{equation*} L(s,\chi _{D}) = \sum _{n=1}^\infty \frac {\chi _D(n)}{n^{s}} = \prod _{p \in P} \left ( 1-\frac {\chi _{D}(p)}{p^s} \right )^{-1}. \end{equation*}

Note that $\chi _D$ takes values only in $\{-1,0,1\}$ and hence $\chi _D^2$ is equal to the principal character of modulus $m$ , which we denote by $\chi _{0,m}$ . We then have the elementary relation

\begin{equation*} \prod _{p \in P} \left ( 1+\frac {\chi _{D}(p)}{p^s} \right ) = \prod _{p \in P} \left ( 1-\frac {\chi _{0,m}(p)}{p^{2s}} \right ) \left ( 1-\frac {\chi _{D}(p)}{p^s} \right )^{-1}, \end{equation*}

which we can simplify using the Riemann zeta function $\zeta (s)$ as follows:

(81) \begin{equation} \prod _{p \in P} \left ( 1+\frac {\chi _{D}(p)}{p^s} \right ) =\frac {L(s,\chi _{D})}{\zeta (2s) \prod _{p\mid m} (1-p^{-2s}) }. \end{equation}

Since $\mathrm{tr}(\gamma )\neq \pm 2$ , the discriminant $\mathcal{D}(\gamma )$ is a not a square and $\mathcal{D}(\gamma )$ is either $0$ or $1$ mod $4$ . Hence, all the above results apply with $D=\mathcal{D}(\gamma )$ . Using (80) and (81), we can now express $F(s;\,\gamma )$ as

(82) \begin{equation} F(s;\,\gamma ) = B(s;\,\gamma ) \prod _{p \in P} \left ( 1+\frac {\chi _{\mathcal{D}(\gamma )}(p)}{p^s} \right ) =\frac {B(s;\,\gamma ) L(s,\chi _{\mathcal{D}(\gamma )})}{\zeta (2s) \prod _{p\mid m} (1-p^{-2s})}, \end{equation}

where the correction term $B(s;\,\gamma )$ is given by the finite product

\begin{equation*} B(s;\,\gamma ) = \prod _{p\in \mathcal{B}(\gamma )} \frac {\left ( 1+\frac {\mathrm{tr}(\nu _p(\gamma ))}{p^s} \right )}{\left ( 1+\frac {\chi _{\mathcal{D}(\gamma )}(p)}{p^s} \right )}. \end{equation*}

Let us first deal with this correction term. We claim that for all $s=\sigma +it$ within the vertical strip $\frac {1}{4}\lt \sigma \lt 1$ , the following estimate holds for any $\epsilon \gt 0$ :

(83) \begin{equation} \vert B(s;\,\gamma )\vert \ll _{\epsilon ,\Gamma } \mathcal{Q}(\gamma )^{1-\sigma +\epsilon }. \end{equation}

To see this, observe that for all $\frac {1}{4}\lt \sigma \lt 1$ and all $p\in \mathcal{B}_2(\gamma )$ we have

\begin{equation*} \vert 1+\frac {\mathrm{tr}(\nu _p(\gamma ))}{p^s}\vert \lt 1+p^{1-\sigma } \lt 2 p^{1-\sigma } \end{equation*}

and for every prime $p,$

\begin{equation*} \vert 1+\frac {\chi _{\mathcal{D}(\gamma )}(p)}{p^s} \vert \gt 1-\frac {1}{p^{1/4}} \gt \frac {1}{10}. \end{equation*}

With these estimates at hand as well as (79) we obtain

\begin{equation*} \vert B(s;\,\gamma )\vert \ll _\Gamma \prod _{p\in \mathcal{B}_2(\gamma )} (20p^{1-\sigma }) \ll 20^{o(\log (\mathcal{Q}(\gamma )))} \mathcal{Q}(\gamma )^{1-\sigma } \ll _{\epsilon } \mathcal{Q}(\gamma )^{1-\sigma +\epsilon }, \end{equation*}

which is what we claimed in (83).

We can now proceed to estimate the contour integral in (78). Recall that $L(s,\chi _{\mathcal{D}(\gamma )})$ is an entire function and that $\zeta (s)$ has a simple pole at $s=1$ with no other poles in $\mathbb{C}$ and no zeros in $\mathrm{Re}(s)\geqslant 1$ . It follows that $F(s;\,\gamma )$ has a meromorphic continuation to $s\in {\mathbb{C}}$ with no poles in the half-plane $\mathrm{Re}(s)\geqslant \frac {1}{2}$ . Furthermore, $\vert \widehat {\psi }(s)\vert$ decays faster than any polynomial as $\vert \mathrm{Im}(s)\vert \to \infty$ , since $\psi$ is a compactly supported, smooth function. We now invoke the Lindelöf hypothesis for $ L(s,\chi _{\mathcal{D}(\gamma )})$ which posits that for all $s=\sigma +it$ with $\sigma = \mathrm{Re}(s) = \frac {1}{2}$ we have

(84) \begin{equation} \vert L(s,\chi _{\mathcal{D}(\gamma )}) \vert \ll _\epsilon \mathcal{D}(\gamma )^{\epsilon }(1+\vert t\vert )^\epsilon . \end{equation}

Moreover, on the 1-line, it is well known that $\frac {1}{\zeta (1+it)} = O(\log (t))$ , see for instance [Reference Montgomery and Vaughan28, Theorem 6.7]. Since the modulus $m$ of $\chi _D$ satisfies $m=O(D)$ , we also have for $s$ with $\mathrm{Re}(s)=\frac {1}{2}$ :

\begin{equation*} \frac {1}{\prod _{p\mid m} (1-p^{-2s})} \ll _\epsilon D^\epsilon \end{equation*}

This shows that along the line $\operatorname {Re}(s) = \tfrac {1}{2}$ , we have

(85) \begin{equation} \frac {1}{\zeta (2s) \prod _{p\mid m} (1-p^{-2s})} \ll _\epsilon D^\epsilon \vert t\vert ^{\epsilon }, \end{equation}

with the convention that the left-hand side is interpreted as zero when $s = \tfrac {1}{2}$ . Combining the three bounds (83), (84), and (85), we obtain for $\mathrm{Re}(s)=\frac {1}{2}$ :

(86) \begin{equation} \vert F(s;\,\gamma ) \vert \ll _{\epsilon , \Gamma } \mathcal{Q}(\gamma )^{\frac {1}{2}+\epsilon } \mathcal{D}(\gamma )^{\epsilon } (1+\vert t\vert )^\epsilon . \end{equation}

Hence, shifting the line of integration on the right of (78) to $\mathrm{Re}(s) = \frac {1}{2}$ and then using (86), we find

(87) \begin{equation} \sum _{\text{ $q$ sqf }} \mathrm{tr}(\nu _q(\gamma )) \psi ( \frac {q}{x} ) = \frac {1}{2\pi i} \int _{(\frac {1}{2})} F(s;\,\gamma ) x^s \widehat {\psi }(s) ds \ll _{\epsilon ,\psi ,\Gamma } x^{\frac {1}{2}} \mathcal{Q}(\gamma )^{\frac {1}{2}+\epsilon } \mathcal{D}(\gamma )^{\epsilon }. \end{equation}

This completes the proof of the estimate (70) conditional on the Lindelöf hypothesis for quadratic L-functions.

Now, we prove the estimate in (71). Fix a hyperbolic element $\gamma \in \Gamma$ with

(88) \begin{equation} \Vert \gamma \Vert \lt \frac {x^2}{12}. \end{equation}

Recall that the notation $p\sim x$ means $p\in (\frac {x}{2}, x]$ . It is easy to verify that the condition (88) implies that $\gamma \not \equiv \pm I \mod p$ for all $p\sim x$ . Assume not. Then, we have $\gamma \equiv \pm I \mod p$ for some $p\gt \frac {x}{2}$ , which by Lemma 4.8 implies that

\begin{equation*} \Vert \gamma \Vert \geqslant \frac {p^2}{3} \gt \frac {x^2}{12}, \end{equation*}

contradicting (88). We can now apply Lemma 4.6. We deduce that for all $x$ sufficiently large we have

(89) \begin{equation} p\sim x \Longrightarrow \mathrm{tr}(\nu _p(\gamma )) = \chi _{\mathcal{D}(\gamma )}(p). \end{equation}

Recall once again that if $D\in \mathbb{N}$ be a non-square such that $D \equiv 0,1 \mod 4$ , then $q\mapsto \chi _D(q)$ is a non-principal Dirichlet character of conductor at most $4D$ . We now invoke a classical result from the book of Iwaniec-Kowalski [Reference Iwaniec and Kowalski18, Theorem 5.15]: conditional on the GRH for $L(s,\chi _D)$ it holds that

\begin{equation*} \sum _{q\leqslant x} \chi _D(q) \Lambda (q) \ll _\epsilon x^{1/2+\epsilon } D^\epsilon , \end{equation*}

where $\Lambda (q)$ is the von Mangoldt function. We can restrict this sum to primes at the cost of an error term of $O(x^{1/2+\epsilon })$ , so the above implies that

(90) \begin{equation} \sum _{\substack {p\leqslant x \\ \text{$p$ prime}}} \chi _D(p) \log (p) \ll _\epsilon x^{1/2+\epsilon } D^\epsilon . \end{equation}

It follows from (89) and (90) applied with $D=\mathcal{D}(\gamma )$ that

\begin{align*} \sum _{\substack {p\sim x \\ \text{$p$ prime}}} \log (p) \mathrm{tr}(\nu _p(\gamma )) &= \sum _{\substack {p\sim x \\ \text{$p$ prime}}} \log (p) \chi _{\mathcal{D}(\gamma )}(p)\\ &= \sum _{\substack {p\leqslant x/2 \\ \text{$p$ prime}}} \log (p) \chi _{\mathcal{D}(\gamma )}(n) - \sum _{\substack {p\leqslant x \\ \text{$p$ prime}}} \log (p) \chi _{\mathcal{D}(\gamma )}(n)\\ &\ll _\epsilon x^{1/2+\epsilon } \mathcal{D}(\gamma )^\epsilon . \end{align*}

Finally, observe that $\mathcal{D}(\gamma ) \ll \Vert \gamma \Vert ^2 \ll x^4$ , so the right-hand side of the previous inequality can be further bounded by $O_\epsilon (x^{1/2+\epsilon }).$ The proof of Proposition 4.5 is finished.