2 Definitions and notation

For a set A, define $ {\mathrm{PLC}}(A)$ , the positive linear combination of A, to be the set given by

$$ \begin{align*}\bigg\{\sum\limits_{a\in A'} \lambda_a a\mid A'\subseteq A, A' \text{ is finite and nonempty, } \lambda_{a}>0 \text{ for all } a\in A'\bigg\};\end{align*} $$

and we define $ {\mathrm{cone}}(A)$ , the positive cone spanned by A, to be the set

$$ \begin{align*}{\mathrm{cone}} (A) ={\mathrm{PLC}}(A)\cup \{0\};\end{align*} $$

and we use the notation of $ {\mathrm{conv}}(A)$ to denote the convex hull of A.

Suppose that $(W, R)$ is a Coxeter system in the sense of [Reference Hiller20] or [Reference Humphreys23] with R being a set of involutions generating W subject only to the condition that $(st)^{m_{st}}=1$ for all $s\neq t\in R$ for some $m_{st}\in \mathbb {N}_{\geq 2}\cup \{\infty \}$ . We stress that the same abstract Coxeter group W may be realized in a number of different ways, that is, different Coxeter systems may give rise to the same abstract group with distinct generating sets and, possibly more importantly, different sets of reflections. For an elementary example, one may consider the dihedral group of order $20$ and the direct product of $\mathbb {Z}_2$ with the dihedral group of order $10$ , in which the same abstract group can be realized differently as Coxeter groups with incompatible sets of reflections. Having this in mind, in this paper, in most cases, we take the preference of referring to a Coxeter system $(W, R)$ instead of just referring to a Coxeter group W; and when we simply use the expression ‘let W be a Coxeter group’, the reader should interpret it as a statement concerning not only the group W, but more importantly, the particular presentation specified by a Coxeter system $(W, R)$ .

Following the tradition of [Reference Krammer25, Reference Krammer26], we make the following definition of a root basis.

When $\Pi $ is a root basis, the triple $\mathscr {C}=(\,V, \, \Pi , \,(\,,\,)\,)$ is called a Coxeter datum. For each nonisotropic $a\in V$ (that is, $(a, a)\neq 0$ ), let $\rho _a$ be the reflection corresponding to a; that is, $\rho _a \in {\mathrm{GL}}(V)$ is defined by $\rho _a x=x-2({(x, a)}/{(a, a)})a$ for all $x\in V$ . Let $G_{\mathscr {C}}$ be the subgroup of $ {\mathrm{GL}}(V)$ generated by $\{\rho _{\alpha _s}\mid s\in R\}=\{\rho _a\mid a\in \Pi \}$ . Let $\phi _{\mathscr {C}}\colon W\to G_{\mathscr {C}} $ be the Tits representation. It is the unique isomorphism with

$$ \begin{align*}\phi_{\mathscr{C}}\colon W\to G_{\mathscr{C}}, \text{ satisfying } \phi_{\mathscr{C}}(r_a)=\rho_a\quad \text{for all } a\in \Pi.\end{align*} $$

Under the Tits representation, there is a $(\,,\,)$ -preserving W-action on V: for each $w\in W$ and $x\in V$ , define $wx\in V$ by $wx=\phi _{\mathscr {C}}(w)x$ . We call $(W, R)$ the abstract Coxeter system associated to the Coxeter datum $\mathscr {C}$ and we call W a Coxeter group of rank $\#R$ (where $\#$ denotes cardinality). We use $\ell \colon W\to \mathbb {N}$ to denote the length function of W (with respect to R).

Let $\Phi \subset V$ be the root system of the Coxeter system $(W, R)$ arising from the Coxeter datum $\mathscr {C}$ . That is,

$$ \begin{align*}\Phi=W\Pi =\{wa\mid w\in W, \, a\in \Pi\}.\end{align*} $$

The set $\Phi ^+=\Phi \cap {\mathrm{PLC}}(\Pi )$ is called the set of positive roots and the set $\Phi ^-=-\Phi ^+$ is called the set of negative roots.

When $\Pi $ is a basis for the space V, for each $x\in V$ , there is a unique expression

$$ \begin{align*}x=\sum_{a\in \Pi}{\mathrm{coeff}}_a(x) a, \quad {\mathrm{coeff}}_a(x)\in \mathbb{R}\quad \text{ for all } a\in \Pi,\end{align*} $$

and we define the support of x to be

$$ \begin{align*}{\mathrm{supp}}(x)=\{a\in \Pi\mid {\mathrm{coeff}}_a(x)\neq 0\}.\end{align*} $$

Define $T=\bigcup _{w\in W} w Rw^{-1}$ . We call T the set of reflections in W. For each ${x\in \Phi }$ , we have $x=w \alpha _s$ for some $w\in W$ and $s\in R$ (or equivalently, $\alpha _s\in \Pi $ ). Define ${r_x =wsw^{-1}\in T}$ . Then, $\phi _{\mathscr {C}}(r_x)=\rho _x$ and we call $r_x$ the reflection corresponding to x. It is readily checked that $T=\{r_x \mid x\in \Phi \}$ .

Define functions $N\colon W\to \mathcal {P}(\Phi ^+)$ and $\overline {N}\colon W\to \mathcal {P}(T)$ (where $\mathcal {P}$ denotes the power set) by requiring $N(w)=\{x\in \Phi ^+\mid wx\in \Phi ^-\}$ and $\overline {N}(w)=\{t\in T \mid \ell (wt)<\ell (w)\}$ for all ${w\in W}$ . Note that then, $\overline {N}(w)=\{r_x\mid x\in N(w)\}$ .

Definition 2.2. A subgroup $W'$ of W is a reflection subgroup of W if $W'=\langle T'\rangle $ for some $T'\subseteq T$ . If $W'$ is a reflection subgroup of W, we write $W'\leq W$ ; furthermore, we let

$$ \begin{align*} R(W')&=\{t\in T\mid \overline{N}(t)\cap W'=\{t\}\} \end{align*} $$

and

$$ \begin{align*} \Pi(W')&=\{x\in \Phi^+\mid r_x\in R(W')\}. \end{align*} $$

We call $R(W')$ the set of canonical generators of $W'$ and we call $\Pi (W')$ the set of canonical roots of $W'$ .

It was shown by Dyer [Reference Dyer9, Theorems 3.3] and Deodhar [Reference Deodhar6, Reference Deodhar7] that $(W', R(W'))$ is a Coxeter system whenever $W'$ is a reflection subgroup of W, and furthermore, [Reference Dyer9, Theorem 4.4] gave a characterization of $\Pi (W')$ : for $\Delta \subseteq \Phi ^+$ , let $W'=\langle r_x\mid x\in \Delta \rangle $ . Then, $\Delta =\Pi (W')$ if and only if

$$ \begin{align*} (a, b)\in \{-\cos(\pi/n)\mid n\in \mathbb{N} \text{ and } n\geq 2\}\cup (-\infty, -1], \end{align*} $$

whenever $a, b\in \Delta $ are distinct. We use this characterization quite often in the rest of this paper.

Let $(\,,\,)'$ be the restriction of $(\,,\,)$ to the subspace $\mathbb {R}(\Pi (W'))$ of V spanned by $\Pi (W')$ . Then, $\mathscr {C'}=(\,\mathbb {R}(\Pi (W')),\, \Pi (W'),\, (\,,\,)'\,)$ is a Coxeter datum with $(W', R(W'))$ being the associated abstract Coxeter system. Thus, the notion of a root system applies to $\mathscr {C'}$ . We let $\Phi (W')$ , $\Phi ^+(W')$ , and $\Phi ^-(W')$ be respectively the set of roots, positive roots, and negative roots for the datum $\mathscr {C'}$ . Then, $\Phi (W')=W'\Pi (W')$ and [Reference Dyer9, Theorem 3.3] yields that

$$ \begin{align*}\Phi(W')=\{x\in \Phi\mid r_x \in W'\},\end{align*} $$

and we call $\Phi (W')$ the root subsystem of the reflection subgroup $W'$ . We denote $\Phi ^+(W')=\Phi (W')\cap {\mathrm{PLC}}(\Pi (W'))$ and $\Phi ^-(W')=-\Phi ^+(W')$ , the sets of positive and negative roots in this subsystem. In this paper, a reflection subgroup $W'$ is called a dihedral reflection subgroup if $\#R(W')=2$ . Moreover, we distinguish two types of infinite dihedral reflection subgroups: infinite affine reflection subgroups and infinite nonaffine dihedral reflection subgroups. Specifically, if $W'$ is an infinite dihedral reflection subgroup of W and $\Pi (W')=\{a, b\}\subset \Phi ^+$ , then

$$ \begin{align*}(a, b)\leq -1.\end{align*} $$

If $(a, b)=-1$ , then $W'$ is an infinite affine dihedral reflection subgroup; whereas if $(a, b)<-1$ , then $W'$ is an infinite nonaffine dihedral reflection subgroup. Note that the two types of infinite dihedral reflection groups are each isomorphic to the abstract infinite dihedral Coxeter group $\widetilde {A}_1$ .

A standard parabolic subgroup $W_M$ of W is defined as follows: for $M\subseteq \Pi $ , let ${R_M:=\{r_a\mid a\in M \}}$ and $W_M :=\langle R_M \rangle $ . Observe that $\Phi (W_M)=\Phi \cap \mathbb {R} M$ . We may also use the notation $W_{R_M}$ in place of $W_M$ . A parabolic subgroup of W is any conjugate of a standard parabolic subgroup of W.

3 Some background results

It is well known that in an irreducible affine Coxeter group W, the bilinear form $(\,,\,)$ afforded by the Tits representation of W has a radical with the following properties.

The following is a key result that affords an elegant method to ascertain whether a Coxeter group W is affine or not, based on the radical associated with its Coxeter datum $\mathscr {C}=(V, \Pi , (\,,\,))$ .

Lemma 3.4 [Reference Krammer26, Lemma 6.1.1].

Let W be an irreducible infinite Coxeter group of finite rank. Then, W is nonaffine implies that

$$ \begin{align*}{\mathrm{Rad}}\cap {\mathrm{cone}}(\Pi)=\{0\}.\end{align*} $$

The next result is also well known.

It is clear from the above definition that for each $x\in \Phi ^+$ , the set $D(x)$ is finite; indeed, if $x {\mathrm{\,dom}}\, y$ and $y\in \Phi ^+$ , then $r_x y\in \Phi ^-$ , whence $y\in N(r_x)$ , a finite set of cardinality $\ell (r_x)$ .

It was shown in [Reference Brink and Howlett4] by Brink and Howlett that when a Coxeter group W is finitely generated, then the set of elementary roots is finite. This finiteness property then enabled Brink and Howlett to establish that all finitely generated Coxeter groups are automatic. Brink later gave a complete construction of $D_0$ for all such Coxeter groups in [Reference Brink3]. Subsequently, in [Reference Fu13], it was shown that in a finitely generated Coxeter group W, the sets $D_n$ are finite for all $n\in \mathbb {N}$ , and furthermore, each $D_n$ is nonempty for all infinite Coxeter groups.

It is readily checked that dominance with respect to any reflection subgroup $W'$ of a Coxeter group W is a partial ordering on $\Phi (W')$ . The following lemma summarizes some basic properties of dominance.

We close this section with the following easy but useful result.

4 Limit roots

Throughout this section, we fix a Coxeter system $(W, R)$ and an associated Coxeter datum $\mathscr {C}=(V, \Pi , (\,,\,))$ with $\Phi $ being the corresponding root system. Furthermore, we impose that $\#\Pi =\#R<\infty $ . It is well known that $\Phi $ is a discrete set (with respect to the standard topology), since W acts discretely on V. Nevertheless, following the approach of studying the so-called normalized roots, as pioneered in [Reference Hohlweg, Labbé and Ripoll21], we may study the asymptotic behaviors that these normalized roots exhibit if we consider the elements of $\Phi $ as representatives of the directions whose corresponding reflections generate the reflection group W. In [Reference Hohlweg, Labbé and Ripoll21], the following novel approach has been adopted to obtain such a set of representative directions: consider a projective version of the infinite root system $\Phi $ by cutting the cone $ {\mathrm{PLC}}(\Pi )$ , in which the positive roots are located, by a suitable affine hyperplane $\mathscr {H}$ ; and in doing so, we obtain the so-called normalized root system $\widehat {\Phi }$ , consisting of the intersections of the rays spanned by the roots in $\Phi ^+$ with the hyperplane $\mathscr {H}$ . The great advantage of this construction is that the properties of a root system then ensure that these normalized roots are contained in the convex hull of $\widehat {\Pi }$ (consisting of the intersections of the rays $\{\mathbb {R} x\}_{x\in \Pi }$ with the hyperplane $\mathscr {H}$ ). Under the assumption that $\#\Pi $ is finite, the convex hull of $\widehat {\Pi }$ forms a compact polytope and, thus, allows $\widehat {\Phi }$ to possess a nonempty set of accumulation points, and by studying these accumulation points, we may gain a glimpse of the distribution of roots in infinite root systems. For this construction to work properly, the hyperplane $\mathscr {H}$ should conform with the following definition.

Let $V_1$ be a transverse hyperplane and let $V_0$ be as in the preceding definition. Let $V_0^{+}$ be the open half space induced by $V_0$ that contains $V_1$ . Observe that $V_0^{+}$ contains $ {\mathrm{PLC}}(\Pi )$ . Since $\Phi ^+\subset {\mathrm{PLC}}(\Pi )\subset V_0^{+}$ and $V_1$ is parallel to the boundary of $V_0^{+}$ , it follows that

$$ \begin{align*}\#(V_1\cap \mathbb{R}_{>0} \beta) =1\end{align*} $$

for each $\beta \in \Phi ^+$ .

Observe that $\pi _{V_1}(-x)= \pi _{V_1}(x)$ for all $x\in V$ , and $\widehat {y} =({y}/({|y|_1}))$ for all $y\in V\setminus V_0$ .

Also observe that $\widehat {\Phi }\subseteq {\mathrm{conv}}(\widehat {\Pi })$ (recall that $ {\mathrm{conv}}(X)$ denotes the convex hull of a set X) and $\widehat {\Pi }=\{\widehat {x}\mid x\in \Pi \}$ . Since W is finitely generated, $\Pi $ is a finite set and then we see that $\widehat {\Phi }$ is contained in the compact set $ {\mathrm{conv}}(\widehat {\Pi })$ . Consequently, if $\Pi $ is finite, then the accumulation points of $\widehat {\Phi }$ are contained in $ {\mathrm{conv}}(\widehat {\Pi })$ .

Remark 4.6. For the rest of this section and the whole of the next section, we adopt the set up as in the preceding example, that is, we assume that $\Pi $ forms a basis for V and we take

$$ \begin{align*}V_1=\left\{v\in V \left\vert \sum_{a\in \Pi} {\mathrm{coeff}}_a(v)\right.\! =1 \right\}.\end{align*} $$

Next, we have an elegant result from [Reference Hohlweg, Labbé and Ripoll21].

Following the convention set in [Reference Hohlweg, Labbé and Ripoll21], we define

$$ \begin{align*}D=\bigcap_{w\in W} w(V\setminus V_0)\cap V_1 = V_1\setminus \bigcup_{w\in W} w V_0,\end{align*} $$

and we define the $\cdot $ -action of W on D as follows: for any $w\in W$ and $x\in D$ ,

$$ \begin{align*}w\cdot x =\widehat{wx}.\end{align*} $$

Observe that the property that $WD=\bigcup _{w\in W}w D \subseteq V\setminus V_0$ guarantees that this $\cdot $ -action of W on D is well defined. Furthermore, we observe that each $w\in W$ acts continuously on D. The next result is taken from [Reference Hohlweg, Labbé and Ripoll21], which summarizes a number of key facts.

Furthermore, it has been observed in [Reference Hohlweg, Labbé and Ripoll21] that the $\cdot $ -action has the following nice geometric description.

Given a transverse hyperplane $V_1$ , the limit roots coming from a given rank- $2$ reflection subgroup can be observed inside $E(\Phi )$ . Take two distinct positive roots a and b, and denote by $W'=\langle r_a, r_b\rangle $ the dihedral reflection subgroup of W generated by the reflections $r_a$ and $r_b$ corresponding to the two positive roots. Let $\Pi (W')=\{a', b'\}$ be the set of canonical roots for the dihedral reflection subgroup $W'$ . Then, [Reference Dyer9, Theorem 4.4] yields that $(a', b')\in (-\infty , -1]\cup \{-\cos (\pi /n)\mid n\in \mathbb {N}_{\geq 2}\}$ . It is clear that $0\notin {\mathrm{PLC}}(\{a', b'\})$ , since $a', b'\in {\mathrm{PLC}}(\Pi )$ and $0\notin {\mathrm{PLC}}(\Pi )$ by the definition of $\Pi $ . Now, let $V'=\mathbb {R} a'\oplus \mathbb {R} b'$ , a two-dimensional subspace of V spanned by $\Pi '=\{a', b'\}$ , and let $(\,,\,)'$ be the restriction of $(\,,\,)$ on $V'$ . Then, $\mathscr {C}'=(V', \Pi ', (\,,\,)')$ is also a Coxeter datum with associated root system $\Phi ' = W' \Pi '$ . Observe that the hyperplane $V_1$ is also transverse with respect to $\Phi '$ . Let $E(W')$ denote the set of limit roots of $W'$ in the root system $\Phi '$ with respect to $V_1$ . The following was observed in [Reference Hohlweg, Labbé and Ripoll21, Section 2.3].

It turns out that by considering the cardinality of $E(W)$ , we may easily ascertain whether W is affine or not.

The last two propositions allow us to deduce that every infinite dihedral reflection subgroup of an irreducible affine Coxeter group must be affine. Conversely, as a corollary to [Reference Dyer, Hohlweg and Ripoll11, Theorem 6.2(a)], one can establish that an infinite nonaffine Coxeter group must possess an infinite dihedral reflection subgroup that is nonaffine. Thus, we have the following proposition.

To close this section on necessary introductory material on limit roots, we further recall the following important result that establishes that when W is a finitely generated irreducible and infinite Coxeter group, the limit set $E(W)$ itself is a W-invariant object (with respect to the $\cdot $ -action of W), and this action is minimally faithful.

With a more technical but otherwise predictable generalization, it can be observed that the irreducibility requirement in part (i) of Theorem 4.13 can be removed.

5 Limit roots relating to affine reflection subgroups

Throughout this section, each Coxeter system $(W, R)$ is understood to have an associated Coxeter datum $\mathscr {C}=(V, \Pi , (\,,\,))$ with $\Pi $ being a finite set and $\Phi $ being the corresponding root system. Further, unless otherwise stated, the Coxeter group W is assumed to be infinite. Keeping the set up as in Remark 4.6, specifically, we assume that $\Pi $ forms a basis for V and we let $V_1$ be the transverse hyperplane as in Remark 4.6, and for any reflection subgroup $W'\leq W$ , let $E(W')$ be the set of limit roots of $W'$ with respect to the transverse hyperplane $V_1$ .

A number of results in this section, including Theorem 5.5, and Propositions 5.14 and 5.25, can be seen as consequences of results contained in [Reference Dyer10]. More specifically, in [Reference Dyer10], a novel approach of realizing the Tits cone of a Coxeter group W in the same vector space V bearing the Tits representation of W (rather than in the algebraic dual of V, as in the more conventional treatment) has been adopted; and a certain functoriality condition was established on the imaginary cones of a Coxeter group W and the reflection subgroups of W, namely, the imaginary cone of a reflection subgroup is contained in the imaginary cone of the ambient group W. These results are significant and they often require working in set-ups different from the more conventional ones, and therefore, for the purposes of self-containedness and ease of reading, we have included independent proofs using more standard methodologies. We sincerely thank an anonymous referee who pointed out these alternative possibilities to us.

Suppose that $W'$ is an irreducible affine reflection subgroup of W and let ${\Pi (W') =\{ a_1, a_2, \ldots , a_s\}}$ . Let $\eta $ be the unique limit root in $E(W')$ . By Lemma 3.3, the limit root $\eta $ is in the radical of the bilinear form $(\,,\,)$ restricted to the subspace $V_{W'}$ spanned by $a_1, a_2, \ldots , a_s$ , and it then follows that $(\eta , x)=0$ for all $x\in \Phi (W')\subset V_{W'}$ . However, it is possible that $(\eta , x)>0$ for roots $x\in \Phi ^+\setminus \Phi (W')$ and such positive roots may potentially form an infinite set. In this section, among other things, we prove that $(\eta , x)>0$ for only finitely many positive roots $x\in \Phi ^+$ . Note that Propositions 3.5, 4.11, and 4.12 together imply that if $W'$ is an irreducible affine reflection subgroup of W, then $W'$ has an affine dihedral reflection subgroup $W"\subseteq W'$ with $E(W")=E(W')$ . Hence, many discussions on general affine reflection subgroups, especially their limit roots, can be simplified to discussions on affine dihedral reflection subgroups.

Before we give a characterization of those limit roots in $E(W)$ arising from affine reflection subgroups, let us first look at some limit roots that possibly do not arise from affine reflection subgroups. The first candidate for such possibly nonaffine limit roots might come from infinite nonaffine dihedral reflection subgroups.

Choose $a, b{\kern-1pt}\in{\kern-1pt} \Phi ^+$ such that $(a, b){\kern-1pt}={\kern-1pt}-{\kern-1pt}\cosh \theta {\kern-1pt}<-{\kern-1pt}1$ for some $\theta{\kern-1pt}>{\kern-1pt}0$ , and let ${W'{\kern-1pt}={\kern-1pt}\langle r_a, r_b\rangle }$ be the dihedral reflection subgroup generated by the reflections corresponding to a and b. Note first that $W'$ is infinite and nonaffine. Proposition 4.10 then yields that $E(W')$ consists of two distinct limit roots. Direct calculations then show that the intersection of the isotropic cone Q with the subspace $\mathbb {R} a\oplus \mathbb {R} b$ consists of two lines:

$$ \begin{align*} \mathbb{R}((\cosh\theta+\sinh\theta)a+b) \end{align*} $$

and

$$ \begin{align*} \mathbb{R}((\cosh\theta-\sinh\theta)a+b) = \mathbb{R}(a+(\cosh\theta+\sinh\theta)b), \end{align*} $$

and if we let $\eta _1$ and $\eta _2$ be as follows, then $E(W') =\{\eta _1, \eta _2\}$ :

$$ \begin{align*} \eta_1:= \frac{(\cosh\theta+\sinh\theta)|a|_1}{(\cosh\theta +\sinh\theta)|a|_1+|b|_1}\widehat{a}+ \frac{|b|_1}{(\cosh\theta +\sinh\theta)|a|_1+|b|_1}\widehat{b} \end{align*} $$

and

$$ \begin{align*} \eta_2:= \frac{(\cosh\theta-\sinh\theta)|a|_1}{(\cosh\theta -\sinh\theta)|a|_1+|b|_1}\widehat{a}+ \frac{|b|_1}{(\cosh\theta -\sinh\theta)|a|_1+|b|_1}\widehat{b}. \end{align*} $$

Furthermore,

$$ \begin{align*} \eta_1 &= \lim_{i\to\infty}\widehat{(r_a r_b)^i a}\in \mathbb{R}((\cosh\theta+\sinh\theta)a+b) \end{align*} $$

and

$$ \begin{align*} \eta_2 &= \lim_{i\to \infty}\widehat{(r_b r_a)^i b} \in \mathbb{R}((\cosh\theta-\sinh\theta)a+b). \end{align*} $$

It follows readily that

$$ \begin{align*} (\eta_1, a) &=\frac{\sinh\theta}{(\cosh\theta +\sinh\theta)|a|_1+|b|_1}>0 \end{align*} $$

and

$$ \begin{align*} (\eta_2, b) &=\frac{\sinh\theta (\cosh\theta-\sinh\theta)}{(\cosh\theta -\sinh\theta)|a|_1+|b|_1}>0. \end{align*} $$

Note that $(r_b r_a)^i\cdot \eta _1 =\eta _1$ and $(r_a r_b)^i \cdot \eta _2 =\eta _2$ for all $i\in \mathbb {N}$ . Consequently, for all $i\in \mathbb {N}$ ,

$$ \begin{align*} (\eta_1,\, (r_a r_b)^i a)>0 \quad\text{and}\quad (\eta_2,\, (r_b r_a)^i b)>0. \end{align*} $$

In particular,

$$ \begin{align*} (\eta_1, x)>0 \text{ and } (\eta_2, y)>0 \text{ for infinitely many } x, y\in \Phi^+. \end{align*} $$

However, the above discussion does not immediately rule out the possibility that an affine dihedral reflection subgroup and an infinite nonaffine dihedral reflection subgroup share the same limit root, a situation illustrated in the following schematic diagram.

In this schematic diagram, the dotted circle represents the normalized isotropic cone, and the normalized root subsystems of two infinite dihedral reflection subgroups, one affine and the other nonaffine, are contained in the two straight lines, with the black dots schematically representing normalized roots and the two red dots representing possible limit roots.

It turns out that this situation does not arise. Indeed, in this section, we give a characterization of the set of limit roots arising from affine reflection subgroups. For each $\eta $ arising from an affine reflection subgroup of an infinite Coxeter group of finite rank, we establish where in $\widehat {Q}$ such an $\eta $ must have been located; and we establish the finiteness of the cardinality of the set $\{x\in \Phi ^+\mid (x, \eta )>0\}$ . We also establish that the support of $\eta $ is connected and remains connected under the $\cdot $ -action of W (recall that a set $M\subseteq \Pi $ is disconnected if there are nonempty proper subsets $M_1$ and $M_2$ of M with $M=M_1\uplus M_2$ such that $(a, b) =0$ for each $a\in M_1$ and $b\in M_2$ ; otherwise M is said to be connected and in that case, we may say that M forms a connected component of $\Pi $ ). Moreover, we establish that a limit root simultaneously satisfying these finiteness and connectedness properties must be arising from an affine reflection subgroup and no other limit roots may satisfy simultaneously these two properties.

Definition 5.1. Let $(W, R)$ be a Coxeter system in which W is an infinite Coxeter group. A limit root $\eta \in E(W)$ is called an affine limit root (or is simply called affine) if there exists an irreducible affine reflection subgroup $W'\le W$ with $E(W')=\{\eta \}$ and we use the set

$$ \begin{align*}E_{{\mathrm{aff}}}:=\{\eta\in E(W')\mid W' \leq W \text{ is affine}\}\end{align*} $$

to denote the set of all affine limit roots in W. Also, we use the following set to denote the set of all nonaffine dihedral limit roots in W:

$$ \begin{align*}E_{{\mathrm{nonaff}}}^2:=\{\eta\in E(W')\mid W' \leq W \text{ is nonaffine dihedral}\}.\end{align*} $$

Note that the calculations immediately before the schematic diagram show that if $\eta \in E_{ {\mathrm{nonaff}}}^2$ , then $(\eta , x)>0$ for infinitely many $x\in \Phi ^+$ and, consequently, if we could show that $(\eta , x)>0$ only for finitely many $x\in \Phi ^+$ whenever $\eta \in E_{ {\mathrm{aff}}}$ , then we would have established that $E_{ {\mathrm{aff}}}\cap E_{ {\mathrm{nonaff}}}^2=\emptyset $ ; thus ruling out the possibility raised in the diagram. To achieve this, we need some preparatory work first.

Definition 5.2. For each $v\in V $ , define $ {\mathrm{Pos}}(v)\subseteq \Phi ^+$ by

$$ \begin{align*}{\mathrm{Pos}}(v)=\{x\in \Phi^+\mid (v, x)>0\}.\end{align*} $$

Further, define

$$ \begin{align*} \mathscr{K}&:=\{v\in{\mathrm{cone}}(\Pi)\mid {\mathrm{Pos}}(v)=\emptyset\}\\ &\,\,=\{v\in{\mathrm{cone}}(\Pi) \mid (v, a)\leq 0 \text{ for all } a\in \Phi^+\}. \end{align*} $$

Next, we recall the following fundamental result for infinite Coxeter groups of finite rank that is going to be a key ingredient in our proof that $E_{ {\mathrm{aff}}}\cap E_{ {\mathrm{nonaff}}}^2=\emptyset $ .

Lemma 5.3 [Reference Björner and Brenti1, Proposition 4.5.5].

The set

$$ \begin{align*}\{(\alpha, \beta)\mid \alpha, \beta\in \Phi, \,\, |(\alpha, \beta)|<1\}\end{align*} $$

is finite. In particular, there exists a fixed $\epsilon>0$ such that $|(\alpha , \beta )|>\epsilon $ whenever $\alpha , \beta \in \Phi $ satisfy $(\alpha , \beta )\neq 0$ .

Now, we are ready to prove that $(\eta , x)>0$ only for finitely many $x\in \Phi ^+$ whenever $\eta \in E_{ {\mathrm{aff}}}$ .

Proof. Let $D\le W$ be an affine dihedral reflection subgroup with $E(D)=\{\eta \}$ and let $a,b\in \Phi ^+$ be such that $\Pi (D)=\{a,b\}$ . Then, $(a, b)=-1$ and $(\eta , a)=(\eta , b)=0$ . Letting $a_k=k(a+b)+a$ and $b_k=k(a+b)+b$ (for $k\in \mathbb {N}_{\geq 0}$ ), we have that $\widehat {a_k}\to \eta $ and $\widehat {b_k}\to \eta $ , and the root subsystem of D satisfies $\Phi ^+(D)=\{a_k, b_k \mid k\ge 0\}$ . Note also that $a_k {\mathrm{\,dom\,}} a$ and $b_k {\mathrm{\,dom\,}} b$ for all $k\ge 0$ .

Let $x\in {\mathrm{Pos}}(\eta )$ . Since $\widehat {a_k}\to \eta $ and $\widehat {b_k}\to \eta $ , for sufficiently large k, we have $(x,a_k)>0$ and $(x,b_k)>0$ . Fixing such a k,

$$ \begin{align*} 0<(x,a_k)+(x,b_k)=(x,(2k+1)(a+b))=(2k+1)(x,a+b). \end{align*} $$

Therefore, $(x,a+b)>0$ . Observe that $(x,a)>-1$ , since otherwise, by Lemma 3.7, $x {\mathrm{\,dom\,}} {-}a {\mathrm{\,dom\,}} {-}a_k$ for all $k\in \mathbb {N}$ , which implies that $(x, a_k)\leq -1$ for all $k\in \mathbb {N}$ , but this means that $(x, \eta )\leq 0$ , which contradicts $x\in {\mathrm{Pos}}(\eta )$ . Similarly, $(x, b)> -1$ .

We now observe that there exists $\epsilon>0$ such that $(x,a+b)>\epsilon $ for all $x\in {\mathrm{Pos}}(\eta )$ . To see this, let F denote the finite set

$$ \begin{align*}F=\{(\alpha, \beta)\mid \alpha, \beta\in \Phi, \,\, |(\alpha, \beta)|<1\},\end{align*} $$

and define $\epsilon '=\min \{r+s\mid r,s\in F, r+s>0 \}$ and $\epsilon "=1+\min F$ . Setting $\epsilon =\min (\epsilon ',\epsilon ")>0$ ,

$$ \begin{align*} (x,a+b)=(x,a)+(x,b)\geq \epsilon>0. \end{align*} $$

Therefore, since

$$ \begin{align*} (x,a_k)&=k(x,a+b)+(x,a) \end{align*} $$

and

$$ \begin{align*} (x,b_k)&=k(x,a+b)+(x,b), \end{align*} $$

we may deduce that there is a fixed $M\in \mathbb {N}$ such that

(5-1) $$ \begin{align} (x,a_k)\ge 1\quad\text{and}\quad (x,b_k)\ge 1\quad \text{for all } x\in {\mathrm{Pos}}(\eta)\, \text{ and for all } k\ge M. \end{align} $$

Now, for a contradiction, suppose that $ {\mathrm{Pos}}(\eta )$ is an infinite set. With M as above and for some fixed $k\ge M$ , (5-1) then implies that there is dominance between the infinitely many $x\in {\mathrm{Pos}}(\eta )$ and each of $a_k$ and $b_k$ . Since a given root can dominate only finitely many positive roots (see the remark immediately after Definition 3.6), it follows that there are infinitely many positive roots $x\in {\mathrm{Pos}}(\eta )$ satisfying $x {\mathrm{\,dom\,}} a_k$ and $x {\mathrm{\,dom\,}} b_k$ , but then each of such x would simultaneously dominate both a and b, and we have a contradiction to Lemma 3.8.

It turns out that for a limit root $\eta \in E(W)$ , the finiteness of $ {\mathrm{Pos}}(\eta )$ is not a sufficient condition to ensure that $\eta \in E_{ {\mathrm{aff}}}$ . To ensure that $\eta $ is the limit root of an affine reflection subgroup of W, more conditions on $\eta $ are required. The following is an example demonstrating this general fact. We gratefully acknowledge the comments made by Professor M. Dyer on an earlier version of this paper and, in particular, we thank him for directing our attention to this example, which first appeared in [Reference Hohlweg, Labbé and Ripoll21, Example 5.8].

Example 5.6. Let $(W, R)$ be a rank- $5$ Coxeter system with an associated Coxeter datum $\mathscr {C}=(V, \Pi , (\,,\,))$ such that $\Pi :=\{a, b, c, d, e\}$ is a basis for V, and $(\,,\,)$ satisfies the requirement that the only nonzero values between a pair of simple roots are the following:

$$ \begin{align*}(a, b) =(d, e)=-1 \quad and \quad (b, c)=(c, d)=-\frac{1}{2}.\end{align*} $$

A direct calculation shows that for each $n\in \mathbb {N}$ ,

$$ \begin{align*}(r_a r_b r_e r_d)^n c= c+ n^2b +n^2 d+(n^2+n)a +(n^2+n)e.\end{align*} $$

Thus,

$$ \begin{align*}\eta_0:=\lim_{n\to \infty} \widehat{(r_a r_b r_e r_d)^n c}=\frac{a+b+d+e}{4}\in E(W).\end{align*} $$

Further, note that $ {\mathrm{Pos}}(\eta _0)=\emptyset $ , but $\eta _0$ is not a limit arising from an affine reflection subgroup of W. Instead, $\eta _0$ can be seen as the normalized sum of the limit roots from two mutually orthogonal affine reflection subgroups, namely, $W_1:=\langle r_a, r_b\rangle $ and $W_2:=\langle r_d, r_e\rangle $ . Moreover, it can be checked that every point on the line segment in the interval between the two affine limit roots $({a+b})/{2}$ and $({d+e})/{2}$ is in fact a limit root. For more details on this interesting behavior, please see [Reference Dyer10, Example 7.12]. It should also be pointed out that even though $ {\mathrm{supp}}(\eta _0)=\{a, b, d,e\}$ , it happens that $\eta _0\notin E(\langle r_a, r_b, r_d, r_e\rangle $ ).

In addition to the finiteness condition of the respective $ {\mathrm{Pos}}$ , a complete characterization of affine limit roots also rests on the number of connected components of their supports. To prepare us for such a characterization, we still need a few observations. We begin by recalling a few results on the imaginary cones associated to Coxeter groups.

Definition 5.7. Let $(W, R)$ be an arbitrary Coxeter system and let $\mathscr {C}=(V, \Pi , (\,,\,))$ be an associated Coxeter datum with $\Phi $ being the corresponding root system. We define the imaginary cone, denoted by $\mathscr {Z}$ , to be

$$ \begin{align*}\mathscr{Z}:=\bigcup_{w\in W} w\mathscr{K},\end{align*} $$

where we recall $\mathscr {K}=\{v\in {\mathrm{cone}}(\Pi )\mid {\mathrm{Pos}}(v)=\emptyset \}$ , as introduced in Definition 5.2.

The concept of the imaginary cone was first introduced in [Reference Kac24] in the context of Kac–Moody Lie algebras as the pointed cone spanned by the positive imaginary roots. This concept is later generalized to Coxeter groups. A definitive reference on the imaginary cones of a Coxeter group can be found in [Reference Dyer10]. We show, among other things, that in a finitely generated Coxeter group W, the only limit roots in $\mathscr {Z}$ are precisely those limit roots relating to affine reflection subgroups of W (more precisely, linear combinations of affine limit roots). Furthermore, we show that every point in the intersection of the imaginary cone and the normalized isotropic cone is, in fact, a limit root (expressible as a linear combination of affine limit roots).

The following characterization of the imaginary cone of a finitely generated Coxeter group is taken from [Reference Fu14].

Proposition 5.8 [Reference Fu14, Lemma 4.4 and Proposition 4.22].

Let $\mathscr {Z}$ be the imaginary cone of a finitely generated Coxeter group W. Then,

$$ \begin{align*}\mathscr{Z}=\{v\in U^*\mid (v, a)\leq 0 \text{ for all but finitely many } a\in \Phi^+\}, \end{align*} $$

where $U^*$ is the dual of the Tits cone given by

$$ \begin{align*}U^*=\bigcap_{w\in W} w ({\mathrm{cone}}(\Pi)).\end{align*} $$

Remark 5.9. Theorem 5.5 here may be alternatively established by considering the imaginary cones associated with the affine reflection subgroups of a finitely generated Coxeter group. Suppose that W is a finitely generated infinite Coxeter group possessing an affine reflection subgroup $W'$ . Propositions 3.5, 4.11, and 4.12 yield that there exists some affine dihedral reflection subgroup $W"$ of $W'$ (and, hence, also a reflection subgroup of W) with $E(W")=E(W')=\{\eta \}$ . Let $V"$ be the two-dimensional subspace spanned by the canonical roots $\Pi (W")$ of the affine dihedral reflection subgroup $W"$ and let $(\,,\,)"$ be the restriction of the bilinear form $(\,,\,)$ to $V"$ . Let $\Phi (W")$ be the root subsystem of the affine dihedral reflection subgroup $W"$ . Let

$$ \begin{align*}\mathscr{K}_{W"}=\{v\in {\mathrm{cone}}(\Pi(W"))\mid (v, a)\leq 0 \text{ for all } a\in \Phi^+(W")\},\end{align*} $$

and let $\mathscr {Z}_{W"}=\bigcup _{w\in W"} w\mathscr {K}_{W"}$ be the imaginary cone of $W"$ with respect to the Coxeter datum $(V", \Pi (W"), (\,\,)")$ . Then,

$$ \begin{align*}\{\eta\}=\mathscr{K}_{W"}\subset \mathscr{Z}_{W"}.\end{align*} $$

It is a remarkable result of [Reference Dyer10, Theorem 6.3] that

$$ \begin{align*}\mathscr{Z}_{W"}\subseteq \mathscr{Z}.\end{align*} $$

Consequently, $\eta {\kern-1pt}\in{\kern-1pt} \mathscr {Z}$ and it follows from Proposition 5.8 immediately that ${\# {\mathrm{Pos}}(\eta ){\kern-1pt}<{\kern-1pt}\infty }$ .

To establish the connection between limit roots of finitely generated infinite Coxeter groups and the associated imaginary cones, we recall the following key result from [Reference Dyer, Hohlweg and Ripoll11].

From Theorem 5.10, it is immediately clear that $E(W)\subseteq \overline {Z}$ , where $\overline {\,\cdot \,}$ denotes the topological closure of $\cdot $ .

It has been shown in [Reference Dyer, Hohlweg and Ripoll11, Lemma 2.4] that if $(W, R)$ is an irreducible Coxeter system in which W is a nonaffine infinite Coxeter group, then $ {\mathrm{Int}}(\mathscr {K})\neq \emptyset $ (where $ {\mathrm{Int}}(\,\cdot \,)$ denotes the topological interior of $\cdot $ ). Note that if $x\in {\mathrm{Int}}(\mathscr {K})$ , then it is clear from the definition of $\mathscr {K}$ that $(x, a)<0$ for all $a\in \Pi $ and, in particular, $(x, x)<0$ . Thus, no limit root can be in the interior of $\mathscr {K}$ and indeed, no limit root can be in the interior of $\mathscr {Z}$ .

By combining Proposition 5.8, and Theorems 4.13 and 5.5, we can establish that affine limit roots are in the imaginary cone (in fact, every limit root $\eta $ with $\# {\mathrm{Pos}}(\eta )<\infty $ is in the imaginary cone).

We are now able to establish that the containment of the imaginary cone in the dual of the Tits cone is strict in the case of nonaffine infinite Coxeter groups of finite rank.

Corollary 5.12. Let $(W, R)$ be a Coxeter system in which W is an irreducible infinite nonaffine Coxeter group and R is a finite generating set. The imaginary cone $\mathscr {Z}$ is a proper subset of the dual of the Tits cone $U^*$ . In particular,

$$ \begin{align*} E_{{\mathrm{nonaff}}}^2 \subseteq U^*\setminus \mathscr{Z}\neq \emptyset.\end{align*} $$

For any finitely generated infinite Coxeter group W, it was proven in [Reference Fu and Reeves16] that there is a W-equivariant map from the Davis complex of W into the imaginary cone of W, and this map is a homeomorphism onto its image. The Davis complex has many known topological properties and the existence of such a map may assist in the discovery of comparable properties of the imaginary cone, whose general topological features are yet to be explored. The construction of this map is built on earlier works in [Reference Krammer25, Reference Krammer26], where a similar map from the Davis complex of W to the dual of the Tits cone of W was constructed. It was not entirely clear that the map obtained in [Reference Fu and Reeves16] was in fact an improvement to the map constructed in [Reference Krammer25, Reference Krammer26]. Indeed, it was not clear that the containment of the imaginary cone in the dual of the Tits cone was strict. This issue is now resolved by Corollary 5.12 in the case of infinite nonaffine Coxeter groups.

Next, we strive to complete a characterization of the limit roots in $E_{ {\mathrm{aff}}}$ . So far, we have affirmed that

$$ \begin{align*}E_{{\mathrm{aff}}}\subseteq \{\eta\in E(W)\mid \#{\mathrm{Pos}}(\eta)<\infty\}.\end{align*} $$

We show that if the group W is not an affine Coxeter group, then this containment is strict. Further, we complete a characterization of the set ${\{\eta \in E(W)\mid \# {\mathrm{Pos}}(\eta )<\infty \}}$ itself and show that this set is actually closely related to affine limit roots. Furthermore, we establish the connection of this set of limit roots with the imaginary cone $\mathscr {Z}$ . In particular, we are able to ascertain that each point in $\mathscr {Z}\cap \widehat {Q}$ is in fact a limit root. To complete these tasks, we still need a few observations on limit roots.

It turns out that in a Coxeter group W of finite rank, if a limit root $\eta \in E(W)$ satisfies the condition that $\# {\mathrm{Pos}}(\eta )<\infty $ , then the W-orbit of $\eta $ contains a unique element in $\mathscr {K}$ . This is a consequence of a more general fact that $\mathscr {K}$ is a fundamental domain for the action of W on $\mathscr {Z}$ . That is, the W-orbit of every element of $\mathscr {Z}$ contains one and only one element of $\mathscr {K}$ .

Proposition 5.14. Keep the notation of Definition 5.7. Let $\eta \in \mathscr {K}$ and let $M$ be the set of all $a\in\Pi$ such that $(a,\eta)=0$ . Then,

(5-2) $$ \begin{align} \{w\in W\mid w\eta\in \mathscr{K}\}=W_M =\{w\in W\mid w\eta=\eta\}. \end{align} $$

That is, $\mathscr {K}$ is a fundamental domain for the action of W on $\mathscr {Z}$ .

In particular, in a Coxeter group W of finite rank, if a limit root $\eta \in E(W)$ satisfies the condition that $\# {\mathrm{Pos}}(\eta )<\infty $ , then the W-orbit of $\eta $ contains a unique element of $\mathscr {K}$ .

Proof. Note that by the definition of $\mathscr {Z}$ , it is clear that the W-orbit of every element of $\mathscr {Z}$ contains some element(s) of $\mathscr {K}$ and we only need to prove (5-2) to establish the uniqueness.

Clearly, $W_M\subseteq \{w\in W\mid w\eta =\eta \}\subseteq \{w\in W\mid w\eta \in \mathscr {K}\}$ . Hence, it is enough to show that $\{w\in W\mid w\eta \in \mathscr {K}\}\subseteq W_M$ .

Let $w\in W$ such that $w\eta =\eta '\in \mathscr {K}$ . If $w\neq 1$ , then we may choose $a\in \Pi $ with ${w=w'r_a}$ and $\ell (w)=\ell (w')+1$ . Note that $w'a\in \Phi ^+$ and, hence,

$$ \begin{align*} 0\geq (w'a, \eta')= (w'a, w'r_a \eta) = (a, r_a\eta) = (r_a a, \eta) =-(a, \eta) \geq 0, \end{align*} $$

since $\eta ,\,\eta '\in \mathscr {K}$ and $(\,,\,)$ is W-invariant. Consequently, $a\in M$ . Setting $w_0=w$ and $w_1 =w'$ , we have $\eta ' =w\eta = w_0\eta =w'r_a\eta =w_1\eta $ and $w_0 W_M =w_1 W_M$ . Now, if $w_1\neq 1$ , then we may repeat the above argument and find some $w_2\in W$ satisfying the following three conditions: $\ell (w_2)=\ell (w_1)-1=\ell (w_0)-2$ ; $w_2 \eta =w_1 \eta =w_0\eta =\eta '$ ; and ${w_2 W_M= w_1 W_M=w_0 W_M}$ . If $w_2\neq 1$ , we can repeat the argument again and so construct a sequence $w_0, w_1, w_2, \cdots \in W$ . Since the length decreases at each step, this sequence must terminate; however, this process can be continued so long as $w_i\neq 1$ . Thus, $w_i=1$ for some finite i and

$$ \begin{align*}wW_M =w_0 W_M =w_1 W_M =w_2 W_M =\cdots =W_M,\end{align*} $$

proving $w\in W_M$ .

Finally, note that under the given conditions, $\# {\mathrm{Pos}}(\eta )<\infty $ implies that $\eta \in \mathscr {Z}$ and the rest of the proposition follows from the above proof.

Proof. Write $x=\sum _{a\in M} {\mathrm{coeff}}_a(x) a$ . Since $0\neq x\in \mathscr {K}\subseteq {\mathrm{PLC}}(\Pi )$ , it follows that $ {\mathrm{coeff}}_a(x)>0$ for all $a\in M$ . Now, $x\in Q$ implies that

(5-3) $$ \begin{align} 0=(x, x)=\sum_{a\in M}{\mathrm{coeff}}_a(x)(x, a). \end{align} $$

Since $x\in \mathscr {K}$ , it follows that $(x, a)\leq 0$ for all $a\in \Pi $ and, hence, (5-3) yields that ${(x, a)=0}$ for all $a\in M$ . Consequently, the restriction of the bilinear form $(\,,\,)$ on the subspace spanned by M has a nonzero radical, $ {\mathrm{Rad}}(M)$ , and moreover, $x\in {\mathrm{Rad}}(M)$ .

We can deduce that the converse of Lemma 5.16 is also true.

Proposition 5.18. Suppose that $\eta \in E(W)$ such that $ {\mathrm{Pos}}(\eta )$ is a finite set. There exists $M\subseteq \Pi $ such that the restriction of the bilinear form $(\,,\,)$ on the subspace spanned by M has a nonzero radical $ {\mathrm{Rad}}(M)$ satisfying the condition that $ {\mathrm{Rad}}(M)\cap {\mathrm{PLC}}(M)\neq \emptyset $ and there exists some $w\in W$ such that

$$ \begin{align*}w\cdot \eta \in {\mathrm{Rad}}(M).\end{align*} $$

The next technical result follows from [Reference Dyer10, Section 8.4], but for the sake of completeness, we include a proof here.

It turns out that $\mathscr {K}\cap \widehat {Q} \subset E(W)$ , and indeed, $\mathscr {Z} \cap \widehat {Q} \subset E(W)$ . We show these once we have a better understanding of those limit roots whose supports are not connected.

Lemma 5.20 has the following easily seen consequence.

Proof. Note that Propositions 4.11 and 4.12 yield that we may choose an affine dihedral reflection subgroup $W"\leq W'$ such that $\{\eta \}=E(W")$ . Set $\Pi (W")=\{a, b\}$ . Then, $a, b\in \Phi ^+$ and $(a, b)=-1$ . Then, $ {\mathrm{supp}}(\eta ) = {\mathrm{supp}}(a)\cup {\mathrm{supp}}(b)$ . Since a and b are roots, it can be easily checked that $ {\mathrm{supp}}(a)$ and $ {\mathrm{supp}}(b)$ are both connected. Observe then that the fact $(a, b)=-1$ implies that $ {\mathrm{supp}}(a)\cup {\mathrm{supp}}(b)$ is also connected. For otherwise,

$$ \begin{align*} a&=\sum_{\alpha\in {\mathrm{supp}}(a)}{\mathrm{coeff}}_{\alpha}(a) \alpha \quad\text{where all the } {\mathrm{coeff}}_{\alpha}(a)>0, \end{align*} $$

and

$$ \begin{align*} b&=\sum_{\beta\in {\mathrm{supp}}(b)}{\mathrm{coeff}}_{\beta}(b) \beta \quad\text{ where all the } {\mathrm{coeff}}_{\beta}(b) >0, \end{align*} $$

and if $ {\mathrm{supp}}(a)\cup {\mathrm{supp}}(b)$ is not connected, then note that $(\alpha , \beta )=0$ for all $\alpha \in {\mathrm{supp}}(a)$ and for all $\beta \in {\mathrm{supp}}(b)$ . However, then

$$ \begin{align*}(a, b)=\sum_{\alpha\in {\mathrm{supp}}(a)} \sum_{\beta\in {\mathrm{supp}}(b)}{\mathrm{coeff}}_{\alpha}(a)\ {\mathrm{coeff}}_{\beta}(b) (\alpha, \beta)=0,\end{align*} $$

which contradicts that $(a, b)=-1$ .

Proof. By Lemma 5.13, the fact $\# {\mathrm{Pos}}(\eta )<\infty $ implies that there exists $w\in W$ such that

$$ \begin{align*}0\neq w\cdot \eta\in \mathscr{K}\cap Q.\end{align*} $$

Let $M:= {\mathrm{supp}}(w\cdot \eta )$ and note that M is connected. Then, Proposition 5.19 yields that $W_M:=\langle r_a\mid a\in M\rangle \leq W$ is an irreducible affine standard parabolic subgroup and $\{w\cdot \eta \}=E(W_M)$ . It follows that $w^{-1}W_M w$ , being a conjugate of an irreducible affine reflection subgroup of W, is itself an irreducible affine reflection subgroup of W and $\{\eta \}=E(w^{-1}W_M w)$ .

Combining Theorem 5.5, Proposition 5.22, and Theorem 5.23, we immediately have the following characterization of affine limit roots in an infinite Coxeter group of finite rank.

In fact, the set of affine limit roots also admits the following alternative characterization.

Proof. Suppose that W has an affine standard parabolic subgroup. Then, there is nothing to prove, for a standard parabolic subgroup of W is, a priori, a reflection subgroup of W.

Conversely, suppose that W has an irreducible affine reflection subgroup $W'$ and suppose that $\{\eta \}=E(W')$ . Then, Theorem 5.25 yields that there exists some $w\in W$ such that $w\cdot \eta \in \mathscr {K}$ and $N:= {\mathrm{supp}}(w\cdot \eta )$ is connected. It follows from Proposition 5.19 that $W_N$ is an irreducible affine reflection subgroup of W. Next, let

$$ \begin{align*}M:=\{a\in \Pi\mid (a, w\cdot\eta)=0\}.\end{align*} $$

Note that Proposition 5.14 then yields that $N\subseteq M$ . Now, if $a\in M\setminus N$ , then $(a, b)=0$ for all $b\in N$ , since otherwise, there exists $b\in N$ with $(a, b)<0$ , and since

$$ \begin{align*}w\cdot \eta ={\mathrm{coeff}}_b(w\cdot \eta) b +\sum_{c\in N\setminus\{b\}}{\mathrm{coeff}}_c(w\cdot \eta) c,\end{align*} $$

with all the coefficients being strictly positive, then it follows from the definition of a root basis that

$$ \begin{align*} (a, w\cdot \eta) &=\left(a, {\mathrm{coeff}}_b(w\cdot \eta) b +\sum_{c\in N\setminus\{b\}}{\mathrm{coeff}}_c(w\cdot\eta) c\right)\\&={\mathrm{coeff}}_b(w\cdot\eta) (a, b) + \sum_{c\in N\setminus\{b\}} {\mathrm{coeff}}_c(w\cdot\eta) (a, c)\\&<0, \end{align*} $$

which contradicts that $a\in M$ . Thus, $M=N\uplus N'$ , where N and $N'$ are disconnected subsets of $\Pi $ , and $W_M$ is the direct product of standard parabolic subgroups $W_N$ and $W_{N'}$ . Now, for any $w'\in W'$ ,

$$ \begin{align*} (ww'w^{-1})\cdot (w\cdot\eta)=(w w' w^{-1} w)\cdot \eta=(w w')\cdot\eta &=w\cdot(w'\cdot\eta)\\ &=w\cdot \eta. \end{align*} $$

That is, $w W' w^{-1}$ stabilizes $w\cdot \eta \in \mathscr {K}$ . It follows from Proposition 5.14 that $w W' w^{-1}\subseteq W_M$ and, indeed, $w W' w^{-1}\subseteq W_N$ .

Observations comparable to Corollary 5.27 may be found in [Reference Dyer10, Corollary 8.8] and [Reference Caprace and Mühlherr5, Theorem 3.3] (ultimately tracing back to [Reference Krammer25, Reference Krammer26] and other unpublished works of Daan Krammer) for affine reflection subgroups of ranks greater than or equal to three. Corollary 5.27 provides a unified approach to deal with arbitrary affine reflection subgroups.

Proof. Since $\# {\mathrm{Pos}}(\eta )<\infty $ , it follows from Lemma 5.13 that there exists some $w\in W$ such that $w\cdot \eta \in \mathscr {K}\cap Q$ . Let $M:= {\mathrm{supp}}(w\cdot \eta )$ . Lemma 5.16 gives that the bilinear form $(\,,\,)$ restricted to the subspace spanned by M has a nonzero radical $ {\mathrm{Rad}}(M)$ such that $w\cdot \eta \in {\mathrm{Rad}}(M)\cap {\mathrm{PLC}}(M)$ . Since $\eta \notin E_{ {\mathrm{aff}}}$ , it follows that $w\cdot \eta \notin E_{ {\mathrm{aff}}}$ , and then Proposition 5.14 and Theorem 5.25 together yield that M is not connected. Now, suppose that $M=M_1\uplus \cdots \uplus M_m$ , where $M_1, \ldots , M_m$ form a complete list of connected components of M. Corollary 5.21 and Lemma 3.4 together imply that the standard parabolic subgroups $W_{M_1}, \ldots , W_{M_m}$ are all irreducible affine reflection subgroups. Let $\{\eta _i\}:=E(W_{M_i})\subset E_{ {\mathrm{aff}}}$ for each $i=1, \ldots , m$ . It follows from Corollary 5.21 that $w\cdot \eta $ can be expressed as a positive linear combination of these $\eta _i$ . That is, $w\cdot \eta =\sum _{i=1}^m \lambda _i \eta _i$ for some $\lambda _i \geq 0$ with $\sum _{i=1}^m \lambda _i=1$ . Consequently,

$$ \begin{align*}\eta =\sum_{i=1}^m \lambda^{\prime}_i w^{-1}\cdot\eta_i\quad \text{where~} \lambda^{\prime}_i={|w\eta|}_1{|w^{-1}\eta_i|}_1\lambda_i,\end{align*} $$

where $w^{-1}\cdot \eta _i$ is the limit root of the affine reflection subgroup $w^{-1} W_{M_i}w$ for each $1, \ldots , m$ .

The following is a well-known result and for completeness, we include a proof here.

Proposition 5.30. Let $(W, R)$ be a Coxeter system in which W is an irreducible infinite nonaffine Coxeter group and R is a finite generating set. Let $M_1, \ldots , M_m$ be connected but pairwise disconnected subsets of $\Pi $ that further satisfy the requirement that their corresponding standard parabolic subgroups $W_{M_1}, \ldots , W_{M_m}$ are irreducible affine reflection subgroups. Let $\eta _1, \ldots , \eta _m$ be the limit roots of $W_{M_1}, \ldots , W_{M_m}$ , respectively. Then,

$$ \begin{align*}\bigg\{\!\sum_{i=1}^m \lambda_i \eta_i \bigg\vert \sum_{i=1}^m \lambda_i =1 \text{ and } 0\leq \lambda_i \leq 1 \text{ for all } i=1, \ldots, m\bigg\}\subseteq E(W).\end{align*} $$

That is, each point in the convex hull of the affine limit roots $\eta _1,\ldots , \eta _m$ is a limit root.

Proof. Fix a positive root x with $ {\mathrm{supp}}(x) = \Pi $ whose existence is guaranteed by Lemma 5.29.

For each $i=1, \ldots , m$ , let $\Phi (W_{M_i})$ denote the root subsystem of the affine reflection subgroup $W_{M_i}$ . For each $M_i$ , fix a simple root $\alpha _i\in M_i$ . By [Reference Howlett, Rowley and Taylor22, Proposition 3.13], we can find some $\beta _i \in \Phi ^+(W_{M_i})$ satisfying $\beta _i \,\underset {\neq }{\mathrm {dom}}\,\alpha _i$ . Since $W_{M_i}$ is affine, it follows that $(\alpha _i, \beta _i) =1$ . Let $\gamma _i :=r_{\alpha _i} \beta _i\in \Phi ^+(W_{M_i})$ and

$$ \begin{align*}(\alpha_i, \gamma_i)=(\alpha_i, r_{\alpha_i} \beta_i)=-(\alpha_i, \beta_i)=-1.\end{align*} $$

Then, by [Reference Dyer9, Theorem 4.4], $\{\alpha _i, \gamma _i\}$ is the set of canonical generators for the dihedral reflection subgroup $\langle r_{\alpha _i}, r_{\gamma _i}\rangle $ . Set $\delta _i =\alpha _i +\gamma _i \in {\mathrm{PLC}}(\Pi )$ . Note that ${(\delta _i, \alpha _i)= (\delta _i, \gamma _i)=0}$ . Direct calculations show that

$$ \begin{align*}\Phi^+(\langle r_{\alpha_i}, r_{\gamma_i}\rangle) =\{n\alpha_i +(n\pm 1)\gamma_i\mid n\in \mathbb{N}\},\end{align*} $$

and hence, $\alpha _i +n\delta _i \in \Phi ^+(\langle r_{\alpha _i}, r_{\gamma _i}\rangle )$ for all $n\in \mathbb {N}$ . Thus, $\widehat {\delta _i}$ is the sole limit root in $E(\langle r_{\alpha _i}, r_{\gamma _i}\rangle )$ . Since $W_{M_i}$ is an irreducible affine reflection subgroup and $\langle r_{\alpha _i}, r_{\gamma _i}\rangle \subseteq W_{M_i}$ , it follows that $\eta _i =\widehat {\delta _i}$ and, in particular, $(\delta _i, a)=0$ for all $a\in M_i$ . Next, note that for each $\ell \in \mathbb {N}$ ,

$$ \begin{align*} r_{\alpha_i+\ell\delta_i} (x)&=x-2(x, \alpha_i+\ell\delta_i) (\alpha_i+\ell\delta_i)\\ &=x-2(x, \alpha_i)\alpha_i -2\ell(x,\delta_i)\alpha_i-2\ell (x, \alpha_i) \delta_i -2\ell^2(x, \delta_i)\delta_i. \end{align*} $$

Observe that this is a positive root (since $ {\mathrm{supp}}(x)=\Pi $ ). Now, we let $p_i:=-2(x, \alpha _i)$ and $q_i:=-2(x, \delta _i)$ . Note that $q_i>0$ (since W is irreducible). Thus, the above can be written as

(5-4) $$ \begin{align} r_{\alpha_i+\ell\delta_i} (x) =x+ p_i \alpha_i +\ell q_i \alpha_i +\ell p_i \delta_i +\ell^2 q_i \delta_i. \end{align} $$

Since $M_1, \ldots , M_m$ are pairwise disconnected subsets of $\Pi $ , it follows that ${(x_i, x_j)=0}$ for all $x_i\in \mathbb {R} M_i$ and $x_j\in \mathbb {R} M_j$ , whenever $i\neq j$ . Thus, from (5-4), we see that applying the pairwise commuting reflections corresponding to the positive roots ${\alpha _1+\ell _1 \delta _1, \,\ldots ,\, \alpha _m+\ell _m\delta _m}$ to x (where $\ell _1, \ldots , \ell _m\in \mathbb {N}$ ) produces the following positive root:

$$ \begin{align*} r_{\alpha_m+\ell_m\delta_m} \cdots r_{\alpha_i+\ell_i\delta_i}\cdots r_{\alpha_1+\ell_1\delta_1} (x)=x &+\sum_{i=1}^m p_i \alpha_i +\sum_{i=1}^m \ell_i q_i \alpha_i\\ &+\sum_{i=1}^m \ell_i p_i \delta_i +\sum_{i=1}^m \ell_i^2 q_i \delta_i. \end{align*} $$

It can be checked that for any $t_i\in (0, 1)$ with $\sum _{i=1}^m t_i=1$ , we may find integral sequences $(\ell _{1_n}), \ldots , (\ell _{m_n})$ with

$$ \begin{align*}\lim_{n\to \infty} \ell_{i_{n}} =\infty\quad \text{and}\quad\lim_{n\to \infty}\frac{\ell_{i_n}^2 }{\sum_{i=1}^m \ell_{i_n}^2 }=t_i.\end{align*} $$

Consequently,

$$ \begin{align*}\bigg\{\!\sum_{i=1}^m \lambda_i \eta_i \bigg\vert \sum_{i=1}^m \lambda_i =1 \text{ and } 0\leq \lambda_i \leq 1 \text{ for all } i=1, \ldots, m\bigg\}\subseteq E(W).\\[-45pt]\end{align*} $$

Theorem 5.31. Let W be an irreducible Coxeter group. Then,

$$ \begin{align*}\mathscr{Z} \cap\widehat{Q} \subseteq E(W).\end{align*} $$

Definition 5.32. Define $E_{{\mathrm{aff-type}}}$ to be

$$ \begin{align*} \{\eta\in E(W)\mid \#{\mathrm{Pos}}(\eta) <\infty \text{ and } {\mathrm{supp}}(w\cdot\eta) \text{ is } \text{disconnected for some } w\in W\},\end{align*} $$

the set of limit roots that are nontrivial sums of affine limit roots.

In a Coxeter system $(W, R)$ in which W is an infinite irreducible nonaffine Coxeter group and R is a finite generating set, it follows readily from Theorem 5.24, and Propositions 5.28 and 5.30 that $E_{ {\mathrm{nonaff}}}^2 \cap (E_{ {\mathrm{aff}}}\uplus E_{ {\mathrm{aff}\text{-}\mathrm{type}}}) =\emptyset $ . Furthermore, by combining Proposition 5.8, Theorem 5.24, Proposition 5.28, and Theorem 5.31, we immediately arrive at the following conclusion.

Corollary 5.33. We have

$$ \begin{align*}\mathscr{Z}\cap \widehat{Q}= E(W)\cap \mathscr{Z}=E_{{\mathrm{aff}}}\uplus E_{{\mathrm{aff}\text{-}\mathrm{type}}}=\{\eta\in E(W)\mid \#{\mathrm{Pos}}(\eta)<\infty\}.\end{align*} $$

6 Connection with dominance chains

We close this paper with an application of the Pos-finite analysis developed in the previous section to the dominance partial ordering. In [Reference Dyer, Hohlweg and Ripoll11, Open Question 7.7], it was asked: if $\{x_i\}_{i\in \mathbb {N}}\subset \Phi ^+$ is an infinite injective sequence of positive roots totally ordered by dominance, that is, $x_{i+1}\,\underset {\neq }{\mathrm {dom}}\, x_i$ for all $i\in \mathbb {N}$ , will the corresponding sequence of normalized roots $\{\widehat {x_i}\}_{i\in \mathbb {N}}$ possess a unique limit root. It has been observed in [Reference Dyer, Hohlweg and Ripoll11, Proposition 6.8] that if such a normalized sequence of an infinite injective sequence of positive roots totally ordered by dominance possess two distinct accumulation points, $\eta _1$ and $\eta _2$ , then $(\eta _1, \eta _2)=0$ and they span a two-dimensional totally isotropic subspace of V. In [Reference Higashitani, Mineyama and Nakashima19, Lemma 2.10], it has been further shown that in all Coxeter groups where the bilinear forms afforded by the Tits realization are of signature $(n-1, 1)$ , such an infinite sequence possesses a unique accumulation point. Moreover, in a forthcoming paper, we show that in all infinite Coxeter groups of ranks $3$ and $4$ , there is also unique accumulation. Based on these findings, we have some confidence that the uniqueness of accumulation can be generally expected for all infinite Coxeter groups of finite rank. Although at this present stage we are unable to resolve this question in its entirety, in this section, nevertheless, we prove that if such a normalized sequence possesses an accumulation point $\eta \in E_{ {\mathrm{aff}}}$ , then this normalized sequence is convergent with $\eta $ being the only accumulation point.

We believe that the significance of resolving this possible uniqueness of accumulation among infinite dominance chains is not to be underestimated. So far, in the exploration of limit roots, examples of a convergent infinite sequence of normalized roots are scarce and indeed, for the time being, we can only rely on an infinite sequence coming from infinite dihedral reflection subgroups or affine reflection subgroups. Needless to say that such examples are severely limited in their scopes, and the need to construct other possibly more nontrivial examples of convergent infinite sequences of normalized roots is somewhat pressing. If one could prove that in a finitely generated infinite Coxeter group, the normalized sequence corresponding to an infinite injective sequence of normalized roots is convergent, then we would have succeeded in some way to construct a nontrivial example of convergence. Furthermore, since dominance partial ordering is compatible with reflection subgroups (containing the reflections corresponding to those roots satisfying a dominance condition), such convergence behaviors work well with reflection subgroups.

We first list a number of observations concerning dominance, establishing certain inequalities that may prove to be useful; we also prove a result establishing that when three distinct roots satisfy dominance, their corresponding reflections generate an affine reflection subgroup. We recall the following technical result taken from [Reference Fu14] (which is a refinement in one direction of [Reference Fu14, Proposition 4.10] that for two roots, x and y, we have $x {\mathrm{\,dom\,}} y$ if and only if $x-y\in \mathscr {Z}\subset U^*$ ).

From the above-mentioned proposition, we may deduce a number of inequalities.

An entirely similar proof to the previous one also establishes the following lemma.

Proposition 6.4. Suppose that $x, y, z$ are roots with $x\,\underset {\neq }{\mathrm {dom}}\, y\,\underset {\neq }{\mathrm {dom}}\, z$ . If further ${(x, z)=(x, y)}$ , then

$$ \begin{align*}(x, z)=(x, y)=(y, z)=1,\end{align*} $$

and the reflection subgroup $W':=\langle \rho _x, \rho _y, \rho _z\rangle $ is affine dihedral.

Proof. First, since $x\,\underset {\neq }{\mathrm {dom}}\, y\,\underset {\neq }{\mathrm {dom}}\, z$ , it follows from [Reference Fu14, Proposition 4.10] that ${x-y\in U^*}$ and $y-z\in U^*$ . Then,

$$ \begin{align*}0\geq (x-y, y-z)=\underbrace{(x, y-z)}_{=(x, y)-(x,z)=0}-\overbrace{(y, y-z)}^{=1-(y, z)\leq 0},\end{align*} $$

where $0\geq (x-y, y-z)$ follows from [Reference Fu15, Theorem 5.8]. Consequently, $(y, y-z)=0$ and so $(y, z)=1$ . Then,

$$ \begin{align*}(x, y-z)=0, \quad (y, y-z)=0, \quad\text{and}\quad(z, y-z)=0.\end{align*} $$

That is, $y-z$ is in the radical of the bilinear form $(\,,\,)$ restricted to the subspace spanned by x, y, and z. Furthermore,

$$ \begin{align*}w\cdot (\widehat{y-z})=\widehat{y-z} \quad \text{ for all } w\in W',\end{align*} $$

which in turn establishes that $E(W')=\{\widehat {y-z}\}$ is a singleton (following [Reference Dyer, Hohlweg and Ripoll11, Theorem 3.1]). Now, suppose for a contradiction that $W'$ is nonaffine. It follows from Proposition 4.12 that there exists a nonaffine dihedral reflection subgroup $W"\subset W'$ with $|E(W")|=2$ , which contradicts $E(W")\subseteq E(W')$ . Consequently, $W'$ is affine, whence

$$ \begin{align*}(x, y)=(y, z)=(x, z)=1.\end{align*} $$

Then, $(x-y, x)=(x-y, y)=(x-y, z)=0$ , and $x-y$ is also in the radical of the bilinear form $(\,,\,)$ restricted to the subspace spanned by x, y, and z. Now, if $W'$ is of rank $3$ , then $\{x, y, z\}$ is linearly independent, and the radical of the bilinear form $(\,,\,)$ restricted to the subspace spanned by x, y, and z is of dimension $2$ spanned by $x-y$ and $y-z$ . However, it is well known (for instance, see [Reference Humphreys23, Section 2.6]) that the radical of the restriction of the bilinear form $(\,,\,)$ on the subspace spanned by x, y, and z is one-dimensional if $W'$ is irreducible affine, and we have a contradiction, and whence $W'$ is dihedral.

Proof. Suppose for a contradiction that there is yet another accumulation point for the normalized sequence $\{\widehat {x_i}\}_{i\in \mathbb {N}}$ , say $\eta '$ . Choose two subsequences $\{y_i\}_{i\in \mathbb {N}}$ and $\{z_i\}_{i\in \mathbb {N}}$ of $\{x_i\}_{i\in \mathbb {N}}$ such that $\lim _{i\to \infty }\widehat {y_i}=\eta $ and $\lim _{i\to \infty }\widehat {z_i}=\eta '$ ; furthermore, we require that

$$ \begin{align*}z_{i+1}\,\underset{\neq}{\mathrm{dom}}\, y_{i+1}\,\underset{\neq}{\mathrm{dom}}\, z_i \quad\text{ for all } i\in \mathbb{N}.\end{align*} $$

It follows from Corollary 5.27 that there exists some $w\in W$ such that $\{w\cdot \eta \}=E(W_I)$ for some irreducible affine standard parabolic subgroup $W_I$ . Note that $\{wx_i\}_{i\geq N}$ for some fixed $N\geq \ell (w)$ is still an infinite injective sequence of positive roots totally ordered by dominance; further, $w\cdot \eta $ and $w\cdot \eta '$ are now accumulation points for $\{\widehat {wy_i}\}_{i\geq N}$ and $\{\widehat {wz_i}\}_{i\geq N}$ , respectively. Since now $ {\mathrm{Pos}}(w\cdot \eta )=\emptyset $ , it follows that

$$ \begin{align*}(w\cdot\eta, \widehat{ wy_i})\leq 0 \quad\text{and}\quad (w\cdot \eta, \widehat{ w z_i})\leq 0 \quad\text{ for all } i\geq N.\end{align*} $$

However, the dominance in the two subsequences implies that, $\text { for all } i\geq N$ ,

$$ \begin{align*} (w\cdot \eta, \widehat{w y_i})&=\lim_{j\to \infty}(\widehat{ w y_j}\, ,\, \widehat{w y_i})\geq 0, \end{align*} $$

and

$$ \begin{align*} (w\cdot \eta, \widehat{w z_i})&=\lim_{j\to \infty}(\widehat{ w y_j}\, ,\, \widehat{w z_i})\geq 0. \end{align*} $$

Thus, $\text { for all } i\geq N$ ,

$$ \begin{align*}(w\cdot \eta, \widehat{w y_i})=(w\cdot \eta, \widehat{w z_i})=0.\end{align*} $$

Note now that I, being a connected component of $\Pi $ , is the set of simple roots for $W_I$ , and furthermore, $I= {\mathrm{supp}}(w\cdot \eta )$ . The latter assertion is well known and it can be seen in the following way: if $ {\mathrm{supp}}(w\cdot \eta ) \neq I$ , then there exists some $a\in I$ while ${a\notin {\mathrm{supp}}(w\cdot \eta )}$ , and the connectedness of I implies that there must be some ${b\in {\mathrm{supp}}(w\cdot \eta )}$ such that $(a, b)<0$ , but then $(w\cdot \eta , b)\neq 0$ , which contradicts that ${w\cdot \eta \in {\mathrm{Rad}}(I)}$ . Now, suppose that $x\in \Phi ^+$ such that $(x, w\cdot \eta )=0$ . The fact that $ {{\mathrm{supp}}(w\cdot \eta )=I}$ readily implies that there is no $b\in {\mathrm{supp}}(x)$ for which $(b, a)\neq 0$ for some $a\in I$ . Thus, all $ {\mathrm{supp}}(w z_i)$ , $\text { for all } i\geq N$ , must either be disconnected from I or contained in I (since the support of a root must be itself connected). By our construction, $ {{\mathrm{supp}}(w y_i)= I}$ for all i large enough. Now, if some $wz_j \notin \Phi (W_I)$ , where j is large enough, then $ {\mathrm{supp}}(wz_j)$ must be disconnected from I, but this contradicts that $w y_j\,\underset {\neq }{\mathrm {dom}}\, w z_j$ . Thus,

$$ \begin{align*}wz_i\in \Phi(W_I) \quad \text{for all } i \text{ large enough}.\end{align*} $$

Since $E(W_I)=\{w\cdot \eta \}$ , it follows that

$$ \begin{align*}w\cdot \eta'=\lim_{i\to \infty} \widehat{w z_i} =w\cdot \eta,\end{align*} $$

whence $\eta '=\eta $ and such a sequence has a unique limit.

In a separate forthcoming paper, we obtain additional results establishing that the support of any accumulation point of a normalized sequence corresponding to an infinite injective sequence of roots totally ordered by dominance is connected (and remains connected upon the dot-action by the group, and therefore, cannot be of affine-type); further, we show that any two potential accumulation points $\eta _1$ and $\eta _2$ from the same normalized sequence corresponding to one infinite injective dominance chain have the same support, and $ {\mathrm{Pos}}(\eta _1)= {\mathrm{Pos}}(\eta _2)$ .