2. Actions of automorphisms of G on ${\rm Sub}_G$

In this section we discuss basic properties of the Chabauty topology on ${\rm Sub}_G$ , and discuss certain elementary results about convergence of sequences in ${\rm Sub}_G$ and the action of ${\rm Aut}(G)$ on ${\rm Sub}_G$ .

Let G be a locally compact (Hausdorff) group. A sub-basis of the Chabauty topology on ${\rm Sub}_G$ is given by the sets of the form $\mathcal{O}_1(K)=\{A\in {\rm Sub}_G\mid A\cap K=\emptyset\}$ and $\mathcal{O}_2(U)=\{A\in {\rm Sub}_G\mid A\cap U\neq\emptyset\}$ , where K is a compact and U is an open subset of G. As observed earlier, ${\rm Sub}_G$ is compact and Hausdorff, and if G is second countable, then both G and ${\rm Sub}_G$ are metrisable. For more details on the Chabauty topology, see [Reference Bridson, De La Harpe and Kleptsyn8, Reference Pourezza and HUBBARD23].

Recall that ${\rm Aut}(G)$ is the group of all automorphisms of G. There is a natural group action of ${\rm Aut}(G)$ on ${\rm Sub}_G$ defined as follows:

\begin{equation*}{\rm Aut}(G)\times{\rm Sub}_G\longrightarrow{\rm Sub}_G,\ (T,H)\longmapsto T(H);\ T\in{\rm Aut}(G),\ H\in{\rm Sub}_G.\end{equation*}

It is easy to see that the map $H\mapsto T(H)$ defines a homeomorphism of ${\rm Sub}_G$ for every $T\in{\rm Aut}(G)$ (see e.g. [Reference Hamrouni and KADRI14, Proposition 2.1]), and the corresponding map ${\rm Aut}(G)\to {\rm Homeo}({\rm Sub}_{\rm G})$ is a group homomorphism.

All our groups G are locally compact (Hausdorff) and we now assume that they are second countable, which ensures that ${\rm Sub}_G$ is metrisable and also that G is metrisable.

We first state a criterion (cf. [Reference Benedetti and Petronio7, p. 161]) for the convergence of a sequence in ${\rm Sub}_G$ in the following lemma, which will be used often.

The following lemma lists some elementary facts about the convergence in ${\rm Sub}_G$ and also the interplay of the action of ${\rm Aut}(G)$ and the convergence.

Proof. (i) Let $h\in H$ . As $H_n\to H$ , Lemma 2·1 (ii) implies that there exists a sequence $\{h_n\}$ such that $h_n\in H_n$ for all n, and $h_n\to h$ . Since $H_n\subset L_n$ , we have that $h_n\in L_n$ , for all n. As $L_n\to L$ , Lemma 2·1 (i) implies that $h\in L$ . Hence $H\subset L$ .

(i) $\Rightarrow$ (ii) as we can put $H_n=H$ in (1), for all n.

(i) $\Rightarrow$ (iii) as we can take $H_n=T_n(H_0)$ and $L_n=T_n(L_0)$ in (1), for all n.

(ii) $\Rightarrow$ (iv) is obvious.

The following elementary result about the relation between the convergence of a sequence of subgroups and that of the corresponding sequence in the quotient group will be useful.

Proof. (i) Suppose $L_n\to L$ . Observe that $\{\pi(L_n)\}$ is relatively compact as ${\rm Sub}_{G/H}$ is compact. Let L ^′ be a limit point of $\{\pi(L_n)\}$ . As ${\rm Sub}_{G/H}$ is metrisable, there exists $\{n_k\}\subset{\mathbb N}$ such that $\pi(L_{n_k})\to L'$ . First we show that $\pi(L)\subset L'$ . By Lemma 2·1 (ii), for any $x\in L$ , there exists $\{x_n\}$ such that $x_n\in L_n$ , $n\in{\mathbb N}$ , and $x_n\to x$ in G. This implies that $\pi(x_n)\to\pi(x)$ (as $\pi$ is continuous). Therefore, $\pi(x)\in L'$ , and hence $\pi(L)\subset L'$ .

Now we show that $L'\subset\pi(L)$ . Suppose $x'\in L'$ . Again by Lemma 2·1 (ii), there exists $\{x^{\prime}_k\}$ such that $x^{\prime}_k\in\pi(L_{n_k})$ , $k\in{\mathbb N}$ , and $x^{\prime}_k\to x'$ in $G/H$ . For $k\in{\mathbb N}$ , let $x_k\in L_{n_k}$ be such that $\pi(x_k)=x^{\prime}_k$ . There exists a sequence $\{h_k\}\subset H$ such that $x_kh_k\to x$ , for some $x\in G$ . Now $x_kh_k\in L_{n_k}$ as $H\subset L_{n_k}$ , for all k, and hence $x\in L$ (by Lemma 2·1 (i)). Moreover, $x'=\pi(x)\in \pi(L)$ , and hence $\pi(L)=L'$ . Since this holds for all limit points of $\{\pi(L_n)\}$ , we have that $\pi(L_n)\to\pi(L)$ .

(ii) Suppose $\pi(L_n)\to L'$ . Here, $\{L_n\}$ is relatively compact as ${\rm Sub}_G$ is compact. Let $L\in{\rm Sub}_G$ be a limit point of $\{L_n\}$ . We show that $L=\pi^{-1}(L')$ . As ${\rm Sub}_G$ is metrisable, there exists a subsequence $\{L_{n_k}\}$ of $\{L_n\}$ such that $L_{n_k}\to L$ . From (i), we get that $\pi(L)=L'$ . As $H\subset L_{n_k}$ , for all k, $H\subset L$ (from Lemma 2·2 (ii)), which in turn implies that $L=\pi^{-1}(L')$ . Since this holds for all the limit points L of $\{L_n\}$ , it follows that $L_n\to\pi^{-1}(L')$

Recall that for a locally compact (Hausdorff) group G, if ${\rm Aut}(G)$ is endowed with the compact-open topology, the map ${\rm Aut}(G)\times G\to G$ , $(T,x)\mapsto T(x)$ , $T\in {\rm Aut}(G)$ , $x\in G$ , is continuous. If ${\rm Aut}(G)$ is endowed with the modified compact-open topology (which is finer than the compact-open topology), then ${\rm Aut}(G)$ is a topological group [Reference Stroppel32, 9·17], and we show that the action of the group ${\rm Aut}(G)$ on ${\rm Sub}_G$ is continuous. Note that for a connected Lie group G with the Lie algebra $\mathcal G$ , ${\rm Aut}(G)$ is a Lie group as a closed subgroup of ${\rm GL}(\mathcal G)$ and, the topology on ${\rm Aut}(G)$ inherited from ${\rm GL}(\mathcal G)$ coincides with the compact-open topology, as well as with the modified compact-open topology [Reference Arens3, Reference Hochschild16].

Proof. For a compact set C and an open set U in G, let $\lfloor C,U\rfloor=\{\phi\in{\rm Aut}(G)\mid \phi(C)\subset U\}$ and $\lfloor C,U\rfloor^{-1}=\{\phi\in{\rm Aut}(G)\mid \phi^{-1}\in \lfloor C,U\rfloor\}$ . They are open in ${\rm Aut}(G)$ and the collection of all such sets form a sub-basis for the modified compact-open topology on ${\rm Aut}(G)$ [Reference Stroppel32].

Let $T\in{\rm Aut}(G)$ and $H\in {\rm Sub}_G$ . Then T(H) belongs to either $O_1(K)$ or $O_2(U)$ for some compact set K or an open set U in G, where $O_1(K)$ and $O_2(U)$ are open sets in the sub-basis of the topology on ${\rm Sub}_G$ . Suppose $T(H)\in O_1(K)=\{L\in{\rm Sub}_G\mid L\cap K=\emptyset\}$ . Then $H\cap T^{-1}(K)=\emptyset$ . As H is closed, $T^{-1}(K)$ is compact and since G is locally compact, there exists an open relatively compact set W containing $T^{-1}(K)$ such that $H\cap \overline{W}=\emptyset$ . Then $\lfloor K,W\rfloor^{-1}\times O_1(\overline{W})$ is open in ${\rm Aut}(G)\times {\rm Sub}_G$ , it contains (T,H) and if $\phi\in \lfloor K,W\rfloor^{-1}$ and $L\in O_1(\overline{W})$ , then $\phi(L)\in O_1(K)$ .

Now suppose $T(H)\in O_2(U)=\{L\in{\rm Sub}_G\mid L\cap U\ne\emptyset\}$ . Since $T(H)\cap U\ne\emptyset$ , there exists $x\in H$ such that $T(x)\in U$ . Since G is locally compact and U is open, there exists an open relatively compact set $W_1$ such that $T(x)\in W_1\subset \overline{W}_1\subset U$ . Let $V=T^{-1}(W_1)$ . Then V is relatively compact. As $x\in V$ and $T(\overline{V})\subset U$ , we get that $(T,H)\in \lfloor\overline{V},U\rfloor\times O_2(V)$ which is open in ${\rm Aut}(G)\times{\rm Sub}_G$ . Moreover, if $\phi\in \lfloor\overline{V},U\rfloor$ and $L\in O_2(V)$ , then $\phi(L)\cap U\ne\emptyset$ . Therefore, the map ${\rm Aut}(G)\times{\rm Sub}_G\to{\rm Sub}_G$ as above is continuous.

3. Distality of automorphisms on G and ${\rm Sub}_G$

In this section, for $T\in{\rm Aut}(G)$ , we show that the distality of the T-action on ${\rm Sub}_G$ implies the distality of the corresponding action on ${\rm Sub}_{G/H}$ for any closed normal T-invariant subgroup H of G. We also compare the distality of the T-action on ${\rm Sub}_G$ and that of T on G. [Reference Shah28, theorem 1·3] shows that an automorphism T of G is distal if and only if for any closed normal T-invariant subgroup H, $T|_H$ is distal and T acts distally on $G/H$ . Lemma 3·1 and Example 3·2 illustrate that only a partial analogue of this theorem holds for the action of automorphisms on ${\rm Sub}_G$ . We prove some results about the distality of automorphisms on a connected Lie group which belong to ${\rm (NC)}$ . We also prove a result on the structure of a nilpotent group admitting a unipotent automorphism which is a generalisation of Kolchin’s Theorem for vector spaces; this will be useful to prove some of the main results. We end the section with a useful lemma about a criterion for the behaviour of an automorphism which implies that it does not belong to ${\rm (NC)}$ .

Proof. Suppose T acts distally on ${\rm Sub}_G$ . As ${\rm Sub}_H$ is closed and T-invariant, it is easy to see that T acts distally on ${\rm Sub}_H$ . Now we show that $\overline{T}$ acts distally on ${\rm Sub}_{G/H}$ . Suppose G is second countable. For $i=1,2$ , suppose $H_i\in{\rm Sub}_{G/H}$ and a sequence $\{n_k\}\subset{\mathbb Z}$ are such that $\overline{T}^{n_k}(H_i)\to L$ in ${\rm Sub}_{G/H}$ . Then $T(H)=H\subset \pi^{-1}(H_i)$ . By Lemma 2·3 (ii), $T^{n_k}(\pi^{-1}(H_i))\to\pi^{-1}(L)$ , for $i=1,2$ . As T is distal on ${\rm Sub}_G$ , $\pi^{-1}(H_1)=\pi^{-1}(H_2)$ , and hence $H_1=H_2$ . Thus, $\overline{T}$ acts distally on ${\rm Sub}_{G/H}$ in this case.

Now suppose G is not second countable. Let $S_H:=\{L\in{\rm Sub}_G\mid H\subset L\}$ . As $T(H)=H$ , it follows that $S_H$ is a closed T-invariant subspace of ${\rm Sub}_G$ . Let $\bar\pi:S_H\to{\rm Sub}_{G/H}$ , $\bar\pi(L)=\pi(L)=L/H$ , $L\in S_H$ . Then $\bar\pi$ is a homeomorphism such that $\overline{T}\circ\bar\pi=\bar\pi\circ T$ . As T acts distally on ${\rm Sub}_G$ , it acts distally on $S_H$ . Now using the homeomorphism $\bar\pi$ , it is easy to see that $\overline{T}$ acts distally on ${\rm Sub}_{G/H}$ .

The converse of the lemma does not hold as illustrated by the following.

Recall that $T\in{\rm Aut}(G)$ belongs to ${\rm (NC)}$ if for any discrete (closed) nontrivial cyclic subgroup A of G, $T^{n_k}(A)\not\to \{e\}$ in ${\rm Sub}_G$ for any sequence $\{n_k\}\subset{\mathbb Z}$ such that ${n_k}\to\infty$ or ${n_k}\to -\infty$ . Note that for T as in Example 3·2 and $G={\mathbb R}^2$ , $T\notin{\rm (NC)}$ . This is because one can choose a discrete cyclic subgroup $A=\{0\}\times{\mathbb Z}$ , for which $T^n(A)=\{(nz,z)\mid z\in {\mathbb Z}\}$ , $n\in{\mathbb Z}$ , and $T^n(A)\to\{(0,0)\}$ as $n\to\pm\infty$ .

A topological group is said to be monothetic if it has a dense cyclic subgroup. Monothetic groups are abelian. In a locally compact group, a closed monothetic subgroup is either compact or a discrete infinite cyclic group. The following lemma will be useful.

Proof. Let $\pi:G\to G/Z$ be the natural projection. Since each $A_i$ is monothetic and Z is a central subgroup of G, it is easy to see that each $\pi^{-1}(A_i)$ is abelian. By Lemma 2·3 (ii), $T^{n_k}(\pi^{-1}(A_i))\to\pi^{-1}(B)$ , $i=1,2$ . Therefore, $\pi^{-1}(B)\in {\rm Sub}^a_G$ . As T acts distally on ${\rm Sub}_G$ , we get that $\pi^{-1}(A_1)=\pi^{-1}(A_2)$ and hence that $A_1=A_2$ . The second assertion follows from the first.

The following useful lemma can be proved easily using Lemma 2·4. The lemma will apply in particular to the case of a connected Lie group G (with the Lie algebra $\mathcal G$ ) as ${\rm Aut}(G)$ is a Lie group whose topology (inherited from ${\rm GL}(\mathcal G)$ ) coincides with the compact-open topology, as well as with the modified compact-open topology.

For a locally compact group G, $T\in{\rm Aut}(G)$ and a compact T-invariant subgroup K of G, $C_K(T)=\{x\in G\mid T^n(x)K\to K\mbox{ in } G/K\mbox{ as } n\to\infty\}$ is known as the K-contraction group of T. The group $C(T)=C_{\{e\}}(T)$ is known as the contraction group of T. It has been shown in [Reference Raja and SHAH26] that T is distal if and only if both C(T) and $C(T^{-1})$ are trivial. For a connected Lie group G, it is well–known that $C_K(T)=C(T)K$ [Reference Hazod and SIEBERT15]; see [Reference Raja and SHAH26] for a more general result on this decomposition.

For an almost connected locally compact group G, let K be the largest compact normal subgroup. Then K is characteristic in G and $G/K$ is a Lie group. As observed in [Reference Raja and SHAH26], every inner automorphism of G acts distally on K. If any $T\in{\rm Aut}(G)$ acts distally on K, then C(T) is closed [Reference Raja and SHAH26, proposition 4·3]. The following useful lemma gives a more general result in the case of connected Lie groups. Note that the largest compact connected central subgroup of G is characteristic in G.

Proof. (i) Let $D=\{d_n\}$ be a countable subgroup in C(T) which is dense in $\overline{C(T)}$ and let $\mu=\sum_{n\in{\mathbb N}} 2^{-n}\delta_{d_n}$ be a probability measure on G. It is easy to see that $T^i(\mu)\to\delta_e$ as $i\to\infty$ . Now we can apply [Reference Dani and SHAH11, theorem 1·1] and get that $\overline{C(T)}= \textrm{supp}\mu\subset C_C(T)$ . As G is a connected Lie group, we have that $C_C(T)=C(T)C$ [Reference Hazod and SIEBERT15, theorem 2·4]. Therefore, we get that $\overline{C(T)}\subset C(T)C$ and (1) follows. Moreover, $C(T)C=\overline{C(T)}C$ and hence C(T)C is closed.

(ii) Now suppose T acts distally on C. Then $C(T)\cap C=\{e\}$ . Let $G'=C(T)C=\overline{C(T)}C$ . Then G ^′ is a closed T-invariant subgroup of G and $G'=C_C(\tau)$ , where $\tau=T|_{G'}$ . As $C(T)=C(\tau)$ , $C(\tau)\cap C=\{e\}$ . By [Reference Hazod and SIEBERT15, corollary 2·7], we get that $C(\tau)$ is closed in G ^′. As G ^′ is closed in G, we get that C(T) is closed. In particular, if G does not have any compact central subgroup of positive dimension, i.e. if C is trivial, then C(T) is closed.

For $T\in{\rm Aut}(G)$ , if the T-action on G is distal, it does not imply that the T-action on ${\rm Sub}_G$ is distal; see Example 3·2. Conversely, the following theorem shows that for a large class of locally compact groups G, the distality of the T-action on ${\rm Sub}_G$ implies the distality of the T-action on G.

Proof. Let G be as above and let $T\in{\rm Aut}(G)$ . Suppose T acts distally on ${\rm Sub}_G$ . Let K be as in the hypothesis. Note that $K^0$ is characteristic in G. In particular, it is T-invariant. We want to first show that the T-action on $G/K^0$ is distal.

Since T acts distally on ${\rm Sub}_G$ , by Lemma 3·1, T acts distally on ${\rm Sub}_{G/K^0}$ . Hence, without loss of any generality, we may assume that K as above is totally disconnected and show that T is distal. By [Reference Raja and SHAH26, theorem 4·1], T is distal if and only if both C(T) and $C(T^{-1})$ are trivial.

Step 1. We first assume that G is second countable. Suppose G is totally disconnected. Then $K=\{e\}$ . If possible, suppose that C(T) is nontrivial. Since G is totally disconnected, there exists a neighbourhood basis of open compact subgroups $\{C_m\}_{m\in{\mathbb N}}$ at the identity e in G. Choose m such that $C(T)\not\subset C_m$ . Let $H=C_m\cap \overline{C(T)}$ . As ${\rm Sub}_G$ is compact, there exists a sequence $\{n_k\}\subset{\mathbb N}$ such that $T^{-n_k}(H)\to L$ for some $L\in{\rm Sub}_G$ . As $\overline{C(T)}$ is T-invariant and $H\subset \overline{C(T)}$ , by Lemma 2·2 (iii) we get that $L\subset \overline{C(T)}$ . Let $x\in C(T)$ . Since $C_m$ is an open subgroup in G and C(T) is T-invariant, we get that $T^n(x)\in C_m\cap C(T)\subset H$ for all large n, and hence, that $x=T^{-n}(T^n(x))\in T^{-n}(H)$ for all large n. Therefore, $x\in L$ and hence, $C(T)\subset L$ . As L is closed, $\overline{C(T)}\subset L$ . Thus, we have that $\overline{C(T)}=L$ , i.e. $T^{-n_k}(H)\to\overline{C(T)}$ . As $\overline{C(T)}$ is T-invariant and $H\ne \overline{C(T)}$ , we get that T does not act distally on ${\rm Sub}_G$ , a contradiction. Therefore, C(T) is trivial. Interchanging T and $T^{-1}$ and arguing as above, we can show that $C(T^{-1})$ is trivial.

Step 2. Suppose G is not totally disconnected. Note that $G^0$ is T-invariant and $G/G^0$ is totally disconnected. Let $\overline{T}:G/G^0\to G/G^0$ be the automorphism corresponding to T. Then from Lemma 3·1, $\overline{T}$ acts distally on ${\rm Sub}_{G/G^0}$ . Hence, from the assertion in Step 1, $C(\overline{T})$ and $C(\overline{T}^{-1})$ are trivial. Therefore, $C(T)\cup C(T^{-1})\subset G^0$ . As K is totally disconnected and normal in $G^0$ , by [Reference Shah29, lemma 2·2], it is central in $G^0$ (this also follows from [Reference Iwasawa18, theorem 1^′]). Now ${\rm Sub}_K={\rm Sub}^a_K$ . From Step 1, if T acts distally on ${\rm Sub}^a_K$ , we get that T acts distally on K. Now it is enough to show that T acts distally on $G/K$ . As T acts distally on ${\rm Sub}_{G/K}$ , replacing G by $G/K$ and T by the corresponding automorphism of $G/K$ , we may assume that G is a connected Lie group without any nontrivial compact normal subgroup. We have also assumed that T acts distally on ${\rm Sub}_G$ . We want to show that T is distal.

Step 3. We prove a more general statement: if G is any connected Lie group, $T\in{\rm Aut}(G)$ acts distally on the largest compact connected central subgroup of G and if $T\in{\rm (NC)}$ , then T is distal.

Assume that G and T are as above. By Lemma 3·5 (ii), both C(T) and $C(T^{-1})$ are closed, and hence, simply connected and nilpotent.

If possible, suppose C(T) is nontrivial. Let $V=Z(C(T))$ , the center of C(T). Then $V\cong{\mathbb R}^m$ , for some $m\in{\mathbb N}$ . Let $T_1=T|_{V}$ be the restriction of T to V. Then $T_1\in{\rm GL}(V)$ and $C(T_1)=V$ , which is turn implies that all the eigenvalues of $T_1$ have absolute value less than 1. Suppose $T_1$ has a real eigenvalue $\lambda$ . Then $0<|\lambda|<1$ and there exists a subspace $W\cong{\mathbb R}$ in V such that $T_1(x)=\lambda x$ for all $x\in W$ . For the discrete subgroup ${\mathbb Z}$ in ${\mathbb R}\cong W$ , it is easy to see that $T_1^{-n}({\mathbb Z})=\lambda^{-n}{\mathbb Z}\to\{0\}$ as $n\to\infty$ in ${\rm Sub}_{\mathbb R}$ [Reference Baik and ClavierL5]. As $T|_W=T_1|_W$ , this leads to a contradiction as $T\in{\rm (NC)}$ .

Now suppose all the eigenvalues of $T_1$ are complex. There exists a $T_1$ -invariant subspace $W'\cong{\mathbb R}^2$ in V such that $T_1|_{W'}$ has a complex eigenvalue of the form $r(\cos\theta+i\sin\theta)$ , where $r\in{\mathbb R}$ and $0<r<1$ . Let $T_2\in {\rm GL}(2,{\mathbb R})$ be such that $T|_{W'}=T_1|_{W'}=T_2$ (under the isomorphism of ${\mathbb R}^2$ with W ^′). Then $T_2=rA_\theta$ , where $A_\theta=AR_\theta A^{-1}$ for some $A\in {\rm GL}(2,{\mathbb R})$ , and $R_\theta$ is the rotation by the angle $\theta$ on ${\mathbb R}^2$ . Here, $A_\theta$ generates a relatively compact group in ${\rm GL}(2,{\mathbb R})$ . Let $Z=\{(m,0)\mid m\in{\mathbb Z}\}$ . Then $Z\in{\rm Sub}_{{\mathbb R}^2}$ . Since $0<r<1$ , we have $r ^{-n}Z\to\{(0,0)\}$ as $n\to\infty$ [Reference Baik and ClavierL5], and that $r\,{\rm Id}\not\in{\rm (NC)}$ . Since $T_2=(r\,{\rm Id})A_\theta=A_\theta(r\,{\rm Id})$ , and $A_\theta$ generates a relatively compact group, by Lemma 3·4, we get that $T_2\not\in{\rm (NC)}$ . As $T_2=T|_{W'}$ , this leads to a contradiction. Hence C(T) is trivial. Replacing T by $T^{-1}$ and arguing as above, we get that $C(T^{-1})$ is also trivial. Therefore, T is distal.

We have proved the first assertion in case G is second countable. Now suppose G is not second countable. As noted before step 1, K can be assumed to be totally disconnected and T acts distally on ${\rm Sub}_G$ . We need to show that T is distal. Suppose $x\in G$ is such that $T^{n_k}(x)\to e$ in G for sone sequence $\{n_k\}\subset {\mathbb Z}$ . Let $L_x$ be the subgroup generated by $O_x=\{T^n(x)\mid n\in{\mathbb Z}\}$ in G and let $H=\overline{L_xG^0}$ . Then H is a closed T-invariant subgroup of G. As $O_x$ , and hence, $L_x$ is countable, we have that $H/H'$ is countable for some open almost connected subgroup H ^′ of H. Since H ^′ is compactly generated and first countable, and hence second countable, we get that H is second countable. As $H^0=G^0$ , K is the largest compact normal subgroup of $H^0$ and K is totally disconnected. Since T acts distally on ${\rm Sub}_G$ and since ${\rm Sub}_H$ is closed in ${\rm Sub}_G$ , we have that $T|_H$ acts distally on ${\rm Sub}_H$ . As H is second countable, we get from above that $T|_H$ is distal. Therefore, $x=e$ and T is distal.

We have proved that if T acts distally on ${\rm Sub}_G$ , then T acts distally on $G/K^0$ , where K is the largest compact normal subgroup of $G^0$ . Moreover, if T also acts distally on $K^0$ , then it is easy to show that T acts distally on G.

We get a stronger result in case of Lie groups as follows.

Proof. As $G/G^0$ is discrete and C as above is characteristic in G, T acts distally on G (resp. $G/C$ ) if and only if T acts distally on $G^0$ (resp. $G^0/C$ ). Moreover, ${\rm Sub}^a_{G^0}$ is closed in ${\rm Sub}^a_G$ and, if $T\in{\rm (NC)}$ (resp. T acts distally on ${\rm Sub}^a_G$ ), then $T|_{G^0}\in{\rm (NC)}$ (resp. $T|_{G^0}$ acts distally on ${\rm Sub}^a_{G^0}$ ). Thus, to prove (a) and (b), we may assume that G is connected and C is central in G. Now (a) follows from the general statement in step 3 of the proof of Theorem 3·6. To prove (b), suppose T acts distally on ${\rm Sub}^a_G$ , where G is connected as assumed above. As C is central in G, by Lemma 3·3 we get that the automorphism of $G/C$ corresponding to T belongs to ${\rm (NC)}$ . As $G/C$ does not have a compact central subgroup of positive dimension, (a) implies that T acts distally on $G/C$ .

A locally compact group G is said to be pointwise distal if every inner automorphism is distal on G. The group G is said to be distal if the conjugacy action of G on G is distal, i.e. for $x\in G$ , the closure of $\{gxg^{-1}\mid g\in G\}$ does not contain the identity e unless $x=e$ . We get the following for inner automorphisms of almost connected groups.

Proof. Let K be the largest compact normal subgroup of G. Let T be an inner automorphism of G. It is easy to see that T acts distally on K (see the last part of the proof of [Reference Raja and SHAH26, theorem 5·2]). Hence C(T) is closed and a simply connected nilpotent Lie group [Reference Raja and SHAH26, proposition 4·3]. In particular, it has no nontrivial compact subgroups. Moreover, if $T\in{\rm (NC)}$ , then $T|_{C(T)}$ also belongs to ${\rm (NC)}$ , and from Corollary 3·7 (a), it follows that $T|_{C(T)}$ is distal i.e. $C(T)=\{e\}$ . Since this is true for all inner automorphisms of G, by [Reference Raja and SHAH26, theorem 4·1], we have that G is pointwise distal, and hence it is distal [Reference Rosenblatt27, theorem 9]. The second assertion follows from the first.

Recall that an automorphism T of a connected Lie group is unipotent if the corresponding map ${\rm d}\, T$ on the Lie algebra is a unipotent linear transformation, i.e. all the eigenvalues of ${\rm d}\, T$ are equal to 1. The following proposition will be very useful. In the special case of vector spaces, it is well known as Kolchin’s Theorem. It is also easy to deduce for compact connected abelian Lie groups by considering the corresponding map on the Lie algebra (see also [Reference Abels2, lemma 2·5]).

Proof. Suppose $T\ne{\rm Id}$ . As G is nilpotent, $G/K$ is simply connected, where K is the largest compact connected central subgroup in G. Then G has a sequence of closed connected normal T-invariant subgroups $\{e\}=Z_0\subset Z_1=K\subset \cdots \subset Z_m=G$ , such that $Z_1/Z_0=K$ and if G is not compact, $Z_i/Z_{i-1}$ is a vector group and it is the center of $G/Z_{i-1}$ , for all $i=2,\ldots, m$ . The automorphism on each $Z_i/Z_{i-1}$ corresponding to T is unipotent. From Kolchin’s Theorem for vector spaces and [Reference Abels2, lemma 2·5] for compact connected abelian Lie groups, it follows that there exists a sequence of closed connected normal subgroups in each $Z_i/Z_{i-1}$ such that T-acts trivially on each successive quotient. Taking the pre-images of these subgroups in $Z_i$ , we see that they are closed and T-invariant. Also since $Z_i/Z_{i-1}$ is central in $G/Z_{i-1}$ , any subgroup of $Z_i$ which contains $Z_{i-1}$ is normal in G. Now we have a finite sequence of closed connected normal T-invariant subgroups $\{e\}=H_0\subset \cdots\subset H_l=G$ , such that T acts trivially on $H_i/H_{i-1}$ , $i=1,\ldots l$ . As $T\ne{\rm Id}$ and T acts trivially on $H_1$ , there exists $i_1$ such that $1\leq i_1< l$ , T acts trivially on $H_{i_1}$ and T does not act trivially on $H_{i_1+1}$ . Let $G_1=H_{i_1}$ . Having chosen $G_k=H_{i_k}$ for $k\geq 1$ , if $G_k\ne G$ (equivalently, $i_k\ne l$ ), we choose $G_{k+1}=H_j$ , where $i_k<j\leq l$ and j is the largest such natural number such that T acts trivially on $H_j/G_k$ . As l is finite, there exists n such that $G_n=H_l=G$ .

The following is known but we give a proof for the sake of completeness. It does not hold for general compact abelian groups as there are some compact totally disconnected abelian groups which are torsion-free, e.g. ${\mathbb Z}_p$ , the ring of p-adic integers in ${\mathbb Q}_p$ , p a prime.

Proof. Note that each $B_n$ is a finite subgroup and as G is abelian, $B_mB_n\subset B_{mn}$ , $m,n\in{\mathbb N}$ . Hence B is a group. It follows from the first statement that B is dense in G.

Let $B'=\cup_n B_{a_n}$ . We may assume that $\{a_n\}$ is strictly increasing and that $a_n\to \infty$ . As $G={\mathbb T}^k=({\mathbb S}^1)^k$ , where ${\mathbb S}^1$ is the unit circle and $k=\dim G$ , it is enough to prove the statement for $G={\mathbb S}^1$ . There exists a continuous real one-parameter subgroup $\{x(t)\}_{t\in{\mathbb R}}$ such that $x(0)=x(1)=e$ , $x(t)\ne e$ for all $t\in ]0,1[$ and $G=\{x(t)\mid t\in[0,1[\}$ . Now $x(1/a_n)\in B_{a_n}$ and $x(1/a_n)\to x(0)=e$ . Let $t\in ]0,1[$ . As $1/a_n\to 0$ , there exist $m_n\in{\mathbb N}$ , $n\in{\mathbb N}$ , such that $m_n<a_n$ for all large n, and $m_n/a_n\to t$ . Therefore, $x(m_n/a_n)\to x(t)$ . As $x(m_n/a_n)\in B_{a_n}$ , we get that B ^′ is dense in G.

Note that in a locally compact group G, the subgroup $G_x$ generated by an element x is either closed and hence a discrete cyclic group, or its closure $\overline{G}_x$ , being a compactly generated (locally compact) abelian group, is compact. If G is a Lie group, then either $G_x$ is discrete or $\overline{G}_x$ is a compact Lie group with finitely many connected components.

We now state and prove a lemma which will play a crucial role in the proofs of main results in Section 4. Since we need to use Lemma 3·11 for the proof, we state it only for Lie groups even though it may hold for some locally compact (non-discrete) groups.

For $T\in{\rm Aut}(G)$ , let $S_T=\{x\in G\mid T(x)=x\}$ be the stabiliser of T. It is a closed subgroup of G.

Proof. Let G, T, H, x and y be as above. Suppose $G_x\cap S_T=\{e\}$ . Let G(x,y) be the subgroup of G generated by x and y. Then G(x,y) is countable and T-invariant, and its closure $\overline{G(x,y)}$ is a T-invariant Lie subgroup, which is separable, and hence second countable. In particular, $\overline{G(x,y)}$ has countably many connected components. Replacing G by $\overline{G(x,y)}$ , T by its restriction to ${\overline{G(x,y)}}$ , and H by $\overline{G(x,y)}\cap H$ , we may assume that G is second countable and $G/H$ is an abelian Lie group which is either discrete or compact. Now ${\rm Sub}_G$ is metrisable. We show that $T\not\in{\rm (NC)}$ . Since H is T-invariant and normal in G, we get for every $n\in{\mathbb Z}$ that $T(x^n)=x^ny_n$ for some $y_n\in H$ , where $y_n\ne e$ unless $x^n=e$ , as $G_x\cap S_T=\{e\}$ .

Step 1. Suppose $G/H$ is discrete, i.e. $G=G_xH$ . We first show that $G_x$ is discrete. Observe that if $G_xH/H$ is finite, then for some $n\in{\mathbb N}$ , $x^n\in G_x\cap H\subset G_x\cap S_T=\{e\}$ , and hence $G_x$ is finite. If possible, suppose $G_x$ is not discrete. Then $G_x$ is infinite and relatively compact. This implies that $G/H=G_xH/H$ is compact, and hence it is finite. Then we get as above that $G_x$ is finite, and it leads to a contradiction. Therefore, $G_x$ is discrete, and hence, closed.

There exists an unbounded sequence $\{n_k\}\subset{\mathbb N}$ such that $T^{n_k}(G_x)\to L$ (say). We show that $L=\{e\}$ . If possible, suppose $a\in L$ is such that $a\ne e$ . Replacing a by $a^{-1}$ if necessary, we get that there exists $\{m_k\}\subset{\mathbb N}$ such that $T^{n_k}(x^{m_k})\to a$ . If $G_x$ is finite, then passing to a subsequence if necessary, we can choose $m_k=m_1$ for all k. Suppose $G_x$ is infinite. As observed above, $G/H$ is also infinite. Let $\pi:G\to G/H$ be the natural projection. Since $T(x)\in xH$ , we have that $\pi(x^{m_k})\to \pi(a)$ . As $G/H$ is infinite and discrete, we get that $\{m_k\}$ is eventually constant. In either case, we have that $T^{n_k}(x^m)\to a$ for some $m\in{\mathbb N}$ . Then $T^{n_k}(x^m)=x^my_m^{n_k}\to a$ . Also, since $a\ne e$ , we get that $x^m\ne e$ and hence $y_m\ne e$ . Therefore, $\{y_m^{n_k}\}_{k\in{\mathbb N}}$ is convergent, and hence $y_m$ is contained in a nontrivial compact subgroup in H. This leads to a contradiction. Therefore, $L=\{e\}$ and $T\not\in{\rm (NC)}$ .

Step 2. Now suppose $G/H$ is not discrete. As $G_xH$ is dense in G, we have that $G/H$ is compact and $G_x$ is infinite. Since $G/H$ is abelian and $H\subset S_T$ , we have that $S_T$ is normal in G. Moreover, $G/S_T$ is infinite, as $G_x$ is infinite and $G_x\cap S_T=\{e\}$ . As G is a Lie group, $G^0$ , and hence, $G^0H$ is an open subgroup in G. Therefore, $G/G^0H$ , being compact, is finite. We may replace x by $x^n$ for some $n\in{\mathbb N}$ and assume that $G=G^0H$ . Here, $G^0/(G^0\cap H)$ is connected, and it is also compact as it is isomorphic to $G/H$ . Since $G/S_T$ is infinite and $H\subset S_T$ , we get that $G/S_T=(G^0H)/S_T$ , and hence $G^0/(G^0\cap S_T)$ is compact and infinite.

Note that $G^0$ is T-invariant, T acts trivially on $H\cap G^0$ and on $G^0/(H\cap G^0)$ . Also, $xh\in G^0$ for some $h\in H$ , $T(xh)\in xH\cap G^0=xh(H\cap G^0)$ and $G_{xh}\cap S_T=\{e\}$ . Moreover $G_{xh}(H\cap G^0)$ is dense in $G^0$ and $G_{xh}$ is infinite, as $G_{xh}H=G_xH$ . We may replace x by xh and assume that $x\in G^0$ , and we may also replace G, H by $G^0$ , $H\cap G^0$ respectively, and assume G is a connected Lie group, $G/H$ is a compact connected abelian Lie group and $G/S_T$ is infinite and compact. Note that $T(g)\in gH$ for all $g\in G$ . Let $Q=\{g^{-1}T(g)\mid g\in G\}$ . Then $e\in Q\subset H$ and, Q is connected as G is so. Therefore, $Q\subset H^0$ , and hence $T(g)\in gH^0$ for all $g\in G$ . Here, $H/H^0$ , being a discrete normal subgroup of $G/H^0$ , is central in $G/H^0$ . Hence, $G/H^0$ is a covering group of $G/H$ , and the later is abelian. Therefore, $G/H^0$ is abelian and, being connected, it is isomorphic to ${\mathbb R}^n\times {\mathbb T}^m$ , for $n,m\geq 0$ and $m+n\in{\mathbb N}$ . Let $\pi_0: G\to G/H^0$ be the natural projection.

Suppose $m\in{\mathbb N}$ . Let $M=\pi_0^{-1}({\mathbb T}^m)$ . Then M is a closed connected T-invariant Lie subgroup and $H^0\subset M$ . By Theorem 3·7 in Ch. XV of [Reference Hochschild17], $M=CH^0$ , where C is a maximal compact connected subgroup of G. Now $M=C\ltimes H^0$ , since $H^0$ does not have any nontrivial compact subgroup. Observe that C is isomorphic to $\pi_0(C)$ , and hence it is nontrivial and abelian. Suppose $M\not\subset S_T$ , i.e. $C\cap S_T\subsetneq C$ . Note that $C\cap S_T$ is a closed (abelian) subgroup of C.

As C is a compact connected abelian Lie group and the set of primes is an unbounded set in ${\mathbb N}$ , by Lemma 3·11, the set of elements of prime orders in C is dense in C. As $C\setminus (C\cap S_T)$ is open in C, there exists $b\in C\setminus (C\cap S_T)$ such that $b^p=e$ for some prime p. Since $C/(C\cap S_T)$ is also a compact connected abelian Lie group, we have that $b^n\in C\setminus (C\cap S_T)$ for all $n<p$ and $T(b^n)=b^nh_n$ for some $h_n\in H$ and $h_n\ne e$ , $1\leq n<p$ . Let $G_b$ be the cyclic subgroup generated by b. Then $G_b\cap S_T=\{e\}$ . Note that $\pi_0(G_b)$ is finite and discrete in $M/H^0$ , and hence in $G/H^0$ . Now arguing as in Step 1 for b, $H^0$ instead of x, H respectively, we get that $T\not\in{\rm (NC)}$ .

Now suppose $m=0$ or $C\subset S_T$ . Then $G/(CH^0)={\mathbb R}^n$ , where $n\in{\mathbb N}$ and, either $C=\{e\}$ or $C\subset S_T$ . As $H\subset S_T\subsetneq G$ and $G/S_T$ , being a compact connected abelian Lie group, is monothetic, there exists $x_1\in G\setminus S_T$ such that its image in $G/S_T$ generates a (dense) infinite group. Hence $G_{x_1}\cap S_T=\{e\}$ , where $G_{x_1}$ is the group generated by $x_1$ in G. Moreover $\pi_0(x_1)$ generates a discrete infinite group in $G/CH^0$ , and hence in $G/H^0$ . Arguing as in Step 1, for $x_1$ , $H^0$ instead of x, H respectively, we get that $T\not\in{\rm (NC)}$ .

4. Characterisation of automorphisms of Lie groups G which act distally on ${\rm Sub}_G$

In this section, we state and prove some of the main results characterising the class of automorphisms which belong to (NC) or act distally on ${\rm Sub}^a_G$ (see Theorems 4·1, 4·3 and 4·4). We also prove Proposition 4·2 for connected solvable Lie groups which will be useful in proving these theorems. After proving Theorems 4·1, we prove Theorems 4·3 and 4·4 which show for all connected Lie groups G that if T is unipotent or if $T\in ({\rm Aut}(G))^0$ , then the following holds: T acts distally on ${\rm Sub}^a_G$ if and only if T is contained in a compact subgroup of ${\rm Aut}(G)$ . We also characterise connected Lie groups G which act distally on ${\rm Sub}^a_G$ (see Corollary 4·5).

Note that Theorem 4·1 does not hold for all connected Lie groups and the condition in the hypothesis is necessary as illustrated by the example in Remark 3·8 of the group $G={\mathbb T}^n$ , $n\geq 2$ , for which every $T\in{\rm Aut}(G)\cong{\rm GL}(n,{\mathbb Z})$ belongs to ${\rm (NC)}$ . However, it admits many distal (unipotent) and non-distal automorphisms, each of which generates a discrete infinite group in ${\rm Aut}(G)$ .

Before proving the theorem, we prove a somewhat stronger result about distality for unipotent automorphisms on connected solvable Lie groups. Note that Proposition 4·2 (i) below will later be generalised to all connected Lie groups, see Theorem 4·3.

Proof. Step 1. Let G be as in the hypothesis. Suppose $T\in{\rm Aut}(G)$ acts distally on ${\rm Sub}^a_G$ . Let K be the largest compact connected central subgroup of G. Then K is characteristic, and in particular, it is T-invariant and normal in G. If possible, suppose $T|_K\ne{\rm Id}$ . By Proposition 3·10, there exist T-invariant compact connected (normal) nontrivial subgroups $K_1\subset K_2$ such that T acts trivially on $K_1$ and on $K_2/K_1$ but it does not act trivially on $K_2$ . We may, without loss of any generality, assume that $K_1$ is the largest connected subgroup in $K_2$ such that $T|_{K_1}={\rm Id}$ . As $K_1$ and $K_2$ are compact connected abelian Lie groups, and $\dim K_1<\dim K_2$ , we can get a nontrivial compact connected subgroup $M\subset K_2$ such that $M\cap K_1=\{e\}$ . As $M\subset K_2$ , for every $x\in M$ , $T(x)=xy$ , for some $y\in K_1$ which depends on x. As M is monothetic, we can choose $g\in M$ , such that g generates a dense subgroup in M. Let $h\in K_1$ be such that $T(g)=gh$ .

We now show that h has infinite order. If possible, suppose h has finite order m (say). Then $T(g^{mn})=g^{mn}$ , $n\in{\mathbb N}$ , and as $\{g^{mn}\mid n\in{\mathbb N}\}$ is dense in M, we get that T acts trivially on M. This leads to a contradiction to our assumption on $K_1$ as $K_1\ne MK_1\subset K_2$ . As $K_1$ is compact, there exists an unbounded sequence $\{n_k\}\subset{\mathbb N}$ such that $h^{n_k}\to h$ . Passing to a subsequence if necessary, we may assume that $T^{n_k}(M)\to L$ for some compact group $L\subset K_2$ . As $\{g^n\}_{n\in{\mathbb N}}$ is dense in M and $T(g)=gh$ , we have that $T^{n_k}(M)\subset MK_h$ , where $K_h$ is the closed subgroup generated by h in $K_1$ . As $T^{n_k}(g)=gh^{n_k}\to gh$ , we get that $gh\in L$ . As M is a connected abelian Lie group, there exists a continuous real one-parameter group $\{g(t)\}_{t\in{\mathbb R}}\subset M$ such that $g(0)=e$ and $g(1)=g$ . Let $x_k=g(1/n_k)\in M$ , $k\in{\mathbb N}$ . We have that $x_k\to e$ and $x_k^{n_k}=g$ . Now

\begin{equation*}gh=T(g)=T(x_k)^{n_k}=(x_ky_k)^{n_k}=gy_k^{n_k}\mbox{ for some }y_k\in K_1.\end{equation*}

In particular, $h=y_k^{n_k}$ for all k, and it implies that $T^{n_k}(gx_k^{-1})=gh^{n_k}h^{-1}x_k^{-1}\to g$ . Therefore, $g\in L$ and hence $h\in L$ . As L is a closed subgroup, we have that $M\subset L$ and $K_h\subset L$ . Since $T^{n_k}(M)\subset MK_h$ for all k, we have that $L=MK_h$ . As $T(MK_h)=MK_h$ , this contradicts the hypothesis that T acts distally on ${\rm Sub}^a_G$ . Therefore, $T|_K={\rm Id}$ . If G is compact, then $G=K$ and $T={\rm Id}$ .

Step 2. Suppose G is not compact. Let $\pi:G\to G/K$ be the natural projection and let $\overline{T}$ be the automorphism of $G/K$ corresponding to T. Then $\overline{T}$ is also unipotent, and by Lemma 3·3, $\overline{T}\in{\rm (NC)}$ in ${\rm Aut}(G/K)$ . Suppose (ii) holds. Since $G/K$ does not have any compact central subgroup of positive dimension, the statement in (ii) implies that $\overline{T}={\rm Id}$ . Now we have that $T|_K={\rm Id}$ and $T(xK)=xK$ for all $x\in G$ . Let N be the nilradical of G, then $K\subset N$ , $N/K$ is simply connected and $G/N$ is abelian and isomorphic to ${\mathbb R}^n\times {\mathbb T}^m$ , where $m,n\geq 0$ . Let R be the closed connected (solvable) normal subgroup containing N, such that $R/N={\mathbb R}^n$ , $K\subset N\subset R$ and $R/K$ is a closed connected solvable normal subgroup of $G/K$ without any nontrivial compact subgroup. As $G/R={\mathbb T}^m$ is compact, we get by [Reference Hochschild17, chapter XV, theorem 3·7], that $G=CR$ , where C is a maximal compact connected subgroup of G which is abelian. As $K\subset C$ , we have $T(C)=C$ . Here, $T|_C$ is also unipotent and from step 1, we get that $T|_C={\rm Id}$ .

Now it is enough to show that $T|_R={\rm Id}$ . For every $x\in R$ , $T(x)\in xK$ . There exists a neighbourhood U of the identity e in G such that every $x\in U$ is exponential. If $T(x)=x$ for every $x\in U\cap R$ , then $T|_R={\rm Id}$ , since $U\cap R$ generates R. If possible, suppose $x\in U\cap R$ is such that $x\not\in K$ and $T(x)\ne x$ . Since x is exponential, it can be embedded in a continuous real one-parameter subgroup $\{x(t)\}_{t\in{\mathbb R}}$ as $x=x(1)$ . We know that $\{x(t)\}_{t\in{\mathbb R}}$ is unbounded (and closed) as $R/K$ has no nontrivial compact subgroups. Note that $T(x(t))=x(t)k(t)$ , $k(t)\in K$ for all $t\in{\mathbb R}$ and $k(1)=k\ne e$ . Let $M'=\overline{\{k(t)\mid t\in{\mathbb R}\}}$ . It is a connected compact subgroup of K. Let $C_x=\{x(t)\}_{t\in{\mathbb R}}$ . It is a closed connected abelian subgroup of R. For some $t_0\in{\mathbb R}$ , $k(t_0)$ has infinite order, and hence, there exists an unbounded sequence $\{n_m\}\subset{\mathbb N}$ such that $k(t_0)^{n_m}\to k(t_0)$ . Passing to a subsequence, we get that $\{k(t_0/l!)^{n_m}\}$ converges for every $l\in{\mathbb N}$ . As ${\rm Sub}^a_G$ is compact, passing to a subsequence if necessary, we get that $\{T^{n_m}(C_x)\}$ converges to a closed abelian subgroup H (say). Now $T^{n_m}(x(t_0))=x(t_0)k(t_0)^{n_m}\to x(t_0)k(t_0)$ . Therefore, $x(t_0)k(t_0)\in H$ . Also, $T^n(x(t/n))=x(t/n)k(t)$ , $n\in{\mathbb N}$ . Therefore, $T^{n_m}(x(t/n_m))=x(t/n_m)k(t)\to k(t)\in H$ for all t, i.e. $M'\subset H$ and hence $x(t_0)\in H$ . Moreover, $T^{n_m}(x(t_0/l!))=x(t_0/l!)k(t_0/l!)^{n_m}\to x(t_0/l!)h_l$ for some $h_l\in M'$ such that $k(t_0/l!)^{n_m}\to h_l$ for all $l\in{\mathbb N}$ . Hence $x(t_0/l!)\in H$ for all $l\in{\mathbb N}$ . As H is a closed subgroup, we get that $C_x\subset H$ , and hence that $C_xM'\subset H$ . Since $C_xM'$ is T-invariant, we get by Lemma 2·2 (iii) that $H=C_xM'$ . As $C_x\ne C_xM'$ , it contradicts the hypothesis that T acts distally on ${\rm Sub}^a_G$ . Therefore, $T(x)=x$ for every $x\in U\cap R$ and hence, $T|_R={\rm Id}$ . This implies that $T={\rm Id}$ assuming that (ii) holds. Therefore, (i) holds if (ii) holds.

We now prove (ii). Suppose G has no compact central subgroup of positive dimension. Then its nilradical N is simply connected. Moreover, ${\rm Aut}(G)$ is almost algebraic [Reference Dani10, theorem 1], i.e. a closed subgroup of a finite index in an algebraic subgroup of ${\rm GL}(\mathcal G)$ , where $\mathcal G$ is the Lie algebra of G. There exists a continuous unipotent one-parameter subgroup $\{T_t\}_{t\in{\mathbb R}}\subset{\rm Aut}(G)$ such that $T=T_1$ . Let $G'=\{T_t\}_{t\in{\mathbb R}}\ltimes G$ . Then G ^′ is a connected solvable Lie group. By [Reference Raghunathan24, proposition 3·11], the commutator subgroup of G ^′ is nilpotent and, as it is contained in G, it is contained in N. This implies that each $T_t$ acts trivially on $G/N$ .

By Proposition 3·10, either $T|_N={\rm Id}$ or there exists a sequence of closed connected normal T-invariant subgroups of N; $\{e\}=G_0\subset\cdots \subset G_m=N$ , $m\geq 2$ , such that T acts trivially on $G_k/G_{k-1}$ , $1\le k\le m$ , and T does not act trivially on $G_{k+1}/G_{k-1}$ , $1\le k\le m-1$ .

Suppose $T\in{\rm (NC)}$ . If possible, suppose $T|_N\ne {\rm Id}$ . Note that $G_1\subset G_2\cap S_T\subsetneq G_2$ , where $S_T$ is the stabiliser of T. There exists $x\in G_2\setminus (G_2\cap S_T)$ such that the image of x in $G_2/(G_2\cap S_T)$ generates an infinite group; this is true since $G_2/G_1$ is (simply) connected and nilpotent. Now $G_x\cap S_T=\{e\}$ , where $G_x$ is the cyclic subgroup generated by x. Moreover, $T(x)=xy$ , $y\ne e$ and $y\in G_1$ . As N is simply connected and nilpotent, $G_1$ does not contain any nontrivial compact subgroup. By Lemma 3·12, it follows that $T|_{G_2}\notin{\rm (NC)}$ , and hence $T|_N\not\in{\rm (NC)}$ , which contradicts the hypothesis. Hence $T|_N={\rm Id}$ . If G is nilpotent, then $T={\rm Id}$ . Now suppose G is not nilpotent.

As observed above, T acts trivially on $G/N$ . If possible, suppose $T\ne{\rm Id}$ . Then $N\subset S_T\ne G$ and, as $[G,G]\subset N$ [Reference Raghunathan24, proposition 3·11], we get that $S_T$ is a proper closed normal subgroup of G and $G/S_T$ is a nontrivial connected abelian group. There exists $x\in G\setminus S_T$ such that the image of x in $G/S_T$ generates an infinite group. In particular, $G_x\cap S_T=\{e\}$ , where $G_x$ is the cyclic group generated by x. As T acts trivially on $G/N$ , $T(x)=xg$ for some $g\in N$ and $g\ne e$ . Since N is simply connected and nilpotent, it has no nontrivial compact subgroup. Since $T|_N={\rm Id}$ , we get by Lemma 3·12 that $T\not\in{\rm (NC)}$ . This contradicts the hypothesis, and hence $T={\rm Id}$ .

Proof of Theorem 4·1. Let G be a connected Lie group such that it does not admit any nontrivial compact connected central subgroup and let $T\in {\rm Aut}(G)$ . It is easy to see that (iv) $\Rightarrow \textrm{(iii)}\Rightarrow \textrm{(ii)}\Rightarrow \textrm{(i)}$ . Now suppose (i) holds, i.e. $T\in{\rm (NC)}$ . To show the equivalence of $\textrm{(i)}-\textrm{(iv)}$ , it is enough to show that (iv) holds; i.e. we need to show that T is contained in a compact subgroup of ${\rm Aut}(G)$ .

Step 1. Let $\mathcal G$ be the Lie algebra of G. For any $\tau\in{\rm Aut}(G)$ , let ${\rm d}\, \tau$ be the corresponding Lie algebra automorphism of $\mathcal G$ . Then we can view ${\rm Aut}(G)$ as a closed subgroup of ${\rm GL}(\mathcal G)$ . By Corollary 3·7 (a), we have that T acts distally on G. By [Reference Abels2, theorem 1·1], the Lie algebra automorphism ${\rm d}\, T$ on the Lie algebra $\mathcal G$ of G corresponding to T is distal and hence, all the eigenvalues of ${\rm d}\, T$ have absolute value 1 [Reference Abels1, theorem 1^′]. Since G has no compact central subgroup of positive dimension, by [Reference Dani10, theorem 1] (see also [Reference Previts and WU22]) ${\rm Aut}(G)$ is an almost algebraic subgroup of ${\rm GL}(\mathcal G)$ , i.e. a closed subgroup of finite index in an algebraic group. Let H be the smallest almost algebraic subgroup containing T in ${\rm Aut}(G)$ . Then H is abelian and it acts distally on G [Reference Abels1, corollary 2·3], and $H=\mathcal K\times\mathcal U$ , where $\mathcal K$ is a compact abelian group and $\mathcal U$ is unipotent and abelian, and in fact, a unipotent one-parameter group $\{T_t\}_{t\in{\mathbb R}}$ [Reference Abels1, corollary 2·5]. Note that all the eigenvalues of each $T_t$ are equal to 1. Then each $T_t$ acts distally on G [Reference Abels1, theorem 1^′]. Let $T=T_sT_u=T_uT_s$ , where $T_s\in \mathcal K$ and $T_u=T_{t_0}\in\mathcal U$ , for some $t_0\in{\mathbb R}$ , $t_0\ne 0$ . By Lemma 3·4, $T_u$ , and hence each $T_t$ , $t\in{\mathbb R}$ , belongs to ${\rm (NC)}$ . Now to show that T is contained in a compact subgroup of ${\rm Aut}(G)$ , it enough to show that $T_u={\rm Id}$ .

Step 2. Let R be the solvable radical of G. Then each $T_t|_R$ is unipotent and it belongs to ${\rm (NC)}$ . Since the largest compact connected central subgroup of R, being characteristic in G, is central in G by [Reference Iwasawa18, theorem 4], and hence it is trivial. Therefore, R does not have any compact central subgroup of positive dimension. By Proposition 4·2, $T_t|_R={\rm Id}$ . If G is solvable then $T_t={\rm Id}$ for each t.

Suppose G is not solvable. We show that $T_t$ acts trivially on $G/R$ . Let $G_s=G/R$ . Then $G_s$ is a connected semisimple Lie group. Moreover, ${\rm Inn}(G_s)$ is the connected component of the identity in ${\rm Aut}(G_s)$ and it is a subgroup of finite index in ${\rm Aut}(G_s)$ . Let $\eta:G\to G_s$ be the natural projection and let $T^{\prime}_t$ be the automorphism of $G_s$ corresponding to $T_t$ for each t. We now show that each $T^{\prime}_t$ is trivial. Since $\{T^{\prime}_t\}$ is unipotent and connected, there exists a Ad-unipotent one-parameter subgroup $\{u_t\}$ in $G_s$ such that $T^{\prime}_t={\rm inn} (u_t)$ , the inner automorphism by $u_t$ in $G_s$ . Suppose $\{u_t\}$ is nontrivial.

Let U be a maximal connected Ad-unipotent subgroup of $G_s$ containing $\{u_t\}$ , i.e. a maximal connected nilpotent subgroup containing $\{u_t\}$ such that for all $u\in U$ , ${\rm Ad}(u)$ is a unipotent linear transformation of the Lie algebra of $G_s$ . Let $G_s=KAN$ be an Iwasawa decomposition, where A is a closed abelian subgroup, N is a maximal connected Ad-unipotent subgroup of $G_s$ which is normalised by A, and AN is a closed solvable subgroup of $G_s$ (see [Reference Knapp20, proof of Theorem 6·46]). Since any two maximal connected Ad-unipotent subgroups are conjugate to each other, we get that N and U are conjugate by an element in a compact set contained in K (see [Reference Dani and SHAH11, Lemma 4·4]). If necessary, we may replace AN by its conjugate and assume that $U=N$ , i.e. $u_t\in N$ for all $t\in{\mathbb R}$ . Then AN is $T^{\prime}_t$ -invariant.

Let $B=\eta^{-1}(AN)$ , where $\eta:G\to G/R$ is as above for the radical R of G. Then B is a closed connected solvable subgroup of G. Note that since AN is $T^{\prime}_t$ -invariant, we have that B is $T_t$ -invariant for each t. Since A and N are simply connected and R has no compact central subgroup of positive dimension, we get that B has no compact central subgroup of positive dimension. As each $T_t|_{B}$ belongs to ${\rm (NC)}$ , by Proposition 4·2, $T_t|_{B}={\rm Id}$ for each t. This implies that $u_t$ centralises AN for all $t\in{\mathbb R}$ . Let ${\mathcal G}_s$ be the Lie algebra of $G_s$ and let ${\mathcal A}$ and ${\mathcal N}$ be the Lie subalgebra of A and N respectively. If $X\in {\mathcal N}$ is such that $\exp tX=u_t$ , $t\in{\mathbb R}$ , then from above, X belongs to the center of ${\mathcal A}\oplus{\mathcal N}$ , the Lie algebra of AN. This leads to a contradiction as the centraliser of ${\mathcal A}$ in ${\mathcal A}\oplus{\mathcal N}$ is ${\mathcal A}$ [Reference Knapp20, lemma 6·50]. Therefore, $u_t=e$ , and hence $T^{\prime}_t={\rm Id}$ , for each t.

Step 3. Now we have from steps 1 and 2 that $T_t|_R={\rm Id}$ and $T_t$ acts trivially on $G/R$ for each t, where R is the radical of G. If R is central in G, then the preceding assertions would imply that each $T_t$ acts trivially on $\overline{[G,G]}$ , the closure of the commutator subgroup of G, and as $G=\overline{[G,G]}R$ , we get that $T_t={\rm Id}$ in this case and, in particular, when G is semisimple. Now we want to prove that $T_t={\rm Id}$ , for each t in the general case.

For the natural projection $\eta:G\to G/R$ , the subgroup AN of $G/R$ and the subgroup $B=\eta^{-1}(AN)$ of G as in step 2, let $B_x=xBx^{-1}$ , for a fixed $x\in G$ . Then $B_x$ is a closed connected solvable subgroup of G, $\eta(B_x)=\eta(x)AN\eta(x)^{-1}$ is also a closed connected solvable group of $G/R$ , and $B_x$ is $T_t$ -invariant, as $T_t$ acts trivially on $G/R$ , for each t. Since $B_x$ is a conjugate of B, it does not have any compact central subgroup of positive dimension. Now by Proposition 4·2, we get that the restriction of each $T_t$ to $B_x$ is trivial. Since this is true for every $x\in G$ and the closed subgroup generated by $H=\cup_{x\in G}\eta(B_x)$ is a closed normal subgroup containing AN in $G/R$ , we have that H contains all the non-compact simple factors of $G/R$ . Therefore, each $T_t$ is trivial on a closed co-compact normal subgroup $\eta^{-1}(H)$ in G which contains R. That is, if $G/R$ has no nontrivial compact simple factors, then $T_t={\rm Id}$ , for each t.

Step 4. Let S ^′ be the product of all compact simple factors of $G/R$ and let $G'=\eta^{-1}(S')$ . Then G ^′ is a closed characteristic (normal) subgroup in G. Let $G'=S_cR$ be the Levi decomposition of G ^′. Then $S_c$ is compact and it is the product of all simple compact factors of a Levi subgroup in G. As $G=G'\eta^{-1}(H)=S_c\eta^{-1}(H)$ , and $T_t$ acts trivially on $\eta^{-1}(H)$ , it is enough to show that $T_t|_{S_c}={\rm Id}$ .

Here, $G'=S_cR$ is a closed subgroup. As G ^′ is characteristic, it is $T_t$ -invariant for each t. Let $[S_c,R]$ be the group generated by $\{xyx^{-1}y^{-1}\mid x\in S_c, y\in R\}$ . By [Reference Hochschild17, chapter XI, theorem 3·2], $\overline{[S_c,R]}\subset N$ ; here N denotes the nilradical of R, which is the same as the nilradical of G. Note that $\overline{[R,R]}\subset N$ [Reference Raghunathan24, proposition 3·11]. Let $G^{\prime}_1$ be the closure of the group [G ^′, G ^′]. Then $S_c\subset G^{\prime}_1\subset S_cN$ . As $G^{\prime}_1$ is $T_t$ -invariant, we have that $T_t(S_c)\subset S_cN$ . Since each $T_t$ acts trivially on $G'/R$ , we have that for $x\in S_c$ , $x^{-1}T_t(x)\in (S_c\cap R)N$ . As $S_c\cap R$ is a finite subgroup and $S_c$ is connected, $T_t(x)\in xN$ for all $x\in S_c$ and for all t. Note that the nilradical N of G is simply connected and has no nontrivial compact subgroups. Let K be a maximal compact connected abelian subgroup of $S_c$ . If possible, suppose for some t, $T_t|_K\ne{\rm Id}$ . Let $S_{T_t}$ be the stabiliser of $T_t$ . Then $K\cap S_{T_t}$ is a proper closed subgroup of K. Now as $K/(K\cap S_{T_t})$ is a nontrivial compact connected abelian group, it is monothetic and hence, there exists $x\in K$ such that the image of x in $K/(K\cap S_{T_t})$ generats a dense infinite group. In particular, $G_x\cap S_{T_t}=\{e\}$ . As $T_t(x)\in xN$ , $T_t|_N={\rm Id}$ and N has no nontrivial compact subgroups, by Lemma 3·12, $T_t\not\in{\rm (NC)}$ , which leads to a contradiction. Therefore, $T_t|_K={\rm Id}$ for all t. Since the union of all maximal compact connected abelian subgroups of $S_c$ is dense in $S_c$ , we have that $T_t|_{S_c}={\rm Id}$ for all t. As observed above, this shows that $T_t={\rm Id}$ for all t. As observed at the end of step 1, this implies that T is contained in a compact subgroup of G. Therefore, $\textrm{(i)}\Rightarrow \textrm{(iv)}$ and $\textrm{(i)} - \textrm{(iv)}$ are equivalent.

We know from Remark 3·8 that all automorphisms of connected compact abelian Lie groups of dimension greater than or equal to 2 belong to class (NC) and, in particular, this hold for nontrivial unipotent automorphisms of such groups. The following theorem shows that a connected Lie group G does not admit any nontrivial unipotent automorphism which acts distally on ${\rm Sub}^a_G$ .

Proof. Let G and T be as in the hypothesis. One way implication is obvious. Now suppose T acts distally on ${\rm Sub}^a_G$ . We show that $T={\rm Id}$ .

Let R be the solvable radical of G. Then R is T-invariant and by Proposition 4·2, $T|_R={\rm Id}$ . Let C be the largest compact connected central subgroup. If C is trivial, then the assertion follows from Theorem 4·1 as the only unipotent automorphism contained in a compact subgroup of ${\rm Aut}(G)$ is the identity map. Suppose C is nontrivial. Let $\overline{T}$ be the automorphisms of $G/C$ corresponding to T. Then by Lemma 3·3, $\overline{T}\in{\rm (NC)}$ . Note that $\overline{T}$ is also unipotent. Now by Theorem 4·1, $\overline{T}$ is contained in a compact subgroup of ${\rm Aut}(G/C)$ , and being unipotent, $\overline{T}$ is trivial. As C is connected and central, it follows that T acts trivially on $\overline{[G,G]}$ . Now as $G=\overline{[G,G]}R$ , we get that $T={\rm Id}$ .

The following theorem shows that a part of Theorem 4·1 can be generalised to all connected Lie groups if we restrict the class of automorphisms to $({\rm Aut}(G))^0$ , the connected component of the identity in ${\rm Aut}(G)$ . Example 4·7 illustrates that Theorem 4·1 can not be generalised fully even if we restrict to $({\rm Aut}(G))^0$ .

Proof. Let G be a connected Lie group and let $T\in({\rm Aut}(G))^0$ . It is enough to show that $\textrm{(ii)}\Rightarrow \textrm{(iv)}$ in Theorem 4·1. Suppose (ii) holds, i.e. T acts distally on ${\rm Sub}^a_G$ . We want to prove that T is contained in a compact subgroup of ${\rm Aut}(G)$ .

Let C be the largest compact connected central subgroup of G. Then C is characteristic. As C is compact and abelian, ${\rm Aut}(C)$ is totally disconnected and hence every element of $({\rm Aut}(G))^0$ acts trivially on C. In particular, $T|_C={\rm Id}$ . Also, by Corollary 3·7 (a), T acts distally on G. Therefore, the eigenvalues of ${\rm d}\, T$ on the Lie algebra $\mathcal G$ of G are of absolute value 1. Moreover, $({\rm Aut}(G))^0$ is almost algebraic as a closed subgroup of ${\rm GL}(\mathcal G)$ [Reference Wigner33, Reference Wigner34]; see also [Reference Dani10, theorem 2]. Let $G_T$ be the smallest almost algebraic group in $({\rm Aut}(G))^0$ containing T. By [Reference Abels1, corollary 2·5], $G_T=\mathcal K\times \mathcal U$ where $\mathcal K$ is compact and abelian, and $\mathcal U$ is unipotent and abelian. In fact, $\mathcal U$ is a unipotent one-parameter group $\{T_t\}$ in $({\rm Aut}(G))^0$ , such that $T=T_sT_u=T_uT_s$ , $T_s\in\mathcal K$ and $T_u=T_1$ is unipotent. By [Reference Shah and YADAV31, lemma 2·2], $T_u$ acts distally on ${\rm Sub}^a_G$ . By Theorem 4·3, $T_u={\rm Id}$ . Therefore, $T=T_s$ is contained in a compact subgroup of $({\rm Aut}(G))^0$ , i.e. (iv) holds.