Proof. Let $\mathfrak {C}$ be the set of closed connected subgroups of G. We have $\overline {\mathfrak {C}} = \mathrm {Sub}^{co}_G$ . By [Reference Benedetti and Petronio6, Proposition E.1.5] (see also [Reference Bridson, de la Harpe and Kleptsyn8, Proposition 3.4(iii)] or also [Reference Wang45, Corollary of Lemma 1.2]), the set $\mathfrak {D}$ of discrete subgroups of G is open in $\mathrm {Sub}_G$ and so ${\{\{e\}\} \subseteq \mathfrak {D}\cap \mathrm {Sub}^{co}_G =\mathfrak {D} \cap \overline {\mathfrak {C}}\subseteq \overline {\mathfrak {D}\cap \mathfrak {C}} = \{\{e\}\}}$ . Hence, the assertions hold.

Proof. Let K be a compact connected abelian subgroup of G. Then, the set F of all finite order elements in K is dense in K and each such element of F is contained in a closed one-parameter subgroup of K. Since K is monothetic (that is, it has a dense cyclic subgroup), we get that K is a limit of a sequence of compact one-parameter subgroups of K in $\mathrm {Sub}_K$ . Therefore, $K\in \mathrm {Sub}^{p}_K\subset \mathrm {Sub}^p_G$ . Thus, statement (1) holds.

Let $\{x_t\}\in {\mathcal {P}}_1(G)\subset \mathrm {Sub}^p_G$ be such that each $x_t$ centralises K. If $\{x_t\}$ is compact, then $\{x_t\}K$ is a compact connected abelian subgroup of G and hence $\{x_t\}K\in \mathrm {Sub}^p_G$ from statement (1). Now, suppose $\{x_t\}$ is isomorphic to ${\mathbb R}$ . Let $k\in K$ be such that it generates a dense cyclic subgroup in K. Since K is exponential, there exists a one-parameter subgroup $\{k_t\}$ that is dense in K and $k=k_1$ . Let $A_n=\{a_t(n)=x_{t/n}k_t\}_{t\in {\mathbb R}}$ . Then, each $A_n$ is a closed one-parameter subgroup in G and, passing to a subsequence if necessary, we get that $A_n\to A$ for some subgroup $A\in \mathrm {Sub}^p_G$ . Note that $\{x_t\}\times K$ is a closed subgroup of G and it contains each $A_n$ . Therefore, $A\subset \{x_t\}\times K$ . Let $t\in {\mathbb R}$ be fixed. Since $a_t(n)=x_{t/n}k_t\to k_t$ as $n\to \infty $ , we get that $k_t\in A$ , and hence $K\subset A$ . Let $b_n=a_{tn}(n)=x_tk_{tn}\in A_n$ , $n\in {\mathbb N}$ . Passing to a further subsequence of $\{b_n\}$ if necessary, we get that $b_n\to x_th$ for some $h\in K$ , and hence, $x_t\in A$ , as $K\subset A$ . Now, $\{x_t\}\subset A$ . Hence, $A=\{x_t\}\times K$ and statement (2) holds.

Proof. Let K and $\pi $ be as above. Note that $K\in \mathrm {Sub}^p_G$ by Lemma 2.3. If $G=K$ , then it is easy to see that statements (1)–(4) are trivially satisfied.

Now, suppose that $G\ne K$ . Since K is compact and central in G, $HK$ is a closed subgroup of G for any closed subgroup H of G. If $H_nK\to L$ in $\mathrm {Sub}_G$ , then since both $\mathrm {Sub}_G$ and K are compact, it is easy to show that for any limit point H of $\{H_n\}$ , $L=HK$ . Therefore, $\{HK\mid H\in \mathrm {Sub}_G\}$ is closed. Now, if $H\in \mathrm {Sub}^p_G$ , then there exists $\{A_n\}\subset {\mathcal {P}}_1(G)$ , such that $A_n\to H$ . By Lemma 2.3(2), $A_nK\in \mathrm {Sub}^p_G$ . As $A_nK\to HK$ and $\mathrm {Sub}^p_G$ is closed, we get that $HK\in \mathrm {Sub}^p_G$ . Now, arguing as above, it is easy to show that $\{HK\mid H\in \mathrm {Sub}^p_G\}$ is a closed subset of $\mathrm {Sub}^p_G$ . Moreover, if $T\in \mathrm {Aut}(G)$ with $T(K)=K$ , then $T(HK)=T(H)K$ for any $H\in \mathrm {Sub}_G$ . Therefore, the sets mentioned in statement (1) are invariant under the action of $\mathrm {Aut}(G)$ if $T(K)=K$ . Thus, statement (1) holds.

If $A\in {\mathcal {P}}_1(G)$ , then $AK$ is closed and $\pi (A)\in {\mathcal {P}}_1(G/K)$ . Conversely, suppose ${B\in {\mathcal {P}}_1(G/K}$ ) is nontrivial. It is easy to see that $B_1:=\pi ^{-1}(B)$ is a closed connected nilpotent Lie subgroup of G. There exists a neighbourhood U of e such that every element of U is contained in a one-parameter subgroup and $U\setminus K$ is nonempty. Choose any $x\in U\setminus K$ . There exists a one-parameter subgroup $\{x_t\}$ with $x_1=x$ . Then, $\pi (\{x_t\})=B$ and $B_1=\{x_t\}K$ . As K is central in G, we get that $B_1$ is, in fact, abelian. If $\{x_t\}$ is closed, then it belongs to ${\mathcal {P}}_1(G)$ and $\pi (\{x_t\})=B$ . Now, suppose it is not closed, then its closure is compact and, hence, $B_1$ is compact. This implies that the set of torsion elements is dense in $B_1$ . Since $U\setminus K$ is open, we can choose $y\in B_1\cap (U\setminus K)$ such that $y^n=e$ for some $n\in {\mathbb N}\setminus \{1\}$ . Let $\{y_t\}$ be a one-parameter subgroup in $B_1$ such that $y_1=y$ . Then, $\pi (\{y_t\})=B$ . Here, $\{y_t\}$ is closed as $y_1^n=e$ . Therefore, ${\mathcal {P}}_1(G/K)=\{\pi (A)\mid A\in {\mathcal {P}}_1(G)\}$ , that is, statement (2) holds.

Let $\pi ':\mathrm {Sub}^p_G\to \mathrm {Sub}^{p}_{G/K}$ be the map induced by $\pi $ . Then, $\pi '$ is a continuous closed map. From statement (2), we get that

$$ \begin{align*}\mathrm {Sub}^{p}_{G/K}= \overline {{\mathcal {P}}_1(G/K)}=\pi '(\overline {{\mathcal {P}}_1(G)})=\pi '(\mathrm {Sub}^p_G)=\{\pi (H)\mid H\in \mathrm {Sub}^p_G\}. \end{align*} $$

Thus, statement (3) holds.

Let ${\mathcal S}(K):=\{HK\mid H\in \mathrm {Sub}^p_G\}$ . By statement (1), ${\mathcal S}(K)$ is closed in $\mathrm {Sub}^p_G$ and, hence, it is compact. Observe that ${\mathcal S}(K)=\{H\in \mathrm {Sub}^p_G\mid HK=KH=H\}$ . By statement (3), $\pi '({\mathcal S}(K))=\pi '(\mathrm {Sub}^p_G)=\mathrm {Sub}^{p}_{G/K}$ and it is obvious that $\pi '|_{{\mathcal S}(K)}$ is continuous and bijective, and hence, it is a homeomorphism. Thus, statement (4) holds.

Proof. Let M be the largest compact connected central subgroup of G. Then, $G/M$ is in class ${\mathcal {C}'}$ . As G is not nilpotent, $G/M$ is also not nilpotent. Since M is characteristic in G, we may replace G by $G/M$ and assume that $G\in {\mathcal {C}'}$ . If T has finite order, then $({d\,} T)^n=\mathrm {Id}$ for some $n\in {\mathbb N}$ , and the assertion follows in this case. Now, suppose T does not have finite order.

Since $G\in {\mathcal {C}'}$ , $\mathrm {Aut}(G)$ is almost algebraic (see [Reference Dani12]) and, hence, it has finitely many connected components. Now, we may replace T by its suitable power and assume that $T\in \mathrm {Aut}(G)^0$ , the connected component of the identity in $\mathrm {Aut}(G)$ , and that T is contained in a nontrivial one-parameter subgroup (say) $\{T_t\}$ of $\mathrm {Aut}(G)^0$ as $T=T_1$ . We show that 1 is an eigenvalue of ${d\,} T$ .

Suppose G is solvable. Let $H=\{T_t\}\ltimes G$ . Then, H is solvable and the nilradical of H, being contained in G, is the same as the nilradical N of G. Therefore, $H/N$ is abelian. Thus, T acts trivially on $G/N$ . As G is not nilpotent, $G\ne N$ , and, hence, 1 is an eigenvalue of ${d\,} T$ .

Now, suppose G is not solvable. Let R be the radical of G. Then, R is characteristic in G and let $\overline {T}_t$ be the automorphism of $G/R$ induced by $T_t$ for each t. It is enough to prove that 1 is an eigenvalue of ${d\,} \overline {T}$ . Hence, we may replace G by $G/R$ and $\{T_t\}$ by $\{\overline {T}_t\}$ and assume that G is semisimple. Now, $\mathrm {Aut}(G)^0$ consists of inner automorphisms of G and $T_t=\mathrm {inn}(x_t)$ , $t\in {\mathbb R}$ , for some one-parameter subgroup $\{x_t\}\subset G$ . If $x_1=e$ , then $T=T_1=\mathrm {Id}$ . Now, suppose $x_1\ne e$ . Then, $\{x_t\}$ is nontrivial. Let $X\in {\mathfrak g}$ be such that $\exp tX=x_t$ , $t\in {\mathbb R}$ . As $T(x_t)=x_t$ , $t\in {\mathbb R}$ , we have that ${d\,} T(X)=X$ and 1 is an eigenvalue of ${d\,} T$ .

Proof. Let M be the largest compact connected central subgroup of G. If $G=M$ , then it is easy to see that statements (1)–(3) are equivalent. Now, suppose $G\ne M$ . Let $\pi :G\to G/M$ be the natural projection and let $T'\in \mathrm {Aut}(G/M)$ be the automorphism induced by T on $G/M$ . As $G/M\in {\mathcal {C}'}$ , using [Reference Dani, Shah, Budzban, Feinsilver and Mukherjea14, Theorem 1.1], one can show that $C(T')$ is closed (see the proof of [Reference Raja and Shah35, Proposition 4.3] for more details). Hence, $\pi ^{-1}(C(T'))=C_M(T):=\{x\in G\mid T^n(x)M\to M\mbox { in }G/M\}$ is closed. Suppose statement (1) holds, that is, T acts distally on M. Then, $C(T)\cap M=C(T|_M)=\{e\}$ . As $C_M(T)$ is closed, by [Reference Hazod and Siebert21, Corollary 2.7], $C(T)$ is closed. Similarly, $C(T^{-1})$ is closed. The rest of the assertion in statement (2) follows from [Reference Siebert43, Corollary 2.4]. This proves $(1)\implies (2)$ .

Now suppose statement (2) holds, that is, $C(T)$ is closed. Let K be any compact T-invariant subgroup of G. Then, $C(T|_K)=C(T)\cap K$ is a compact subgroup of $C(T)$ . Hence, $C(T|_K)$ is trivial (see [Reference Hazod and Siebert21, Corollary 2.5]). Similarly, $C((T|_K)^{-1})$ is trivial and T acts distally on K (see [Reference Jaworski26]). Thus, statement (3) holds. Then, $(3)\implies (1)$ is obvious.

Proof. Let $T\in \mathrm {Aut}(G)$ and let $\pi :G\to G/K$ be the natural projection. Here, $\overline {T}(\pi (H))=\pi (T(H))$ for any $H\in \mathrm {Sub}_G$ , and $\mathrm {Sub}^{p}_{G/K}$ is $\overline {T}$ invariant. Consider the set ${\mathcal S}(K)=\{H\in \mathrm {Sub}^p_G\mid HK=H\}$ . It is a compact T-invariant subset of $\mathrm {Sub}^p_G$ and it is homeomorphic to $\mathrm {Sub}^{p}_{G/K}$ under the map induced by $\pi $ on ${\mathcal S}(K)$ (see Lemma 2.4(4)). Suppose T acts distally on $\mathrm {Sub}^p_G$ . Then, the T-action on ${\mathcal S}(K)$ is distal. As $\pi \circ T=\overline {T}\circ \pi $ , we get that $\overline {T}$ acts distally on $\mathrm {Sub}^{p}_{G/K}$ .

Proof. In a simply connected nilpotent Lie group, all one-parameter subgroups are closed and there is a one-to-one correspondence between nontrivial (closed) one-parameter subgroups of G and one-dimensional subspaces of ${\mathfrak g}$ . The correspondence is via the exponential map $\exp : {\mathfrak g}\to G$ with $\log $ as its inverse. Namely, for $\{x_t\}$ , $x_t=\exp tX$ , for some $X\in {\mathfrak g}$ and $\log x_t=tX$ , $t\in {\mathbb R}$ . Also, the exponential map induces a homeomorphism from ${\mathcal {P}}_1({\mathfrak g})\to {\mathcal {P}}_1(G)$ . As $\exp \circ \,{d\,} T=T\circ \exp $ , it follows that T acts distally on ${\mathcal {P}}_1(G)=\mathrm {Sub}^p_G$ if and only if ${d\,} T$ acts distally on ${\mathcal {P}}_1({\mathfrak g})=\mathrm {Sub}^{p}_{\mathfrak g}$ .

Proof of Theorem 1.1  $(1)$ and $(2)$ for nilpotent groups.

Let G be a (nontrivial) connected Lie group without any nontrivial compact connected central subgroup, that is, G belongs to class ${\mathcal {C}'}$ . Let $T\in \mathrm {Aut}(G)$ .

$(1):$ Suppose G is abelian. Then, it is a vector group isomorphic to ${\mathbb R}^d$ for some ${d\in {\mathbb N}}$ . Note that ${\mathcal D}=\{r\,\mathrm {Id}\mid r\in {\mathbb R}\setminus \{0\}\}$ , where $\mathrm {Id}$ is the identity matrix in $\mathrm {GL}(d,{\mathbb R})$ . If $d=1$ , then $\mathrm {GL}(1,{\mathbb R})$ is abelian and the assertion follows trivially. Now, suppose $d\ge 2$ . Then, $\mathrm {Sub}^p_G$ is isomorphic to $\mathbb {RP}^{d-1} \sqcup \{\{e\}\}$ and $T\in \mathrm {GL}(d,{\mathbb R})$ . Both $\mathbb { RP}^{d-1}$ and $\{\{e\}\}$ are compact and T invariant. Hence, T acts distally on $\mathrm {Sub}^{p}_{{\mathbb R}^d}$ if and only if T acts distally on $\mathbb {RP}^{d-1}$ . One can define an action of T on $\mathbb S^{d-1}$ , the boundary of the unit sphere in ${\mathbb R}^d$ , as $\overline {T}(x)=T(x)/\|T(x)\|$ , where $\|\cdot \|$ denotes the usual norm on ${\mathbb R}^d$ . Here, $\overline {T}$ is a homeomorphism and $\overline {T}(-x)= -\overline {T}(x)$ . As ${\mathbb {RP}}^{d-1}$ is a quotient space of $\mathbb S^{d-1}$ , the action of T on ${\mathbb {RP}}^{d-1}$ is the same as the action of $\overline {T}$ on ${\mathbb S}^{d-1}/\sim $ , where $\sim $ is the equivalence relation that identifies x with $-x$ for each $x\in \mathbb S^{d-1}$ . It is easy to see that T acts distally on ${\mathbb {RP}}^{d-1}$ if and only if $\overline {T}$ acts distally on $\mathbb S^{d-1}$ . The latter statement is equivalent to the following: $T \in {\mathcal K} {\mathcal D}$ , where ${\mathcal K}$ is a compact subgroup of $\mathrm {GL}(d,{\mathbb R})$ and ${\mathcal D}$ is the centre of $\mathrm {GL}(d,{\mathbb R})$ (this follows from [Reference Shah and Yadav39, Theorem 1]; see also [Reference Shah and Yadav41]).

Conversely, for $G={\mathbb R}^d$ and $T\in {\mathcal K}{\mathcal D}$ , with a compact subgroup ${\mathcal K}$ of $\mathrm {GL}(d,{\mathbb R})$ and the centre ${\mathcal D}$ of $\mathrm {GL}(d,{\mathbb R})$ , the action of T on $\mathrm {Sub}^p_G$ is the same as that of S on $\mathrm {Sub}^p_G$ for some $S\in {\mathcal K}$ since ${\mathcal D}$ acts trivially on $\mathrm {Sub}^p_G$ . As the compact subgroups of $\mathrm {Aut}(G)$ act distally on any closed invariant subspace of $\mathrm {Sub}_G$ ( $\mathrm {Sub}^p_G$ in this case), we have that T acts distally on $\mathrm {Sub}^p_G$ . This completes the proof of Theorem 1.1(1).

$(2):$ Suppose G belongs to the class ${\mathcal {C}'}$ and it is nonabelian and nilpotent. Then, G is simply connected as G has no compact central subgroup of positive dimension. Suppose T acts distally on $\mathrm {Sub}^p_G$ . We want to show that T is contained in a compact subgroup of $\mathrm {Aut}(G)$ . Since G belongs to the class ${\mathcal {C}'}$ , $\mathrm {Aut}(G)$ is almost algebraic as a subgroup of $\mathrm {GL}({\mathfrak g})$ (see [Reference Dani12]).

As G is simply connected and nilpotent, by Lemma 3.6, ${d\,} T$ acts distally on $\mathrm {Sub}^{p}_{\mathfrak g}$ . Then, from statement (1) above, we have that ${d\,} T\in {\mathcal K}\{r\,\mathrm {Id}\mid r\ne 0\}$ for some compact subgroup ${\mathcal K}$ of $\mathrm {GL}({\mathfrak g})$ and $\mathrm {Id}$ is the identity map on ${\mathfrak g}$ . Here, ${d\,} T=\psi \,\circ \, r\,\mathrm {Id}$ for some $r\in {\mathbb R}\setminus \{0\}$ and $\psi \in {\mathcal K}$ . If $r=\pm 1$ , then ${d\,} T^2\in {\mathcal K}$ and T generates a relatively compact subgroup of $\mathrm {Aut}(G)$ . If possible, suppose $r\ne \pm 1$ . Since $\mathrm {Aut}(G)$ is almost algebraic, there exists $n\in {\mathbb N}$ such that ${d\,} T^n$ is contained in a connected almost algebraic subgroup ${\mathbb R}\times \mathbb T^k$ of $\mathrm {Aut}(G)^0$ (which is identified with its image in $\mathrm {GL}({\mathfrak g})$ ), where $\mathbb T^k=(\mathbb S^1)^k$ for some $k\in {\mathbb N}$ . Therefore, $\psi ^n\,\circ \, r^n\,\mathrm {Id}\in {\mathbb R}\times \mathbb T^k\subset \mathrm {Aut}(G)^0$ . Hence, both $\psi ^n$ and $r^n\,\mathrm {Id}\in \mathrm {Aut}(G)^0$ . However, $r^n\,\mathrm {Id}$ is not a Lie algebra automorphism of ${\mathfrak g}$ as $r^n\ne 1$ ; otherwise, $r^n[X,Y]=r^n\mathrm {Id}[X,Y]=[r^n\mathrm {Id}(X),r^n\mathrm {Id}(Y)]=r^{2n}[X,Y]$ , which would imply that $[X,Y]=0$ for all $X,Y\in {\mathfrak g}$ , which leads to a contradiction as G is not abelian. Thus, $r=\pm 1$ and T is contained in a compact subgroup of $\mathrm {Aut}(G)$ . The converse statement is obvious. Thus, statement (2) holds if G is nilpotent.

Proof of Theorem 1.2.

Let G be a connected Lie group that is not a vector group. Let M be the largest compact connected central subgroup of G. Suppose $T\in \mathrm {Aut}(G)$ be such that it acts distally on M. Suppose T acts distally on $\mathrm {Sub}^p_G$ . Then, we want to show that T acts distally on G. The assertion holds trivially if $G=M$ .

Step 1: Suppose G is simply connected and nilpotent. Since G is not a vector group, G is nonabelian and by the proof of Theorem 1.1(2) for the nilpotent case, T is contained in a compact subgroup of $\mathrm {Aut}(G)$ . Hence, T acts distally on G.

Step 2: Now suppose G is not a simply connected nilpotent group. By [Reference Raja and Shah35, Theorem 4.1], it is enough to prove that the contraction groups $C(T)$ and $C(T^{-1})$ are trivial. As T acts distally on the maximal compact connected central subgroup M, by Lemma 3.4, both $C(T)$ and $C(T^{-1})$ are closed, simply connected and nilpotent subgroups. Suppose $C(T)$ is nontrivial. If possible, suppose $C(T)$ is nonabelian. As it is T-invariant and $T|_{C(T)}$ acts distally on $\mathrm {Sub}^{p}_{C(T)}$ , we get from Step 1 that $T|_{C(T)}$ acts distally on $C(T)$ , which is a contradiction. Thus, $C(T)$ is abelian and $C(T)$ is isomorphic to ${\mathbb R}^d$ as it is simply connected. By Theorem 1.1(1), $T_1=T|_{C(T)}=\phi \circ \alpha \,I$ for some $\phi $ contained in a compact subgroup of $\mathrm {GL}(d,{\mathbb R})$ and $\alpha \in {\mathbb R}$ , $0< |\alpha |<1$ ; here, I denotes the identity matrix in $\mathrm {GL}(d,{\mathbb R})$ , replacing T by $T^2$ , we may assume that $\alpha>0$ .

Step 3: Suppose G is nilpotent but not simply connected. Then, M as above is nontrivial. Since T acts distally on M, by [Reference Abels2, Lemma 2.5], we have that $T^k|_M$ is unipotent for some $k\in {\mathbb N}$ , that is, ${d\,} T$ has an eigenvalue that is a root of unity. If G is not nilpotent, by Lemma 3.2, ${d\,} T$ has an eigenvalue that is a root of unity. We may replace T by $T^m$ , for some $m\in {\mathbb N}$ and assume that 1 is an eigenvalue of ${d\,} T$ . Let $X\in {\mathfrak g}$ , $X\ne 0$ be such that ${d\,} T(X)=X$ and let $A=\{\exp tX\}$ , a one-parameter subgroup, which is either closed or its closure $\bar A$ is compact. Since $T|_A=\mathrm {Id}$ , we have that $T|_{\bar A}=\mathrm {Id}$ and it follows that $\bar A$ normalises $C(T)$ and $\bar A\cap C(T)=\{e\}$ .

We now show that $C(T)$ normalises $\bar A$ , which would imply that $\bar A\, C(T)$ is abelian as $C(T)$ and A are so. Here, $T(\bar A)=\bar A$ and $\bar A\in \mathrm {Sub}^p_G$ . Now, for any nonzero $c\in C(T)$ , we have that

$$ \begin{align*} T^n(c\bar A c^{-1})=T^n(c)\,\bar A\,T^n(c^{-1})\to\bar A.\end{align*} $$

Since T acts distally on $\mathrm {Sub}^p_G$ , we get that $c\bar Ac^{-1}=\bar A$ for all $c\in C(T)$ . Thus, $C(T)$ normalises $\bar A$ and, hence, $\bar A\, C(T)$ is abelian.

Let $H=\overline {A\times C(T)}$ . Then, H is a closed T-invariant abelian subgroup and $\bar A\subset H$ . First, suppose H is a vector group. By Theorem 1.1(1), $T|_H\in {\mathcal K}{\mathcal D}$ for some compact subgroup ${\mathcal K}$ in $\mathrm { GL}(H)$ and the centre ${\mathcal D}$ of $\mathrm {GL}(H)$ , and thus all the eigenvalues of $T|_H$ have the same absolute value. Since $T|_{\bar A}=\mathrm {Id}$ and $T|_{C(T)}$ have eigenvalues of absolute value $\alpha \ne 1$ , this leads to a contradiction. Thus, $C(T)$ must be trivial in this case.

Step 4: Now suppose H is not a vector group. Since $C(T)$ is closed and the image of A is dense in $H/C(T)$ , we get that $H=K\times C(T)$ for the maximal compact subgroup K of H. As T acts trivially on $H/C(T)$ and K is T-invariant, we get that $T|_K=\mathrm {Id}$ .

Let $\psi , S: H\to H$ be such that $S|_K=\mathrm {Id}$ , $S|_{C(T)}=\alpha \,\mathrm {Id}$ , $\psi |_K=\mathrm {Id}$ and $\psi |_{C(T)}=\phi $ , where $\alpha $ and $\phi $ are as in Step 2. Now, $T|_{H}=S\psi =\psi S$ and $\psi $ is contained in a compact subgroup of $\mathrm {Aut}(H)$ . Therefore, S acts distally on $\mathrm {Sub}^{p}_{H}$ (see [Reference Shah and Yadav40, Lemma 2.2]) and $C(S)=C(T)$ .

Let $\{z_t=x_tc_t\}$ be a closed one-parameter subgroup, where $\{x_t\}\subset K$ is closed and isomorphic to ${\mathbb S}^1$ and $\{c_t\}\subset C(S)=C(T)$ is a nontrivial (closed) one-parameter subgroup such that $S(c_t)=c_{\alpha t}$ , $t\in {\mathbb R}$ (as addition is the group operation on $C(T)={\mathbb R}^d$ , we choose $c_t=tc$ , $t\in {\mathbb R}$ , for some nonzero $c\in {\mathbb R}^d$ ). Let $\{n_k\}$ be an unbounded sequence in ${\mathbb N}$ such that $S^{n_k}(\{z_t\})\to C$ for some C in $\mathrm {Sub}^{p}_{H}$ . Note that $\{z_t\}\subset \{x_t\}\times \{c_t\}$ and the latter is closed and S-invariant. So, $C\subset \{x_t\}\times \{c_t\}$ . For a fixed $t\in {\mathbb R}$ , as $c_t\in C(S)$ , we get that

$$ \begin{align*} S^{n_k}(z_t)=x_t S^{n_k}(c_t)\to x_t,\end{align*} $$

as $n_k\to \infty $ . Therefore, $\{x_t\}\subset C$ . Again, for a fixed $t\in {\mathbb R}$ ,

$$ \begin{align*}S^{n_k}(z_{\alpha^{-n_k}t})= x_{\alpha^{-n_k}t}S^{n_k}(c_{\alpha^{-n_k}t})=x_{\alpha^{-n_k}t}c_t\to x_sc_t\end{align*} $$

along a subsequence of $\{n_k\}$ for some $s\in {\mathbb R}$ . Therefore, $x_sc_t$ and, hence, $c_t$ belongs to C. Thus, $\{c_t\}\subset C$ and $C=\{x_t\}\times \{c_t\}$ and, hence, C is S-invariant. This implies that $S^{n_k}(S(\{z_t\}))\to C$ and, hence, that $S(\{z_t\})=\{z_t\}$ since S acts distally on $\mathrm {Sub}^p_G$ . As $\{z_t\}$ is isomorphic to ${\mathbb R}$ , there exists $\beta \in {\mathbb R}\setminus \{0\}$ such that for all $t\in {\mathbb R}$ ,

$$ \begin{align*} S(z_t)=z_{\beta t}=x_{\beta t}c_{\beta t}\ \mbox{ and }\ S(z_t)=S(x_tc_t)=x_tc_{\alpha t}.\end{align*} $$

As $\{x_t\}\times \{c_t\}$ is a direct product and $\{c_t\}$ is isomorphic to ${\mathbb R}$ , it follows that $\beta =\alpha $ and $x_t=x_{\alpha t}$ , that is, $x_{(1-\alpha )t}=e$ for all $t\in {\mathbb R}$ . As $\alpha \ne 1$ , $x_t=e$ for each t, which is a contradiction. Therefore, $C(T)$ is trivial.

Similarly, we can show that $C(T^{-1})$ is trivial and, hence, T acts distally on G.

Proof. Since T acts trivially on $G/L$ and L, we have that T acts distally on G. In fact, all the eigenvalues of ${d\,} T$ are equal to 1 and T is unipotent. Suppose T acts distally on $\mathrm {Sub}^p_G$ . Let U be a neighbourhood of the identity e such that every element of U is contained in a one-parameter subgroup. Since G is connected, U generates G and it is enough to show that $T(x)=x$ for all $x\in U$ .

Step 1: If possible, suppose $T(x)\ne x$ for some $x\in U$ . There exists a one-parameter subgroup $\{x_t\}$ in G such that $x_1=x$ . Let $H=\overline {\{x_t\}L}$ . Then, H is a closed connected solvable T-invariant subgroup of G and $H/L$ is abelian. Let $\pi :G\to G/L$ be the natural projection. Then, $\pi (H)=\overline {\pi (\{x_t\})}$ . If $\pi (\{x_t\})$ is unbounded, it is closed and it is isomorphic to ${\mathbb R}$ and in this case, $\{x_t\}$ is also unbounded and closed. Moreover, $H=\{x_t\}L$ .

Now, suppose $\pi (\{x_t\})$ is relatively compact. As H is solvable, there exists a maximal compact (connected, abelian) subgroup K in H such that $H=KL$ . As $T(x)\ne x$ and $T|_L=\mathrm {Id}$ , we have that $T(h)\ne h$ for some $h\in K$ , and hence, $T(h)\ne h$ for some finite order element h in $K\cap U$ . There exists a closed (compact) one-parameter subgroup $\{h_t\}$ in K such that $h_1=h$ . Replacing $\{x_t\}$ by $\{h_t\}$ , we have in this case too that H is closed and $H=\{x_t\}L$ .

Now, we have that $H=\{x_t\}L$ is closed, where either $\pi (\{x_t\})$ (as well as $\{x_t\}$ ) is isomorphic to ${\mathbb R}$ or $\{x_t\}$ is isomorphic to ${\mathbb S}^1$ and $T(x_1)\ne x_1$ . As T acts trivially on $G/L$ , we have that $T(H)=H$ and $T(x_t)=x_ty_t$ for some $y_t\in L$ , $t\in {\mathbb R}$ , where $y_1\ne e$ .

Step 2: Let $A:=\{x_t\}$ and let $\{n_k\}\subset {\mathbb N}$ be an unbounded sequence such that $T^{n_k}(A)\to B$ for some $B\in \mathrm {Sub}^{p}_H$ . By Lemma 2.2, B is not discrete as A is connected and nontrivial. Let $b\in B$ be such that $b\ne e$ . There exist $t_k\in {\mathbb R}$ , $k\in {\mathbb N}$ , such that

$$ \begin{align*} T^{n_k}(x_{t_k})=x_{t_k}y_{t_k}^{n_k}\to b.\end{align*} $$

As $y_t\in L$ , $t\in {\mathbb R}$ , we get that $\pi (x_{t_k})\to \pi (b)$ . If $H/L$ is isomorphic to ${\mathbb R}$ , then we get that the sequence $\{t_k\}$ is bounded. In the second case where $\{x_t\}$ is compact, we can choose $\{t_k\}$ to be bounded. Passing to a subsequence if necessary, we may assume that $t_k\to t_0$ . Therefore, $x_{t_k}\to x_{t_0}$ and $y_{t_k}\to y_{t_0}$ , and $b=x_{t_0}y'$ , where $y_{t_k}^{n_k}\to y'$ in L.

Step 3: We first assume that L is simply connected. We show that $y_{t_0}=e$ in this case. Let ${\mathfrak l}$ be the Lie algebra of L. Since L is simply connected and nilpotent, $\exp : {\mathfrak l}\to L$ is a homeomorphism with $\log $ as its inverse. Let $y_{t_k}=\exp Y_k$ for some $Y_k\in {\mathfrak l}$ . Then, $y_{t_k}^{n_k}=\exp (n_kY_k)$ and as $y_{t_k}^{n_k}\to y'$ , we have that $n_kY_k\to \log y'$ in ${\mathfrak l}$ . As ${\mathfrak l}$ is isomorphic to ${\mathbb R}^d$ for some $d\in {\mathbb N}$ , it follows that $Y_k\to 0$ in ${\mathfrak l}$ and, hence, $y_{t_k}\to e$ , that is, $y_{t_0}=e$ . Now,

$$ \begin{align*} T(b)=T(x_{t_0})y'=x_{t_0}y_{t_0}y'=x_{t_0}y'=b.\end{align*} $$

Thus, $T(B)=B$ . As T acts distally on $\mathrm {Sub}^p_G$ , we have that $T(A)=A$ . Since A is a one-parameter subgroup in G and T is unipotent, we get that $T(x_t)=x_t$ , $t\in {\mathbb R}$ . This leads to a contradiction as $T(x_1)\ne x_1$ . Therefore, $T(x)=x$ for all $x\in U$ and, hence, for all $x\in G$ . Thus, $T=\mathrm {Id}$ if L is simply connected.

Step 4: Suppose that L is not simply connected. Let M be the maximal compact (normal) subgroup of L. Then, M is central in G and $L/M$ is simply connected. Let $\varrho :G\to G/M$ be the natural projection and let $\overline {T}$ be the automorphism on $G/M$ induced by T. Then, $\overline {T}$ acts trivially on both $\varrho (L)$ and $\varrho (G)/\varrho (L)$ . By Proposition 3.5, $\overline {T}$ acts distally on $\mathrm {Sub}^{p}_{G/M}$ . From Step 3, we get that $\overline {T}$ acts trivially on $G/M$ , that is, $T(g)\in gM$ , $g\in G$ . In particular, for $\{x_t\}$ as above, $T(x_t)=x_ty_t$ , where $y_t\in M$ for all $t\in {\mathbb R}$ and $y_1\ne e$ . Since M is central in G, $y_tx_s=x_sy_t$ , $t, s\in {\mathbb R}$ , and $\{y_t\}$ is a one-parameter subgroup of the compact central subgroup M. Note that $\{y_t\}$ is closed if $\{x_t\}$ is compact.

As $\mathrm {Sub}^p_G$ is compact, there exists an unbounded sequence $\{n_k\}$ in ${\mathbb N}$ such that $T^{n_k}(\{x_t\})\to C$ for some $C\in \mathrm {Sub}^p_G$ . We show that $C=\{x_t\}\times \overline {\{y_t\}}$ . As $\{x_t\}\subset \{x_t\}\times \overline {\{y_t\}}$ and the latter is T-invariant, we get that $C\subset \{x_t\}\times \overline {\{y_t\}}$ . For a fixed $t\in {\mathbb R}$ ,

$$ \begin{align*} T^{n_k}(x_{t/n_k})=x_{t/n_k}y_{t/n_k}^{n_k}=x_{t/n_k}y_t\to y_t\end{align*} $$

as $n_k\to \infty $ . Therefore, $y_t\in C$ for each t, and $\overline {\{y_t\}}\subset C$ as C is closed. Also, note that $T^{n_k}(x_t)=x_ty_t^{n_k}\to x_ty$ along a subsequence of $\{n_k\}$ for some $y\in \overline {\{y_t\}}\subset C$ . Hence, $x_t\in C$ , $t\in {\mathbb R}$ . Thus, $\{x_t\}\times \overline {\{y_t\}}\subset C$ . Now, $T(C)=C$ and we get that $T^{n_k}(T(\{x_t\}))\to C$ . As T acts distally on $\mathrm {Sub}^p_G$ , we have that $T(\{x_t\})=\{x_t\}$ and since T is unipotent, $T(x_t)=x_t$ for each t. However, $T(x_1)=x_1y_1\ne x_1$ . This leads to a contradiction. Therefore, $T(x)=x$ for each $x\in U$ and, hence, for each $x\in G$ . Thus, $T=\mathrm {Id}$ .

Proof of Theorem 1.3.

Let G be a connected Lie group and let $T\in \mathrm {Aut}(G)$ . Suppose T is unipotent and it acts distally on $\mathrm {Sub}^p_G$ . We show that $T=\mathrm {Id}$ .

Step 1: Suppose G is nilpotent. If possible, suppose $T\ne \mathrm {Id}$ . Since T is unipotent, by [Reference Shah and Yadav42, Proposition 3.10], there exists an increasing sequence of closed connected normal T-invariant subgroups $\{e\} = G_0 \subset G_1 \subset \cdots \subset G_n = G$ , $n \geq 2$ , such that T acts trivially on each successive quotient group $G_i/G_{i-1}$ , $1\leq i\leq n$ , and T does not act trivially on $G_{i+1}/G_{i-1}$ , $1\leq i \leq n -1$ . Then, $G_2\ne G_1$ . As T acts distally on $\mathrm {Sub}^p_G$ , it acts distally on $\mathrm {Sub}^{p}_{G_2}$ . Now, by Proposition 4.2, $T|_{G_2}=\mathrm {Id}$ , which leads to a contradiction. Therefore, $T=\mathrm {Id}$ when G is nilpotent.

Step 2: Suppose G is not nilpotent. Let N be the nilradical of G. As T acts distally on $\mathrm {Sub}^{p}_N$ , it follows from Step 1 that $T|_N=\mathrm {Id}$ . Suppose G is solvable. As noted in Remark 3.3, we have that the automorphism induced by T on $G/N$ is a finite order automorphism and it is trivial since it is unipotent. Now, by Proposition 4.2, $T=\mathrm {Id}$ .

Step 3: Suppose G is not solvable. Let R be the radical of G. Then, as T acts distally on $\mathrm {Sub}^{p}_R$ , we have from Step 2 that $T|_R=\mathrm {Id}$ .

Now, $G/N$ is reductive. Let $\pi :G\to G/N$ be the natural projection. Then, ${\pi (G)=SR'}$ , an almost direct product of a semisimple Levi subgroup S of $\pi (G)$ and the radical $R'=\pi (R)$ of $\pi (G)$ , where $R'$ is central in $\pi (G)$ . Let ${\overline {T}}$ be the automorphism of $\pi (G)$ induced by T. Then, ${\overline {T}}$ acts trivially on $R'$ . Also, ${\overline {T}}$ keeps S invariant as it is the commutator subgroup of $\pi (G)$ . Now, if ${\overline {T}}$ acts trivially on $\pi (G)=G/N$ , then by Proposition 4.2, $T=\mathrm {Id}$ . If possible suppose ${\overline {T}}$ does not act trivially on $\pi (G)$ . As ${\overline {T}}$ is unipotent, there exists a nontrivial unipotent element u in S such that ${\overline {T}}=\mathrm {inn}(u)$ . Considering the Iwasawa decomposition $S=KAU$ , where $u\in U$ , we have that u is not in the centre of $AU$ (see [Reference Knapp29, Lemma 6.50]). Now, $AUR'$ is connected, solvable and ${\overline {T}}$ invariant. Let B be the closure of $\pi ^{-1}(AUR')$ . It is a closed connected solvable T-invariant subgroup of G. As T acts distally on $\mathrm {Sub}^{p}_B$ , by Step 2, $T|_B=\mathrm {Id}$ . This implies that ${\overline {T}}$ acts trivially on $AU$ and that u centralises $AU$ , which is a contradiction. Thus, ${\overline {T}}$ acts trivially on $\pi (G)=G/N$ , and as noted above, by Proposition 4.2, we get that $T=\mathrm {Id}$ .

The converse statement in the theorem is obvious.

Proof of Theorem 1.1(2) for non-nilpotent case.

Let G be a connected Lie group in class ${\mathcal {C}'}$ . Then, $\mathrm {Aut}(G)$ is almost algebraic as a subgroup of $\mathrm {GL}({\mathfrak g})$ (via the correspondence $T\leftrightarrow {d\,} T$ ) (see [Reference Dani12]). Suppose $T\in \mathrm {Aut}(G)$ acts distally on $\mathrm {Sub}^p_G$ . Suppose G is not nilpotent. In particular, it is not a vector group. By Theorem 1.2, T acts distally on G. Then, for some $n\in {\mathbb N}$ , $T^n$ is contained in a connected abelian almost algebraic subgroup of $\mathrm {Aut}(G)$ and it has the form ${\mathcal K}\times \,{\mathcal U}$ , where ${{\mathcal K},\,{\mathcal U}\subset \mathrm {Aut}(G)}$ , ${\mathcal K}$ is an abelian compact connected group and ${\mathcal U}$ is an abelian unipotent connected group; that is, it consists of unipotent elements (see [Reference Abels1, Corollaries 2.3 and 2.5]). Now, $T^n=S\circ \psi =\psi \circ S$ , where $S\in {\mathcal U}$ is unipotent and $\psi \in {\mathcal K}$ . Hence, S acts distally on $\mathrm {Sub}^p_G$ (see [Reference Shah and Yadav40, Lemma 2.2]). By Theorem 1.3, $S=\mathrm {Id}$ . Thus, $T^n=\psi \in {\mathcal K}$ and, hence, T is contained in a compact subgroup of $\mathrm {Aut}(G)$ . The converse is obvious (see [Reference Palit and Shah32, Lemma 4.2]).

Proof. Let A be any maximal torus in a connected Lie group G. It is easy to see that for any closed connected normal subgroup L of G, $A\cap L$ is a maximal torus of L.

Let ${\mathcal K}$ be a compact connected abelian subgroup of $\mathrm {Aut}(G)$ and let $H={\mathcal K}\ltimes G$ ; where the action of ${\mathcal K}$ on G is by automorphisms. Then, H is a connected Lie group and G is normal in H. Let R be the radical of G. Then, R is characteristic in G and, hence, normal in H. Let A be a maximal torus in H containing ${\mathcal K}$ . Arguing as above, we get that $A\cap G$ (respectively $A\cap R$ ) is a maximal torus in G (respectively R). As ${\mathcal K}\subset A$ and A is abelian, we get that every automorphism in ${\mathcal K}$ acts trivially on $A\cap G$ (respectively $A\cap R$ ).

Proof of Theorem 1.4.

Let G be a connected Lie group and let $T\in \mathrm {Aut}(G)$ . The statements $(5)\implies (4)\implies (3)\implies (2)\implies (1)$ are obvious. Now, it is enough to show that $(1)\implies (5)$ when G is not a vector group and T acts distally on the largest compact connected central subgroup (say) M of G. Suppose statement (1) holds, that is, T acts distally on $\mathrm {Sub}^p_G$ . Then, we want to show that T is contained in a compact subgroup of $\mathrm {Aut}(G)$ . If M is trivial, then G belongs to the class ${\mathcal {C}'}$ and the assertion follows from Theorem 1.1(2). Now, suppose M is nontrivial. By Theorem 1.2, T acts distally on G. Then, by [Reference Raja and Shah34, Theorem 3.1], $\overline {T}$ acts distally on $G/M$ , where $\overline {T}$ is the automorphism of $G/M$ induced by T.

Suppose $G=M$ . Since T acts distally on M, by [Reference Abels2, Lemma 2.5], $T^n$ is unipotent for some $n\in {\mathbb N}$ . As T acts distally on $\mathrm {Sub}^p_G$ , so does $T^n$ , and by Theorem 1.3, $T^n=\mathrm {Id}$ . Hence, the assertion holds when G is compact and abelian.

Now, suppose M is nontrivial and $G\ne M$ . We show that there exists a maximal torus of G that is $T^k$ -invariant for some $k\in {\mathbb N}$ . Let $\pi :G\to G/M$ be the natural projection. By Proposition 3.5, $\overline {T}$ acts distally on $\mathrm {Sub}^{p}_{G/M}$ .

Suppose $G/M$ is a vector group. Then by Theorem 1.1(1), $\overline {T}=\psi \circ \,\alpha \,\mathrm {Id}$ for some $\alpha \in {\mathbb R}\setminus \{0\}$ , and $\psi $ generates a relatively compact subgroup in $\mathrm {Aut}(G/M)$ . Since $\overline {T}$ acts distally on $G/M$ , we get that $|\alpha |=1$ and, hence, $\overline {T}$ is contained in a compact subgroup of $\mathrm {Aut}(G/M)$ . Now, suppose $G/M$ is not a vector group. As $G/M$ belongs to the class ${\mathcal {C}'}$ , by Theorem 1.1(2), $\overline {T}$ is contained in a compact subgroup of $\mathrm {Aut}(G/M)$ . So, in either case, we have that $\overline {T}$ is contained in a compact subgroup of $\mathrm {Aut}(G/M)$ .

Let ${\mathcal K}$ be the connected component of the identity in the closed (compact) subgroup generated by $\overline {T}$ in $\mathrm {Aut}(G/M)$ . Then, $(\overline {T})^k\in {\mathcal K}$ for some $k\in {\mathbb N}$ . Now, by Lemma 4.3, $(\overline {T})^k$ keeps a maximal torus (say) A of $G/M$ invariant and it acts trivially on A. Let $K=\pi ^{-1}(A)$ . Then, K is a maximal torus of G and $T^k(K)=K$ .

Note that K is compact, connected and abelian. Moreover, as T acts distally on G, so does $T^k$ and the restriction of $T^k$ to K is also distal. Arguing as above for K instead of M, we get that for some $m\in {\mathbb N}$ , $T^m|_K$ is unipotent and, as it acts distally on $\mathrm {Sub}^{p}_K$ , it acts trivially on K (by Theorem 1.3). Without loss of any generality, we may replace T by $T^m$ and assume that T fixes every element of K, that is, $T\in F_K(G) $ , where the set $F_K(G) :=\{\varrho \in \mathrm {Aut}(G)\mid \varrho (x)=x \mbox { for all } x\in K\}$ . We know that $F_K(G)$ is almost algebraic (see [Reference Dani and Shah15]). As T acts distally on G and $T\in F_K(G)$ , for some $n\in {\mathbb N}$ , $T^n$ is contained in a connected almost algebraic subgroup ${\mathcal B}\times {\mathcal U}$ of $F_K(G)$ , where ${\mathcal B}$ is a compact group and ${\mathcal U}$ consists of unipotent elements (see [Reference Abels1, Corollaries 2.3 and 2.5]). Now, we have that $T=\phi \circ S=S\circ \phi $ such that $\phi \in {\mathcal B}$ and $S\in {\mathcal U}$ . As ${\mathcal B}$ is compact, we have that S acts distally on $\mathrm {Sub}^p_G$ . By Theorem 1.3, $S=\mathrm {Id}$ . Then, $T=\phi \in {\mathcal B}$ and statement (5) holds.

Proof of Theorem 1.5.

Let G be a connected Lie group and let ${\mathcal H}$ be a subgroup of $\mathrm {Aut}(G)$ . Suppose ${\mathcal H}$ acts distally on the largest compact connected central subgroup of G. Then, so does $\overline {{\mathcal H}}$ . Note that ${\mathcal H}$ acts distally on a compact ${\mathcal H}$ -invariant subspace of $\mathrm {Sub}_G$ if and only if $\overline {{\mathcal H}}$ does (both the above statements follow from [Reference Ellis16, Theorem 1]). Also, statement (7) holds for ${\mathcal H}$ if and only if it holds when ${\mathcal H}$ is replaced by $\overline {{\mathcal H}}$ , as ${\mathcal K}{\mathcal D}$ is closed in $\mathrm {GL}(d,{\mathbb R})$ for ${\mathcal K},{\mathcal D}$ as in statement (7). Therefore, to prove the assertions, we may assume that ${\mathcal H}$ is closed.

Note that every compact subgroup of $\mathrm {Aut}(G)$ acts distally on $\mathrm {Sub}_G$ (see, for example, [Reference Shah and Yadav40, Lemma 2.2]). Now $(6)\implies (5)\implies (4)\implies (3)\implies (2)\implies (1)$ are obvious.

First, suppose that G is not a vector group. We show that $(1)\implies (6)$ . Suppose statement (1) holds, that is, every $T\in {\mathcal H}$ acts distally on $\mathrm {Sub}^p_G$ . By Theorem 1.4, every $T\in {\mathcal H}$ is contained in a compact subgroup of $\mathrm {Aut}(G)$ . Recall that $\mathrm {Aut}(G)$ is identified with a closed subgroup of $\mathrm {GL}({\mathfrak g})$ via the map $T\mapsto {d\,} T$ , where ${\mathfrak g}$ is the Lie algebra of G and it is a d-dimensional vector space for some $d\in {\mathbb N}$ (see [Reference Chevalley9]; see also [Reference Hochschild22]). Therefore, ${\mathcal H}$ is also a closed subgroup of $\mathrm {GL}({\mathfrak g})$ and the latter is isomorphic to $\mathrm { GL}(d,{\mathbb R})$ , and we get by [Reference Fresnel and van der Put17, Theorem 1.1] (see also [Reference Shah and Yadav39, Proposition 2]) that ${\mathcal H}$ is compact. Thus statement (6) holds and statements (1)–(6) are equivalent. Thus, if G is not a vector group, the assertions in the theorem hold for this case.

Now, suppose $G={\mathbb R}^d$ for some $d\in {\mathbb N}$ . Suppose statement (3) holds, that is, ${\mathcal H}$ acts distally on $\overline {\mathrm {Sub}^c_G}$ . Then, $T\in \rm {(NC)}$ for every $T\in {\mathcal H}$ . Since G belongs to the class ${\mathcal {C}'}$ , by [Reference Shah and Yadav42, Proposition 2], T is contained in a compact subgroup of $\mathrm {Aut}(G)$ for every $T\in {\mathcal H}$ . Since ${\mathcal H}$ is closed, arguing as above using [Reference Fresnel and van der Put17, Theorem 1.1] or [Reference Shah and Yadav39, Proposition 2], we get that ${\mathcal H}$ is compact. Hence, statement (6) holds. Thus, statements (3)–(6) are equivalent when G is a vector group. Now we show that statements (1), (2) and (7) are equivalent. Suppose statement (7) holds. Note that ${\mathcal D}=\{r\,\mathrm {Id}\in \mathrm {GL}(d,{\mathbb R})\mid r\in {\mathbb R}\setminus \{0\}\}$ . Therefore, ${\mathcal D}$ acts trivially on $\mathrm {Sub}^p_G$ and the action of ${\mathcal K}{\mathcal D}$ on $\mathrm {Sub}^p_G$ is the same as that of the compact group ${\mathcal K}$ on $\mathrm {Sub}^p_G$ . Therefore, the action of ${\mathcal K}{\mathcal D}$ and, hence, the action of ${\mathcal H}$ on $\mathrm {Sub}^p_G$ are distal. Thus, $(7)\implies (2)$ . We have already noted that $(2)\implies (1)$ .

Now, we show that $(1)\implies (7)$ . Suppose every $T\in {\mathcal H}$ acts distally on $\mathrm {Sub}^p_G$ . Let $T\in {\mathcal H}$ be fixed. By Theorem 1.1(1), $T\in {\mathcal B}{\mathcal D}$ for some compact group ${\mathcal B}\subset \mathrm {GL}(d,{\mathbb R})$ . Let ${\mathcal H}_1:=\overline {[{\mathcal H},{\mathcal H}]}$ . Then, ${\mathcal H}_1\subset {\mathcal H}$ as the latter is closed. Now, let $T\in {\mathcal H}_1$ . Then, $\det T=1$ . From above, we have that $T=\phi \,\circ \,r\,\mathrm {Id}$ for some $\phi \in {\mathcal B}$ and some $r\in {\mathbb R}\setminus \{0\}$ . Since ${\mathcal B}$ is a compact group, $\det \phi =\pm 1$ and, hence, $r=\pm 1$ . Therefore, T generates a relatively compact group for every $T\in {\mathcal H}_1$ . As ${\mathcal H}_1$ is a closed subgroup of $\mathrm {GL}(d,{\mathbb R})$ by [Reference Fresnel and van der Put17, Theorem 1.1] or [Reference Shah and Yadav39, Proposition 2], ${\mathcal H}_1$ is compact. Let $N({\mathcal H}_1)$ be the normaliser of ${\mathcal H}_1$ in $\mathrm {GL}(d,{\mathbb R})$ . Then, $N({\mathcal H}_1)$ is a closed algebraic subgroup of $\mathrm {GL}(d,{\mathbb R})$ and ${\mathcal H},{\mathcal D}\subset N({\mathcal H}_1)$ . Moreover, $N({\mathcal H}_1)\subset N({\mathcal H}_1{\mathcal D})$ . Being algebraic, $N({\mathcal H}_1)$ has finitely many connected components and, hence, so does $N({\mathcal H}_1)/{\mathcal H}_1{\mathcal D}$ . In particular, any closed abelian subgroup of $N({\mathcal H}_1)/{\mathcal H}_1{\mathcal D}$ is compactly generated (see [Reference Hofmann and Neeb24]). Let ${\rho : N({\mathcal H}_1)\to N({\mathcal H}_1)/{\mathcal H}_1{\mathcal D}}$ be the natural projection. Then, $\rho ({\mathcal H})$ is abelian and, hence, so is $\overline {\rho ({\mathcal H})}$ . Therefore, $\overline {\rho ({\mathcal H})}$ is compactly generated and it has a unique maximal compact subgroup, say $\mathcal {L}$ .

Now, let $T\in {\mathcal H}$ . Then, $T=\psi \,\circ \,r\,\mathrm {Id}$ for some $r\in {\mathbb R}\setminus \{0\}$ and some $\psi $ , which generates a relatively compact group and $r\,\mathrm {Id}\in {\mathcal D}$ . Then, $\rho (T)=\rho (\psi )$ also generates a relatively compact group in $\overline {\rho ({\mathcal H})}$ and, hence, it is contained in $\mathcal {L}$ . Thus, $\rho ({\mathcal H})\subset \mathcal {L}$ and $\overline {\rho ({\mathcal H})}$ is compact. As ${\mathcal H}_1$ is compact, it follows that $\overline {{\mathcal H}{\mathcal D}}/{\mathcal D}$ is compact. Since ${\mathcal D}$ has two connected components, we get that $\overline {{\mathcal H}{\mathcal D}}/{\mathcal D}^0$ is also compact and it is a Lie group. As ${\mathcal D}^0$ is a vector group that is central in $\overline {{\mathcal H}{\mathcal D}}$ , by [Reference Iwasawa25, Lemma 3.7], $\overline {{\mathcal H}{\mathcal D}}={\mathcal K}\times {\mathcal D}^0$ , where ${\mathcal K}$ is the maximal compact subgroup of $\overline {{\mathcal H}{\mathcal D}}$ . In particular, ${\mathcal H}\subset {\mathcal K}{\mathcal D}^0$ . Hence, statement (7) holds. Thus, if G is a vector group, statements (1), (2) and (7) are equivalent. This completes the proof.

Proof of Corollary 1.6.

Let G be a connected Lie group. Statement (5) implies that $\mathrm {Inn}(G)$ is compact and, hence, it follows that $(5)\implies (4)$ . Also, $(4)\implies (3)\implies (1)$ and $(4)\implies (2)\implies (1)$ are obvious. It is enough to show that $(1)\implies (5)$ .

Suppose statement (1) holds; that is, every inner automorphism of G acts distally on $\mathrm {Sub}^p_G$ . If G is either compact or a vector group, then statement (5) holds trivially. Now, suppose G is neither compact nor a vector group. Since every inner automorphism acts trivially on the centre of G, we have by Theorem 1.2 that it acts distally on G. Therefore, G is distal (see [Reference Rosenblatt36]), that is, the action of $\mathrm {Inn}(G)$ on G is distal. It follows from [Reference Rosenblatt36, Theorem 9] and [Reference Jenkins27, Corollary 2.1(ii)] that $G/R$ is compact, where R is the radical of G. Let N be the nilradical of G and let $x\in N$ . Then, $\mathrm {inn}(x)$ is unipotent and by Theorem 1.3, $\mathrm {inn}(x)=\mathrm {Id}$ . This implies that x belongs to the centre of G for every $x\in N$ . Thus, N is abelian and central in G. Now, for the radical R, we know that $[R,R]\subset N$ is central in G and, hence, R is nilpotent and it is the same as N and it is abelian and central in G. Now, $G/N$ is compact, where N is central in G, that is, $N={\mathbb R}^n\times M$ for some $n\in {\mathbb N}$ , where M is compact. Here, $V:={\mathbb R}^n$ is a vector subgroup, which is central. By [Reference Iwasawa25, Lemma 3.7], $G={\mathbb R}^n\times K$ , where K is the maximal compact subgroup of G. Thus, statement (5) holds.