1 Introduction
1.1 General background
Let
$(V,\langle \cdot \,,\cdot \rangle )$
be a euclidean vector space of dimension
$n$
and let
$A$
be an abelian semigroup. Let
${\mathcal{K}}(V)$
be the space of compact convex subsets of
$V$
. A valuation on
$V$
is a map
$\unicode[STIX]{x1D707}:{\mathcal{K}}(V)\rightarrow A$
satisfying
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D707}(K\cup L)+\unicode[STIX]{x1D707}(K\cap L)=\unicode[STIX]{x1D707}(K)+\unicode[STIX]{x1D707}(L), & & \displaystyle \nonumber\end{eqnarray}$$
whenever
$K,L,K\cup L\in {\mathcal{K}}(V)$
. If
$A$
is a topological abelian semigroup, then
$\unicode[STIX]{x1D707}$
is a continuous valuation if it is continuous with respect to the topology on
${\mathcal{K}}(V)$
induced by the Hausdorff metric.
Natural properties of valuations are translation-invariance (i.e.,
$\unicode[STIX]{x1D707}(K+v)=\unicode[STIX]{x1D707}(K)$
for all
$K$
and
$v\in V$
) and rotation-invariance (i.e.,
$\unicode[STIX]{x1D707}(gK)=\unicode[STIX]{x1D707}(K)$
for all
$K$
and
$g\in \text{SO}(n)$
). Hadwiger’s seminal theorem states that the space of real-valued, continuous, rotation- and translation-invariant valuations is spanned by the intrinsic volumes
$\unicode[STIX]{x1D707}_{0},\ldots ,\unicode[STIX]{x1D707}_{n}$
. We refer to [Reference Klain and Rota18] for a proof of this theorem and a detailed account on the related kinematic formulas in euclidean vector spaces.
Intrinsic volumes may be defined via a tube formula (called Steiner’s formula, see [Reference Klain and Rota18, Reference Schneider22, Reference Steiner28]) as follows. Denote by
$B_{2}^{m}$
the euclidean unit ball of dimension
$m$
and by
$\unicode[STIX]{x1D705}_{m}:=\unicode[STIX]{x1D70B}^{m/2}/\unicode[STIX]{x1D6E4}(1+m/2)$
its volume. Then
$\unicode[STIX]{x1D714}_{m}:=m\unicode[STIX]{x1D705}_{m}$
is the volume of the unit sphere of dimension
$m-1$
, denoted by
$S^{m-1}$
. Given
$K\in {\mathcal{K}}(V)$
and
$\unicode[STIX]{x1D70C}>0$
, let
$K+\unicode[STIX]{x1D70C}B_{2}^{n}$
be the Minkowski sum of
$K$
and a
$\unicode[STIX]{x1D70C}$
-ball, which we call a
$\unicode[STIX]{x1D70C}$
-tube around
$K$
. Then
$\operatorname{vol}(K+\unicode[STIX]{x1D70C}B_{2}^{n})$
is a polynomial in
$\unicode[STIX]{x1D70C}$
, whose coefficients (up to a normalization) are the intrinsic volumes:
$$\begin{eqnarray}\displaystyle \operatorname{vol}(K+\unicode[STIX]{x1D70C}B_{2}^{n})=\mathop{\sum }_{k=0}^{n}\unicode[STIX]{x1D707}_{k}(K)\unicode[STIX]{x1D705}_{n-k}\unicode[STIX]{x1D70C}^{n-k}. & & \displaystyle \nonumber\end{eqnarray}$$
Intrinsic volumes appear in the following kinematic formulas due to Chern, Blaschke and Santaló. Let
$\overline{\text{SO}(n)}:=V\rtimes \text{SO}(n)$
be the special euclidean group (i.e., the group of isometries of
$V$
preserving orientation), endowed with the product of the Lebesgue measure on
$V$
and the probability measure on
$\text{SO}(n)$
. Given
$K,L\in {\mathcal{K}}(V)$
, the intersectional kinematic formulas assert that for
$0\leqslant i\leqslant n$
(see, e.g., [Reference Klain and Rota18, Reference Schneider22])
$$\begin{eqnarray}\int _{\overline{\text{SO}(n)}}\unicode[STIX]{x1D707}_{i}(K\cap \bar{g}L)\,d\bar{g}=\left[\begin{array}{@{}c@{}}n+i\\ i\end{array}\right]\mathop{\sum }_{k+l=n+i}\left[\begin{array}{@{}c@{}}n+i\\ k\end{array}\right]^{-1}\unicode[STIX]{x1D707}_{k}(K)\unicode[STIX]{x1D707}_{l}(L).\end{eqnarray}$$
The expression on the right-hand side involves flag coefficients defined by
$$\begin{eqnarray}\displaystyle \left[\begin{array}{@{}c@{}}n\\ k\end{array}\right]:=\binom{n}{k}\frac{\unicode[STIX]{x1D705}_{n}}{\unicode[STIX]{x1D705}_{k}\unicode[STIX]{x1D705}_{n-k}}. & & \displaystyle \nonumber\end{eqnarray}$$
The additive kinematic formulas, also called rotation sum formulas (see [Reference Schneider22]), are given by
$$\begin{eqnarray}\int _{\text{SO}(n)}\unicode[STIX]{x1D707}_{i}(K+gL)\,dg=\left[\begin{array}{@{}c@{}}2n-i\\ n-i\end{array}\right]\mathop{\sum }_{k+l=i}\left[\begin{array}{@{}c@{}}2n-i\\ n-k\end{array}\right]^{-1}\unicode[STIX]{x1D707}_{k}(K)\unicode[STIX]{x1D707}_{l}(L),\end{eqnarray}$$
with
$0\leqslant i\leqslant n$
.
Both formulas admit local versions. In order to describe them, let us recall the notion of a support measure (we refer to [Reference Schneider22] for a detailed study). Let
$SV:=V\times S^{n-1}$
be the unit sphere bundle over
$V$
. Denote by
${\mathcal{B}}(SV)$
the Borel subsets of
$SV$
. For a given
$\unicode[STIX]{x1D702}\in {\mathcal{B}}(SV)$
, consider the part of the
$\unicode[STIX]{x1D70C}$
-tube around
$K$
defined by
$$\begin{eqnarray}\displaystyle M_{\unicode[STIX]{x1D70C}}(K,\unicode[STIX]{x1D702}):=\{x\in V:0<d(x,K)\leqslant \unicode[STIX]{x1D70C};(p(K,x),u(K,x))\in \unicode[STIX]{x1D702}\}. & & \displaystyle \nonumber\end{eqnarray}$$
Here
$p(K,x)$
is the unique nearest point to
$x$
in
$K$
and
$u(K,x):=(x-p(K,x))/\Vert x-p(K,x)\Vert$
is a unit normal vector to
$K$
at
$p(K,x)$
. The volume of
$M_{\unicode[STIX]{x1D70C}}(K,\unicode[STIX]{x1D702})$
is again a polynomial in
$\unicode[STIX]{x1D70C}$
, whose coefficients (up to a normalization) are the support measures of
$K$
:
$$\begin{eqnarray}\displaystyle \operatorname{vol}M_{\unicode[STIX]{x1D70C}}(K,\unicode[STIX]{x1D702})=\frac{1}{n}\mathop{\sum }_{k=0}^{n-1}\unicode[STIX]{x1D6E9}_{k}(K,\unicode[STIX]{x1D702})\binom{n}{k}\unicode[STIX]{x1D70C}^{n-k}. & & \displaystyle \nonumber\end{eqnarray}$$
For a fixed
$K$
,
$\unicode[STIX]{x1D6E9}_{k}(K,\cdot )$
is a measure on
$SV$
. For a fixed
$\unicode[STIX]{x1D702}$
,
$\unicode[STIX]{x1D6E9}_{k}(\cdot ,\unicode[STIX]{x1D702})$
is a measurable valuation. Moreover,
$K\mapsto \unicode[STIX]{x1D6E9}_{k}(K,\cdot )$
is continuous, where the space of measures is endowed with the weak topology.
The support measures
$\unicode[STIX]{x1D6E9}_{k}$
are
$\operatorname{SO}(n)$
-invariant, i.e.,
$\unicode[STIX]{x1D6E9}_{k}(gK,g\unicode[STIX]{x1D702})=\unicode[STIX]{x1D6E9}_{k}(K,\unicode[STIX]{x1D702})$
for every
$g\in \operatorname{SO}(n),K\in {\mathcal{K}}(V)$
and
$\unicode[STIX]{x1D702}\in {\mathcal{B}}(SV)$
. For characterization theorems in the spirit of Hadwiger, see [Reference Schneider21] and [Reference Schneider22, p. 226]. The marginals of support measures are curvature measures and area measures:
$$\begin{eqnarray}\displaystyle & \displaystyle C_{k}(K,\unicode[STIX]{x1D6FD}):=\unicode[STIX]{x1D6E9}_{k}(K,\unicode[STIX]{x1D6FD}\times S^{n-1}),\quad \unicode[STIX]{x1D6FD}\in {\mathcal{B}}(V), & \displaystyle \nonumber\\ \displaystyle & \displaystyle S_{k}(K,\unicode[STIX]{x1D6FD}):=\unicode[STIX]{x1D6E9}_{k}(K,V\times \unicode[STIX]{x1D6FD}),\quad \unicode[STIX]{x1D6FD}\in {\mathcal{B}}(S^{n-1}), & \displaystyle \nonumber\end{eqnarray}$$
where
$0\leqslant k\leqslant n-1$
. Another common normalization is the following:
$$\begin{eqnarray}\displaystyle & \displaystyle \unicode[STIX]{x1D6F7}_{k}(K,\cdot ):=\frac{1}{n\unicode[STIX]{x1D705}_{n-k}}\binom{n}{k}C_{k}(K,\cdot ), & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D6F9}_{k}(K,\cdot ):=\frac{1}{n\unicode[STIX]{x1D705}_{n-k}}\binom{n}{k}S_{k}(K,\cdot ). & \displaystyle \nonumber\end{eqnarray}$$
One completes the definition by setting
$\unicode[STIX]{x1D6F7}_{n}(K,\unicode[STIX]{x1D6FD}):={\mathcal{H}}^{n}(K\cap \unicode[STIX]{x1D6FD})$
. Then
$\unicode[STIX]{x1D6F7}_{k}(K,V)=\unicode[STIX]{x1D707}_{k}(K)$
for
$0\leqslant k\leqslant n$
and
$\unicode[STIX]{x1D6F9}_{k}(K,S^{n-1})=\unicode[STIX]{x1D707}_{k}(K)$
for
$0\leqslant k\leqslant n-1$
.
The local form of formula (1) can best be described in terms of the curvature measures
$\unicode[STIX]{x1D6F7}_{k}$
. For
$\unicode[STIX]{x1D6FD},\unicode[STIX]{x1D6FD}^{\prime }\in {\mathcal{B}}(V)$
, we have
$$\begin{eqnarray}\displaystyle & & \displaystyle \int _{\overline{\text{SO}(n)}}\unicode[STIX]{x1D6F7}_{i}(K\cap \bar{g}L,\unicode[STIX]{x1D6FD}\cap \bar{g}\unicode[STIX]{x1D6FD}^{\prime })\,d\bar{g}\nonumber\\ \displaystyle & & \displaystyle \quad =\left[\begin{array}{@{}c@{}}n+i\\ i\end{array}\right]\mathop{\sum }_{k+l=n+i}\left[\begin{array}{@{}c@{}}n+i\\ k\end{array}\right]^{-1}\unicode[STIX]{x1D6F7}_{k}(K,\unicode[STIX]{x1D6FD})\unicode[STIX]{x1D6F7}_{l}(L,\unicode[STIX]{x1D6FD}^{\prime }),\end{eqnarray}$$
with
$0\leqslant i\leqslant n$
. The additive kinematic formula (2) admits a localization in terms of the area measures
$S_{k}$
, for
$\unicode[STIX]{x1D6FD},\unicode[STIX]{x1D6FD}^{\prime }\in {\mathcal{B}}(S^{n-1})$
, as follows:
$$\begin{eqnarray}\displaystyle & & \displaystyle \int _{\text{SO}(n)}S_{i}(K+gL,\unicode[STIX]{x1D6FD}\cap g\unicode[STIX]{x1D6FD}^{\prime })\,dg\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\unicode[STIX]{x1D714}_{n}}\mathop{\sum }_{k+l=i}\binom{i}{k}S_{k}(K,\unicode[STIX]{x1D6FD})S_{l}(L,\unicode[STIX]{x1D6FD}^{\prime }),\quad 0\leqslant i\leqslant n-1.\nonumber\end{eqnarray}$$
Formulas of this type can also be proved for subgroups of the euclidean motion group
$\overline{\text{SO}(n)}$
. For the important hermitian case, where the rotation group is replaced by the unitary rotation group
$\overline{\text{U}(m)}$
, the global kinematic formulas were obtained in [Reference Bernig and Fu6]; the local intersectional kinematic formulas were derived in [Reference Bernig, Fu and Solanes7]; and the local additive formulas were found in [Reference Wannerer29].
Let us next recall the construction of flag-type support, area and curvature measures via local tube formulas. We refer to [Reference Hug, Türk and Weil14] for a survey and to [Reference Goodey, Hinderer, Hug, Rataj and Weil11–Reference Hug, Rataj and Weil13] for more recent developments. Let
$\operatorname{Gr}_{p}(V)$
denote the Grassmann variety of
$p$
-dimensional subspaces in
$V$
and
$\overline{\operatorname{Gr}}_{p}(V)$
the set of all affine
$p$
-planes in
$V$
. Then
$\operatorname{Gr}_{p}(V)$
admits an
$\text{SO}(n)$
-invariant measure, while
$\overline{\operatorname{Gr}}_{p}(V)$
admits an
$\overline{\text{SO}(n)}$
-invariant measure. These measures are unique up to a normalization, and we choose the standard normalization from [Reference Schneider and Weil23, Theorem 13.2.12]. For a given
$\bar{E}\in \overline{\operatorname{Gr}}_{p}(V)$
, denote by
$E\in \operatorname{Gr}_{p}(V)$
the linear space parallel to
$\bar{E}$
.
Fix
$K\in {\mathcal{K}}(V)$
and
$\unicode[STIX]{x1D70C}>0$
. Let
$\bar{E}\in \overline{\operatorname{Gr}}_{p}(V)$
be such that
$\bar{E}\cap K=\emptyset$
. For almost all such
$\bar{E}$
, there exists a unique pair of nearest points
$p(K,\bar{E})\in K$
and
$l(K,\bar{E})\in \bar{E}$
. Then
$u(K,\bar{E}):=(l(K,\bar{E})-p(K,\bar{E}))/d(K,\bar{E})\in S^{n-1}$
is a normal vector to
$K$
at
$p(K,\bar{E})$
. Given a Borel subset
$\unicode[STIX]{x1D702}$
of
$V\times S^{n-1}\times \operatorname{Gr}_{p}(V)$
, the local parallel set of
$K$
in
$\overline{\operatorname{Gr}}_{p}(V)$
is defined by
$$\begin{eqnarray}\displaystyle M_{\unicode[STIX]{x1D70C}}^{(p)}(K,\unicode[STIX]{x1D702}):=\{\bar{E}\in \overline{\operatorname{Gr}}_{p}(V):0<d(K,\bar{E})\leqslant \unicode[STIX]{x1D70C},(p(K,\bar{E}),u(K,\bar{E}),E)\in \unicode[STIX]{x1D702}\}. & & \displaystyle \nonumber\end{eqnarray}$$
The volume of
$M_{\unicode[STIX]{x1D70C}}^{(p)}(K,\unicode[STIX]{x1D702})\subset \overline{\operatorname{Gr}}_{p}(V)$
,
$0\leqslant p\leqslant n-1$
, is a polynomial in
$\unicode[STIX]{x1D70C}$
,
$$\begin{eqnarray}\displaystyle \operatorname{vol}M_{\unicode[STIX]{x1D70C}}^{(p)}(K,\unicode[STIX]{x1D702})=\mathop{\sum }_{k=0}^{n-p-1}\unicode[STIX]{x1D6EF}_{k}^{(p)}(K,\unicode[STIX]{x1D702})\unicode[STIX]{x1D705}_{n-p-k}\unicode[STIX]{x1D70C}^{n-p-k}, & & \displaystyle \nonumber\end{eqnarray}$$
whose coefficients
$\unicode[STIX]{x1D6EF}_{k}^{(p)}(K,\cdot ),0\leqslant k\leqslant n-p-1$
are the flag-type support measures (see [Reference Hug, Türk and Weil14]). Another normalization given in [Reference Hug, Türk and Weil14] is the following:
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6E9}_{k}^{(p)}:=\unicode[STIX]{x1D714}_{n-p-k}\binom{n-p-1}{k}^{-1}\unicode[STIX]{x1D6EF}_{k}^{(p)}. & & \displaystyle \nonumber\end{eqnarray}$$
Let us mention some properties of flag-type support measures. For a fixed
$\unicode[STIX]{x1D702}$
, the map
$\unicode[STIX]{x1D6EF}_{k}^{(p)}(\cdot ,\unicode[STIX]{x1D702})$
is a measurable valuation. The map
$K\mapsto \unicode[STIX]{x1D6EF}_{k}^{(p)}(K,\cdot )$
is continuous. The group
$\overline{\text{SO}(n)}$
acts on
$V\times S^{n-1}\times \operatorname{Gr}_{p}(V)$
by
$\overline{g}\cdot (x,v,E)=(gx+t,gv,gE)$
, where
$\overline{g}=(g,t)$
with
$g\in \operatorname{SO}(n)$
and
$t\in V$
. The maps
$\unicode[STIX]{x1D6EF}_{k}^{(p)}$
are
$\overline{\text{SO}(n)}$
-invariant by construction, i.e., for
$\bar{g}\in \overline{\text{SO}(n)}$
,
$\unicode[STIX]{x1D6EF}_{k}^{(p)}(\bar{g}K,\bar{g}\unicode[STIX]{x1D702})=\unicode[STIX]{x1D6EF}_{k}^{(p)}(K,\unicode[STIX]{x1D702})$
.
Similarly to curvature and area measures, which appear as marginals of support measures, flag-type curvature and flag-type area measures are defined by
$$\begin{eqnarray}\displaystyle \begin{array}{@{}rll@{}}\unicode[STIX]{x1D6F7}_{k}^{(p)}(K,\unicode[STIX]{x1D6FD})\ & :=\unicode[STIX]{x1D6EF}_{k}^{(p)}(K,\unicode[STIX]{x1D6FD}\times S^{n-1}),\quad & \unicode[STIX]{x1D6FD}\in {\mathcal{B}}(V\times \operatorname{Gr}_{p}(V)),\\ \unicode[STIX]{x1D6F9}_{k}^{(p)}(K,\unicode[STIX]{x1D6FD})\ & :=\unicode[STIX]{x1D6EF}_{k}^{(p)}(K,V\times \unicode[STIX]{x1D6FD}),\quad & \unicode[STIX]{x1D6FD}\in {\mathcal{B}}(S^{n-1}\times \operatorname{Gr}_{p}(V)).\end{array} & & \displaystyle \nonumber\end{eqnarray}$$
We will use the following normalization of flag-type area measures, as used in [Reference Hinderer, Hug and Weil12]:
$$\begin{eqnarray}\displaystyle S_{k}^{(p)}:=\unicode[STIX]{x1D714}_{n-p-k}\binom{n-p-1}{k}^{-1}\unicode[STIX]{x1D6F9}_{k}^{(p)}. & & \displaystyle \nonumber\end{eqnarray}$$
Set
$$\begin{eqnarray}\displaystyle & \displaystyle F(n,p+1):=\{(v,E)\in S^{n-1}\times \operatorname{Gr}_{p+1}(V):v\in E\}, & \displaystyle \nonumber\\ \displaystyle & \displaystyle F^{\bot }(n,p):=\{(v,E)\in S^{n-1}\times \operatorname{Gr}_{p}(V):v\bot E\}. & \displaystyle \nonumber\end{eqnarray}$$
The diffeomorphisms
$F^{\bot }(n,p)\cong F(n,p+1),(v,E)\mapsto (v,\mathbb{R}v\oplus E)$
and
$F(n,p+1)\cong F(n,n-p),(v,E)\mapsto (v,\mathbb{R}v\oplus E^{\bot })$
are
$\text{O}(n)$
-equivariant. By construction the measure
$S_{k}^{(p)}(K,\cdot )$
is concentrated on
$F^{\bot }(n,p)$
.
We now give an explicit expression of flag-type area measures on polytopes, which follows from [Reference Hinderer, Hug and Weil12]. Given a polytope
$P$
, denote by
${\mathcal{F}}_{k}(P)$
the set of all
$k$
-dimensional faces of
$P$
. For a face
$F$
, let
$N(P,F)$
denote the normal cone of
$P$
at
$F$
and let
$\text{n}(P,F):=N(P,F)\cap S^{n-1}$
. Let
$F$
also denote the linear space parallel to the face
$F$
of the same dimension. The cosine of the angle between two subspaces
$E$
and
$F$
is denoted by
$|\text{cos}(E,F)|$
, see §4 for its definition.
Proposition 1.1 [Reference Hinderer, Hug and Weil12, Theorem 3.8].
Let
$P$
be a polytope,
$0\leqslant p\leqslant n-1$
,
$0\leqslant k\leqslant n-p-1$
and
$\unicode[STIX]{x1D6FD}\in {\mathcal{B}}(F^{\bot }(n,p))$
. Then
$$\begin{eqnarray}\displaystyle & & \displaystyle S_{k}^{(p)}(P,\unicode[STIX]{x1D6FD})=\binom{n-p-1}{k}^{-1}\frac{\unicode[STIX]{x1D714}_{n-p}}{\unicode[STIX]{x1D714}_{n}}\nonumber\\ \displaystyle & & \displaystyle \quad \times \mathop{\sum }_{F\in {\mathcal{F}}_{k}(P)}\operatorname{vol}_{k}(F)\int _{\operatorname{n}(P,F)}\int _{\operatorname{Gr}_{p+1}(v)}\mathbf{1}_{(v,E\cap v^{\bot })\in \unicode[STIX]{x1D6FD}}\cos ^{2}(E^{\bot },F)\,dE\,dv.\nonumber\end{eqnarray}$$
Here
$\operatorname{Gr}_{p+1}(v)$
denotes the Grassmannian of all
$(p+1)$
-planes containing
$v$
, endowed with an invariant probability measure
$dE$
.
These flag area measures appear naturally in several contexts: extension of valuations [Reference Hinderer, Hug and Weil12]; integral formulas for projection functions of convex bodies [Reference Goodey, Hinderer, Hug, Rataj and Weil11]; integral formulas for mixed volumes of convex bodies [Reference Hug, Rataj and Weil13].
1.2 Results of the present paper
We start with a general definition.
Definition 1.2. Let
$0\leqslant p\leqslant n-1$
. A flag area measure on
$V$
is a translation-invariant valuation
$\unicode[STIX]{x1D6F7}$
with values in the space of signed measures on the flag manifold
$F(n,p+1)$
. The space of continuous flag area measures is denoted by
$\operatorname{FlagArea}^{(p)}$
and its
$k$
-homogeneous elements by
$\operatorname{FlagArea}_{k}^{(p)}$
. For a group
$G$
acting linearly on
$V$
,
$\unicode[STIX]{x1D6F7}$
is called covariant if
$\unicode[STIX]{x1D6F7}(gK)=g_{\ast }\unicode[STIX]{x1D6F7}(K)$
for all
$g\in G$
. The subspace of
$G$
-covariant continuous flag area measures is denoted by
$\operatorname{FlagArea}^{(p),G}$
.
Without further assumptions, like, for example, a version of the local definedness from [Reference Schneider20, Satz 2] or [Reference Schneider21, Theorem 6.1], this definition is probably too general to be useful. In §2 we introduce a notion of smoothness of flag area measures, which is stronger than continuity. The main purpose of this article is to classify
$\operatorname{SO}(n)$
-covariant smooth flag area measures.
To describe our results more precisely, we need the notion of Jordan angles between subspaces. Let
$W$
be a euclidean vector space and
$0\leqslant k,p\leqslant \dim W$
. The orthogonal group
$\text{O}(W)$
acts diagonally on the product
$\operatorname{Gr}_{p}(W)\times \operatorname{Gr}_{k}(W)$
. The orbits under this action are characterized in terms of Jordan angles
$\unicode[STIX]{x1D703}_{1},\ldots ,\unicode[STIX]{x1D703}_{m}$
, where
$m=\min \{k,\dim W-k,p,\dim W-p\}$
. Given
$E\in \operatorname{Gr}_{p}(W)$
and
$F\in \operatorname{Gr}_{k}(W)$
, we denote by
$$\begin{eqnarray}\unicode[STIX]{x1D70E}_{i}(E,F):=\unicode[STIX]{x1D70E}_{i}(\cos ^{2}\unicode[STIX]{x1D703}_{1},\ldots ,\cos ^{2}\unicode[STIX]{x1D703}_{m})\end{eqnarray}$$
the
$i$
th elementary symmetric polynomial. For instance,
$\unicode[STIX]{x1D70E}_{m}(E,F)$
is the square of the usual cosine between
$E$
and
$F$
.
In the special case when
$\dim W=2a$
is even and
$E,F$
are oriented subspaces with
$\dim E=\dim F=a$
, there is an additional
$\operatorname{SO}(W)$
-invariant,
$\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)$
. We refer to §4 for the definition and properties of Jordan angles and of
$\tilde{\unicode[STIX]{x1D70E}}_{a}$
.
As our first main result, we construct a three-parameter family of flag area measures
$S_{k}^{(p),i}$
, which contains the two-parameter family
$S_{k}^{(p)}$
as a subfamily.
Theorem 1 (Construction of flag area measures).
For every
$0\leqslant p$
,
$k\leqslant n-1,0\leqslant i\leqslant m:=\min \{k,n-k-1,p,n-p-1\}$
, there exists a unique continuous translation-invariant flag area measure such that for a polytope
$P\subset V$
and
$\unicode[STIX]{x1D6FD}\in {\mathcal{B}}(F(n,p+1))$
,
$$\begin{eqnarray}\displaystyle S_{k}^{(p),i}(P,\unicode[STIX]{x1D6FD}) & = & \displaystyle c_{n,k,p,i}\mathop{\sum }_{F\in {\mathcal{F}}_{k}(P)}\operatorname{vol}_{k}(F)\nonumber\\ \displaystyle & & \displaystyle \times \int _{\operatorname{n}(P,F)}\int _{\operatorname{Gr}_{p+1}(v)}\mathbf{1}_{(v,E)\in \unicode[STIX]{x1D6FD}}\unicode[STIX]{x1D70E}_{i}(E^{\bot },F)\,dE\,dv,\end{eqnarray}$$
where
$$\begin{eqnarray}\displaystyle c_{n,k,p,i}:=\binom{n-1}{k}^{-1}\binom{m}{i}^{-1}\binom{|k-(n-1-p)|+m}{i}^{-1}\binom{n-1}{i}. & & \displaystyle \nonumber\end{eqnarray}$$
The Jordan angles are computed with respect to the
$(n-1)$
-dimensional space
$W:=v^{\bot }$
.
For an odd
$n$
, there exists an additional unique continuous translation-invariant flag area measure
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}$
such that for a polytope
$P\subset V$
and
$\unicode[STIX]{x1D6FD}\in {\mathcal{B}}(F(n,(n+1)/2))$
,
$$\begin{eqnarray}\displaystyle \tilde{S}_{(n-1)/2}^{((n-1)/2)}(P,\unicode[STIX]{x1D6FD}) & = & \displaystyle \mathop{\sum }_{F\in {\mathcal{F}}_{(n-1)/2}(P)}\operatorname{vol}_{(n-1)/2}(F)\nonumber\\ \displaystyle & & \displaystyle \times \int _{\operatorname{n}(P,F)}\int _{\operatorname{Gr}_{(n+1)/2}(v)}\mathbf{1}_{(v,E)\in \unicode[STIX]{x1D6FD}}\tilde{\unicode[STIX]{x1D70E}}_{(n-1)/2}(E^{\bot },F)\,dE\,dv.\qquad\end{eqnarray}$$
The flag area measures from Theorem 1 can also be computed explicitly for a smooth compact convex body
$K$
. Let
$x\in \unicode[STIX]{x2202}K$
be a boundary point and let
$E\in \operatorname{Gr}_{p+1}(V)$
be a plane containing the normal vector
$\unicode[STIX]{x1D708}(x)$
to
$K$
at the point
$x$
. There are three linear maps on the tangent space
$T_{x}\unicode[STIX]{x2202}K$
: the shape operator
$S_{x}$
and the orthogonal projections
$\unicode[STIX]{x1D6F1}_{E}$
and
$\unicode[STIX]{x1D6F1}_{E^{\bot }}$
onto
$E\cap T_{x}\unicode[STIX]{x2202}K$
and
$E^{\bot }$
, respectively. We orient the spaces
$E\cap T_{x}\unicode[STIX]{x2202}K$
and
$E^{\bot }$
in such a way that
$\mathbb{R}\unicode[STIX]{x1D708}(x)\oplus (E\cap T_{x}\unicode[STIX]{x2202}K)\oplus E^{\bot }\cong V$
as oriented vector spaces. Let
$D$
denote the mixed discriminant. We derive the following formulas.
Theorem 2. Let
$k,p,i,\unicode[STIX]{x1D6FD}$
be as in Theorem 1. Then for a smooth compact convex body
$K$
,
$$\begin{eqnarray}\displaystyle S_{k}^{(p),i}(K,\unicode[STIX]{x1D6FD}) & = & \displaystyle c_{n,k,p,i}\binom{n-1}{k}\mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}-i}\binom{\min \{k,p\}-a}{i}\binom{k}{a}\nonumber\\ \displaystyle & & \displaystyle \times \int _{\unicode[STIX]{x2202}K}\int _{\operatorname{Gr}_{p+1}(\unicode[STIX]{x1D708}(x))}\mathbf{1}_{(\unicode[STIX]{x1D708}(x),E)\in \unicode[STIX]{x1D6FD}}\nonumber\\ \displaystyle & & \displaystyle \times \,D(S_{x}[n-k-1],\unicode[STIX]{x1D6F1}_{E}[a],\unicode[STIX]{x1D6F1}_{E^{\bot }}[k-a])\,dE\,d{\mathcal{H}}^{n-1}(x).\quad\end{eqnarray}$$
If
$n$
is odd and
$p=k=(n-1)/2$
,
$$\begin{eqnarray}\displaystyle \tilde{S}_{(n-1)/2}^{((n-1)/2)}(K,\unicode[STIX]{x1D6FD}) & = & \displaystyle (-1)^{(n-1)/2}\int _{\unicode[STIX]{x2202}K}\int _{\operatorname{Gr}_{p+1}(\unicode[STIX]{x1D708}(x))}\mathbf{1}_{(\unicode[STIX]{x1D708}(x),E)\in \unicode[STIX]{x1D6FD}}\nonumber\\ \displaystyle & & \displaystyle \times \,\text{det}(\unicode[STIX]{x1D6F1}_{E^{\bot }}\circ S_{x}:E\cap T_{x}\unicode[STIX]{x2202}K\rightarrow E^{\bot })\,dE\,d{\mathcal{H}}^{n-1}(x).\nonumber\end{eqnarray}$$
Remark.
(i) We will see that in the case
$k\leqslant p$
, equation (6) simplifies to $$\begin{eqnarray}\displaystyle S_{k}^{(p),i}(K,\unicode[STIX]{x1D6FD}) & = & \displaystyle c_{n,k,p,i}\binom{n-1}{k}\binom{k}{i}\nonumber\\ \displaystyle & & \displaystyle \times \int _{\unicode[STIX]{x2202}K}\int _{\operatorname{Gr}_{p+1}(\unicode[STIX]{x1D708}(x))}\mathbf{1}_{(\unicode[STIX]{x1D708}(x),E)\in \unicode[STIX]{x1D6FD}}\nonumber\\ \displaystyle & & \displaystyle \times \,D(S_{x}[n-k-1],\text{Id}[k-i],\unicode[STIX]{x1D6F1}_{E^{\bot }}[i])\,dE\,d{\mathcal{H}}^{n-1}(x).\nonumber\end{eqnarray}$$
(ii) In §5 we will prove an integral formula which generalizes equation (6) to arbitrary convex bodies. It involves generalized principal curvatures and the generalized shape operator.
The next theorem summarizes the main properties of flag area measures. Properties (i)–(v) are easy to prove. The proof of (vi) requires some deep results by James about the distribution of Jordan angles and by Aomoto about Selberg-type integrals.
Theorem 3 (Properties of flag area measures).
The flag area measures
$S_{k}^{(p),i}$
and
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}$
satisfy the following.
(i) For
$k\leqslant n-p-1$
, we have
$m=\min \{k,p\}$
and
$S_{k}^{(p),m}=(\unicode[STIX]{x1D714}_{n}/\unicode[STIX]{x1D714}_{n-p})S_{k}^{(p)}$
.(ii) For a fixed
$\unicode[STIX]{x1D6FD}$
,
$S_{k}^{(p),i}(\cdot ,\unicode[STIX]{x1D6FD})$
and
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}(\cdot ,\unicode[STIX]{x1D6FD})$
are translation-invariant valuations homogeneous of degree
$k$
and
$(n-1)/2$
, respectively.(iii) For all
$g\in \text{O}(n)$
,
$S_{k}^{(p),i}(gK,g\unicode[STIX]{x1D6FD})=S_{k}^{(p),i}(K,\unicode[STIX]{x1D6FD})$
.(iv) For every
$K\in {\mathcal{K}}(V)$
, the measures
$S_{k}^{(p),i}(K,\cdot )$
are positive.(v) For all
$g\in \text{O}(n)$
,
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}(gK,g\unicode[STIX]{x1D6FD})=\det g\cdot \tilde{S}_{(n-1)/2}^{((n-1)/2)}(K,\unicode[STIX]{x1D6FD})$
.(vi) Let
$\unicode[STIX]{x1D70B}:F(n,p+1)\rightarrow S^{n-1}$
be the projection onto the first factor. For each
$\unicode[STIX]{x1D6FD}\in {\mathcal{B}}(S^{n-1})$
, we have $$\begin{eqnarray}\displaystyle S_{k}^{(p),i}(K,\unicode[STIX]{x1D70B}^{-1}(\unicode[STIX]{x1D6FD}))=S_{k}(K,\unicode[STIX]{x1D6FD})\quad \text{and}\quad \tilde{S}_{(n-1)/2}^{((n-1)/2)}(K,\unicode[STIX]{x1D70B}^{-1}(\unicode[STIX]{x1D6FD}))=0. & & \displaystyle \nonumber\end{eqnarray}$$
To prove Theorem 1 we construct a sequence of
$\overline{\text{SO}(n)}$
-invariant differential forms on the product
$V\times F(n,p+1)$
. The flag area measures are obtained by integration over the pull-back of the normal cycle of
$K$
under the projection map
$V\times F(n,p+1)\rightarrow V\times S^{n-1},(x,v,E)\mapsto (x,v)$
. The flag area measures obtained in this way will be called smooth and the corresponding space will be denoted by
$\text{FlagArea}^{(p),\text{sm}}$
(cf. Definition 2.2).
Theorem 4. Let
$0\leqslant p,k\leqslant n-1$
. For
$(p,k)\neq ((n-1)/2,(n-1)/2)$
a basis of
$\operatorname{FlagArea}_{k}^{(p),\text{sm},\text{SO}(n)}$
is given by
$$\begin{eqnarray}S_{k}^{(p),i}\quad \text{with }0\leqslant i\leqslant m=\min \{k,n-k-1,p,n-p-1\}.\end{eqnarray}$$
For
$n$
odd and
$(p,k)=((n-1)/2,(n-1)/2)$
, a basis of
$\operatorname{FlagArea}_{(n-1)/2}^{((n-1)/2),\text{sm},\text{SO}(n)}$
is given by
$$\begin{eqnarray}\displaystyle S_{(n-1)/2}^{((n-1)/2),i}\quad \text{with }0\leqslant i\leqslant \frac{n-1}{2},\quad \text{and}\quad \tilde{S}_{(n-1)/2}^{((n-1)/2)}. & & \displaystyle \nonumber\end{eqnarray}$$
Our construction of flag area measures can be generalized in several ways. First, one could use other—partial or complete—flag manifolds instead of
$F(n,p+1)$
. Second, the
$\operatorname{SO}(n)$
-invariance may be relaxed or dropped. This will lead to a very general class of smooth flag area measures that are translation-invariant, but not necessarily
$\operatorname{SO}(n)$
-invariant. We think that these general flag area measures deserve further study.
Plan of the paper. In §2, we introduce the notion of smooth flag area measures. These are given by forms on
$V\times F(n,p+1)$
. We describe the forms on this space that induce the trivial flag area measure.
In §3, we determine the dimension of the space of
$\operatorname{SO}(n)$
-covariant smooth flag area measures that are homogeneous of degree
$k$
,
$0\leqslant k\leqslant n-1$
.
In §4, we recall the notion of Jordan angles between two subspaces of a euclidean vector space and compute the average value of elementary symmetric functions over the squared cosines of the Jordan angles between two subspaces.
The main results of this paper are proved in §5. We give an explicit construction of the differential forms inducing linearly independent flag area measures. We then write the value of our flag area measures on a compact convex body in terms of generalized principal curvatures and the generalized shape operator. Theorems 1 and 2 are easy consequences of this general formula.
2 Smooth flag area measures
In the following, given a smooth manifold
$M$
, we denote the space of differential forms on
$M$
by
$\unicode[STIX]{x1D6FA}^{\ast }(M)$
, the tangent (respectively cotangent) bundle by
$TM$
(respectively
$T^{\ast }M$
) and for
$p\in M$
, the tangent (respectively cotangent) space of
$M$
at
$p$
, by
$T_{p}M$
(respectively
$T_{p}^{\ast }M$
).
Let us recall the definition of the fiber integration, also called the push-forward of differential forms. We follow the sign convention in [Reference Álvarez Paiva and Fernandes1]. For another sign convention see, e.g., [Reference Berline, Getzler and Vergne3].
Definition 2.1. Let
$M$
and
$B$
be oriented manifolds. Let
$\unicode[STIX]{x1D6F1}:M\rightarrow B$
be a fiber bundle with a compact fiber of dimension
$r$
oriented by the local product orientation. Let
$d\geqslant 0$
. The fiber integration of a form
$\unicode[STIX]{x1D702}\in \unicode[STIX]{x1D6FA}^{d+r}(M)$
of degree
$d+r$
is the form
$\unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D702}\in \unicode[STIX]{x1D6FA}^{d}(B)$
of degree
$d$
defined by
$$\begin{eqnarray}\unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D702}|_{y}(w_{1},\ldots ,w_{d}):=\int _{\unicode[STIX]{x1D6F1}^{-1}(y)}\unicode[STIX]{x1D702}_{w_{1},\ldots ,w_{d}},\quad y\in B,\end{eqnarray}$$
where
$\unicode[STIX]{x1D702}_{w_{1},\ldots ,w_{d}}\in \unicode[STIX]{x1D6FA}^{r}(M)$
is defined as follows: for
$x\in M$
with
$\unicode[STIX]{x1D6F1}(x)=y$
and for
$W_{j}\in T_{x}M$
with
$d\unicode[STIX]{x1D6F1}(W_{j})=w_{j}$
,
$$\begin{eqnarray}\displaystyle (\unicode[STIX]{x1D702}_{w_{1},\ldots ,w_{d}})|_{x}:=\unicode[STIX]{x1D702}|_{x}(W_{1},\ldots ,W_{d},-). & & \displaystyle \nonumber\end{eqnarray}$$
$W_{j}$
are called lifts of
$w_{j}$
. The lifts
$W_{j}$
are not unique. However, the right-hand side of equation (7) is independent of their choices.
An equivalent definition is as follows. Let
$\unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FD}$
denote the pull-back of a form
$\unicode[STIX]{x1D6FD}$
under the map
$\unicode[STIX]{x1D6F1}$
. It has the same degree as
$\unicode[STIX]{x1D6FD}$
. The fiber integration,
$\unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D702}$
, is uniquely defined by the equation
$$\begin{eqnarray}\displaystyle \int _{B}\unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D702}\wedge \unicode[STIX]{x1D6FD}:=(-1)^{r(\dim B-d)}\int _{M}\unicode[STIX]{x1D702}\wedge \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FD}, & & \displaystyle \nonumber\end{eqnarray}$$
to be satisfied for every compactly supported differential form
$\unicode[STIX]{x1D6FD}\in \unicode[STIX]{x1D6FA}^{\ast }(B)$
.
The following projection formula can be easily derived from the definition of the push-forward (cf. [Reference Berline, Getzler and Vergne3, equation (1.16)]): for every
$\unicode[STIX]{x1D702}\in \unicode[STIX]{x1D6FA}^{\ast }(M)$
and
$\unicode[STIX]{x1D6FD}\in \unicode[STIX]{x1D6FA}^{\ast }(B)$
,
$$\begin{eqnarray}\unicode[STIX]{x1D6F1}_{\ast }(\unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FD}\wedge \unicode[STIX]{x1D702})=\unicode[STIX]{x1D6FD}\wedge \unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D702}.\end{eqnarray}$$
Let us return to our setting of an
$n$
-dimensional euclidean vector space
$V$
. Recall our notation for the sphere bundle over
$V$
,
$SV=V\times S^{n-1}$
. And let
$\unicode[STIX]{x1D70B}:SV\rightarrow V$
be the projection map. Then
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FC}|_{(x,v)}(w):=\langle v,d\unicode[STIX]{x1D70B}_{(x,v)}w\rangle ,\quad (x,v)\in SV,w\in T_{(x,v)}SV, & & \displaystyle \nonumber\end{eqnarray}$$
defines a contact form on
$SV$
.
To every compact convex subset
$K\subset V$
one can associate its normal cycle
$\operatorname{nc}(K)$
, which is an
$(n-1)$
-dimensional Legendrian cycle in
$SV$
. Consequently we can integrate any form
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{n-1}(SV)$
over
$\operatorname{nc}(K)$
. We obtain
$$\begin{eqnarray}\int _{\operatorname{nc}(K)}\unicode[STIX]{x1D6FC}\wedge \unicode[STIX]{x1D70F}=0\quad \text{and}\quad \int _{\operatorname{nc}(K)}d\unicode[STIX]{x1D70F}=0\quad \text{for every }\unicode[STIX]{x1D70F}\in \unicode[STIX]{x1D6FA}^{n-2}(SV).\end{eqnarray}$$
These two equations further imply that
$\int _{\operatorname{nc}(K)}d\unicode[STIX]{x1D6FC}\wedge \unicode[STIX]{x1D702}=0$
for
$\unicode[STIX]{x1D702}\in \unicode[STIX]{x1D6FA}^{n-3}(SV)$
. We refer to [Reference Zähle30] for the construction of the normal cycle of a convex body and its main properties.
Let
$0\leqslant p\leqslant n-1$
. Consider the partial flag manifold
$F(n,p+1)=\{(v,E):v\in S^{n-1},E\in \operatorname{Gr}_{p+1}(V),v\in E\}$
. Denote by
$\unicode[STIX]{x1D6F1}$
the projection map
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}:V\times F(n,p+1)\rightarrow V\times S^{n-1},(x,v,E)\mapsto (x,v). & & \displaystyle \nonumber\end{eqnarray}$$
Set
$r:=p(n-p-1)$
to be the dimension of the fiber of
$\unicode[STIX]{x1D6F1}$
and set
$s:=n-1+r$
.
Definition 2.2. A flag area measure
$\unicode[STIX]{x1D6F7}\in \operatorname{FlagArea}^{(p)}$
is called smooth if there exists a translation-invariant form
$\unicode[STIX]{x1D70F}\in \unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}$
such that for any
$f\in C^{\infty }(F(n,p+1))$
and for any
$K\in {\mathcal{K}}(V)$
,
$$\begin{eqnarray}\displaystyle \int _{F(n,p+1)}f\,d\unicode[STIX]{x1D6F7}(K,\cdot )=\int _{\operatorname{nc}(K)}\unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D70F}). & & \displaystyle \nonumber\end{eqnarray}$$
The space of smooth flag area measures is denoted by
$\operatorname{FlagArea}^{(p),\text{sm}}$
.
Note that since smooth functions are dense in the space of continuous functions, a flag area measure
$\unicode[STIX]{x1D6F7}$
is determined by the integrals
$\int _{F(n,p+1)}f\,d\unicode[STIX]{x1D6F7}(K,~\cdot )$
with smooth functions
$f$
.
To each translation-invariant
$s$
-form
$\unicode[STIX]{x1D70F}$
on
$V\times F(n,p+1)$
we associate a smooth flag area measure
$\unicode[STIX]{x1D6F7}$
as described in Definition 2.2. We now determine the kernel of the map
$\unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}\rightarrow \operatorname{FlagArea}^{(p),\text{sm}}:\unicode[STIX]{x1D70F}\mapsto \unicode[STIX]{x1D6F7}$
.
The space
$\unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}$
admits a filtration as follows. For
$(x,v,E)\in V\times F(n,p+1),0\leqslant j\leqslant s$
, we define
$$\begin{eqnarray}\displaystyle & & \displaystyle \mathfrak{F}_{x,v,E}^{s,j}:=\{\unicode[STIX]{x1D719}\in \wedge ^{s}T_{(x,v,E)}^{\ast }(V\times F(n,p+1)):\nonumber\\ \displaystyle & & \displaystyle \quad \qquad \qquad \forall f_{1},\ldots ,f_{j}\in T_{(x,v,E)}\unicode[STIX]{x1D6F1}^{-1}(\unicode[STIX]{x1D6F1}(x,v,E)),\unicode[STIX]{x1D719}(f_{1},\ldots ,f_{j},-)=0\},\nonumber\end{eqnarray}$$
and
$$\begin{eqnarray}\displaystyle & & \displaystyle \mathfrak{F}^{s,j}:=\{\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}:\nonumber\\ \displaystyle & & \displaystyle \quad \qquad \quad \forall (x,v,E)\in V\times F(n,p+1),\unicode[STIX]{x1D714}|_{(x,v,E)}\in \mathfrak{F}_{x,v,E}^{s,j}\}.\nonumber\end{eqnarray}$$
Then
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}=\mathfrak{F}^{s,r+1}\supset \mathfrak{F}^{s,r}\supset \cdots \supset \mathfrak{F}^{s,0}=\{0\}. & & \displaystyle \nonumber\end{eqnarray}$$
In the proof of the following theorem we will work with smooth convex bodies. Let us recall some general and well-known facts concerning these bodies. For a smooth compact convex body
$K$
with strictly positive curvature, the normal cycle of
$K$
is the graph of
$(\text{id},\unicode[STIX]{x1D708}):\unicode[STIX]{x2202}K\rightarrow SV$
, where
$\unicode[STIX]{x1D708}:\unicode[STIX]{x2202}K\rightarrow S^{n-1}$
is the Gauss map.
Fix
$x_{0}\in \unicode[STIX]{x2202}K$
. Set
$U:=T_{x_{0}}\unicode[STIX]{x2202}K$
and
$W:=U\oplus U$
. The shape operator, also called the Weingarten map,
$d\unicode[STIX]{x1D708}_{x_{0}}:U=T_{x_{0}}\unicode[STIX]{x2202}K\rightarrow T_{\unicode[STIX]{x1D708}(x_{0})}S^{n-1}\cong T_{x_{0}}\unicode[STIX]{x2202}K=U$
is self-adjoint (see, e.g., [Reference Spivak27]). Hence there is an orthonormal basis
$u_{1},\ldots ,u_{n-1}$
of
$U$
(the principal curvature directions) and
$\unicode[STIX]{x1D706}_{1},\ldots ,\unicode[STIX]{x1D706}_{n-1}>0$
(the principal curvatures) such that
$T_{(x_{0},\unicode[STIX]{x1D708}(x_{0}))}\operatorname{nc}(K)$
is spanned by the vectors
$w_{i}:=(u_{i},\unicode[STIX]{x1D706}_{i}u_{i})\in W$
.
Conversely, let
$(x_{0},v_{0})\in SV$
be given. Set
$U:=v_{0}^{\bot }$
and
$W:=U\oplus U$
endowed with the symplectic form
$\unicode[STIX]{x1D703}((u_{1},u_{2}),(\tilde{u} _{1},\tilde{u} _{2}))=\langle u_{1},\tilde{u} _{2}\rangle -\langle \tilde{u} _{1},u_{2}\rangle$
,
$u_{i},\tilde{u} _{i}\in U$
. Let
$u_{1},\ldots ,u_{n-1}$
be an orthogonal basis of
$U$
and
$\unicode[STIX]{x1D706}_{1},\ldots ,\unicode[STIX]{x1D706}_{n-1}>0$
. The subspace
$L$
of
$W$
spanned by
$(u_{i},\unicode[STIX]{x1D706}_{i}u_{i})$
,
$i=1,\ldots ,n-1$
, is a Lagrangian space. Lagrangian spaces of this type will be called strictly positive. Under these conditions, there exists a smooth compact convex body
$K$
with strictly positive curvature such that
$x_{0}\in \unicode[STIX]{x2202}K,\unicode[STIX]{x1D708}(x_{0})=v_{0}$
and
$T_{(x_{0},\unicode[STIX]{x1D708}(x_{0}))}\operatorname{nc}(K)=L$
. For instance,
$$\begin{eqnarray}\displaystyle K:=\bigg\{(x_{1},\ldots ,x_{n}):-1\leqslant x_{n}\leqslant -\frac{1}{2}\mathop{\sum }_{i=1}^{n-1}\unicode[STIX]{x1D706}_{i}x_{i}^{2}\bigg\} & & \displaystyle \nonumber\end{eqnarray}$$
with respect to the coordinate system given by
$(x_{0};u_{1},\ldots ,u_{n-1},v_{0})$
has this property.
Theorem 2.3. A form
$\unicode[STIX]{x1D70F}\in \unicode[STIX]{x1D6FA}^{s}(V\times F(n,p+1))^{\text{tr}}$
induces the trivial flag area measure if and only if
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D70F}\in \langle \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC},\mathfrak{F}^{s,r}\rangle . & & \displaystyle \nonumber\end{eqnarray}$$
Proof. Let us first show that the displayed space is a subset of the kernel. Let
$\unicode[STIX]{x1D70F}=\unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC}\wedge \unicode[STIX]{x1D70C}$
for some form
$\unicode[STIX]{x1D70C}$
. Using equation (8), we obtain
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D70F})=\unicode[STIX]{x1D6F1}_{\ast }(\unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC}\wedge f\unicode[STIX]{x1D70C})=\unicode[STIX]{x1D6FC}\wedge \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D70C}). & & \displaystyle \nonumber\end{eqnarray}$$
Hence the integral of this form over the normal cycle vanishes by equation (9). The same argument shows that the forms
$\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC}\wedge \unicode[STIX]{x1D70C}$
belong to the kernel. If
$\unicode[STIX]{x1D70F}\in \mathfrak{F}^{s,r}$
, then
$\unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D70F})=0$
by the definition of the push-forward.
Let us prove the other inclusion. Suppose that
$\unicode[STIX]{x1D70F}$
induces the trivial flag area measure. Fix
$(x_{0},v_{0},E_{0})\in V\times F(n,p+1)$
and set
$U:=v_{0}^{\bot }$
, which is an
$(n-1)$
-dimensional euclidean subspace of
$V$
. Take a sequence of functions
$h_{j}\in C^{\infty }(F(n,p+1))$
with
$h_{j}(v_{0},E_{0})=1,h_{j}\geqslant 0$
, whose supports shrink to
$\{(v_{0},E_{0})\}$
. Then for all
$K$
, we have
$$\begin{eqnarray}\int _{\operatorname{nc}(K)}\unicode[STIX]{x1D6F1}_{\ast }(h_{j}\unicode[STIX]{x1D70F})=0.\end{eqnarray}$$
Let us fix a strictly positive Lagrangian subspace
$L$
of
$W:=U\oplus U$
spanned by
$w_{i}:=(u_{i},\unicode[STIX]{x1D706}_{i}u_{i})$
as above. As noted above, there exists a smooth convex body
$K$
with strictly positive curvature having
$x_{0}\in \unicode[STIX]{x2202}K,\unicode[STIX]{x1D708}(x_{0})=v_{0}$
and
$T_{(x_{0},v_{0})}\operatorname{nc}(K)=L$
. Let
$$\begin{eqnarray}\displaystyle A:=d\unicode[STIX]{x1D6F1}|_{(x_{0},v_{0},E_{0})}:T_{(x_{0},v_{0},E_{0})}(V\times F(n,p+1))\rightarrow T_{(x_{0},v_{0})}(V\times S^{n-1}). & & \displaystyle \nonumber\end{eqnarray}$$
Let
$W_{1},\ldots ,W_{n-1}$
be lifts of
$w_{1},\ldots ,w_{n-1}$
, i.e.,
$W_{i}\in T_{(x_{0},v_{0},E_{0})}(V\times F(n,p+1))$
and
$A(W_{i})=w_{i}$
,
$i=1,\ldots ,n-1$
. Then equation (10), the definition of the push-forward and a continuity argument imply that
$$\begin{eqnarray}\unicode[STIX]{x1D70F}|_{(x_{0},v_{0},E_{0})}(W_{1},\ldots ,W_{n-1},f_{1},\ldots ,f_{r})=0\quad \text{for all }f_{1},\ldots ,f_{r}\in \ker A.\end{eqnarray}$$
We may decompose
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D70F}|_{(x_{0},v_{0},E_{0})}\equiv A^{\ast }\unicode[STIX]{x1D719}\wedge \unicode[STIX]{x1D705}\quad \text{mod}\;\mathfrak{F}_{x_{0},v_{0},E_{0}}^{s,r}, & & \displaystyle \nonumber\end{eqnarray}$$
where
$0\neq \unicode[STIX]{x1D705}\in \wedge ^{r}(\ker A)^{\ast }$
and
$\unicode[STIX]{x1D719}\in \wedge ^{n-1}T_{(x_{0},v_{0})}^{\ast }(V\times S^{n-1})$
. For instance, we may use a local trivialization of
$\unicode[STIX]{x1D6F1}$
, decompose
$\unicode[STIX]{x1D70F}|_{(x_{0},v_{0},E_{0})}$
according to bidegrees and pick the highest degree part. Then equation (11) implies that
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D719}(w_{1},\ldots ,w_{n-1})=0, & & \displaystyle \nonumber\end{eqnarray}$$
i.e.,
$\unicode[STIX]{x1D719}$
vanishes on all strictly positive Lagrangian subsets of
$W$
.
Now we apply a technical Lemma 2.4 below to the symplectic vector space
$(W=U\oplus U,\unicode[STIX]{x1D703}:=-d\unicode[STIX]{x1D6FC}_{(x_{0},v_{0})}|_{W})$
. Note that
$T_{(x_{0},v_{0})}(V\times S^{n-1})=\mathbb{R}v_{0}\oplus v_{0}^{\bot }\oplus v_{0}^{\bot }=\mathbb{R}v_{0}\oplus W$
and that the projection onto
$\mathbb{R}v_{0}$
equals
$\unicode[STIX]{x1D6FC}|_{(x_{0},v_{0})}$
. Hence
$\unicode[STIX]{x1D719}\in \langle \unicode[STIX]{x1D6FC}|_{(x_{0},v_{0})},d\unicode[STIX]{x1D6FC}|_{(x_{0},v_{0})}\rangle$
. It follows that
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D70F}|_{(x_{0},v_{0},E_{0})}\in \langle A^{\ast }\unicode[STIX]{x1D6FC}|_{(x_{0},v_{0},E_{0})},A^{\ast }d\unicode[STIX]{x1D6FC}|_{(x_{0},v_{0},E_{0})},\mathfrak{F}_{x_{0},v_{0},E_{0}}^{s,r}\rangle . & & \displaystyle \nonumber\end{eqnarray}$$
Since
$x_{0},v_{0},E_{0}$
were arbitrary, this implies that
$\unicode[STIX]{x1D70F}\in \langle \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC},\mathfrak{F}^{s,r}\rangle$
, finishing the proof.◻
We complete the proof by an algebraic lemma, which is adapted from [Reference Bernig and Bröcker4, Lemma 1.4]. Let
$U$
be a euclidean vector space of dimension
$m$
and let
$W:=U\oplus U$
be endowed with its natural symplectic form
$\unicode[STIX]{x1D703}$
,
$\unicode[STIX]{x1D703}((u_{1},u_{2}),(\tilde{u} _{1},\tilde{u} _{2}))=\langle u_{1},\tilde{u} _{2}\rangle -\langle \tilde{u} _{1},u_{2}\rangle$
,
$u_{i},\tilde{u} _{i}\in U$
.
A subspace of the form
$L:=\text{span}\{w_{1},\ldots ,w_{k}\}\subset W$
with
$w_{i}:=(u_{i},\unicode[STIX]{x1D706}_{i}u_{i})$
,
$i=1,\ldots ,k$
, where
$u_{1},\ldots ,u_{k}$
are non-zero and pairwise orthogonal in
$U$
and
$\unicode[STIX]{x1D706}_{i}>0$
is called a strictly positive isotropic subspace.
Lemma 2.4. Let
$0\leqslant k\leqslant 2m$
and suppose that
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6EC}^{k}W^{\ast }$
vanishes on all strictly positive isotropic subspaces of dimension
$k$
. Then
$\unicode[STIX]{x1D714}$
is a multiple of the symplectic form
$\unicode[STIX]{x1D703}$
.
Proof. We use induction on the dimension
$m$
of
$U$
. The base case
$m=1$
is trivial.
Suppose
$k>m$
. Then basic symplectic algebra implies that every form
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6EC}^{k}W^{\ast }$
is a multiple of
$\unicode[STIX]{x1D703}$
and there is nothing to prove.
Assume
$k\leqslant m$
. Let
$u_{1},\ldots ,u_{m}$
be an orthonormal basis of
$U$
. Define a basis of
$W$
by
$e_{i}:=(u_{i},0),f_{i}:=(0,u_{i})$
. Let
$e_{1}^{\ast },\ldots ,e_{m}^{\ast },f_{1}^{\ast },\ldots ,f_{m}^{\ast }\in W^{\ast }$
be the dual basis. Then the natural symplectic form on
$W$
is
$\unicode[STIX]{x1D703}=\sum _{i=1}^{m}e_{i}^{\ast }\wedge f_{i}^{\ast }$
. Let
$U_{0}$
be the euclidean subspace spanned by
$u_{1},\ldots ,u_{m-1}$
and
$W_{0}:=U_{0}\oplus U_{0}$
with its natural symplectic form
$\unicode[STIX]{x1D703}_{0}$
. Then
$\unicode[STIX]{x1D703}=\unicode[STIX]{x1D703}_{0}+e_{m}^{\ast }\wedge f_{m}^{\ast }$
. We decompose
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D714}=\unicode[STIX]{x1D714}_{0}+\unicode[STIX]{x1D714}_{1}\wedge e_{m}^{\ast }+\unicode[STIX]{x1D714}_{2}\wedge f_{m}^{\ast }+\unicode[STIX]{x1D714}_{3}\wedge e_{m}^{\ast }\wedge f_{m}^{\ast }, & & \displaystyle \nonumber\end{eqnarray}$$
with
$\unicode[STIX]{x1D714}_{0}\in \unicode[STIX]{x1D6EC}^{k}W_{0}^{\ast },\unicode[STIX]{x1D714}_{1},\unicode[STIX]{x1D714}_{2}\in \unicode[STIX]{x1D6EC}^{k-1}W_{0}^{\ast },\unicode[STIX]{x1D714}_{3}\in \unicode[STIX]{x1D6EC}^{k-2}W_{0}^{\ast }$
. Let
$\unicode[STIX]{x1D706}_{1},\ldots ,\unicode[STIX]{x1D706}_{k-1},\unicode[STIX]{x1D706}_{m}>0$
and let
$v_{1},\ldots ,v_{k-1}\in U_{0}\setminus \{0\}$
be pairwise orthogonal. Then
$v_{1},\ldots ,v_{k-1},u_{m}$
are pairwise orthogonal in
$U$
and the vectors
$w_{i}:=(v_{i},\unicode[STIX]{x1D706}_{i}v_{i})$
,
$i=1,\ldots ,k-1$
and
$w_{m}:=(u_{m},\unicode[STIX]{x1D706}_{m}u_{m})$
span a strictly positive isotropic space. By assumption,
$$\begin{eqnarray}\displaystyle 0=\unicode[STIX]{x1D714}(w_{1},\ldots ,w_{k-1},w_{m})=\unicode[STIX]{x1D714}_{1}(w_{1},\ldots ,w_{k-1})+\unicode[STIX]{x1D706}_{m}\unicode[STIX]{x1D714}_{2}(w_{1},\ldots ,w_{k-1}). & & \displaystyle \nonumber\end{eqnarray}$$
With
$\unicode[STIX]{x1D706}_{m}>0$
being arbitrary, it follows that
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D714}_{1}(w_{1},\ldots ,w_{k-1})=0,\qquad \unicode[STIX]{x1D714}_{2}(w_{1},\ldots ,w_{k-1})=0. & & \displaystyle \nonumber\end{eqnarray}$$
By the induction hypothesis,
$\unicode[STIX]{x1D714}_{1},\unicode[STIX]{x1D714}_{2}$
are multiples of
$\unicode[STIX]{x1D703}_{0}$
. Since
$\unicode[STIX]{x1D703}_{0}\wedge e_{m}^{\ast }=\unicode[STIX]{x1D703}\wedge e_{m}^{\ast }$
,
$\unicode[STIX]{x1D703}_{0}\wedge f_{m}^{\ast }=\unicode[STIX]{x1D703}\wedge f_{m}^{\ast }$
, we obtain that
$\unicode[STIX]{x1D714}_{1}\wedge e_{m}^{\ast }+\unicode[STIX]{x1D714}_{2}\wedge f_{m}^{\ast }$
is a multiple of
$\unicode[STIX]{x1D703}$
.
Let
$v_{1},\ldots ,v_{k}\in U_{0}\setminus \{0\}$
be pairwise orthogonal unit vectors and
$w_{i}:=(v_{i},\unicode[STIX]{x1D706}_{i}v_{i})$
with
$\unicode[STIX]{x1D706}_{i}>0$
. Then
$0=\unicode[STIX]{x1D714}_{0}(w_{1},\ldots ,w_{k})$
and the induction hypothesis implies that
$\unicode[STIX]{x1D714}_{0}=\unicode[STIX]{x1D714}_{00}\wedge \unicode[STIX]{x1D703}_{0}$
with
$\unicode[STIX]{x1D714}_{00}\in \unicode[STIX]{x1D6EC}^{k-2}W_{0}^{\ast }$
. Let
$\unicode[STIX]{x1D706}>0$
and
$\tilde{w}_{k-1}:=(v_{k-1}+u_{m},\unicode[STIX]{x1D706}(v_{k-1}+u_{m}))\in W,\tilde{w}_{k}:=(v_{k-1}-u_{m},v_{k-1}-u_{m})\in W$
. Then, since
$\unicode[STIX]{x1D714}_{1}\wedge e_{m}^{\ast }+\unicode[STIX]{x1D714}_{2}\wedge f_{m}^{\ast }$
is a multiple of
$\unicode[STIX]{x1D703}$
,
$$\begin{eqnarray}\displaystyle 0 & = & \displaystyle (\unicode[STIX]{x1D714}_{0}+\unicode[STIX]{x1D714}_{3}\wedge e_{m}^{\ast }\wedge f_{m}^{\ast })(w_{1},\ldots ,w_{k-2},\tilde{w}_{k-1},\tilde{w}_{k})\nonumber\\ \displaystyle & = & \displaystyle (\unicode[STIX]{x1D706}-1)(\unicode[STIX]{x1D714}_{3}-\unicode[STIX]{x1D714}_{00})(w_{1},\ldots ,w_{k-2}).\nonumber\end{eqnarray}$$
By the induction hypothesis
$\unicode[STIX]{x1D714}_{3}-\unicode[STIX]{x1D714}_{00}=\unicode[STIX]{x1D703}_{0}\wedge \unicode[STIX]{x1D719}_{0}$
with
$\unicode[STIX]{x1D719}_{0}\in \unicode[STIX]{x1D6EC}^{k-4}W_{0}^{\ast }$
. Hence
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D714}_{0}+\unicode[STIX]{x1D714}_{3}\wedge e_{m}^{\ast }\wedge f_{m}^{\ast } & = & \displaystyle \unicode[STIX]{x1D714}_{00}\wedge \unicode[STIX]{x1D703}_{0}+(\unicode[STIX]{x1D714}_{00}+\unicode[STIX]{x1D703}_{0}\wedge \unicode[STIX]{x1D719}_{0})\wedge e_{m}^{\ast }\wedge f_{m}^{\ast }\nonumber\\ \displaystyle & = & \displaystyle (\unicode[STIX]{x1D714}_{00}+\unicode[STIX]{x1D719}_{0}\wedge e_{m}^{\ast }\wedge f_{m}^{\ast })\wedge \unicode[STIX]{x1D703}.\nonumber\end{eqnarray}$$
This shows that
$\unicode[STIX]{x1D714}_{0}+\unicode[STIX]{x1D714}_{3}\wedge e_{m}^{\ast }\wedge f_{m}^{\ast }$
is divisible by
$\unicode[STIX]{x1D703}$
.◻
3 Dimension computation
In this section we determine the dimension of the space of smooth,
$k$
-homogeneous, translation- and
$\operatorname{SO}(n)$
-covariant valuations with values in the space of signed measures on the partial flag manifold
$F(n,p+1)$
, denoted by
$\operatorname{FlagArea}_{k}^{(p),\text{sm},\text{SO}(n)}$
.
Theorem 3.1. Let
$0\leqslant p,k\leqslant n-1$
. Then
$$\begin{eqnarray}\displaystyle \dim \operatorname{FlagArea}_{k}^{(p),\text{sm},\text{SO}(n)} & = & \displaystyle \min \{k,n-k-1,p,n-p-1\}+1\nonumber\\ \displaystyle & & \displaystyle +\,\left\{\begin{array}{@{}ll@{}}\displaystyle 1\quad & \text{if }p=k={\displaystyle \frac{n-1}{2}},\\ 0\quad & \text{otherwise}.\end{array}\right.\nonumber\end{eqnarray}$$
Proof. If
$p=0$
or
$p=n-1$
, then
$F(n,p+1)\cong S^{n-1}$
and the result follows from Schneider’s characterization of area measures [Reference Schneider21]. For the rest of the proof, we assume that
$1\leqslant p\leqslant n-2$
.
Since
$F(n,p+1)\cong F(n,q+1)$
with
$q:=n-p-1$
, there is an isomorphism
$$\begin{eqnarray}\displaystyle \operatorname{FlagArea}_{k}^{(p),\text{sm},\overline{\operatorname{SO}(n)}}\cong \operatorname{FlagArea}_{k}^{(q),\text{sm},\overline{\operatorname{SO}(n)}}. & & \displaystyle \nonumber\end{eqnarray}$$
We may thus assume that
$1\leqslant p\leqslant q$
.
According to Definition 2.2, a smooth flag area measure is obtained by integration over the normal cycle of the push-forward of a form
$\unicode[STIX]{x1D70F}$
on
$V\times F(n,p+1)$
. Since all our forms will be obtained as a push-forward of appropriate forms on
$\overline{\operatorname{SO}(n)}=V\rtimes \operatorname{SO}(n)$
, let us describe the different spaces on which we construct the forms and the maps between them.
Let
$G:=\operatorname{SO}(n),\overline{G}:=\overline{\operatorname{SO}(n)}$
, and
$H:=S(\text{O}(p)\times \text{O}(q))$
. Then
$F(n,p+1)\cong G/H$
and
$V\times F(n,p+1)\cong \overline{G}/H$
. Fix an orthonormal basis
$e_{1},\ldots ,e_{n}$
of
$V$
. For an element
$g\in G$
, we let
$g_{i}=ge_{i}$
,
$1\leqslant i\leqslant n$
, denote the column vectors of
$g$
. Let
$$\begin{eqnarray}\displaystyle \tilde{\unicode[STIX]{x1D6F1}}:V\rtimes \operatorname{SO}(n)\rightarrow V\times F(n,p+1) & & \displaystyle \nonumber\end{eqnarray}$$
be the projection map, taking
$(x,g)\mapsto (x,g_{1},\text{span}\{g_{1},\ldots ,g_{p+1}\})$
. The fiber of this map is diffeomorph to
$H$
and has dimension
$\tilde{r}:=(p(p-1)+q(q-1))/2$
.
Let
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}:V\times F(n,p+1)\rightarrow V\times S^{n-1} & & \displaystyle \nonumber\end{eqnarray}$$
be the projection map, taking
$(x,v,E)\mapsto (x,v)$
. The fiber of this map is diffeomorph to
$\operatorname{Gr}_{p}(\mathbb{R}^{n-1})$
and has dimension
$r:=p(n-1-p)=pq$
.
Finally, let
$$\begin{eqnarray}\displaystyle \hat{\unicode[STIX]{x1D6F1}}:V\rtimes \operatorname{SO}(n)\rightarrow V\times S^{n-1} & & \displaystyle \nonumber\end{eqnarray}$$
be the projection map, taking
$(x,g)\mapsto (x,g_{1})$
. The fiber of this map is diffeomorph to
$\operatorname{SO}(n-1)$
and has dimension
$\hat{r}:=(n-1)(n-2)/2$
. Thus we obtain the following commutative diagram:
A basis of the algebra of left-invariant forms on
$\overline{G}$
is given by the
$1$
-forms
$\unicode[STIX]{x1D70E}_{i},1\leqslant i\leqslant n$
and
$\unicode[STIX]{x1D714}_{ij}=-\unicode[STIX]{x1D714}_{ji},1\leqslant i<j\leqslant n$
, see §5.1 for their definition. These
$1$
-forms span the dual space
$\overline{\mathfrak{g}}^{\ast }$
. Denote by
$X_{i},1\leqslant i\leqslant n,X_{ij},1\leqslant i<j\leqslant n$
the corresponding basis of the Lie algebra
$\overline{\mathfrak{g}}$
.
Denote the linear span of
$X_{1}$
by
$V_{0}$
; the linear span of
$X_{i}$
with
$2\leqslant i\leqslant p+1$
by
$V_{\unicode[STIX]{x1D70E}}$
and the linear span of
$X_{i}$
with
$p+2\leqslant i\leqslant n$
by
$W_{\unicode[STIX]{x1D70E}}$
. The linear span of
$X_{1j}$
with
$2\leqslant j\leqslant p+1$
will be denoted by
$V_{\unicode[STIX]{x1D714}}$
; the linear span of
$X_{1j}$
with
$p+2\leqslant j\leqslant n$
by
$W_{\unicode[STIX]{x1D714}}$
; the linear span of
$X_{ij}$
with
$2\leqslant i<j\leqslant p+1$
by
$U_{p}$
; the linear span of
$X_{ij}$
with
$2\leqslant i<p+2\leqslant j\leqslant n$
by
$U_{pq}$
and the linear span of
$X_{ij}$
with
$p+2\leqslant i<j\leqslant n$
by
$U_{q}$
. Note that
$\ker d\tilde{\unicode[STIX]{x1D6F1}}=U_{p}\oplus U_{q}$
,
$\ker d\unicode[STIX]{x1D6F1}=U_{pq}$
and
$\ker d\hat{\unicode[STIX]{x1D6F1}}=U_{p}\oplus U_{q}\oplus U_{pq}$
. Schematically, the Lie algebra
$\overline{\mathfrak{g}}$
looks as follows:
$$\begin{eqnarray}\displaystyle \overline{\mathfrak{g}}=\left(\begin{array}{@{}cccc@{}}0 & & & \\ V_{0} & 0 & & \\ V_{\unicode[STIX]{x1D70E}} & V_{\unicode[STIX]{x1D714}} & U_{p} & \\ W_{\unicode[STIX]{x1D70E}} & W_{\unicode[STIX]{x1D714}} & U_{pq} & U_{q}\end{array}\right). & & \displaystyle \nonumber\end{eqnarray}$$
A smooth flag area measure is generated from a translation-invariant form
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{n-1+r}(\overline{G}/H)^{\text{tr}}$
. A
$k$
-homogeneous smooth flag area measure corresponds to a form of bidegree
$(k,n-1+r-k)$
,
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G}/H)^{\text{tr}}$
, where the bidegree is taken with respect to the product structure of
$\overline{G}/H=V\times G/H$
. For a
$G$
-covariant flag area measure we may assume by averaging with respect to the Haar measure on the compact group
$G$
that the form
$\unicode[STIX]{x1D714}$
is
$G$
-invariant.
Thus let
$\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G}/H)^{\overline{G}}$
and set
$\tilde{\unicode[STIX]{x1D714}}:=\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D714}\in \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G})$
. The form
$\tilde{\unicode[STIX]{x1D714}}$
is left
$\overline{G}$
-invariant and right
$H$
-invariant. It vanishes on each tangent vector to the fiber of
$\tilde{\unicode[STIX]{x1D6F1}}$
, i.e., on
$U_{p}\oplus U_{q}$
. Conversely, every left
$\overline{G}$
-invariant and right
$H$
-invariant form on
$\overline{G}$
, which vanishes on
$U_{p}\oplus U_{q}$
, induces a
$\overline{G}$
-invariant form on
$\overline{G}/H$
.
Since
$(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\circ \unicode[STIX]{x1D6F1}^{\ast })\unicode[STIX]{x1D6FC}=\hat{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D6FC}=\unicode[STIX]{x1D70E}_{1}$
, the quotient of the space of
$\overline{G}$
-invariant forms on
$\overline{G}/H$
by multiples of
$\unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC}$
is identified with the space
$$\begin{eqnarray}\displaystyle & & \displaystyle (\wedge ^{k}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-1+r-k}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast }\oplus U_{pq}^{\ast }))^{H}\nonumber\\ \displaystyle & & \displaystyle \quad =\bigoplus _{i=r-k}^{r}[\wedge ^{k}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-1+r-k-i}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })\otimes \wedge ^{i}U_{pq}^{\ast }]^{H}.\nonumber\end{eqnarray}$$
If
$\tilde{\unicode[STIX]{x1D714}}$
belongs to the sum of terms with
$i<pq=r$
, then
$\unicode[STIX]{x1D714}\in \mathfrak{F}^{n-1+r,r}$
and hence
$\tilde{\unicode[STIX]{x1D714}}$
induces the trivial flag area measure by Theorem 2.3. Moreover,
$\wedge ^{pq}U_{pq}^{\ast }$
is the trivial one-dimensional representation of
$H$
. We obtain
$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G}/H)^{\overline{G}}/\langle \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC},\mathfrak{F}^{n-1+r,r}\rangle \nonumber\\ \displaystyle & & \displaystyle \quad \cong [\wedge ^{k}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-1-k}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })]^{H}.\nonumber\end{eqnarray}$$
We also have to quotient out the form
$\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC}$
. Note that
$(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\circ \unicode[STIX]{x1D6F1}^{\ast })\,d\unicode[STIX]{x1D6FC}=\hat{\unicode[STIX]{x1D6F1}}^{\ast }d\unicode[STIX]{x1D6FC}=-\sum _{i=2}^{n}\unicode[STIX]{x1D70E}_{i}\wedge \unicode[STIX]{x1D714}_{1i}$
. This is a symplectic form on the
$2(n-1)$
-dimensional vector space
$(V_{\unicode[STIX]{x1D70E}}\oplus W_{\unicode[STIX]{x1D70E}})\oplus (V_{\unicode[STIX]{x1D714}}\oplus W_{\unicode[STIX]{x1D714}})$
. By basic symplectic linear algebra (cf. [Reference Huybrechts15, Proposition 1.2.30]), multiplication by this form induces an injective map
$$\begin{eqnarray}\displaystyle & & \displaystyle L:\underbrace{[\wedge ^{k-1}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-2-k}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })]^{H}}_{=:B_{k,p,q}}\nonumber\\ \displaystyle & & \displaystyle \quad \rightarrow \underbrace{[\wedge ^{k}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-1-k}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })]^{H}}_{=:A_{k,p,q}}.\nonumber\end{eqnarray}$$
Therefore we may view
$B_{k,p,q}$
as a subspace of
$A_{k,p,q}$
and obtain
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G}/H)^{\overline{G}}/\langle \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC},\mathfrak{F}^{n-1+r,r}\rangle \cong A_{k,p,q}/B_{k,p,q}. & & \displaystyle \nonumber\end{eqnarray}$$
By Theorem 2.3, it follows that
$$\begin{eqnarray}\displaystyle \dim \operatorname{FlagArea}_{k}^{(p),\text{sm},\text{SO}(n)} & = & \displaystyle \dim \unicode[STIX]{x1D6FA}^{k,n-1+r-k}(\overline{G}/H)^{\overline{G}}/\langle \unicode[STIX]{x1D6F1}^{\ast }\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6F1}^{\ast }d\unicode[STIX]{x1D6FC},\mathfrak{F}^{n-1+r,r}\rangle \nonumber\\ \displaystyle & = & \displaystyle a_{k,p,q}-b_{k,p,q},\nonumber\end{eqnarray}$$
where
$$\begin{eqnarray}\displaystyle a_{k,p,q} & := & \displaystyle \dim A_{k,p,q}\nonumber\\ \displaystyle & = & \displaystyle \dim [\wedge ^{k}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-k-1}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })]^{H}\nonumber\\ \displaystyle & = & \displaystyle \mathop{\sum }_{i,j}\dim [\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{k-i}W_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast }\otimes \wedge ^{n-k-1-j}W_{\unicode[STIX]{x1D714}}^{\ast }]^{H}\nonumber\end{eqnarray}$$
with
$\max \{0,k-q\}\leqslant i\leqslant \min \{k,p\},\max \{0,p-k\}\leqslant j\leqslant \min \{n-k-1,p\}$
and
$$\begin{eqnarray}\displaystyle b_{k,p,q} & := & \displaystyle \dim B_{k,p,q}\nonumber\\ \displaystyle & = & \displaystyle \dim [\wedge ^{k-1}(V_{\unicode[STIX]{x1D70E}}^{\ast }\oplus W_{\unicode[STIX]{x1D70E}}^{\ast })\otimes \wedge ^{n-k-2}(V_{\unicode[STIX]{x1D714}}^{\ast }\oplus W_{\unicode[STIX]{x1D714}}^{\ast })]^{H}\nonumber\\ \displaystyle & = & \displaystyle \mathop{\sum }_{i,j}\dim [\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{k-1-i}W_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast }\otimes \wedge ^{n-k-2-j}W_{\unicode[STIX]{x1D714}}^{\ast }]^{H}\nonumber\end{eqnarray}$$
with
$\max \{0,k-1-q\}\leqslant i\leqslant \min \{k-1,p\},\max \{0,p-k-1\}\leqslant j\leqslant \min \{n-k-2,p\}$
.
Let us first consider the action of the subgroup
$H^{\prime }:=\text{SO}(p)\times \text{SO}(q)\subset H$
. We have
$$\begin{eqnarray}\displaystyle & & \displaystyle [\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{k-i}W_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast }\otimes \wedge ^{n-k-1-j}W_{\unicode[STIX]{x1D714}}^{\ast }]^{H^{\prime }}\nonumber\\ \displaystyle & & \displaystyle \quad =[\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast }]^{\operatorname{SO}(p)}\otimes [\wedge ^{k-i}W_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{n-k-1-j}W_{\unicode[STIX]{x1D714}}^{\ast }]^{\operatorname{SO}(q)}.\end{eqnarray}$$
We decompose
$$\begin{eqnarray}\displaystyle (\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast })^{\operatorname{SO}(p)}=\bigoplus _{\unicode[STIX]{x1D716}=0,1}(\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast })^{\operatorname{SO}(p),\unicode[STIX]{x1D716}}, & & \displaystyle \nonumber\end{eqnarray}$$
where
$(\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast })^{\operatorname{SO}(p),\unicode[STIX]{x1D716}}$
is the space of
$\operatorname{SO}(p)$
-invariant elements such that
$g\in \text{O}(p)$
acts by
$\text{det}(g)^{\unicode[STIX]{x1D716}}$
. An easy exercise in representation theory (see [Reference Bernig and Faifman5, Lemma 5.1] and [Reference Fu9, Lemma 0.4.3]) is to show that
$$\begin{eqnarray}\displaystyle & \displaystyle (\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast })^{\operatorname{SO}(p),+}=\unicode[STIX]{x1D6FF}_{i}^{j}, & \displaystyle \nonumber\\ \displaystyle & \displaystyle (\wedge ^{i}V_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{j}V_{\unicode[STIX]{x1D714}}^{\ast })^{\operatorname{SO}(p),-}=\unicode[STIX]{x1D6FF}_{i+j}^{p}. & \displaystyle \nonumber\end{eqnarray}$$
More precisely, for
$i=j$
, the space
$(\wedge ^{i}V^{\ast }\otimes \wedge ^{i}V^{\ast })^{\operatorname{SO}(p),+}$
is spanned by the
$i$
th power of the symplectic form on
$V$
and for
$i+j=p$
,
$(\wedge ^{i}V^{\ast }\otimes \wedge ^{p-i}V^{\ast })^{\operatorname{SO}(p),-}$
is spanned by the determinant.
The space
$[\wedge ^{k-i}W_{\unicode[STIX]{x1D70E}}^{\ast }\otimes \wedge ^{n-k-1-j}W_{\unicode[STIX]{x1D714}}^{\ast }]^{\operatorname{SO}(q)}$
can be decomposed in an analogous way. A
$H^{\prime }$
-invariant element is invariant under the larger group
$H$
if and only if the factors in the decomposition (12) have the same
$\unicode[STIX]{x1D716}$
. Therefore
$$\begin{eqnarray}\displaystyle a_{k,p,q}=\mathop{\sum }_{i=\max \{0,k-q\}}^{\min \{k,p\}}\mathop{\sum }_{j=\max \{0,p-k\}}^{\min \{n-k-1,p\}}(\unicode[STIX]{x1D6FF}_{i}^{j}\unicode[STIX]{x1D6FF}_{k-i}^{n-k-1-j}+\unicode[STIX]{x1D6FF}_{i+j}^{p}\underbrace{\unicode[STIX]{x1D6FF}_{k-i+n-k-1-j}^{q}}_{=\unicode[STIX]{x1D6FF}_{i+j}^{p}}). & & \displaystyle \nonumber\end{eqnarray}$$
The first summand is non-zero only if
$i=j$
and
$k-i=n-k-1-j$
, i.e., if
$i=j$
and
$k=n-k-1$
. In this case, both sums range from
$0$
to
$p$
. By treating similarly the second summand, we have
$$\begin{eqnarray}\displaystyle a_{k,p,q}=\unicode[STIX]{x1D6FF}_{k}^{n-k-1}(p+1)+\min \{k,n-1-k,p,q\}+1. & & \displaystyle \nonumber\end{eqnarray}$$
Similar arguments yield
$$\begin{eqnarray}\displaystyle b_{k,p,q} & = & \displaystyle \mathop{\sum }_{i=\max \{0,k-1-q\}}^{\min \{k-1,p\}}\mathop{\sum }_{j=\max \{0,p-k-1\}}^{\min \{n-k-2,p\}}\unicode[STIX]{x1D6FF}_{i}^{j}\unicode[STIX]{x1D6FF}_{k-1-i}^{n-k-2-j}\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6FF}_{k}^{n-k-1}\min \bigg\{\frac{n-1}{2},p+1\bigg\}.\nonumber\end{eqnarray}$$
Overall we obtain that
$$\begin{eqnarray}\displaystyle & & \displaystyle \dim \operatorname{FlagArea}_{k}^{(p),\text{sm},\text{SO}(n)}=a_{k,p,q}-b_{k,p,q}\nonumber\\ \displaystyle & & \displaystyle \quad =\min \{k,n-k-1,p,q\}+1+\unicode[STIX]{x1D6FF}_{k}^{n-k-1}\bigg[(p+1)-\min \bigg\{\frac{n-1}{2},p+1\bigg\}\bigg]\nonumber\\ \displaystyle & & \displaystyle \quad =\min \{k,n-k-1,p,q\}+1+\left\{\begin{array}{@{}ll@{}}1\quad & p=k={\displaystyle \frac{n-1}{2}},\\ 0\quad & \text{otherwise},\end{array}\right.\nonumber\end{eqnarray}$$
as claimed. ◻
4 Jordan angles
In this section we introduce the notion of angles between subspaces and discuss their properties. Let
$W$
be a euclidean vector space of dimension
$n^{\prime }$
and let
$E$
and
$F$
be subspaces of
$W$
of dimensions
$p$
and
$k$
, respectively. We start by recalling the notion of the cosine between subspaces. Let us denote by
$B_{E,F}$
the orthogonal projection from
$E$
onto
$F$
. In the case
$p\leqslant k$
, let
$S$
be a domain in
$E$
of volume
$1$
. The absolute value of the cosine between the subspaces
$E,F$
is defined as the
$p$
-dimensional volume of the orthogonal projection of
$S$
onto
$F$
:
$$\begin{eqnarray}|\text{cos}(E,F)|:=\operatorname{vol}_{p}(B_{E,F}S).\end{eqnarray}$$
In the case
$p>k$
, we define the cosine between
$E$
and
$F$
analogously, by considering the orthogonal projection from
$F$
onto
$E$
.
The cosine is invariant under the diagonal action of
$\text{O}(n^{\prime })$
on the product of two Grassmannians, but it is not enough to separate the orbits. For that we use a more general notion of Jordan angles between subspaces, sometimes also called critical angles or principal angles, see, e.g., [Reference Gallagher and Proulx10, Reference Jordan17]. To define Jordan angles associated with a pair of subspaces
$(E,F)$
, we need to start with appropriate bases in our subspaces. Those bases are given by the following simple lemma.
Lemma 4.1 [Reference Gallagher and Proulx10, Lemma 1].
Let
$E$
and
$F$
be subspaces of an
$n^{\prime }$
-dimensional euclidean space
$W$
of dimensions
$p$
and
$k$
, respectively. Set
$m:=\min \{k,n^{\prime }-k,p,n^{\prime }-p\}$
. Then there is an orthonormal basis
$e_{1},\ldots ,e_{p}$
of
$E$
, an orthonormal basis
$f_{1},\ldots ,f_{k}$
of
$F$
and
$\unicode[STIX]{x1D70B}/2\geqslant \unicode[STIX]{x1D703}_{1}\geqslant \cdots \geqslant \unicode[STIX]{x1D703}_{m}\geqslant 0$
such that:
(i)
$\langle e_{i},f_{j}\rangle =0$
for
$i\neq j$
;(ii)
$\langle e_{i},f_{i}\rangle =\cos \unicode[STIX]{x1D703}_{i}$
for
$1\leqslant i\leqslant m$
;(iii)
$\langle e_{i},f_{i}\rangle =1$
for
$m+1\leqslant i\leqslant \min \{k,p\}$
.
The unique angles
$\unicode[STIX]{x1D703}_{1},\ldots ,\unicode[STIX]{x1D703}_{m}$
are called Jordan angles between the subspaces
$E$
and
$F$
.
Two pairs
$(E,F),(E^{\prime },F^{\prime })\in \operatorname{Gr}_{p}(W)\times \operatorname{Gr}_{k}(W)$
belong to the same
$\text{O}(n^{\prime })$
-orbit if and only if the Jordan angles between
$E$
and
$F$
and between
$E^{\prime }$
and
$F^{\prime }$
are the same.
In the following lemma we construct convenient bases for the subspaces
$E,F,E^{\bot }$
and
$F^{\bot }$
simultaneously.
Lemma 4.2. With
$E,F,\{e_{i}\},\{f_{j}\}$
as in the previous lemma, there are orthonormal bases
$h_{1},\ldots ,h_{n^{\prime }-k}$
of
$F^{\bot }$
and
$g_{1},\ldots ,g_{n^{\prime }-p}$
of
$E^{\bot }$
such that:
(i)
$\langle e_{i},h_{i}\rangle =\langle g_{i},f_{i}\rangle =\sin \unicode[STIX]{x1D703}_{i},\langle g_{i},h_{i}\rangle =-\text{cos}\,\unicode[STIX]{x1D703}_{i}$
for
$1\leqslant i\leqslant m$
;(ii)
$g_{i}=h_{i},m+1\leqslant i\leqslant \min \{n^{\prime }-k,n^{\prime }-p\}$
;(iii) if
$p\geqslant k$
, then
$h_{n^{\prime }-p+i}=e_{k+i},1\leqslant i\leqslant p-k$
;(iv) if
$k\geqslant p$
, then
$g_{n^{\prime }-k+i}=f_{p+i},1\leqslant i\leqslant k-p$
;(v) all other scalar products are zero.
In particular, the Jordan angles between
$E$
and
$F^{\bot }$
are given by
$\unicode[STIX]{x1D703}_{j}^{\prime }=\unicode[STIX]{x1D70B}/2-\unicode[STIX]{x1D703}_{m-j+1},j=1,\ldots ,m$
.
Proof. Let
$0\leqslant l\leqslant m$
be the largest index with
$\unicode[STIX]{x1D703}_{l}\neq 0$
. Then a basis of
$E\cup F$
is given by
$$\begin{eqnarray}\displaystyle \{e_{i}:i=1,\ldots ,l\}\cup \{f_{i}:i=1,\ldots ,l\}\cup \{e_{i}=f_{i}:i=l+1,\ldots ,\min \{k,p\}\} & & \displaystyle \nonumber\end{eqnarray}$$
together with
$$\begin{eqnarray}\displaystyle \{e_{i}:k+1\leqslant i\leqslant p\}\quad \text{if }p>k\quad \text{or}\quad \{f_{i}:p+1\leqslant i\leqslant k\}\quad \text{if }p<k. & & \displaystyle \nonumber\end{eqnarray}$$
Hence
$\dim (E\cup F)=\max \{k,p\}+l$
.
For
$1\leqslant i\leqslant l$
, let
$h_{i}$
be the unit vector in the oriented two-dimensional plane spanned by the vectors
$e_{i},f_{i}$
obtained from
$f_{i}$
by a rotation by
$-(\unicode[STIX]{x1D70B}/2)$
and let
$g_{i}$
be the unit vector obtained by rotating
$e_{i}$
by
$\unicode[STIX]{x1D70B}/2$
. Then
$\langle h_{i},e_{i}\rangle =\langle f_{i},g_{i}\rangle =\sin \unicode[STIX]{x1D703}_{i}$
and
$\langle h_{i},g_{i}\rangle =-\text{cos}\,\unicode[STIX]{x1D703}_{i}$
.
For
$p\leqslant k$
,
$\dim (E^{\bot }\cap F^{\bot })=\dim (E\cup F)^{\bot }=n^{\prime }-\dim (E\cup F)=n^{\prime }-k-l$
. Let
$h_{i}$
,
$i=l+1,\ldots ,n^{\prime }-k$
be an orthonormal basis of
$E^{\bot }\cap F^{\bot }$
. Vectors
$\{h_{i}\}$
form a basis of
$F^{\bot }$
. To complete a basis of
$E^{\bot }$
, set
$g_{i}:=-h_{i}$
,
$i=l+1,\ldots ,m$
,
$g_{i}:=h_{i}$
,
$i=m+1,\ldots ,n^{\prime }-k$
and
$g_{n^{\prime }-k+i}:=f_{p+i}$
,
$i=1,\ldots ,k-p$
.
Similarly, for
$p\geqslant k$
,
$\dim (E^{\bot }\cap F^{\bot })=n^{\prime }-p-l$
. Let
$g_{i}$
,
$i=l+1,\ldots ,n^{\prime }-p$
be an orthonormal basis of
$E^{\bot }\cap F^{\bot }$
. Vectors
$\{g_{i}\}$
form a basis of
$E^{\bot }$
. To complete a basis of
$F^{\bot }$
, set
$h_{i}:=-g_{i}$
,
$i=l+1,\ldots ,m$
,
$h_{i}:=g_{i}$
,
$i=m+1,\ldots ,n^{\prime }-p$
and
$h_{n^{\prime }-p+i}:=e_{k+i}$
,
$i=1,\ldots ,p-k$
.◻
Definition 4.3. Let
$E\in \operatorname{Gr}_{p}(W),F\in \operatorname{Gr}_{k}(W)$
have Jordan angles
$\unicode[STIX]{x1D703}_{1},\ldots ,\unicode[STIX]{x1D703}_{m}$
, where
$m=\min \{k,n^{\prime }-k,p,n^{\prime }-p\}$
. We define
$\unicode[STIX]{x1D70E}_{i}(E,F)$
to be the
$i$
th elementary symmetric function in
$\cos ^{2}\unicode[STIX]{x1D703}_{1},\ldots ,\cos ^{2}\unicode[STIX]{x1D703}_{m}$
, for
$0\leqslant i\leqslant m$
.
The following properties are obvious:
(i)
$\unicode[STIX]{x1D70E}_{i}$
is
$\text{O}(n^{\prime })$
-invariant, i.e.,
$\unicode[STIX]{x1D70E}_{i}(gE,gF)=\unicode[STIX]{x1D70E}_{i}(E,F)$
for all
$g\in \text{O}(n^{\prime })$
;(ii)
$\unicode[STIX]{x1D70E}_{i}(E,F)=\unicode[STIX]{x1D70E}_{i}(F,E)$
;(iii)
$\unicode[STIX]{x1D70E}_{m}(E,F)$
equals the squared cosine between
$E$
and
$F$
; and
$\unicode[STIX]{x1D70E}_{m}(E^{\bot },F)$
equals the squared sine between
$E$
and
$F$
.
The next result gives the probability distribution of the Jordan angles between a fixed plane and a random plane in
$W$
.
Theorem 4.4 [Reference James16, §6].
Let
$W$
be an
$n^{\prime }$
-dimensional euclidean vector space. Let
$F\in \operatorname{Gr}_{k}(W)$
be fixed and let
$E\in \operatorname{Gr}_{p}(W)$
be chosen randomly according to the
$\operatorname{SO}(n^{\prime })$
-invariant probability measure. Denote the Jordan angles between
$E$
and
$F$
by
$\unicode[STIX]{x1D703}_{i}$
,
$1\leqslant i\leqslant m$
, and set
$x_{i}:=\cos ^{2}(\unicode[STIX]{x1D703}_{i})$
. Then the probability density of the
$(x_{1},\ldots ,x_{m})$
is proportional to
$$\begin{eqnarray}\displaystyle \mathop{\prod }_{j=1}^{m}x_{j}^{(|p-k|-1)/2}(1-x_{j})^{(|n^{\prime }-p-k|-1)/2}\mathop{\prod }_{1\leqslant i<j\leqslant m}(x_{j}-x_{i}). & & \displaystyle \nonumber\end{eqnarray}$$
In [Reference James16, §6], this formula is shown under the assumption
$k\leqslant p\leqslant n^{\prime }/2$
, but using Lemma 4.2, the other cases can be checked as well.
We note that the integral of a symmetric function in
$x_{1},\ldots ,x_{m}$
with respect to this density over
$\{0\leqslant x_{1}\leqslant \cdots \leqslant x_{m}\leqslant 1\}$
can be written as an integral over
$[0,1]^{m}$
by dividing the integral by
$m!$
and replacing the factor
$\prod _{1\leqslant i<j\leqslant m}(x_{j}-x_{i})$
by
$\prod _{1\leqslant i<j\leqslant m}|x_{j}-x_{i}|$
. Such integrals were studied by Selberg [Reference Selberg24] and others, see the survey [Reference Forrester and Ole Warnaar8]. In this paper, we will need the following integral of Selberg type.
Theorem 4.5 [Reference Aomoto2, Theorem 2].
Let
$m\in \mathbb{N}$
,
$\unicode[STIX]{x1D706}>0$
,
$\unicode[STIX]{x1D706}^{\prime },\unicode[STIX]{x1D706}^{\prime \prime }>-1$
and let
$f:\mathbb{R}^{m}\rightarrow \mathbb{R}$
. Define
$$\begin{eqnarray}\displaystyle J_{f}=\int _{[0,1]^{m}}f(x_{1},\ldots ,x_{m})\mathop{\prod }_{j=1}^{m}x_{j}^{\unicode[STIX]{x1D706}^{\prime }}(1-x_{j})^{\unicode[STIX]{x1D706}^{\prime \prime }}\mathop{\prod }_{1\leqslant i<j\leqslant m}|x_{i}-x_{j}|^{\unicode[STIX]{x1D706}}\,dx_{1}\ldots dx_{m}. & & \displaystyle \nonumber\end{eqnarray}$$
If
$f(x_{1},\ldots ,x_{m}):=\prod _{j=1}^{m}(x_{j}-t)$
for
$t\in \mathbb{R}$
, then
$$\begin{eqnarray}\frac{J_{f}}{J_{1}}=\mathop{\sum }_{r=0}^{m}(-t)^{m-r}\binom{m}{r}\mathop{\prod }_{j=1}^{r}\frac{\unicode[STIX]{x1D706}^{\prime }+1+\frac{1}{2}(m-j)\unicode[STIX]{x1D706}}{\unicode[STIX]{x1D706}^{\prime }+\unicode[STIX]{x1D706}^{\prime \prime }+2+\unicode[STIX]{x1D706}(m-j/2-\frac{1}{2})}.\end{eqnarray}$$
Corollary 4.6. Let
$W$
be an
$n^{\prime }$
-dimensional euclidean vector space. Let
$F\in \operatorname{Gr}_{k}(W)$
be fixed and let
$E\in \operatorname{Gr}_{p}(W)$
be chosen randomly according to the
$\operatorname{SO}(n^{\prime })$
-invariant probability measure. Then the expectation of
$\unicode[STIX]{x1D70E}_{i}(E,F),0\leqslant i\leqslant m$
is given by
$$\begin{eqnarray}\displaystyle \binom{m}{i}\binom{|p-k|+m}{i}\binom{n^{\prime }}{i}^{-1}. & & \displaystyle \nonumber\end{eqnarray}$$
Proof. By Theorems 4.4 and 4.5, we have to compute
$J_{\unicode[STIX]{x1D70E}_{i}}/J_{1}$
, where
$\unicode[STIX]{x1D706}=1$
,
$\unicode[STIX]{x1D706}^{\prime }=(|p-k|-1)/2$
,
$\unicode[STIX]{x1D706}^{\prime \prime }=(|n^{\prime }-p-k|-1)/2$
. Since
$$\begin{eqnarray}f(x_{1},\ldots ,x_{m})=\mathop{\prod }_{j=1}^{m}(x_{j}-t)=\mathop{\sum }_{r=0}^{m}(-t)^{m-r}\unicode[STIX]{x1D70E}_{r}(x_{1},\ldots ,x_{m}),\end{eqnarray}$$
Theorem 4.5 implies that
$$\begin{eqnarray}\displaystyle \frac{J_{\unicode[STIX]{x1D70E}_{i}}}{J_{1}} & = & \displaystyle \binom{m}{i}\mathop{\prod }_{j=1}^{i}\frac{\unicode[STIX]{x1D706}^{\prime }+1+\frac{1}{2}(m-j)\unicode[STIX]{x1D706}}{\unicode[STIX]{x1D706}^{\prime }+\unicode[STIX]{x1D706}^{\prime \prime }+2+\unicode[STIX]{x1D706}(m-j/2-\frac{1}{2})}\nonumber\\ \displaystyle & = & \displaystyle \binom{m}{i}\binom{2\unicode[STIX]{x1D706}^{\prime }+m+1}{i}\binom{2\unicode[STIX]{x1D706}^{\prime }+2\unicode[STIX]{x1D706}^{\prime \prime }+2m+2}{i}^{-1}\nonumber\\ \displaystyle & = & \displaystyle \binom{m}{i}\binom{|p-k|+m}{i}\binom{|p-k|+|n^{\prime }-p-k|+2m}{i}^{-1}\nonumber\\ \displaystyle & = & \displaystyle \binom{m}{i}\binom{|p-k|+m}{i}\binom{n^{\prime }}{i}^{-1},\nonumber\end{eqnarray}$$
where the last line follows by
$$\begin{eqnarray}\displaystyle & & \displaystyle |p-k|+|n^{\prime }-p-k|+2m\nonumber\\ \displaystyle & & \displaystyle \quad =\max \{p-k,k-p\}+\max \{n^{\prime }-p-k,p+k-n^{\prime }\}\nonumber\\ \displaystyle & & \displaystyle \qquad +\,2\min \{k,n^{\prime }-k,p,n^{\prime }-p\}\nonumber\\ \displaystyle & & \displaystyle \quad =\max \{n^{\prime }-2k,2p-n^{\prime },n^{\prime }-2p,2k-n^{\prime }\}+2\min \{k,n^{\prime }-k,p,n^{\prime }-p\}\nonumber\\ \displaystyle & & \displaystyle \quad =2(\max \{k,n^{\prime }-k,p,n^{\prime }-p\}+\min \{k,n^{\prime }-k,p,n^{\prime }-p\})-n^{\prime }\nonumber\\ \displaystyle & & \displaystyle \quad =n^{\prime }.\Box \nonumber\end{eqnarray}$$
We end this section with a definition of the angle between two subspaces both having dimension equal to the half of the dimension of the ambient euclidean space.
Definition 4.7. Let
$W$
be an oriented euclidean vector space of dimension
$n^{\prime }=2a$
and let
$E,F\in \operatorname{Gr}_{a}(W)$
. Fix some orientations of
$E$
and
$F$
, and endow
$F^{\bot }$
with the orientation such that
$W\cong F^{\bot }\oplus F$
is orientation preserving.
Let
$B_{E,F}$
(respectively
$B_{E,F^{\bot }}$
) denote the orthogonal projection from
$E$
onto
$F$
(respectively from
$E$
onto
$F^{\bot }$
). We define
$$\begin{eqnarray}\displaystyle \tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)=\text{det}(B_{E,F})\text{det}(B_{E,F^{\bot }}), & & \displaystyle \nonumber\end{eqnarray}$$
where
$\text{det}(B_{E,F})$
denotes the determinant of the map
$B_{E,F}$
.
It is easy to see that
$\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)$
is independent of the choice of the orientations of
$E$
and
$F$
, but changes its sign when we reverse the orientation of
$W$
.
Moreover, it is easy to check that
$$\begin{eqnarray}\displaystyle |\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)|=\cos (E,F)\cos (E,F^{\bot }) & & \displaystyle \nonumber\end{eqnarray}$$
and that
$\tilde{\unicode[STIX]{x1D70E}}_{a}$
is invariant under the diagonal action of
$\operatorname{SO}(2a)$
. For
$g\in \text{O}(2a)$
, we have
$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D70E}}_{a}(gE,gF)=\det g\,\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F).\end{eqnarray}$$
Moreover,
$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)=(-1)^{a}\tilde{\unicode[STIX]{x1D70E}}_{a}(E^{\bot },F),\end{eqnarray}$$
and
$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D70E}}_{a}(E,F)=(-1)^{a}\tilde{\unicode[STIX]{x1D70E}}_{a}(F,E).\end{eqnarray}$$
Both equations follow by using that the orientations of
$E\oplus E^{\bot }$
and of
$E^{\bot }\oplus E$
differ by a factor
$(-1)^{a}$
, and the fact that
$|\text{det}(B_{E,F})|=|\text{det}(B_{E^{\bot },F^{\bot }})|$
, since the block matrix with blocks
$B_{E,F},B_{E^{\bot },F},B_{E,F^{\bot }}$
, and
$B_{E^{\bot },F^{\bot }}$
is a special orthogonal matrix.
5 Construction of invariant flag area measures
The aim of this section is to prove the main results of this paper: Theorems 1, 2, 3, and 4. The idea is to follow closely the proof of the dimension formula from §3 and to construct the invariant forms in an explicit way.
5.1 Construction of forms
Let
$\overline{G}=\overline{\operatorname{SO}(n)}=V\rtimes \operatorname{SO}(n)$
be the euclidean motion group and let
$\unicode[STIX]{x1D70B}_{1}:\overline{G}\rightarrow V$
,
$\unicode[STIX]{x1D70B}_{2}:\overline{G}\rightarrow \operatorname{SO}(n)$
be the projections onto the first and second factor. The Maurer-Cartan form on
$\overline{G}$
takes values in the Lie algebra
$\bar{\mathfrak{g}}=V\rtimes \mathfrak{so}(n)$
(see, e.g., [Reference Spivak26]).
Let us fix an orthonormal basis
$e_{1},\ldots ,e_{n}$
of
$V$
. Let
$\unicode[STIX]{x1D70E}_{i}$
,
$\unicode[STIX]{x1D714}_{ij}$
be the components of the Maurer-Cartan form with respect to this basis. Then
$\unicode[STIX]{x1D714}_{ij}=-\unicode[STIX]{x1D714}_{ji}$
. The forms
$\unicode[STIX]{x1D714}_{ij}$
,
$1\leqslant i<j\leqslant n$
, together with
$\unicode[STIX]{x1D70E}_{i}$
,
$1\leqslant i\leqslant n$
form a basis of the space of left-invariant
$1$
-forms on
$\overline{G}$
.
For
$\bar{g}\in \overline{G}$
, let
$\unicode[STIX]{x1D70B}_{2}(\bar{g})=:g=(g_{1},\ldots ,g_{n})\in G=\operatorname{SO}(n)$
be its rotational part, where
$g_{i}=ge_{i}$
,
$1\leqslant i\leqslant n$
, denotes the
$i$
th column of
$g$
. We may consider
$g_{i}$
as a vector-valued function on
$\overline{G}$
. By definition (see [Reference Santaló19]),
$$\begin{eqnarray}\displaystyle & \displaystyle \unicode[STIX]{x1D714}_{ij}|_{\bar{g}}(v)=\langle g_{i},dg_{j}(v)\rangle , & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D70E}_{i}|_{\bar{g}}(v)=\langle g_{i},d\unicode[STIX]{x1D70B}_{1}(v)\rangle ,\quad v\in T_{\bar{g}}\overline{G}. & \displaystyle \nonumber\end{eqnarray}$$
Let
$1\leqslant k,p\leqslant n-1$
. Set
$q:=n-1-p$
. The partial flag manifold
$F(n,p+1)$
can be identified with the homogeneous space
$G/H$
, where
$H=S(\text{O}(p)\times \text{O}(q))$
, as described in §3.
Let
$\unicode[STIX]{x1D6F1},\tilde{\unicode[STIX]{x1D6F1}},\hat{\unicode[STIX]{x1D6F1}}$
be the maps from §3. We denote the volume form on the corresponding fiber by
$\unicode[STIX]{x1D70C},\tilde{\unicode[STIX]{x1D70C}}$
, and
$\hat{\unicode[STIX]{x1D70C}}$
.
For
$\max \{0,k-q\}\leqslant a\leqslant \min \{k,p\}$
, we define
$\hat{\unicode[STIX]{x1D70F}}_{a}\in \unicode[STIX]{x1D6FA}^{n-1}(\overline{G})$
to be the coefficient of
$\unicode[STIX]{x1D6FC}^{a}\unicode[STIX]{x1D6FD}^{k-a}$
in the expansion of
$$\begin{eqnarray}\displaystyle \hat{\unicode[STIX]{x1D70F}}_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}} & := & \displaystyle (\unicode[STIX]{x1D6FC}\unicode[STIX]{x1D70E}_{2}+\unicode[STIX]{x1D714}_{2,1})\wedge \cdots \wedge (\unicode[STIX]{x1D6FC}\unicode[STIX]{x1D70E}_{p+1}+\unicode[STIX]{x1D714}_{p+1,1})\nonumber\\ \displaystyle & & \displaystyle \wedge \,(\unicode[STIX]{x1D6FD}\unicode[STIX]{x1D70E}_{p+2}+\unicode[STIX]{x1D714}_{p+2,1})\wedge \cdots \wedge (\unicode[STIX]{x1D6FD}\unicode[STIX]{x1D70E}_{n}+\unicode[STIX]{x1D714}_{n,1}).\nonumber\end{eqnarray}$$
If
$W_{2},\ldots ,W_{n}$
are tangent vectors to
$\overline{G}$
, then
$\hat{\unicode[STIX]{x1D70F}}_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}(W_{2},\ldots ,W_{n})$
equals the determinant of the matrix whose entries are
$(\unicode[STIX]{x1D6FC}\unicode[STIX]{x1D70E}_{i}+\unicode[STIX]{x1D714}_{i,1})(W_{j})$
for
$2\leqslant i\leqslant p+1,2\leqslant j\leqslant n$
and
$(\unicode[STIX]{x1D6FD}\unicode[STIX]{x1D70E}_{i}+\unicode[STIX]{x1D714}_{i,1})(W_{j})$
for
$p+2\leqslant i\leqslant n,2\leqslant j\leqslant n$
. In the exceptional case
$2p=2k=n-1$
, we define
$$\begin{eqnarray}\hat{\unicode[STIX]{x1D70F}}_{\text{ex}}:=\unicode[STIX]{x1D70E}_{p+2}\wedge \cdots \wedge \unicode[STIX]{x1D70E}_{n}\wedge \unicode[STIX]{x1D714}_{p+2,1}\wedge \cdots \wedge \unicode[STIX]{x1D714}_{n,1}\in \unicode[STIX]{x1D6FA}^{n-1}(\overline{G}).\end{eqnarray}$$
We denote
$$\begin{eqnarray}\hat{\unicode[STIX]{x1D714}}_{a}:=\hat{\unicode[STIX]{x1D70F}}_{a}\wedge \unicode[STIX]{x1D70C}\in \unicode[STIX]{x1D6FA}^{n-1+r}(\overline{G}),\qquad \max \{0,k-q\}\leqslant a\leqslant \min \{k,p\},\end{eqnarray}$$
and, if
$n$
is odd,
$$\begin{eqnarray}\hat{\unicode[STIX]{x1D714}}_{\text{ex}}:=\hat{\unicode[STIX]{x1D70F}}_{\text{ex}}\wedge \unicode[STIX]{x1D70C}\in \unicode[STIX]{x1D6FA}^{n-1+r}(\overline{G}).\end{eqnarray}$$
It is obvious that
$\hat{\unicode[STIX]{x1D714}}_{a}$
and
$\hat{\unicode[STIX]{x1D714}}_{\text{ex}}$
are invariant under
$H$
and that they vanish on vectors, which are tangent to the fiber of
$\tilde{\unicode[STIX]{x1D6F1}}$
. Hence there are unique forms
$\unicode[STIX]{x1D714}_{a},\unicode[STIX]{x1D714}_{\text{ex}}\in \unicode[STIX]{x1D6FA}^{n-1+r}(V\times F(n,p+1))$
with
$\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D714}_{a}=\hat{\unicode[STIX]{x1D714}}_{a},\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D714}_{\text{ex}}=\hat{\unicode[STIX]{x1D714}}_{\text{ex}}$
.
From the projection formula (8) and the fact that
$\tilde{\unicode[STIX]{x1D70C}}$
is the volume form on the fiber of
$\tilde{\unicode[STIX]{x1D6F1}}$
, it follows that for each
$\unicode[STIX]{x1D6FD}\in \unicode[STIX]{x1D6FA}^{\ast }(V\times F(n,p+1))$
$$\begin{eqnarray}\displaystyle \hat{\unicode[STIX]{x1D6F1}}_{\ast }(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D6FD}\wedge \tilde{\unicode[STIX]{x1D70C}}) & = & \displaystyle \unicode[STIX]{x1D6F1}_{\ast }(\tilde{\unicode[STIX]{x1D6F1}}_{\ast }(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }\unicode[STIX]{x1D6FD}\wedge \tilde{\unicode[STIX]{x1D70C}}))\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6F1}_{\ast }(\unicode[STIX]{x1D6FD}\wedge \tilde{\unicode[STIX]{x1D6F1}}_{\ast }\tilde{\unicode[STIX]{x1D70C}})\nonumber\\ \displaystyle & = & \displaystyle \operatorname{vol}(H)\unicode[STIX]{x1D6F1}_{\ast }\unicode[STIX]{x1D6FD},\end{eqnarray}$$
where we used that
$\tilde{\unicode[STIX]{x1D6F1}}_{\ast }\tilde{\unicode[STIX]{x1D70C}}=\int _{H}\tilde{\unicode[STIX]{x1D70C}}=\operatorname{vol}H$
(see, for instance, [Reference Santaló19, (12.11)]).
By (7), the push-forward of some form
$\unicode[STIX]{x1D702}\in \unicode[STIX]{x1D6FA}^{n-1+\hat{r}}(\overline{G})$
at the point
$(x,v)\in V\times S^{n-1}$
, evaluated at vectors
$w_{1},\ldots ,w_{n-1}\in T_{(x,v)}(V\times S^{n-1})$
, is given by
$$\begin{eqnarray}\hat{\unicode[STIX]{x1D6F1}}_{\ast }\unicode[STIX]{x1D702}|_{(x,v)}(w_{1},\ldots ,w_{n-1})=\int _{\hat{\unicode[STIX]{x1D6F1}}^{-1}(x,v)}\unicode[STIX]{x1D702}|_{(x,g)}(W_{1},\ldots ,W_{n-1},-),\end{eqnarray}$$
where
$g\in G$
is such that
$\hat{\unicode[STIX]{x1D6F1}}(x,g)=(x,v)$
and
$W_{j}\in T_{(x,g)}\overline{G}$
are lifts of
$w_{j}$
.
Definition 5.1. Define the smooth flag area measure
$S_{k}^{(p),i}$
by the linear combination
$$\begin{eqnarray}\unicode[STIX]{x1D714}:=c_{n,k,p,i}\mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}}\binom{\min \{k,p\}-a}{i}\unicode[STIX]{x1D714}_{a};\end{eqnarray}$$
and, if
$n$
is odd and
$p=k=(n-1)/2$
, the smooth flag area measure
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}$
by the form
$\unicode[STIX]{x1D714}_{ex}$
.
5.2 An integral formula for flag area measures
We first recall some notions from [Reference Zähle30]. In the following, we use the convention
$$\begin{eqnarray}\frac{\infty }{\sqrt{1+\infty ^{2}}}=\lim _{\unicode[STIX]{x1D705}\rightarrow \infty }\frac{\unicode[STIX]{x1D705}}{\sqrt{1+\unicode[STIX]{x1D705}^{2}}}=1.\end{eqnarray}$$
Let
$K$
be a convex body in
$V$
, let
$x\in \unicode[STIX]{x2202}K$
and let
$(x,v)\in \operatorname{nc}(K)$
. Then there is a positive orthonormal basis
$a_{i}=a_{i}(K;x,v)$
,
$i=2,\ldots ,n$
of
$v^{\bot }$
and real numbers
$\unicode[STIX]{x1D705}_{i}=\unicode[STIX]{x1D705}_{i}(K;x,v)\in [0,\infty ]$
,
$i=2,\ldots ,n$
such that the vectors 
form a positive orthonormal basis of
$T_{(x,v)}\operatorname{nc}(K)$
. The
$\unicode[STIX]{x1D705}_{i}$
are called generalized curvatures, the
$a_{i}$
are the generalized curvature directions.
The space spanned by the generalized curvature directions with generalized curvature
$\unicode[STIX]{x1D705}\in [0,\infty ]$
is unique. If all
$\unicode[STIX]{x1D705}_{i}$
are finite, we can define a linear operator
$S_{x,v}:v^{\bot }\rightarrow v^{\bot }$
by
$S_{x,v}a_{i}=\unicode[STIX]{x1D705}_{i}a_{i}$
. We call it the generalized shape operator.
We will write formulas involving
$S_{x,v}$
even if some of the
$\unicode[STIX]{x1D705}_{i}$
are infinite. The corresponding term is then to be understood in the sense of a limit.
If
$K$
is smooth and
$v=\unicode[STIX]{x1D708}(x)$
is the outer normal vector, then
$\unicode[STIX]{x1D705}_{2},\ldots ,\unicode[STIX]{x1D705}_{n}\in [0,\infty )$
are the principal curvatures of the boundary and the
$a_{i}$
are the principal curvature directions and
$S_{x,v}:T_{x}\unicode[STIX]{x2202}K\rightarrow T_{x}\unicode[STIX]{x2202}K$
is the usual shape operator.
Theorem 5.2. Let
$k,p,i,\unicode[STIX]{x1D6FD}$
be as in Theorem 1. Then for a compact convex body
$K$
,
In the case
$k\leqslant p$
, equation (24) simplifies to
If
$n$
is odd and
$p=k=(n-1)/2$
,
Proof. Let
$K$
be a convex body in
$V$
, let
$x\in \unicode[STIX]{x2202}K$
and let
$(x,v)\in \operatorname{nc}(K)$
. We denote by
$\unicode[STIX]{x1D705}_{i}:=\unicode[STIX]{x1D705}_{i}(K;x,v)$
,
$i=2,\ldots ,n$
, the generalized principal curvatures of
$K$
at
$(x,v)$
with associated generalized principal directions
$a_{i}:=a_{i}(K;x,v)$
. We order them in such a way that
$v,a_{2},\ldots ,a_{n}$
is a positive orthonormal basis of
$V$
.
Let
$(x,g)\in \overline{G}$
with
$\hat{\unicode[STIX]{x1D6F1}}(x,g)=(x,v)$
and let
$g_{1},\ldots ,g_{n}$
, as before, denote the columns of
$g$
. Let
$E\in \operatorname{Gr}_{p+1}(v)$
be the linear span of
$g_{1},\ldots ,g_{p+1}$
. Let
. Then
$w_{2},\ldots ,w_{n}$
form a positive orthonormal basis of
$T_{(x,v)}\operatorname{nc}(K)$
.
By
$W_{i}\in T_{(x,g)}\overline{G}$
denote a lift of
$w_{i}$
,
$2\leqslant i\leqslant n$
, i.e.,
$d\hat{\unicode[STIX]{x1D6F1}}|_{(x,g)}(W_{i})=w_{i}$
.
By definition,
and
Let
$f$
be a smooth function on
$F(n,p+1)$
. We first compute
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D714}_{a})|_{(x,v)}(w_{2},\ldots ,w_{n}). & & \displaystyle \nonumber\end{eqnarray}$$
By equation (20),
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D714}_{a})=\frac{1}{\operatorname{vol}H}\hat{\unicode[STIX]{x1D6F1}}_{\ast }(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }f\wedge \hat{\unicode[STIX]{x1D714}}_{a}\wedge \tilde{\unicode[STIX]{x1D70C}}). & & \displaystyle \nonumber\end{eqnarray}$$
Recall that
$\unicode[STIX]{x1D70C},\tilde{\unicode[STIX]{x1D70C}},\hat{\unicode[STIX]{x1D70C}}$
denote the volume forms on the fibers of
$\unicode[STIX]{x1D6F1},\tilde{\unicode[STIX]{x1D6F1}},\hat{\unicode[STIX]{x1D6F1}}$
. Using equations (21) and (18), we have
$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D714}_{a})|_{(x,v)}(w_{2},\ldots ,w_{n})\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\operatorname{vol}H}\hat{\unicode[STIX]{x1D6F1}}_{\ast }(\tilde{\unicode[STIX]{x1D6F1}}^{\ast }f\wedge \hat{\unicode[STIX]{x1D714}}_{a}\wedge \tilde{\unicode[STIX]{x1D70C}})|_{(x,v)}(w_{2},\ldots ,w_{n})\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\operatorname{vol}H}\int _{\hat{\unicode[STIX]{x1D6F1}}^{-1}(x,v)}\tilde{\unicode[STIX]{x1D6F1}}^{\ast }f\cdot (\hat{\unicode[STIX]{x1D714}}_{a}\wedge \tilde{\unicode[STIX]{x1D70C}})|_{(x,g)}(W_{2},\ldots ,W_{n},-)\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\operatorname{vol}H}\int _{\hat{\unicode[STIX]{x1D6F1}}^{-1}(x,v)}\tilde{\unicode[STIX]{x1D6F1}}^{\ast }f\cdot (\hat{\unicode[STIX]{x1D70F}}_{a}\wedge \unicode[STIX]{x1D70C}\wedge \tilde{\unicode[STIX]{x1D70C}})|_{(x,g)}(W_{2},\ldots ,W_{n},-)\nonumber\\ \displaystyle & & \displaystyle \quad =\frac{1}{\operatorname{vol}H}\int _{\hat{\unicode[STIX]{x1D6F1}}^{-1}(x,v)}\tilde{\unicode[STIX]{x1D6F1}}^{\ast }f\cdot (\hat{\unicode[STIX]{x1D70F}}_{a})|_{(x,g)}(W_{2},\ldots ,W_{n})\cdot \hat{\unicode[STIX]{x1D70C}}.\nonumber\end{eqnarray}$$
Since
$\hat{\unicode[STIX]{x1D70F}}_{a}$
and
$\hat{\unicode[STIX]{x1D70C}}$
are invariant under the action of
$H$
, the expression we are integrating is clearly invariant under the action of
$H$
on
$\hat{\unicode[STIX]{x1D6F1}}^{-1}(x,v)$
. Since we are integrating over the left invariant volume form on
$\operatorname{SO}(n-1)$
, given as a product of the volume form on
$H$
and on
$\operatorname{Gr}_{p+1}(v)$
we can interpret the integral as an integral over
$\operatorname{SO}(n-1)/H\cong \operatorname{Gr}_{p+1}(v)$
. In doing so, we get a factor
$\operatorname{vol}(H)$
(see [Reference Sepanski25, Theorem 1.48]). Hence
$$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6F1}_{\ast }(f\unicode[STIX]{x1D714}_{a})|_{(x,v)}(w_{2},\ldots ,w_{n})=\int _{\operatorname{Gr}_{p+1}(v)}f(v,E)\cdot \hat{\unicode[STIX]{x1D70F}}_{a}|_{(x,g)}(W_{2},\ldots ,W_{n})\,dE. & & \displaystyle \nonumber\\ \displaystyle & & \displaystyle\end{eqnarray}$$
By definition of
$\hat{\unicode[STIX]{x1D70F}}_{a}$
, equations (26) and (27),
$\hat{\unicode[STIX]{x1D70F}}_{a}|_{(x,g)}(W_{2},\ldots ,W_{n})$
equals the coefficient of
$\unicode[STIX]{x1D6FC}^{a}\unicode[STIX]{x1D6FD}^{k-a}$
in the expansion of the determinant of the matrix
$(M_{i,j})_{2\leqslant i,j\leqslant n}$
with
We define
$(n-1)\times (n-1)$
-matrices
$A,B,C$
by
$$\begin{eqnarray}\displaystyle A_{i,j} & := & \displaystyle \left\{\begin{array}{@{}ll@{}}\langle g_{j},a_{i}\rangle ,\quad & 2\leqslant j\leqslant p+1,\\ 0,\quad & p+2\leqslant j\leqslant n,\end{array}\right.\nonumber\\ \displaystyle B_{i,j} & := & \displaystyle \left\{\begin{array}{@{}ll@{}}0,\quad & 2\leqslant j\leqslant p+1,\\ \langle g_{j},a_{i}\rangle ,\quad & p+2\leqslant j\leqslant n,\end{array}\right.\nonumber\\ \displaystyle C_{i,j} & := & \displaystyle \unicode[STIX]{x1D705}_{i}\langle g_{j},a_{i}\rangle ,2\leqslant j\leqslant n.\nonumber\end{eqnarray}$$
Then
Thus, the coefficient of
$\unicode[STIX]{x1D6FC}^{a}\unicode[STIX]{x1D6FD}^{k-a}$
of the determinant of
$M$
equals the mixed discriminant
Notice that
$A,B,C$
are the matrices of the orthogonal projection to
$E\cap v^{\bot }$
, the orthogonal projection to
$E^{\bot }$
, and the generalized shape operator, all computed with respect to the bases
$\{a_{2},\ldots ,a_{n}\},\{g_{2},\ldots ,g_{n}\}$
.
Equation (24) now follows from Definition 5.1.
Let us show that equation (24) simplifies to equation (25) in the case
$k\leqslant p$
. Since
$\text{Id}=\unicode[STIX]{x1D6F1}_{E}+\unicode[STIX]{x1D6F1}_{E^{\bot }}$
, we have
$$\begin{eqnarray}\displaystyle & & \displaystyle \binom{k}{i}D(S_{x,v}[n-k-1],\text{Id}[k-i],\unicode[STIX]{x1D6F1}_{E^{\bot }}[i])\nonumber\\ \displaystyle & & \displaystyle \quad =\mathop{\sum }_{a=0}^{k-i}\binom{k}{i}\binom{k-i}{a}D(S_{x,v}[n-k-1],\unicode[STIX]{x1D6F1}_{E}[a],\unicode[STIX]{x1D6F1}_{E^{\bot }}[k-a])\nonumber\\ \displaystyle & & \displaystyle \quad =\mathop{\sum }_{a=0}^{k-i}\binom{k-a}{i}\binom{k}{a}D(S_{x,v}[n-k-1],\unicode[STIX]{x1D6F1}_{E}[a],\unicode[STIX]{x1D6F1}_{E^{\bot }}[k-a])\nonumber\\ \displaystyle & & \displaystyle \quad =\mathop{\sum }_{a=k-m}^{k-i}\binom{k-a}{i}\binom{k}{a}D(S_{x,v}[n-k-1],\unicode[STIX]{x1D6F1}_{E}[a],\unicode[STIX]{x1D6F1}_{E^{\bot }}[k-a]),\nonumber\end{eqnarray}$$
where the last line follows from the fact that if
$0\leqslant a<k-m$
, then
$k-a>n-p-1$
and the mixed discriminant vanishes since
$\unicode[STIX]{x1D6F1}_{E^{\bot }}$
has rank
$n-p-1$
.
In the exceptional case
$p=q=k=(n-1)/2$
, the argument is similar, using
5.3 Proof of Theorems 1 and 2
Proof of Theorem 2.
The normal cycle of a smooth convex body
$K$
is the image of the smooth map
$\unicode[STIX]{x2202}K\rightarrow SV,x\mapsto (x,\unicode[STIX]{x1D708}(x))$
, where
$\unicode[STIX]{x1D708}:\unicode[STIX]{x2202}K\rightarrow S^{n-1}$
is the Gauss map. To transform the integral over the normal cycle into an integral over the boundary, we note that the Jacobian of this map is 
.
Proof of Theorem 1.
First observe that for
$p=0$
,
$S_{k}^{(0),0}=S_{k}$
(the usual surface area measure), which satisfies the formula. Fix
$1\leqslant p,k\leqslant n-1$
.
Let
$x\in \operatorname{int}F$
, where
$F$
is a face of
$P$
of dimension
$\ell$
. Let
$(x,v)\in \operatorname{nc}(P)$
. Every vector tangent to
$F$
is a generalized curvature direction with generalized curvature
$0$
. Every vector in
$F^{\bot }\cap v^{\bot }$
is a generalized curvature direction with generalized curvature
$+\infty$
. We may therefore choose
$a_{i}:=v_{i},i=2,\ldots ,n$
, where
$v=v_{1},v_{2},\ldots ,v_{n}$
is a positive orthonormal basis of
$V$
with
$v_{2},\ldots ,v_{n-\ell }$
spanning
$F^{\bot }\cap v^{\bot }$
and
$v_{n-\ell +1},\ldots ,v_{n}$
spanning
$F$
. Then
$\unicode[STIX]{x1D705}_{2}=\cdots =\unicode[STIX]{x1D705}_{n-\ell }=+\infty$
and
$\unicode[STIX]{x1D705}_{n-\ell +1}=\cdots =\unicode[STIX]{x1D705}_{n}=0$
.
Define the matrices
$A,B,C$
as in the proof of Theorem 5.2. Then,
where
$\tilde{A},\tilde{B},\tilde{C}$
are obtained from
$A,B,C$
by multiplying the
$i$
th row by
.
The last
$\ell$
rows of
$\tilde{C}$
vanish, hence the mixed discriminant vanishes if
$\ell >k$
.
Similarly, the first
$n-\ell -1$
rows in
$\tilde{A}$
and
$\tilde{B}$
vanish, hence the mixed discriminant vanishes if
$\ell <k$
.
Let us next consider the case
$\ell =k$
. The matrix
$M$
from equation (29) is then given by
$$\begin{eqnarray}\displaystyle M_{i,j}:=\left\{\begin{array}{@{}ll@{}}\langle g_{j},v_{i}\rangle ,\quad & 2\leqslant i\leqslant n-k,\\ \unicode[STIX]{x1D6FC}\langle g_{j},v_{i}\rangle ,\quad & n-k+1\leqslant i\leqslant n,2\leqslant j\leqslant p+1,\\ \unicode[STIX]{x1D6FD}\langle g_{j},v_{i}\rangle ,\quad & n-k+1\leqslant i\leqslant n,p+2\leqslant j\leqslant n.\end{array}\right. & & \displaystyle \nonumber\end{eqnarray}$$
It is easy to see that if we define a matrix
$M^{\prime }$
in an analogous way, but using other orthonormal bases of
$E\cap v^{\bot },E^{\bot },F^{\bot }\cap v^{\bot },F$
, then
$\det M^{\prime }=\unicode[STIX]{x1D716}\det M$
, where
$\unicode[STIX]{x1D716}=\pm 1$
depends on whether the orientations on
$(E\cap v^{\bot })\oplus E^{\bot }$
and
$(F^{\bot }\cap v^{\bot })\oplus F$
agree or not.
We use bases
$e_{1},\ldots ,e_{p};g_{1},\ldots ,g_{n-1-p};f_{1},\ldots ,f_{k};h_{1},\ldots ,h_{n-1-k}$
as in Lemma 4.2. Rearranging the rows and columns, the matrix
$M^{\prime }$
has a diagonal block shape, with
$m$
blocks of the type
$\big(\begin{smallmatrix}\sin \unicode[STIX]{x1D703}_{i}^{\prime } & -\text{cos}\,\unicode[STIX]{x1D703}_{i}^{\prime }\\ \unicode[STIX]{x1D6FC}\cos \unicode[STIX]{x1D703}_{i}^{\prime } & \unicode[STIX]{x1D6FD}\sin \unicode[STIX]{x1D703}_{i}^{\prime }\end{smallmatrix}\big)$
,
$\min \{k,p\}-m$
diagonal entries
$\unicode[STIX]{x1D6FC}$
;
$\max \{k,p\}-p$
diagonal entries
$\unicode[STIX]{x1D6FD}$
, and all other diagonal entries
$1$
.
Hence
$$\begin{eqnarray}\displaystyle \det M^{\prime }=\pm \unicode[STIX]{x1D6FC}^{\min \{k,p\}-m}\unicode[STIX]{x1D6FD}^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(\unicode[STIX]{x1D6FC}\cos ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }+\unicode[STIX]{x1D6FD}\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }). & & \displaystyle \nonumber\end{eqnarray}$$
Since
$\det M^{\prime }=\unicode[STIX]{x1D716}$
in the case
$\unicode[STIX]{x1D6FC}=\unicode[STIX]{x1D6FD}=1$
(in this case
$M^{\prime }$
is just the transformation matrix between the two bases), we actually have
$$\begin{eqnarray}\displaystyle \det M^{\prime }=\unicode[STIX]{x1D716}\unicode[STIX]{x1D6FC}^{\min \{k,p\}-m}\unicode[STIX]{x1D6FD}^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(\unicode[STIX]{x1D6FC}\cos ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }+\unicode[STIX]{x1D6FD}\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }), & & \displaystyle \nonumber\end{eqnarray}$$
and hence
$$\begin{eqnarray}\displaystyle \det M & = & \displaystyle \unicode[STIX]{x1D716}\det M^{\prime }\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}(E,F)\nonumber\\ \displaystyle & := & \displaystyle \unicode[STIX]{x1D6FC}^{\min \{k,p\}-m}\unicode[STIX]{x1D6FD}^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(\unicode[STIX]{x1D6FC}\cos ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }+\unicode[STIX]{x1D6FD}\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }),\nonumber\end{eqnarray}$$
where
$\unicode[STIX]{x1D703}_{1}^{\prime },\ldots ,\unicode[STIX]{x1D703}_{m}^{\prime }$
are the principal angles between
$E\cap v^{\bot }$
and
$F$
.
Let
$\unicode[STIX]{x1D70E}^{a}$
be the coefficient of
$\unicode[STIX]{x1D6FC}^{a}\unicode[STIX]{x1D6FD}^{k-a}$
in
$\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}$
. Note that
$\unicode[STIX]{x1D70E}^{a}=0$
if
$a<\min \{k,p\}-m$
or if
$k-a<\max \{k,p\}-p$
. We have
$$\begin{eqnarray}\displaystyle \mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}}\unicode[STIX]{x1D6FC}^{a}\unicode[STIX]{x1D6FD}^{k-a}\unicode[STIX]{x1D70E}^{a} & = & \displaystyle \unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D6FC},\unicode[STIX]{x1D6FD}}\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6FC}^{\min \{k,p\}-m}\unicode[STIX]{x1D6FD}^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(\unicode[STIX]{x1D6FC}\cos ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }+\unicode[STIX]{x1D6FD}\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime })\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6FC}^{\min \{k,p\}-m}\unicode[STIX]{x1D6FD}^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(\unicode[STIX]{x1D6FC}+(\unicode[STIX]{x1D6FD}-\unicode[STIX]{x1D6FC})\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }).\nonumber\end{eqnarray}$$
Substituting
$\unicode[STIX]{x1D6FC}:=1,\unicode[STIX]{x1D6FD}:=t+1$
for some variable
$t$
, we obtain that
$$\begin{eqnarray}\displaystyle \mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}}(t+1)^{k-a}\unicode[STIX]{x1D70E}^{a}=(t+1)^{\max \{k,p\}-p}\mathop{\prod }_{i=1}^{m}(1+t\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime }). & & \displaystyle \nonumber\end{eqnarray}$$
We divide both sides by
$(t+1)^{\max \{k,p\}-p}$
and obtain
$$\begin{eqnarray}\displaystyle \mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}}(t+1)^{\min \{k,p\}-a}\unicode[STIX]{x1D70E}^{a}=\mathop{\prod }_{i=1}^{m}(1+t\sin ^{2}\unicode[STIX]{x1D703}_{i}^{\prime })=\mathop{\prod }_{i=1}^{m}(1+t\cos ^{2}\unicode[STIX]{x1D703}_{i}), & & \displaystyle \nonumber\end{eqnarray}$$
with
$\unicode[STIX]{x1D703}_{1},\ldots ,\unicode[STIX]{x1D703}_{m}$
being the principal angles between
$E^{\bot }$
and
$F$
.
Comparing the coefficient of
$t^{i}$
on both sides yields
$$\begin{eqnarray}\displaystyle \mathop{\sum }_{a=\min \{k,p\}-m}^{\min \{k,p\}-i}\binom{\min \{k,p\}-a}{i}\unicode[STIX]{x1D70E}^{a}=\unicode[STIX]{x1D70E}_{i}(\cos ^{2}\unicode[STIX]{x1D703}_{1},\ldots ,\cos ^{2}\unicode[STIX]{x1D703}_{m})=\unicode[STIX]{x1D70E}_{i}(E^{\bot },F). & & \displaystyle \nonumber\end{eqnarray}$$
Taking into account equations (28) and (22) finishes the proof of equation (4).
Let us finally study the case
$2p=2k=n-1$
, adapting the argument from above. Define the linear operator
$\tilde{S}_{x,v}$
by
Then, using equations (15) and (16),
5.4 Proof of Theorem 3
(i) To show Statement (i), it is enough to use the expression for
$S_{k}^{(p)}$
given in Proposition 1.1 and recall that, as shown in §4, we have
$\cos (E^{\bot },F)^{2}=\unicode[STIX]{x1D70E}_{\min \{k,p\}}(E^{\bot },F)$
.
(ii) Translation invariance and homogeneity follow from the corresponding properties of the forms
$\unicode[STIX]{x1D714}_{a}$
.
(iii) Direct from the fact that the principal angles between a pair of subspaces are invariant under
$\text{O}(n)$
.
(iv) Since the elementary symmetric function of positive numbers is positive, the integrand is positive.
(v) By a change of variables, Statement (v) follows if, for every
$g\in \text{O}(n)$
, we have
$$\begin{eqnarray}\displaystyle \tilde{\unicode[STIX]{x1D70E}}_{(n-1)/2}((gE)^{\bot },gF)=\det g\,\tilde{\unicode[STIX]{x1D70E}}_{(n-1)/2}(E^{\bot },F). & & \displaystyle \nonumber\end{eqnarray}$$
If
$g\in \operatorname{SO}(n)$
, then
$g$
maps positive orthonormal bases to such and the above equation follows. To see what happens for
$g\in \text{O}(n)\setminus \operatorname{SO}(n)$
, it is enough to look at
$g=-\text{Id}$
(which has determinant
$-1$
since
$n$
is odd). Under this reflection, the orientation of the space
$W=v^{\bot }$
changes and hence the sign of
$\tilde{\unicode[STIX]{x1D70E}}_{(n-1)/2}(E^{\bot },F)$
changes.
(vi) Let
$P$
be a polytope in
$V$
and
$\unicode[STIX]{x1D6FD}\in {\mathcal{B}}(S^{n-1})$
. Then, the right-hand side in equation (4) equals
$$\begin{eqnarray}\displaystyle c_{n,k,p,i}\mathop{\sum }_{F\in {\mathcal{F}}_{k}(P)}\operatorname{vol}_{k}(F)\int _{\operatorname{n}(P,F)}\bigg(\int _{\operatorname{Gr}_{p+1}(v)}\unicode[STIX]{x1D70E}_{i}(E^{\bot },F)\,dE\bigg)\mathbf{1}_{v\in \unicode[STIX]{x1D6FD}}\,dv. & & \displaystyle \nonumber\end{eqnarray}$$
The term in brackets is a constant, which can be computed with Corollary 4.6.
Since
$E^{\bot }$
ranges over an
$(n-p-1)$
-plane in the
$(n-1)$
-dimensional space
$W:=v^{\bot }$
, we have
$n^{\prime }=n-1$
. Hence, with
$m:=\min \{k,n-k-1,p,n-p-1\}$
,
$$\begin{eqnarray}\displaystyle S_{k}^{(p),i}(P,\unicode[STIX]{x1D70B}^{-1}(\unicode[STIX]{x1D6FD})) & = & \displaystyle c_{n,k,p,i}\binom{m}{i}\binom{|n-p-k-1|+m}{i}\binom{n-1}{i}^{-1}\nonumber\\ \displaystyle & & \displaystyle \cdot \mathop{\sum }_{F\in {\mathcal{F}}_{k}(P)}\operatorname{vol}_{k}(F)\int _{\operatorname{n}(P,F)}\mathbf{1}_{v\in \unicode[STIX]{x1D6FD}}\,dv\nonumber\\ \displaystyle & = & \displaystyle S_{k}(P,\unicode[STIX]{x1D6FD}).\nonumber\end{eqnarray}$$
By approximation, this formula holds for arbitrary convex bodies.
Finally, to prove that
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}(K,\unicode[STIX]{x1D70B}^{-1}(\unicode[STIX]{x1D6FD}))=0$
, we remark that
$(K,\unicode[STIX]{x1D6FD})\mapsto \unicode[STIX]{x1D6F7}(K,\unicode[STIX]{x1D6FD}):=\tilde{S}_{(n-1)/2}^{((n-1)/2)}(K,\unicode[STIX]{x1D70B}^{-1}(\unicode[STIX]{x1D6FD}))$
satisfies the conditions (B1)–(B5) of Schneider’s characterization result [Reference Schneider20, Satz 2] and is therefore a linear combination of the euclidean area measures. In particular,
$\unicode[STIX]{x1D6F7}(gK,g\unicode[STIX]{x1D6FD})=\unicode[STIX]{x1D6F7}(K,\unicode[STIX]{x1D6FD})$
for each
$g\in \text{O}(n)$
. On the other hand, by (iv) we also have
$\unicode[STIX]{x1D6F7}(gK,g\unicode[STIX]{x1D6FD})=\det g\,\unicode[STIX]{x1D6F7}(K,\unicode[STIX]{x1D6FD})$
for all
$g\in \text{O}(n)$
. Both equations can hold simultaneously only if
$\unicode[STIX]{x1D6F7}\equiv 0$
.◻
5.5 Proof of Theorem 4
Since
$S_{k}^{(p),i}$
with
$0\leqslant i\leqslant m$
are elements of the space
$\operatorname{FlagArea}_{k}^{(p),\text{sm},\operatorname{SO}(n)}$
and this space is of dimension
$m+1$
, it remains to prove that these elements are linearly independent. Otherwise, there would be some fixed
$k,p$
, and constants
$c_{i}$
such that
$$\begin{eqnarray}\displaystyle \mathop{\sum }_{i=0}^{m}c_{i}S_{k}^{(p),i}=0. & & \displaystyle \nonumber\end{eqnarray}$$
Take a polytope of dimension
$k$
. Let
$F$
be its only
$k$
-face. Fix a unit vector
$v$
orthogonal to
$F$
and a
$(p+1)$
-dimensional space
$E$
containing
$v$
. Taking
$(f_{j})_{j}$
a sequence of smooth functions on
$F(n,p+1)$
with
$f_{j}(v,E)=1$
and whose supports shrink to
$(v,E)$
, we obtain that
$$\begin{eqnarray}\displaystyle 0=\mathop{\sum }_{i=0}^{m}c_{i}\unicode[STIX]{x1D70E}_{i}(E^{\bot },F)=\mathop{\sum }_{i=0}^{m}c_{i}\unicode[STIX]{x1D70E}_{i}(\cos ^{2}\unicode[STIX]{x1D703}_{1},\ldots ,\cos ^{2}\unicode[STIX]{x1D703}_{m}), & & \displaystyle \nonumber\end{eqnarray}$$
where
$\unicode[STIX]{x1D703}_{1},\ldots ,\unicode[STIX]{x1D703}_{m}$
are the Jordan angles between
$E^{\bot }$
and
$F$
. Since we may choose
$E$
arbitrarily, the numbers
$\cos ^{2}\unicode[STIX]{x1D703}_{1},\ldots ,\cos ^{2}\unicode[STIX]{x1D703}_{m}$
are arbitrary numbers in
$\{0\leqslant x_{1}\leqslant \cdots \leqslant x_{m}\leqslant 1\}$
. With the
$\unicode[STIX]{x1D70E}_{i}$
being algebraically independent, it follows that each coefficient vanishes.
This proves that the
$S_{k}^{(p),i},0\leqslant i\leqslant m$
are linearly independent. Since these flag area measures are
$\text{O}(n)$
-covariant, while
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}$
is not
$\text{O}(n)$
-covariant, there can also be no linear relation involving
$\tilde{S}_{(n-1)/2}^{((n-1)/2)}$
.◻
Acknowledgements
The first named author was supported by DFG grants AB 584/1-1 and 584/1-2. The second named author was supported by DFG grant BE 2484/5-2. The third named author was supported by FAPA grant P19.160422.007. The authors thank the Oberwolfach Research Institute for Mathematics for its hospitality and support, where this project was started and worked on in various stages of its implementation during several stays at the institute in the framework of the Oberwolfach Leibniz Fellowship of the third named author. We want to thank the anonymous referee for several suggestions, which led to an improvement of the manuscript, in particular Theorem 5.2.
