1. Introduction
Double Eisenbud–Popescu–Walter (EPW) sextics are an important family of hyper-Kähler manifolds, i.e. compact simply connected Kähler manifolds with a unique, up to scalar, holomorphic two-form, which is everywhere non-degenerate. The linebase example of hyper-Kähler manifolds are 
$K3$ surfaces; a classical example is double covers of 
$\mathbb{P}^2$. Double EPW sextics are a generalization of it, as they come with a structure of double covers of special sextic hypersurfaces in 
$\mathbb{P}^5$, the so-called EPW sextics.
In §2, we present the main objects that come into play. The basis of the theory of double EPW sextics has been developed by O’Grady in an influential series of articles. Already from [Reference O’Grady16], an important connection is observed between (double) EPW sextics and ordinary two-dimensional Gushel–Mukai (GM) varieties, complete intersections of a linear space and a quadric hypersurface inside a Grassmannian. This link has been successively extended to higher-dimensional GM varieties by Iliev and Manivel in [Reference Iliev and Manivel5] and then developed in detail by Debarre and Kuznetsov, in a series of articles that lay the groundwork for an extensive study of the beautiful and intricate interplay between the two families.
A natural question about this relation is: can we give conditions on a GM variety to be associated with a double EPW sextic which is smooth? Indeed, whenever the double cover is smooth, it is a hyper-Kähler manifold.
In §3, we answer this question for two-dimensional GM varieties, which are Brill–Noether general 
$\langle 10 \rangle$-polarized 
$K3$ surfaces. We provide an answer in terms of geometry and in terms of period: the following result summarizes theorems 3.2 and 3.3.
Theorem 1.1.
The double EPW sextic associated with a strongly smooth 
$K3$ surface 
$S=\mathbb{P}^6\cap G(2,V_5)\cap Q$ is smooth if and only if S contains neither lines nor quintic elliptic pencils. Equivalently, a 
$\langle 10 \rangle$-polarized 
$K3$ surface is associated with a double EPW sextic which is a hyper-Kähler manifold if and only if it does not lie in six (explicitly described) divisors in the corresponding moduli space.
This result relies on the tools provided by Debarre and Kuznetsov which we presented in §2 as well as on a careful description of the involution acting on the moduli space of EPW sextics, described for the first time by O’Grady [Reference O’Grady17]. It is interesting to note that smoothness was already known in the very general case, but theorem 1.1 holds without any general assumption: eliminating the hypothesis of very generality is usually a challenging problem and we believe that our result is interesting in this spirit.
As an application of theorem 1.1, in §4, we produce symplectic actions for various groups on families of double EPW sextics which are hyper-Kähler, from lattice-theoretic considerations on automorphisms of 
$K3$ surfaces.
Section 5 is devoted to finding bounds for the automorphism group of GM varieties and on actions of groups on EPW sextics. For example, in proposition 5.2, we show that the automorphism group of S is a finite subgroup of 
$PGL(2,\mathbb{C})$ and can only be symplectic; we also provide some results for GM varieties in higher dimension. Most of these results are obtained by studying the automorphism group of Fano varieties containing a GM variety.
A previous version of this article contained some results about lifting automorphisms of EPW sextics to their double cover; in the meanwhile, a much stronger result has been obtained by Kuznetsov in [Reference Debarre and Mongardi29], see proposition 2.10. We used this result to simplify the proof of proposition 4.2 and to prove the symplecticity of the automorphism group of a strongly smooth 
$K3$ surface in 
$\mathbb{P}^6$, see proposition 5.2.
1.1. Notations
Let X be a topological space. A property holds for 
$x\in X$ general if the condition is satisfied by all the points inside an open subset of X. A property holds for 
$x\in X$ very general if the condition is satisfied by all the points in the complement of a countable union of closed subspaces inside X.
Given a complex vector space V, the Grassmannian of k-dimensional vector subspaces in V will be denoted by 
$G(k,V)$; every Grassmannian we consider will be embedded in the projective space 
$\mathbb{P}(\bigwedge^k V)$ via the Plücker embedding. Non-zero decomposable vectors in 
$\mathbb{P}(\bigwedge^k V)$ are elements that lie in 
$G(k,V)$.
Definition 1.2.
Consider 
$V_5\cong\mathbb{C}^5$. Following [Reference Iliev and Manivel5], for 
$v\in V_5-\lbrace 0 \rbrace$, we call Pfaffian quadric Pv the symmetric bilinear form 
$x \mapsto vol (v\wedge x \wedge x )$ on 
$\bigwedge^2 V_5$.
The Grassmannian is the intersection of all the Pfaffian quadrics; since 
$\ker(P_v)=v\wedge V_5$, every Pfaffian quadric Pv has rank 6.
An automorphism α on a hyper-Kähler manifold X is symplectic if it acts trivially on a 2-form on X, non-symplectic otherwise. We always consider 
$H^2(X,\mathbb{Z})$ endowed with the Beauville–Bogomolov (BB) form. A 
$\langle 2t \rangle$-polarization is a polarization whose square is 2t with respect to the BB form.
We write 
$S^{[2]}$ for the Hilbert square on a 
$K3$ surface S. For any 
$H\in \operatorname{NS}(S)$, we denote by H 2 the induced class on 
$S^{[2]}$ and by 
$\delta\in \operatorname{NS}(S^{[2]})$ the class of the divisor such that 2δ parametrizes non-reduced subschemes.
2. EPW sextics and GM varieties
2.1. EPW sextics
O’Grady provided slightly different points of view on the construction of EPW sextics, see for example [Reference O’Grady16, Reference O’Grady19] and [Reference O’Grady18]; here we follow mainly [Reference O’Grady18], with a view to [Reference Debarre and Kuznetsov23], by Debarre and Kuznetsov.
Let V 6 be a six-dimensional vector space over 
$\mathbb{C}$, on which we fix a volume form 
$\bigwedge^6 V_6\xrightarrow{\sim} \mathbb{C}$, which in turns induces a symplectic form ω on 
$\bigwedge^3 V_6$.
In [Reference Eisenbud, Popescu and Walter7, example 9.3], Eisenbud, Popescu, and Walter introduced the Lagrangian subbundle F of 
$\mathcal{O}_{\mathbb{P}(V_6)}\otimes \bigwedge^3 V_6$, whose fiber over 
$[v]$ is 
$F_v=v\wedge \bigwedge^2 V_6$.
Given 
$v\in V_6-\lbrace 0 \rbrace$, we can fix a decomposition 
$V_6\cong \mathbb{C}v\oplus V_5$ for some hyperplane 
$V_5\subset V_6$, which induces 
$\bigwedge^3 V_6\cong \bigwedge^3 V_5 \oplus F_v$. Every element of Fv can be written in the form 
$v\wedge \eta$ for some 
$\eta\in \bigwedge^2 V_5$ and this induces an isomorphism of vector spaces
\begin{align}
\rho\colon F_v &\xrightarrow{\sim} \bigwedge^2 V_5
\end{align}
\begin{align}
v\wedge \eta &\mapsto \eta.
\end{align}This observation will be useful later.
We call 
$\mathbb{LG}(\bigwedge^3 V_6)\subset G(10,\bigwedge^3 V_6)$ the symplectic Grassmannian, parametrizing Lagrangian subspaces with respect to the symplectic form on 
$\bigwedge^3 V_6$; since two volume forms differ by a non-zero constant, 
$\mathbb{LG}(\bigwedge^3 V_6)$ does not depend on the choice of the volume form. From now on, 
$A\in \mathbb{LG}(\bigwedge^3 V_6)$ will be a Lagrangian subspace in 
$\bigwedge^3 V_6$ such that 
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$.
For a fixed A, we consider the map of vector bundles 
$\lambda_A\colon F\to \mathcal{O}_{\mathbb{P}(V_6)} \otimes A^\vee$ such that 
$(\lambda_A)_{[v]}(x)=\omega(x,-)_{|_{A}}$ for every 
$[v]\in \mathbb{P}(V_6)$. Since A is a Lagrangian subspace, the map is injective on 
$[v]\in \mathbb{P}(V_6)$ if and only if 
$F_v\cap A=0$. Since 
$\mathrm{rk}(F)=\mathrm{rk}(\mathcal{O}_{\mathbb{P}(V_6)}\otimes A^\vee)$, we can consider the map 
$ \operatorname{det} (\lambda_A)\colon \operatorname{det} (F)\to \operatorname{det} (\mathcal{O}_{\mathbb{P}(V_6)}\otimes A^\vee).$
Definition 2.1.
The determinantal variety 
$Y_A= Z( \operatorname{det} (\lambda_A))$ is a sextic hypersurface (see [Reference O’Grady16, (1.8)]), called EPW sextic.
We also associate with A a stratification of 
$\mathbb{P}(V_6)$: for 
$k\geq 0$ we define
\begin{equation}
Y^{\geq k}_A=\lbrace [v] \in \mathbb{P}(V_6) | \dim(F_v\cap A)\geq k \rbrace;
\end{equation}
\begin{equation*}Y^{k}_A=\lbrace [v] \in \mathbb{P}(V_6) | \dim(F_v\cap A)= k \rbrace.\end{equation*} The sets 
$Y^{\geq k}_A$ are degeneracy loci and are thus endowed with a natural structure of scheme. For every k, the variety 
$Y^{\geq k+1}_A$ is a closed subvariety of 
$Y^{\geq k}_A$ and clearly 
$Y_A=Y_A^{\geq 1}$.
Proposition 2.2.
If 
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$, then YA is integral and normal and 
$\operatorname{Sing}(Y_A)=Y^{\geq 2}_A$. Moreover, 
$Y^{\geq 2}_A$ is a normal integral surface whose singular locus is 
$Y^{3}_A$, which in turn is finite; when smooth, the surface 
$Y^{\geq 2}_A$ is of general type. Finally 
$Y^{\geq 4}_A=\emptyset$.
Proof. Integrality and normality for YA follow from 
$\operatorname{Sing}(Y_A)=Y^{\geq 2}_A$. For the rest, see [Reference O’Grady18, corollary 2.5], [Reference O’Grady18, proposition 2.9], [Reference O’Grady19, claim 3.7], [Reference Debarre and Kuznetsov22, theorem B.2], and [Reference Ferretti3, proposition 1.10].
The situation can be much more complicated for EPW sextics associated with a Lagrangian subspace A which contains a non-zero decomposable vector; however, for our scope, we will not need to deal with them.
2.2. The dual EPW sextic
One interesting feature of EPW sextics is that they admit a dual counterpart. The volume form on V 6 induces a volume form on 
$ V^\vee_6$ too, and thus a symplectic form on 
$\bigwedge^3 V^\vee_6\cong (\bigwedge^3 V_6)^\vee$, hence for 
$A\in \mathbb{LG}(\bigwedge^3 V_6)$ we also have
\begin{equation*}A^\perp=\lbrace \phi\in (\bigwedge^3 V_6)^\vee| \phi(a)=0 \,\mbox{for every }a\in A\rbrace\in \mathbb{LG}((\bigwedge^3 V_6)^\vee);\end{equation*} 
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$ if and only if 
$\mathbb{P}(A^\perp)\cap G(3,V^\vee_6)=\emptyset$, see [Reference O’Grady18, Section 2.6].
Definition 2.3.
We call dual EPW sextic the EPW sextic associated with 
$A^\perp$.
When 
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$, the hypersurfaces 
$Y_A\subset \mathbb{P}(V_6)$ and 
$Y_{A^\perp}\subset \mathbb{P}(V^\vee_6)$ are projectively dual; this has been proved by O’Grady, see also [Reference Debarre and Kuznetsov22, proposition B.3].
As showed in [Reference O’Grady18, Section 2.6], the EPW strata associated with 
$A^\perp$ can also be described as
\begin{equation*}Y^{\geq k}_{A^\perp}=\lbrace [V_5] \in \mathbb{P}(V^\vee_6) | \dim(\bigwedge^3 V_5 \cap A)\geq k \rbrace\end{equation*}Definition 2.4.
We denote by
\begin{equation*}\Sigma=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | \mathbb{P}(A)\cap G(3,V_6)\neq \emptyset \rbrace \end{equation*}the set of Lagrangian subspaces A admitting a non-zero decomposable vector, by
\begin{equation*}\Delta=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | Y^{\geq 3}_A\neq \emptyset\rbrace\end{equation*}the set of Lagrangian subspaces whose third stratum is not empty, and by
\begin{equation*}\Pi=\lbrace A\in \mathbb{LG}(\bigwedge^3 V_6) | Y^{\geq 3}_{A^\perp}\neq \emptyset\rbrace \end{equation*} the set of Lagrangian subspaces 
$A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ such that the third stratum associated with the dual Lagrangian 
$A^\perp$ is not empty.
Following O’Grady, we write 
$\mathbb{LG}(\bigwedge^3 V_6)^0=\mathbb{LG}(\bigwedge^3 V_6)- (\Delta\cup \Sigma)$.
The subsets Σ, 
$\Delta, \Pi$ descend to distinct irreducible divisors in 
$\mathbb{LG}(\bigwedge^3 V_6)//$
$PGL(6)$, see [Reference O’Grady18, proposition 3.1, Item 1] and [Reference O’Grady19, proposition 2.2]. We denote their quotients in the same way.
Proposition 2.5. [Reference Debarre and Kuznetsov22], proposition B.8
If 
$A\notin \Sigma$, the automorphism group of YA is 
$\lbrace \alpha\in \operatorname{PGL}(V_6)$ such that 
$ \left(\bigwedge^3 \alpha\right)(A)=A \rbrace$. In particular, every 
$\alpha\in \operatorname{Aut}(Y_A)$ fixes 
$Y_A^k$ for every 
$k\geq 0$.
From now on we always identify 
$PGL(V_6)$ and 
$PGL(V^\vee_6)$ through the natural isomorphism sending 
$\phi\in \operatorname{PGL}(V_6)$ to 
$(\phi^{-1})^\vee$, which maps 
$[V_5]$ to 
$[\phi(V_5)]$.
Corollary 2.6.
If 
$A\notin \Sigma$, then 
$\operatorname{Aut}(Y_A)=\operatorname{Aut}(Y_{A^\perp})$.
2.3. Double EPW sextics
The importance of EPW sextics stems mostly from the fact that, in the general case, they admit a ramified double cover which is a hyper-Kähler manifold, as was proved by O’Grady in [Reference O’Grady16]. We also refer to [Reference Debarre and Kuznetsov23, theorem 5.2, Item 1].
We denote by 
$\mathcal{R}=\operatorname{Coker} (\lambda_A)_{|_{Y_A}}$ the first Lagrangian cointersection sheaf; it is a reflexive sheaf, see [Reference O’Grady16, proposition 4.3, Item 1].
Theorem 2.7. (O’Grady)
Consider 
$A\in \mathbb{LG}(\bigwedge^3 V_6)$ such that 
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$. Then there is a unique double cover 
$f_A\colon X_A\to Y_A$ with branch locus 
$Y^{\geq 2}_A$ such that
\begin{equation}
(f_A)_*\mathcal{O}_{X_A}\cong \mathcal{O}_{Y_A} \oplus \mathcal{R}(-3).
\end{equation} The variety XA is integral and normal, and its singular locus is 
$f^{-1}_A(Y^3_A)$.
A double cover of YA can be constructed even when 
$\mathbb{P}(A)\cap G(3,V_6)\neq \emptyset$, provided that 
$Y_A\neq \mathbb{P}(V_6)$. However, in this case, XA is never smooth.
Definition 2.8.
We call double EPW sextic the double cover XA, and we denote by ιA the associated covering involution.
Theorem 2.9 [Reference O’Grady19], theorem 4.25
Suppose that XA is smooth, i.e. 
$A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$. Then XA is a hyper-Kähler manifold equivalent by deformation to the Hilbert square on a K3 surface. The involution ιA is non-symplectic.
The ample class 
$D_A=f^*_A\mathcal{O}_{Y_A}(1)\in \operatorname{NS}(X_A)$ is the only primitive polarization on XA coming from YA, by Lefschetz hyperplane theorem. It has square 2 with respect to the BB form; the general element, inside the moduli space 
$\mathcal{M}_2$ (see appendix A for a definition) is isomorphic to 
$(X_A,D_A)$ for some 
$A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$.
We denote by 
$\operatorname{Aut}_{D_A}(X_A)\subseteq \operatorname{Aut}(X_A)$ the group of automorphisms fixing DA, or equivalently commuting with ιA. There is a short exact sequence
\begin{equation}
1\to \lbrace \operatorname{id},\iota_A \rbrace\to \operatorname{Aut}_{D_A}(X_A) \to \operatorname{Aut}(Y_A) \to 1.
\end{equation}Proposition 2.10. [Reference Debarre and Mongardi29], proposition A.2
The sequence (5) splits, so 
$\operatorname{Aut}_{D_A}(X_A)\cong \operatorname{Aut}(Y_A)\times \lbrace \operatorname{id},\iota_A \rbrace$. When XA is smooth, under this isomorphism 
$\operatorname{Aut}(Y_A)$ is the group of symplectic automorphisms fixing the polarization DA.
2.4. GM varieties
In this section, we introduce GM varieties and in the next one we explain how they admit an associated EPW sextic.
We fix a five-dimensional complex vector space V 5. For the following definition and results, we refer to [Reference Debarre and Kuznetsov22, definition 2.1].
Definition 2.11.
Consider a vector subspace 
$W\subseteq \bigwedge^2 V_5$ of dimension 
$\dim W\geq 6$ and a quadric hypersurface 
$Q\subset \mathbb{P}(W)$. We call ordinary GM intersection the scheme
\begin{equation*} Z= \mathbb{P}(W)\cap G(2,V_5)\cap Q . \end{equation*} It is an ordinary GM variety if Z is integral and 
$\dim(Z)=\dim(W)-5$.
When Z is a GM variety, it is a complete intersection inside 
$G(2,V_5)$ and has degree 10 in 
$\mathbb{P}(W)$.
If 
$\dim(Z)\geq 3$, Z is a Fano variety of index n − 2 (see for example [Reference Debarre and Kuznetsov22, theorem 2.3]). If 
$\dim(Z)=2$, let H be the polarization given by 
$\mathcal{O}_{\mathbb{P}(W)}(1)$: the polarized variety (Z, H) is a Brill–Noether general 
$K3$ surface, see [Reference Mukai30], also [Reference Knutsen and Johnsen4, theorem 10.3]. The converse holds under a technical condition, which is strongly smoothness.
Definition 2.12.
Let 
$Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be an ordinary n-dimensional GM variety. The Grassmannian hull of Z is the 
$(n+1)$-dimensional intersection 
$M_Z=\mathbb{P}(W)\cap G(2,V_5)$. We say that Z is strongly smooth if both Z and MZ are dimensionally transverse and smooth.
Although GM curves exist, here we never deal with them, since they are never strongly smooth or, equivalently, their associated double EPW sextics are never smooth, as we will see later (theorem 2.17).
If Z is strongly smooth, its Grassmannian hull has Picard rank 1 by the Lefschetz hyperplane theorem. For every Brill–Noether general 
$K3$ surface (S, H), the projective model 
$\phi_{|H|}(S)$ is a smooth ordinary GM variety of dimension 2, provided that it is strongly smooth (see remark 2.19), whereas smoothness and strongly smoothness are equivalent when 
$n\geq 3$.
Definition 2.13.
An isomorphism between 
$Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ and 
$Z^{\prime}=\mathbb{P}(W^{\prime})\cap G(2,V^{\prime}_5)\cap Q^{\prime}$ is a linear map 
$\phi \colon \mathbb{P}(W)\to \mathbb{P}(W^{\prime})$ such that 
$\phi(Z)=Z^{\prime}$. We denote by 
$\operatorname{Aut}(Z,\mathbb{P}(W))$ the group of automorphisms of Z.
By the Lefschetz hyperplane theorem, a smooth GM variety Z of dimension at least 3 has Picard rank 1. So, in this case, 
$\operatorname{Aut}(Z,\mathbb{P}(W))$ is the whole automorphism group of Z as an abstract variety.
Debarre and Kuznetsov provided an intrinsic characterization of normal GM varieties in [Reference Debarre and Kuznetsov22, theorem 2.3]; this leads to the definition of GM data, which are a set of linear data which can be associated with any normal GM variety. These collections of objects are very useful to handle.
Definition 2.14.
Ordinary GM data 
$(W,V_6,V_5,\mu,q,\epsilon)$ of dimension n consists of (we set 
$L=(V_6/V_5)^\vee$ for readability)
• a
$(n+5)$-dimensional vector space W;• a six-dimensional vector space V 6;
• a hyperplane V 5 of V 6;
• an injective linear map
$\mu\colon W\otimes L \hookrightarrow \bigwedge^2 V_5$;• a linear map
$q\colon V_6\to S^2 W^\vee$;• a linear isomorphism
$\epsilon\colon \bigwedge^5 V_5\to L^{\otimes 2}$,
such that, for all 
$v\in V_5$, 
$w_1, w_2\in W$,
\begin{equation}
q(v)(w_1,w_2)=\epsilon(v\wedge \mu(w_1)\wedge \mu(w_2)).
\end{equation}Definition 2.15.
An isomoprhism between two ordinary GM data sets 
$(W,V_6,V_5,\mu,q,\epsilon)$ and 
$(W^{\prime},V^{\prime}_6,V^{\prime}_5,\mu^{\prime},q^{\prime},\epsilon^{\prime})$ is a triple of linear isomorphisms 
$\phi_W\colon W\to W^{\prime}$, 
$\phi_V\colon V_6\to V^{\prime}_6$, 
$\phi_L\colon L\to L^{\prime}$ such that 
$\phi_V(V_5)=V^{\prime}_5$, 
$\epsilon^{\prime}\circ \bigwedge^5 \phi_V=\phi^{\otimes 2}_L\circ \epsilon$, and the following diagrams commute
Debarre and Kuznetsov showed in [Reference Debarre and Kuznetsov22, Section 2.1] how to associate a set of GM data 
$(W,V_6,V_5,\mu,q,\epsilon)$ to a normal GM variety and to GM data a GM intersection which a priori may not be a normal variety; when it is the case, the two constructions are mutually inverse and behave well with respect to the definition of isomorphisms. To keep the exposition readable, we refer to their article for the explicit correspondence.
Finally, we point out that another class of GM intersections, special GM intersections, exist, see for example [Reference Debarre and Kuznetsov22, Section 2.5].
2.5. A correspondence between data sets
Debarre and Kuznetsov defined another set of data. Every time we consider a six-dimensional complex vector space V 6, we endow it with a volume form, which induces a symplectic form on 
$\bigwedge^3 V_6$.
Definition 2.16.
An ordinary Lagrangian datum is a collection 
$(V_6,V_5,A)$, where
• V 6 is a six-dimensional complex vector space,
•
$V_5\subset V_6$ is a hyperplane,•
$A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ is a Lagrangian subspace.
The ordinary Lagrangian data 
$(V_6,V_5,A)$ and 
$(V^{\prime}_6,V^{\prime}_5,A^{\prime})$ are isomorphic if there is a linear isomorphism 
$\phi:V_6\to V^{\prime}_6$ such that 
$\phi(V_5)=V^{\prime}_5$ and 
$(\bigwedge^3 \phi)(A)=A^{\prime}$.
The next result, by Debarre and Kuznetsov, sums up the results by O’Grady, [Reference O’Grady16, Section 5 and Section 6], [Reference O’Grady19, Section 4], in dimension 2 and Iliev and Manivel, [Reference Iliev and Manivel5, Section 2] in dimension 5 and [Reference Iliev and Manivel5, Section 4] in dimension 
$3,4$.
Theorem 2.17.
For 
$n\in \lbrace 1,\ldots,5 \rbrace$, there is a bijection between the set of iso-morphism classes of Lagrangian data sets 
$(V_6,V_5,A)$, with
•
$\mathbb{P}(A)\cap G(3,V_6)=\emptyset$•
$[V_5]\in Y^{5-n}_{A^\perp}$
and isomorphism classes of strongly smooth ordinary GM varieties of dimension n. In particular, there are no strongly smooth ordinary GM curves.
Proof. See [Reference Debarre and Kuznetsov22, theorem 3.6] and [Reference Debarre and Kuznetsov22, theorem 3.16]. For n = 1, recall that 
$A^\perp$ admits non-zero decomposable elements if and only if A does. Then we only need to observe that 
$Y^{\geq 4}_{A^\perp}=\emptyset$ by proposition 2.2.
It is interesting to note that the construction works for non-strongly smooth GM varieties, but in that case A always admits non-zero decomposable elements.
Definition 2.18.
Given some Lagrangian data 
$(V_6,V_5,A)$, the associated ordinary GM variety is the GM intersection Z obtained by theorem 2.17 from 
$(V_6,V_5,A)$. When not otherwise specified, we write 
$Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ where Q is the projective quadric associated with q(x) for any 
$x\in V_6-V_5$.
Given a strongly smooth ordinary GM variety Z, we denote by A(Z) the Lagrangian subspace associated with Z by theorem 2.17.
Remark 2.19.
Strongly smooth special GM varieties of dimension 2 do not exist by [Reference Debarre and Kuznetsov22, remark 3.17] and proposition 2.2.
For any 
$(V_6,V_5,A)$ and 
$p\geq 2$, there is a short exact sequence
\begin{equation}
0 \to \bigwedge^p V_5 \to \bigwedge^p V_6 \xrightarrow{\lambda_p} \bigwedge^{p-1} V_5\otimes L^\vee \to 0.
\end{equation}The following two remarks will be useful in the proof of theorem 3.2 in the frame of the study of two-dimensional GM varieties: although the relation between the latter and EPW sextics is already clear in [Reference O’Grady16] and [Reference O’Grady19], the use of GM data allows a very precise study.
Remark 2.20.
Given some Lagrangian data 
$(V_6,V_5,A)$, theorem 2.17 allows to describe explicitly the associated ordinary GM variety 
$Z\subseteq \mathbb{P}(\bigwedge^2 V_5)$. We fix 
$x\in V_6-V_5$, hence an isomorphism 
$\bigwedge^3 V_6 \cong \bigwedge^3 V_5\oplus F_x$. Under this isomorphism and (1), the sequence (8) for p = 3 becomes
\begin{equation}
0 \to \bigwedge^3 V_5 \to \bigwedge^3 V_6\cong \bigwedge^3 V_5\oplus F_x \xrightarrow{\lambda_3} \bigwedge^{2} V_5 \to 0 .
\end{equation} For p = 4, the sequence (8) can be rewritten as 
$ 0 \to \bigwedge^4 V_5 \to \bigwedge^4 V_6 \xrightarrow{\lambda_4} \bigwedge^{3} V_5 \to 0 $ as well. We will thus consider W as a subspace of 
$\bigwedge^2 V_5$ and we identify 
$\bigwedge^5 V_5$ with 
$\mathbb{C}$ through the volume form on V 6. Then 
$W=\lambda_3(A)$. As for the quadric hypersurface, we call Q(x) the projective quadric hypersurface associated with 
$q(x)\in \operatorname{Sym}^2W^\vee$. We have 
$Z=M_Z\cap Q(x)$. Consider 
$ w\in W$: since 
$W=\lambda_3(A)$, there is 
$\eta\in \bigwedge^3 V_5$ such that 
$\eta+x\wedge w\in A$. By definition of 
$q: V_6\to \operatorname{Sym}^2 W^\vee$ given in theorem 2.17, the element 
$w=\lambda_3(\eta+x\wedge w)$ lies in Q(x) if and only if 
$\eta\wedge w=\lambda_4(x\wedge \eta)\wedge w=0$.
Remark 2.21.
Consider instead some ordinary GM intersection 
$Z=\mathbb{P}(W)\cap G(2,V_5)\cap Q$, with associated GM data 
$(W,V_6,V_5,\mu,q,\epsilon)$. We consider again 
$W\subseteq \bigwedge^2 V_5\otimes L^\vee$ and 
$x\in V_6- V_5$ such that 
$Q=Q(x)$. As above 
$\bigwedge^3 V_6 \cong \bigwedge^3 V_5\oplus F_x$ and (9) holds. Also 
$\bigwedge^3 V_5\cong (\bigwedge^2 V_5)^\vee$ via 
$\epsilon:\bigwedge^5 V_5\to L^{\otimes 2}$ and A(Z) is
\begin{align*} \Bigg\{
(\eta,w)\in \bigwedge^3 V_5\oplus W | w\in W ,
\epsilon(\eta\wedge \mu(\_ ))_{|_W}& = -(q(x)(w,-)\otimes [x])\\ &\quad\mbox{ as elements of } W^\vee
\Bigg\} .\end{align*} Now we forget ϵ, we consider again 
$W\subseteq \bigwedge^2 V_5$ and 
$\bigwedge^3 V_5$ as 
$(\bigwedge^2 V_5)^\vee$. We have then a splitting sequence 
$0\to W^\perp \to (\bigwedge^2 V_5)^\vee \to W^\vee \to 0$. Putting all together, we can rewrite 
$A(Z)\subset
W^\perp \oplus W^\vee\oplus F_x$ as
\begin{equation}
\left\lbrace
(\xi,-q(x)(w,-),x\wedge w)\in W^\perp \oplus W^\vee\oplus F_x \quad |\quad \xi\in W^\perp,
w\in W
\right\rbrace .
\end{equation} From this description, under the identification 
$\bigwedge^3 V_5\cong (\bigwedge^2 V_5)^\vee$, we have 
$A(Z)\cap \bigwedge^3 V_5=W^\perp$ by [Reference Debarre and Kuznetsov22, proposition 3.13, Item a].
Consider the linear map 
$x\wedge W\to W^\vee$ that sends 
$x\wedge w$ to 
$-q(x)(w,-)$: its graph 
$\Gamma_x\subset W^\vee\oplus W$ induces a decomposition 
$A(Z)=W^\perp\oplus \Gamma_x\subset W^\perp \oplus W^\vee\oplus F_x$.
3.
$K3$ surfaces whose associated double EPW sextic is smooth
We denote by 
$\mathcal{K}_{10}$ the moduli space of 
$\langle 10 \rangle$-polarized 
$K3$ surfaces.
Definition 3.1.
The divisor 
$D_{x,y}\subset \mathcal{K}_{10}$ is the locus of pairs (S, H) such that there exists a primitive sublattice 
$\mathbb{Z}H+\mathbb{Z}D\subseteq \operatorname{NS}(S)$ whose Gram matrix is 
$$\left[\begin{array}{cc}
10 & x \\
x & y \\
\end{array}\right].$$
In this section, we want to find precise conditions on a Brill–Noether general, 
$\langle 10 \rangle$-polarized, 
$K3$ surface such that the associated double EPW sextic 
$X_{A(S)}$ is a hyper-Kähler manifold.
We answer this question in two ways, as a condition on curves on the embedded 
$K3$ surface (theorem 3.2) and as a divisorial condition on 
$\mathcal{K}_{10}$ (theorem 3.3). In particular, thanks to theorem 3.3, finding whether 
$X_A(S)$ is smooth becomes a lattice-theoretic problem on 
$\operatorname{NS}(S)$.
Theorem 3.2.
Let 
$S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a 
$\langle 10\rangle$-polarized 
$K3$ surface. The double cover 
$X_{A(S)}$ of the associated EPW sextic 
$Y_{A(S)}$ is smooth if and only if S is strongly smooth and contains neither lines nor quintic elliptic pencils.
We recall that 
$(S,H)\in \mathcal{K}_{10}$ is Brill–Noether general if and only if the projective model of (S, H) is a smooth ordinary GM variety of dimension 2.
From now on, we see a Brill–Noether general 
$(S,H)\in \mathcal{K}_{10}$ as a smooth ordinary GM two-dimensional variety 
$S = \mathbb{P}(W) \cap G(2,V_5) \cap Q$, with Grassmannian hull 
$M_S=\mathbb{P}(W) \cap G(2,V_5)$.
Theorem 3.3.
Let 
$(S,H)\in \mathcal{K}_{10}$ be a 
$\langle 10 \rangle$-polarized 
$K3$ surface. If (S, H) is Brill–Noether general, let 
$A(S)\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}$ be the Lagrangian subspace associated with (S, H). Then
(1) (S, H) is Brill–Noether general if and only if
$(S,H)\notin D_{h,0}$ for
$h\in \lbrace 1,2,3 \rbrace$;(2) if (S, H) is Brill–Noether general, then (S, H) is strongly smooth if and only if
$A(S)\notin \Sigma$, if and only if
$(S,H)\notin D_{4,0}$.
Moreover, if (S, H) is strongly smooth, then 
$A(S)\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0$ if and only if 
$(S,H)\notin D_{x,y}$, with 
$(x,y)\in \lbrace(1,-2),(5,0)\rbrace$. In particular
(3)
$Y^3_{A(S)}\cap \mathbb{P}(V_5)=\emptyset$ if and only if
$(S,H)\notin D_{1,-2}$;(4)
$Y^3_{A(S)} - \mathbb{P}(V_5)=\emptyset$ if and only if
$(S,H)\notin D_{5,0}$.
Grassmannian hulls MS of strongly smooth Brill–Noether general 
$K3$ surfaces are isomorphic, see [Reference Iskovskikh and Prokhorov13] or [Reference O’Grady19, proposition 5.2, Item 3)]. We need a characterization of the lines inside MS. For 
$v\in V_5-\lbrace 0 \rbrace$ and 
$v\in V_3\subset V_5$ with 
$V_3\cong \mathbb{C}^3$, we set 
$L_{v,V_3}=\lbrace v\wedge t |t\in V_3 \rbrace$. Every line in 
$G(2,V_5)$, hence in MS, is of the form 
$\mathbb{P}(L_{v,V_3})$.
Lemma 3.4.
Consider 
$v\in V_5-\lbrace 0 \rbrace$. There exists a three-dimensional vector space 
$V_3\ni v$ such that the line 
$\mathbb{P}(L_{v,V_3})\subset G(2,V_5)$ is contained in MS if and only if 
$\ker(P_v)\cap W$ has dimension exactly 2. In this case, 
$L_{v,V_3}=\ker(P_v)\cap W$.
Proof. We fix a decomposition 
$V_5= \mathbb{C}v\oplus U$; we know that 
$\ker(P_v)=v\wedge U$, hence 
$L_{v,V_3}\subset\ker(P_v)$ for any 
$V_3\ni v$. If 
$\ker(P_v)\cap W$ has dimension 2, its projectivization is a line on MS of the form 
$\mathbb{P}(L_{v,V_3})$. Conversely, if 
$\mathbb{P}(L_{v,V_3})\subset M_S$ then 
$L_{v,V_3}\subset W\cap \ker(P_v)$, in particular 
$\dim(W\cap \ker(P_v))\geq 2$. The dimension is exactly 2, or MS would contain a plane, which is absurd since MS has Picard rank 1.
When a GM variety Z is strongly smooth, for 
$v\in V_5-\lbrace 0\rbrace$, the kernel of the Pfaffian quadric 
$(P_v)_{|_W}$ is 
$\ker(P_v)\cap W$ and its corank is at least 
$\dim(W)-6$. For 
$x\notin V_6$, the kernel of q(x) is computed in [Reference Debarre and Kuznetsov22, proposition 3.13, Item b].
Proof of theorem 3.2
The proof consists of three parts. We already know, from §2.5, that strongly smoothness is a necessary condition to have 
$X_{A(S)}$ smooth. We need to prove 
$A(S)\notin \Delta$, i.e. 
$Y^3_{A(S)}=\emptyset$.
S contains no line if and only if 
$Y^3_{A(S)}\cap \mathbb{P}(V_5)=\emptyset$. Fix 
$x\in V_6-V_5$, so that 
$S=M_S\cap Q(x)$, with Q(x) the projective quadric associated with q(x); as in remark 2.20, we have 
$\bigwedge^3 V_6\cong (\bigwedge^2 V_5)^\vee\oplus F_x$ and a linear map 
$\lambda_3 \colon (\bigwedge^2 V_5)^\vee\oplus F_x \to \bigwedge^2 V_5$ sending 
$(\phi, x\wedge w)$ to w.
For 
$v\in V_5-\lbrace 0 \rbrace$, we fix a decomposition 
$V_5=\mathbb{C} v \oplus U$. Since 
$F_v=(v\wedge \bigwedge^2 U) \oplus v\wedge U$, we can write
\begin{equation*} \lambda_3(A)=W \quad\quad \lambda_3(F_v)=v\wedge U \quad\quad \ker((P_v)_{|_W})=(v\wedge U)\cap W . \end{equation*} We have 
$A\cap (\bigwedge^2 V_5)^\vee=W^\perp$ by remark 2.21; we can also write 
$F_v\cap (\bigwedge^2 V_5)^\vee =v\wedge \bigwedge^2 U=\ker(P_v)^\perp$ and by (9),
\begin{equation}
\dim(A\cap F_v)=\dim (\lambda_3 (A\cap F_v))+\dim(A\cap F_v\cap (\bigwedge^2 V_5)^\vee).
\end{equation} We have 
$\lambda_3 (A\cap F_v)\subseteq \ker(P_v) \cap W$. The dimension of 
$\ker(P_v) \cap W$ is at most 2, see lemma 3.4, in particular:
(1) If
$\dim(\ker(P_v) \cap W)=2$, then
$\ker(P_v)+W$ has dimension 9, hence there is only one hyperplane containing both of them, so
$\dim(W^\perp\cap \ker(P_v)^\perp)=1$;(2) If
$\dim(\ker(P_v) \cap W)=1$, then
$\ker(P_v)+W=\bigwedge^2 V_5$ and
$W^\perp\cap \ker(P_v)^\perp=0$.
By (11), since 
$A\cap F_v\cap (\bigwedge^2 V_5)^\vee= W^\perp\cap \ker(P_v)^\perp$ in order to obtain 
$\dim(A\cap F_v)=3$, we must consider all the 
$v\in V_5$ for which (1) above holds. For such a v, we have 
$\lambda_3(A\cap F_v)=\lambda_3(A)\cap \lambda_3(F_v)$, so that the RHS of (11) can be 3. The line 
$\ell=\mathbb{P}(\lambda_3(A\cap F_v))$ lies in MS, so we only need to prove 
$\ell\subset Q(x)$. By remark 2.20, we know that 
$q(x)(w,w)=0$ if and only if there exists 
$\eta\in \bigwedge^3 V_5$ such that 
$\eta+x\wedge w\in A$ and 
$\eta\wedge w=0$. Consider any 
$v\wedge u$ inside 
$\lambda_3(A\cap F_v)$: by hypothesis, there exists 
$\eta\in \bigwedge^3 V_5$ such that 
$\eta+x\wedge v\wedge u\in A\cap F_v$. This means that 
$\eta\in F_v$, so 
$\eta\wedge v\wedge u=0$. In particular 
$v\wedge u\in Q(x)$, hence the whole line 
$\mathbb{P}(\lambda_3(A\cap F_v))$ is contained in S.
On the other hand, if 
$\mathbb{P}(L_{v,V_3})\subset S$ for some 
$v\in V_5$, by lemma 3.4, we also have 
$L_{v,V_3}=\ker(P_v)\cap W\cong \mathbb{C}^2$, hence there is some non-zero 
$\eta\in W^\perp\cap \ker(P_v)^\perp$. Then to prove 
$[v]\in Y^3_A\cap \mathbb{P}(V_5)$, we need 
$\lambda_3(A\cap F_v)=\lambda_3(A)\cap \lambda_3(F_v)=W\cap\ker(P_v)$.
Consider 
$v\wedge u\in W\cap\ker(P_v)$. We use (10): under the decomposition 
$(\bigwedge^2 V_5)^\vee \cong W^\perp\oplus W^\vee$, the element
\begin{equation*}\alpha=(0,-q(x)(v\wedge u,-),x\wedge v\wedge u)\in W^\perp\oplus W^\vee \oplus F_x\end{equation*} lies in A and 
$\lambda_3(\alpha)=v\wedge u$. To prove that 
$\alpha\in F_v$, so that we have 
$v\wedge u\in \lambda_3(A\cap F_v)$, we show 
$\alpha\wedge \eta=0$ for every 
$\eta\in F_v$, then we can conclude, since Fv is Lagrangian. As α is zero on the first component, we are left to prove 
$0=\alpha\wedge \eta=-q(x)(v\wedge u,\eta)$ for 
$\eta\in F_v\cap W=L_{v,V_3}$. By hypothesis 
$\mathbb{P}(L_{v,V_3})\subset S\subset Q(x)$: this happens if and only if 
$q(x)_{|_{L_{v,V_3}\times L_{v,V_3}}}=0$ and in particular 
$q(x)(v\wedge u,-)=0$ on 
$L_{v,V_3}$.
If 
$Y^3_{A(S)}-\mathbb{P}(V_5)\neq \emptyset$, then S contains a quintic elliptic pencil. Consider 
$[x]\in Y^3_A - \mathbb{P}(V_5)$, so that S is the transverse intersection of MS and Q(x): by [Reference Debarre and Kuznetsov22, proposition 3.13, Item b], the corank of q(x) is 3.
The quadric Q(x) is a cone of 
$\mathbb{P}(\ker (q(x)))=\mathbb{P}(A\cap F_x)$ over a smooth quadric surface 
$\mathcal{Q}\subset\mathbb{P}^3$; the latter has two families of lines on it and two lines inside 
$\mathcal{Q}$ intersect if and only if they do not lie in the same family. Let 
$\lbrace\ell_t\rbrace_{t\in \mathbb{P}^1}$ be one of the two families. The cone of 
$\mathbb{P}(\ker (q(x)))$ over 
$\ell_t$ is a 
$\mathbb{P}^4$ which we denote by πt. By transversality, the intersection 
$M_S \cap \pi_t = S\cap \pi_t$ is a degree-5 curve on S.
Consider now 
$(\pi_t\cap S)$ and 
$(\pi_{t^{\prime}}\cap S)$ for 
$t\neq t^{\prime}$. We have 
$\pi_t\cap \pi_{t^{\prime}}=\mathbb{P}(\ker (q(x)))$, as 
$\ell_t$ and 
$\ell_{t^{\prime}}$ are in the same family, so 
$(\pi_t\cap S)\cap (\pi_{t^{\prime}}\cap S)=\mathbb{P}(\ker (q(x)))\cap S$. But the latter is empty, since 
$\mathbb{P}(\ker (q(x)))=\operatorname{Sing}(Q(x))$ and S is smooth. So the pencil 
$\lbrace S\cap \pi_t\rbrace_{t\in \mathbb{P}^1}$ is elliptic.
If S contains a quintic elliptic pencil, then 
$Y^3_{A(S)}-\mathbb{P}(V_5)\neq \emptyset$. We call 
$E\in \operatorname{NS}(S)$ the class of the elliptic pencil: the sublattice 
$$\mathbb{Z} H+\mathbb{Z} E=K\subset \operatorname{NS}(S)$$ has Gram matrix 
$$\begin{pmatrix}
10 & 5\\
5 & 0
\end{pmatrix}$$. The orthogonal complement of H in K is generated by 
$\kappa=H-2E$; we denote by κ 2 the corresponding class in 
$\operatorname{NS}(S^{[2]})$.
From now on, we consider notations and results from appendix A. Following that, we call 
$e,f$ two canonical generators of a copy of U; moreover, 
$e_1,f_1$ will be the canonical generators of a second copy of U, orthogonal to the first one. By Eichler Criterion [Reference Gritsenko, Hulek and Sankaran33, proposition 3.3], there is an isometry 
$\psi:H^2(S^{[2]},\mathbb{Z})\to \Lambda$ such that 
$\psi(H_2-2\delta)=h=e+f$ and 
$\psi(\kappa_2)=g+2\ell-2e_1-8f_1$; the involution j on the moduli space exchanges g and 
$\ell$ in 
$\Lambda_h$, so that 
$j(\kappa_2)=g+2\ell-2e_1-8f_1$ is an algebraic class for 
$X_{A(S)}$, which lies in 
$D_A^\perp$. It has square 10 and divisibility 2 in 
$H^2(X_{A(S)},\mathbb{Z})$, so DA lies in a flopping wall and is not ample, see [Reference Debarre and Macrì28, theorem 5.1, Item b].
Hence 
$Y_{A(S)}^3\neq \emptyset$ and more precisely 
$Y_{A(S)}^3 - \mathbb{P}(V_5)\neq \emptyset$, since points in 
$Y_{A(S)}^3 \cap \mathbb{P}(V_5)$ correspond to lines in S, hence sublattices in 
$\operatorname{NS}(S)$ whose Gram matrix is 
$$\begin{pmatrix}
10 & 1\\
1 & -2
\end{pmatrix}$$. It is then sufficent to observe that such a sublattice is neither an overlattice nor a sublattice of 
$H\mathbb{Z}+E\mathbb{Z}$.
We can now prove the second version of the characterization.
Proof of theorem 3.3
For (1) see [Reference Greer, Li and Tian9, lemma 2.8]. We do not need to ask 
$(S,H)\notin D_{5,2}$, as in [Reference Greer, Li and Tian9], since this is a divisor inside the complement of 
$$\mathcal{K}_{10}$$ in its closure; to see that, one checks that the orthogonal complement of H in 
$\mathbb{Z} H+\mathbb{Z} D$ has square −2 and concludes by [Reference Gritsenko, Hulek and Sankaran35, theorem 2.7]. For (2) see [Reference Greer, Li and Tian9, lemma 2.7]. The rest of the statement comes directly from theorem 3.2. Lines on 
$S=\mathbb{P}(W)\cap G2,V_5)\cap Q$ correspond to sublattices of 
$\operatorname{NS}(S)$ whose Gram matrix is 
$$\left[\begin{array}{cc}
10 & 1 \\
1 & -2
\end{array}\right]$$, To prove (3), it is then sufficient to show that this lattice admits no non-trivial overlattice. For (4), the proof is the same, with the lattice whose Gram matrix is 
$$\left[\begin{array}{cc}
10 & 5 \\
5 & 0
\end{array}\right]$$.
Finally, the description of lines in S can be made more precise.
Proposition 3.5.
Let 
$S= \mathbb{P}(W) \cap G(2,V_5) \cap Q$ be a strongly smooth 
$K3$ surface. Let A(S) be the associated Lagrangian subspace. Then
\begin{equation*} |Y^3_{A(S)}\cap \mathbb{P}(V_5)|= |\lbrace \ell \subset S| \ell\ \mbox{line}\rbrace|. \end{equation*}Proof. This is an easy consequence of the first part of the proof above. To show the one-to-one correspondence, one observes that two lines 
$\mathbb{P}(L_{v,V_3})$ and 
$\mathbb{P}(L_{v^{\prime},V^{\prime}_3})$ intersect in at most one point if 
$[v]\neq [v^{\prime}]$.
4. Inducing automorphisms on double EPW sextics
We explain here how we can use our main result, and particularly theorem 3.3, to deduce the existence of automorphisms on families of smooth double EPW sextics. This was actually the first motivation for our study of strongly smooth GM surfaces, since the automorphisms of S act on the associated EPW sextic.
Proposition 4.1. Debarre and Kuznetsov
For Z a strongly smooth GM variety, there is a natural inclusion of 
$\operatorname{Aut}(Z,\mathbb{P}(W))$ in 
$\operatorname{Aut}(Y_{A(Z)})\cong \operatorname{Aut}(Y_{A(Z)^\perp})$ as the stabilizer of 
$[V_5]\in Y_{A(Z)^\perp}$.
Proof. See [Reference Debarre and Kuznetsov22, proposition 3.21, Item c]: the result still holds if we replace ‘dimension 
$\geq 3$’ in the statement with ‘X strongly smooth’.
O’Grady showed that, for 
$S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ very general, the associated double EPW is smooth; however, the group 
$\operatorname{Aut}(S,\mathbb{P}(W))$ is trivial for S very general! In addition, all those automorphisms turn out to be symplectic, see propositions 2.10 and 5.2, and families of 
$K3$ surfaces in 
$\mathcal{K}_{10}$ carrying symplectic actions typically have big codimension.
So, to find automorphisms on some S that induce automorphisms on a smooth double EPW sextic, the tricky part is to control the generality. By theorem 3.3, we can easily deduce the existence of automorphisms on some smooth double EPW sextic, appealing to the classification of automorphisms on 
$K3$ surfaces. Let 
$(V_6,V_5,A)$ be the Lagrangian data associated with S: by theorem 2.17, 
$[V_5]\in Y^3_{A^\perp}$, hence A always lies in Π.
In the following statement, we denote by Dn the dihedral group of order 2n. In the proof, we use some results from the next section; we decided to postpone them to keep the exposition more cohesive.
Proposition 4.2.
Let G be one of the following groups,
\begin{equation*}
\mathbb{Z}/n\mathbb{Z} \,\mbox{for }n\in\lbrace 2,3,4 \rbrace,\quad (\mathbb{Z}/2\mathbb{Z})^2, \quad D_n\,\mbox{for }n\in\lbrace 4,5,6\rbrace.
\end{equation*} There is a family of Lagrangian subspaces in 
$\Pi - (\Sigma \cup \Delta)$ such that, for any A in the family, the associated double EPW sextic XA is smooth and admits a symplectic action of G which commutes with the covering involution.
For a general element of the family, 
$\operatorname{Aut}_{D_{A}}(X_{A})=\iota_{A}\times G$.
Proof. A 
$\langle 10 \rangle$-polarized 
$K3$ surface (S, H) admits a symplectic action of G which fixes H if and only if its Néron-Severi group contains 
$\mathcal{L}_G=\mathbb{Z} H\oplus \Omega_G$, where 
$\Omega_G$ is a negative-definite lattice associated with G, which turns out to be the coinvariant lattice for the action of G in cohomology. This has been proved by Nikulin [Reference Nikulin31], but see also [Reference Garbagnati and Sarti1, proposition 6.3].
We start by considering 
$G=\mathbb{Z}/n\mathbb{Z}$ with 
$n=2,\ldots 6$, 
$(\mathbb{Z}/2\mathbb{Z})^2$ or D 4 and we want to prove that there is a family of 
$K3$ surfaces for which the very general element S has 
$\operatorname{NS}(S)=\mathcal{L}_G$. Then we will have 
$G=\operatorname{Aut}(S,\mathbb{P}(W))$ for 
$G\neq \mathbb{Z}/n\mathbb{Z}$ with 
$n=5,6$, and 
$\operatorname{Aut}(S,\mathbb{P}(W))=D_n$ otherwise: in the latter case, we appeal to [Reference Garbagnati2, propositions 8.1 and 9.1] and then proposition 5.2, in the former simply to proposition 5.2.
By the surjectivity of the period map for marked 
$K3$ surfaces, proving the existence of the family amounts to showing that there is a primitive embedding 
$\mathcal{L}_G$ in 
$\Lambda_{K3}=U^{\oplus 3}\oplus E_8(-1)^{\oplus 2}$. By [Reference Garbagnati2, propositions 6.2 and 7.7], there is a family of elliptic 
$K3$ surfaces whose Néron–Severi group is 
$U\oplus \Omega_G$. Now, any embedding of U in an even lattice M induces a splitting 
$M=U\oplus U^\perp$; in particular, in our case, we have 
$\Omega_G\subset U^{\oplus 2}\oplus E_8(-1)^{\oplus 2}\subset \Lambda_{K3}$. We fix a canonical basis 
$\lbrace e,f \rbrace$ for the distinguished copy of U and we consider the lattice 
$\mathbb{Z}(e+5f)\oplus \Omega_G$, which is a copy of 
$\mathcal{L}_G$ and is primitive in 
$\Lambda_{K3}$, since the two summands lies in two different direct summands of the big lattice. This give us a maximal family of 
$K3$ surfaces with a symplectic action of G.
For the very general element (S, H) in the family, the 
$\langle 10 \rangle$-polarization has divisibility 10 in the Néron–Severi group; in particular, (S, H) is Brill–Noether general and strongly smooth by theorem 3.3. Since these properties fail on a finite number of divisors in the moduli space, this means that also the general element in the family enjoys them (these families have positive dimension). For the same lattice-theoretic reasons, (S, H) will not lie in 
$D_{h,0}$ for 
$h=4,5$ nor 
$D_{1,-2}$, so its associated double EPW sextic is smooth and carries a symplectic action of G fixing the polarization DA, by propositions 4.1 and 2.10.
Proposition 5.5 ensures that the group of automorphisms fixing DA on the associated EPW sextic is isomorphic to 
$\operatorname{Aut}(S,\mathbb{P}(W))$ for a general (S, H), since in that case the projective model does not contain any line or conic.
5. Bounds on automorphisms of GM varieties
5.1. Bounds for Brill–Noether general K3 surfaces
In the next two sections, we provide some bounds on the automorphism group of GM varieties. Grassmannian hulls will play a fundamental role and the key point is the following observation.
Lemma 5.1.
Let Z be a strongly smooth GM variety with Grassmannian hull MZ. Then 
$\operatorname{Aut}(Z,\mathbb{P}(W))\subset \operatorname{Aut}(M_Z)$.
Proof. Since S spans 
$\mathbb{P}(W)$, the restriction 
$\rho\colon\lbrace \alpha \in \operatorname{Aut}(\mathbb{P}(W)) | \alpha(S)=S\rbrace\to \operatorname{Aut}(S,\mathbb{P}(W))$ is an isomorphism of groups. By [Reference Debarre and Kuznetsov22, corollary 2.11], any 
$\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))$ is induced by an automorphism 
$(g_W,g_{V},g_L)$ of ordinary GM data, where the isomorphism 
$[g_W]\in \operatorname{PGL}(W\otimes L)=\operatorname{PGL}(W)$ is the restriction of 
$\left[\bigwedge^2 \left({g_V}_{|_{V_5}}\right)\right]$ to 
$\mathbb{P}(W)$. Hence ρ factors as 
$\lbrace \alpha \in \operatorname{Aut}(\mathbb{P}(W)) | \alpha(S)=S\rbrace\xrightarrow{\rho_1} \operatorname{Aut}(M_S) \xrightarrow{\rho_2} \operatorname{Aut}(S,\mathbb{P}(W))$. Since ρ is injective, the same holds for ρ 1.
Let 
$S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a Brill–Noether general 
$K3$ surface. The automorphism group 
$\operatorname{Aut}(S,\mathbb{P}(W))$ is the group of the automorphisms of the abstract surface S fixing H; this group is finite, see [Reference Huybrechts6, chapter 5, proposition 3.3].
O’Grady [Reference O’Grady19] studied in detail the six-dimensional complete linear system 
$|H_2-2\delta|$. This linear system is naturally isomorphic to the space of quadrics in 
$\mathbb{P}(W)$ containing S and the associated map admits a very explicit description ([Reference O’Grady19, (4.2.5)]),
\begin{align}
\phi\colon S^{[2]} & \dashrightarrow |H_2-2\delta|^\vee\cong \mathbb{P}^5
\end{align}
\begin{align}
Z & \mapsto \lbrace \mathcal{Q}\in |V_6| \mbox{such that } \langle Z \rangle\subset \mathcal{Q} \rbrace .
\end{align} The map ϕ factors as a small contraction 
$c\colon S^{[2]}\to X_{A(S)^\perp}$ and 
$f\colon X_{A(S)^\perp}\dashrightarrow \mathbb{P}^5$. The latter can be identified to the double cover 
$f_{A(S)\perp}$ up to a finite number of flops [Reference O’Grady19, theorem 4.15]; in particular, it is generically 2-to-1 and induces a birational involution ι on 
$S^{[2]}$ which acts as minus the reflection with respect to the class 
$H_2-2\delta$ in cohomology [Reference O’Grady19, proposition 4.20]. The birational transformation c induces an identification of 
$\operatorname{Aut}_{D_{A(S)^\perp}}(X_{A(S)^\perp})$ with 
$\operatorname{Bir}_{H_2-2\delta}(S^{[2]})$, the group of birational endomorphisms fixing 
$H_2-2\delta$ and (5) translates as
\begin{equation*} 1\to \lbrace \operatorname{id},\iota \rbrace\to \operatorname{Bir}_{H_2-2\delta}(S^{[2]}) \xrightarrow{\lambda} \operatorname{Aut}(Y_{A(S)^\perp}) \to 1. \end{equation*} We denote by 
$\mathfrak{G}_n$ the symmetric group on n objects and by 
$\mathfrak{A}_n$ the alternating group on n objects.
Proposition 5.2.
If S is strongly smooth, 
$\operatorname{Aut}(S,\mathbb{P}(W))$ acts symplectically on S and is isomorphic to one of the following groups:
\begin{equation}
\begin{gathered}
\mathbb{Z}/n\mathbb{Z} \,\mbox{for }n= 1,2,3,4, \quad \quad D_n \,\mbox{for } n= 2,\ldots,6 \\
\mathfrak{A}_4, \quad \quad \quad \mathfrak{G}_4,\quad \quad \quad \mathfrak{A}_5.\quad \quad
\end{gathered}
\end{equation}Proof. We prove simplecticity. We know that (5) splits, identifying the lifting of 
$\operatorname{Aut}(Y_{A(S)^\perp})$, which we denote by 
$\mathcal{S}$, with the group of automorphisms acting trivially on 
$H^{0,2}(X_{A(S)^\perp})$. Since c is an isomorphism in codimension one, 
$\mathcal{S}$ seen as a subgroup of 
$\operatorname{Bir}_{H_2-2\delta}(S^{[2]})$ acts symplectically on 
$S^{[2]}$. Inside 
$\mathcal{S}$ there is G, the lifting of 
$\operatorname{Aut}(S,\mathbb{P}(W))\subset \operatorname{Aut}(Y_{A(S)^\perp})$. We denote by 
$G^{\prime}\subset \operatorname{Bir}_{H_2-2\delta}(S^{[2]})$ the group of natural automorphisms 
$\alpha^{[2]}$ on 
$S^{[2]}$ induced by 
$\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))$: if we can prove 
$G=G^{\prime}$ we are done, since 
$\alpha^{[2]}$ is symplectic if and only if α is.
Fix some 
$\alpha\in \operatorname{Aut}(S,\mathbb{P}(W))\subset \operatorname{Aut}(Y_{A(S)^\perp})$. A straightforward computation, using (12) and (7), gives 
$\phi\circ\alpha^{[2]}=\alpha\circ \phi$, hence 
$\lambda(G)=\lambda(G^{\prime})$. For 
$\tilde{\alpha}\in G$ the lifting of α, we have then 
$\tilde{\alpha}=\alpha^{[2]}$ or 
$\tilde{\alpha}=\alpha^{[2]}\circ \iota$. Note that 
$\alpha^{[2]}$ fixes the class H 2, while ι only fixes 
$H_2-2\delta$ and its multiples [Reference O’Grady15, proposition 4.21 (b)], so the first equality holds if and only if 
$\tilde{\alpha}$ fixes H 2.
We turn to 
$|H_2|$, as a linear system on 
$X_{A(S)^\perp}$, via the small contraction c; it is the pullback via the Hilbert–Chow contraction of 
$H^{\otimes 2}$ on the symmetric product 
$S^{(2)}$,
\begin{equation*} H^0(S^{[2]},H_2)=H^0(S^{(2)},H^{\otimes 2})=\operatorname{Sym}^2 W^\vee.\end{equation*} The action of the lifting 
$\tilde{\alpha}$ of α to 
$X_{A(S)^\perp}$ is induced by the action of 
$SL(V_6)$ [Reference Debarre and Kuznetsov25, proof of proposition A.2]; thus, 
$\tilde{\alpha}$ acts on the global sections of H 2 since it induces a linear automorphism of W, see Eq. (7). So 
$\tilde{\alpha}^*H_2=H_2$, which provides us 
$\tilde{\alpha}=\alpha^{[2]}$ as desired.
To obtain the bound we observe that, by lemma 5.1, G is a finite subgroup of 
$\operatorname{Aut}(M_S)$ and the latter is isomorphic to 
$\operatorname{PGL}(2,\mathbb{C})$ [Reference Piontkowski and Van de Ven14, theorem 7.5], since the strongly smoothness of S implies the condition of generality for MS in [Reference Piontkowski and Van de Ven14], cf. [Reference Debarre and Kuznetsov22, proposition 2.22]. So G is either cyclic, dihedral, 
$\mathfrak{A}_4$, 
$\mathfrak{G}_4$, or 
$\mathfrak{A}_5$. For cyclic groups, the order is at most 8 by [Reference Nikulin31, theorem 4.5] and actually when the order is 
$n=7,8$ the lattice 
$\Omega_G^\perp$ (same notation as in proposition 4.2) does not contain an element of square 10, as a computation modulo 2n shows, so we can rule them out. Moreover, if G contains an order 
$n=5,6$ element, then 
$D_n\subset G$ by [Reference Garbagnati2, proposition 8.1]. Finally, 
$n\leq 6$ for Dn by [Reference Xiao10].
Remark 5.3.
In proposition 4.2, we provide some families of 
$K3$ surfaces with prescribed 
$\operatorname{Aut}(S,\mathbb{P}(W))$. According to proposition 5.2, the possible cases left are 
$\operatorname{Aut}(S,\mathbb{P}(W))\in \lbrace D_3,\mathfrak{A}_4,\mathfrak{G}_4,\mathfrak{A}_5 \rbrace$.
When S contains no line, 
$\phi:S^{[2]}\dashrightarrow |H_2-2\delta|$ behaves particularly well. Let 
$\mathcal{C}$ be the set of smooth conics inside S: by [Reference O’Grady19, claim 4.19], 
$|H_2-2\delta|$ is base-point-free and
\begin{equation}
Y^3_{A(S)^\perp}=\bigcup_{C\in \mathcal{C}} \phi(C^{(2)}).
\end{equation} When moreover S is strongly smooth, we are able to provide a bound for the automorphism group of the associated EPW sextic 
$Y_{A(S)}$, which is particularly useful when there is no conic on S; we used proposition 5.5 under this condition in the proof of proposition 4.2, in the previous section.
Definition 5.4.
We denote by 
$\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))\leq \operatorname{Aut}(S,\mathbb{P}(W))$ the subgroup of automorphisms acting trivially on 
$\mathcal{C}$.
Clearly 
$\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))$ appears in (14).
Proposition 5.5.
Let 
$S=\mathbb{P}(W)\cap G(2,V_5)\cap Q$ be a strongly smooth Brill–Noether general 
$K3$ surface with N smooth conics on it. Suppose that S contains no line: the automorphism group of 
$Y_{A(S)}$ sits in an exact sequence
\begin{equation}
1\to \operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W)) \to \operatorname{Aut}(Y_{A(S)}) \xrightarrow{g} \mathfrak{G}_{N+1}
\end{equation} where g sends 
$\alpha\in \operatorname{Aut}(Y_{A(S)})$ to its action on 
$Y^3_{{A(S)}^\perp}$, whose cardinality is N + 1.
Proof. The morphism g in (16) is well-defined by corollary 2.6 and its kernel is the subgroup of automorphisms of YA fixing pointwise 
$Y^3_{A^\perp}$. We are only left to prove that 
$\operatorname{Aut}_\mathcal{C}(S,\mathbb{P}(W))=\ker(g)$, we actually prove a bit more. Indeed, by (15), the fibres over 
$Y^3_{A(S)^\perp}$ are given by PS and 
$C^{(2)}$ for 
$C\in \mathcal{C}$; by the same computations of proposition 5.2, the action of α on 
$\mathcal{C}$ induces a permutation on 
$\lbrace P_S,C^{(2)}_1, \ldots , C^{(2)}_N \rbrace$ hence on 
$Y^3_{A(S)^\perp}$, which is the same as the one induced by α as an element of 
$\operatorname{Aut}(Y_{A(S)})$ (note that, since α acts on MS too by lemma 5.1, 
$\alpha^{[2]}$ always fixes PS).
Remark 5.6.
The morphism g in (16) is not necessarily surjective. Indeed, let 
$(S,H)\in\mathcal{K}_{10}$ be a very general element of 
$D_{2,-2}$ (see definition 3.1). Then S contains exactly one smooth conic, say C. An involution of 
$Y_{A(S)}$ exchanging the two elements inside 
$Y^3_{A(S)^\perp}$ would lift to a symplectic involution on 
$X_{A(S)}$ (which is smooth for (S, H) general, see theorem 3.3), but then 
$X_{A(S)}$ would have Picard rank at least 9 by [Reference Mongardi12, corollary 5.2]. This is impossible since 
$X_{A(S)}$ and 
$S^{[2]}$ have the same Picard rank, cf. lemma A.4.
Furthermore, there exists 
$(S^{\prime},H^{\prime})\in D_{2,-2}$ such that 
$(S^{[2]}, H_2-2\delta)$ and 
$((S^{\prime})^{[2]},$ 
$H^{\prime}_2-2\delta^{\prime})$ are birational as varieties with a big and nef line bundle or equivalently 
$A(S)=A(S^{\prime})$ and 
$P_{S^{\prime}}$ is sent to 
$\phi(C^{(2)})$ via the map associated with 
$|H^{\prime}-2\delta^{\prime}|$. The argument above proves that (S, H) and 
$(S^{\prime},H^{\prime})$ are not isomorphic.
5.2. Bounds in greater dimension
In §5.1, we dealt with two-dimensional GM varieties. Now we consider GM varieties of dimension 3 and 4. By [Reference Debarre and Kuznetsov22, proposition 3.21], their automorphism group is finite and trivial in the general case.
Remark 5.7.
By proposition 4.1, whenever the associated Lagrangian A(Z) does not lie in 
$\Delta\cap \Pi$, automorphisms of prime order of Z have order at most 11, because the automorphism lifts to 
$X_{A(Z)}$ and 
$X_{A(Z)^\perp}$, of which at least one is smooth, and a symplectic automorphism of prime order on a deformation of a Hilbert scheme of two points has order at most 11, see [Reference Mongardi11, corollary 2.13].
Proposition 5.8.
Let Z be a smooth ordinary GM variety of dimension 3 and 
$G\subset \operatorname{Aut}(Z)$. Then there exists a group H among
\begin{equation}
1, \quad \quad A_4, \quad \quad \mathfrak{G}_4, \quad \quad A_5, \quad \quad \mathbb{Z}/k\mathbb{Z} \quad\mbox{or} \quad D_k \quad\mbox{for }k\geq 2
\end{equation} and 
$r\geq 1$ such that G sits in an exact sequence 
$1 \to \mathbb{Z}/r\mathbb{Z} \to G \to H \to 1 $.
Proof. The Grassmannian hull MZ is smooth since Z is three-dimensional and 
$\operatorname{Aut}(Z,\mathbb{P}(W))=\operatorname{Aut}(Z)$ since Z has Picard rank one. By [Reference Piontkowski and Van de Ven14, theorem 6.6], there is a short exact sequence
\begin{equation}
1\to \mathbb{C}^4 \rtimes \mathbb{C}^* \to \operatorname{Aut}(M_Z)\xrightarrow{\pi} \operatorname{PGL}(2,\mathbb{C}) \to 1 ,
\end{equation} thus G is an extension of 
$H=\pi(G)$ by 
$N=G\cap (\mathbb{C}^4 \rtimes \mathbb{C}^*)$. The group G is finite by lemma 5.1, so 
$H\subset \operatorname{PGL}(2,\mathbb{C})$ has to appear in (17). As for N, it is finite inside 
$\mathbb{C}^4 \rtimes \mathbb{C}^*$, so it is isomorphic to a finite subgroup of 
$\mathbb{C}^*$.
Remark 5.9.
In [Reference Debarre and Mongardi29], Debarre and Mongardi produced an explicit example of EPW sextic with automorphism group isomorphic to 
$\operatorname{PSL}(2,\mathbb{F}_{11})$. This allows them to find examples of smooth ordinary GM threefolds with automorphism group isomorphic to 
$\mathbb{Z}/11\mathbb{Z}$, D 6, 
$\mathbb{Z}/6\mathbb{Z}$, 
$\mathbb{Z}/3\mathbb{Z}$, D 5, D 2, 
$\mathbb{Z}/2\mathbb{Z}$.
The spin group 
$\operatorname{Spin}(n)$, for 
$n\geq 2$, is the universal cover of the special orthogonal group 
$\operatorname{SO}(n)$. It is possible to study finite subgroups of 
$\operatorname{Spin}(5)$ using this definition: finite subgroups of 
$\operatorname{SO}(5)$ have been classified, see [Reference Mecchia and Zimmermann21, corollary 2].
The quotient by 
$\lbrace \pm 1\rbrace$ in the result below comes from the fact that, for any 
$$\alpha\in \operatorname{PGL}(V_5)$$ acting on the Grassmannian hull MZ, a representative in 
$$\operatorname{GL}(V_5)$$ of the form 
$$\begin{pmatrix}
\bar{T} & \underline{0}\\
{^t\underline{b}} & b
\end{pmatrix}$$ is chosen, such that 
$$\operatorname{det}(\bar{T})=1$$. A representative in this form for α is unique, up to multiplication by −1. See [Reference Piontkowski and Van de Ven14, proposition 5.2] for details.
Proposition 5.10.
Let Z be a smooth ordinary GM variety of dimension 4 and 
$G\subset \operatorname{Aut}(Z)$. Then G is the quotient by 
$\lbrace \pm 1\rbrace$ of a group 
$\tilde{G}$ such that there is an exact sequence
\begin{equation*} 1\to \mathbb{Z}/r\mathbb{Z} \to \tilde{G} \to H \to 1 \end{equation*} for some 
$r\geq 1$ and H a finite subgroup inside 
$\operatorname{Spin}(5)$.
Proof. As in proposition 5.8, we just have to impose finiteness for a subgroup of 
$\operatorname{Aut}(M_Z)$. By [Reference Piontkowski and Van de Ven14, proposition 5.2], the group G is an extension of a subgroup of 
$\left( \operatorname{Sp}(4,\mathbb{C})\times \mathbb{C}^* \right) /\lbrace \pm 1\rbrace$ by some 
$N\subset \mathbb{C}^4$. Actually 
$N=\lbrace 1 \rbrace$ as G is finite, hence G is isomorphic to the quotient of a finite subgroup of 
$\operatorname{Sp}(4,\mathbb{C})\times \mathbb{C}^*$. Any finite subgroup of 
$\operatorname{Sp}(4,\mathbb{C})$ lies inside its maximal compact subgroup, which is the compact symplectic group 
$\operatorname{Sp}(2)$. In turn, 
$\operatorname{Sp}(2)$ is isomorphic to 
$\operatorname{Spin}(5)$.
Remark 5.11.
In [Reference Debarre and Mongardi29], Debarre and Mongardi produced smooth ordinary GM fourfolds with automorphism group isomorphic to D 6, 
$\mathbb{Z}/3\mathbb{Z}$, D 5, 
$\mathbb{Z}/5\mathbb{Z}$, D 2, 
$\mathbb{Z}/2\mathbb{Z}$.
Corollary 5.12.
Consider 
$A\in \operatorname{\mathbb{LG}(\bigwedge^3 V_6)} - \Sigma$ and 
$G\subseteq \operatorname{Aut}(Y_A)$. Suppose that G fixes a point 
$[v]\in Y^k_A$ or a point 
$[V_5]\in Y^k_{A^\perp}$ for 
$k\in \lbrace1,2\rbrace$.
(1) If k = 2, then G sits in an exact sequence
$1 \to \mathbb{Z}/r\mathbb{Z} \to G \to H \to 1 $ for some
$r\geq 1$ and H a group in (5.8).(2) If k = 1, then G is the quotient by
$\lbrace \pm 1\rbrace$ of a group
$\tilde{G}$, where
$\tilde{G}$ is an extension of some finite subgroup of
$\operatorname{Spin}(5)$ by a cyclic group
$\mathbb{Z}/r\mathbb{Z}$.
Proof. We consider the case of G fixing 
$[V_5]\in Y^k_{A^\perp}$, the dual case follows by corollary 2.6. We denote by Z the ordinary GM variety associated with the Lagrangian data 
$(V_6,V_5,A)$: by theorem 2.17, it is a strongly smooth GM variety of dimension 
$5-k$. By proposition 4.1, there is an inclusion 
$G\subset \operatorname{Aut}(Z,\mathbb{P}(W))$, then the result follows from propositions 5.8 and 5.10.
If the associated Lagrangian A(Z) lies in Δ or Σ we can be more precise.
Definition 5.13.
A plane 
$\pi\subset G(2,V_5)$ is a σ-plane if it is of the form 
$\mathbb{P}(v\wedge V_4)$ for some four-dimensional vector space 
$V_4\subset V_5$ and 
$v\in V_4-\lbrace 0\rbrace$. A τ-quadric surface in 
$G(2,V_5)$ is a linear section of 
$G(2,V_4)$ for some four-dimensional 
$V_4\subset V_5$.
Corollary 5.14.
Let Z be a smooth ordinary GM variety of dimension 4 and associated Lagrangian subspace A(Z). If one of the following holds:
• Z contains a σ-plane,
• Z contains a τ-quadric surface,
then there is an exact sequence 
$1\to N \to \operatorname{Aut}(Z) \to \mathfrak{G}_{n} $, where 
$n=|Y^3_{A(Z)}|$ (resp. 
$n=|Y^3_{A(Z)^\perp}|$) if Z contains a σ-plane (resp. a τ-quadric surface), and N is one of the groups in the list at (14).
Proof. By [Reference Debarre and Kuznetsov24, theorem 4.3, Item c] for σ-planes, [Reference Debarre and Kuznetsov24, remark 5.29] and [Reference Debarre, Iliev and Manivel27, Section 7.3] for τ-quadric surfaces, the associated Lagrangian subspace A(Z) lies in Δ or Π. So there is a strongly smooth Brill–Noether general 
$K3$ surface whose associated Lagrangian subspace is A(Z). Then we use proposition 5.5.
The coarse moduli space of smooth GM varieties of dimension 4 is constructed in [Reference Debarre and Kuznetsov26]. Inside it, the family of ordinary GM varieties containing a σ-plane (resp. a τ-quadric surface) has codimension 2 (resp. 1), see [Reference Debarre and Kuznetsov24, remark 5.29]; the general members of both families are rational, see [Reference Debarre, Iliev and Manivel27, propositions 7.1 and 7.4].
Acknowledgements
Most of the results of this article are part of my PhD thesis. I am grateful to Alessandra Sarti for her constant support, for the supervision of this work and for many useful discussions. I would like to thank Giovanni Mongardi, Enrico Fatighenti, and Simone Novario for many interesting discussions. Finally, I would also like to thank the anonymous referee for many helpful comments and suggestions.
Funding
The author is supported by the ERC Synergy Grant ERC-2020-SyG-854361-HyperK.
Appendix A. An involution of the moduli space
We collect some results which are useful for the proof of theorem 3.2, but they are different in flavour from the rest of the article; for details about the construction in this section we refer to [Reference Debarre and Macrì28, Section 3.2].
We denote by 
$\Lambda=U^{\oplus 3}\oplus E_8(-1)^{\oplus 2}\oplus \langle -2 \rangle$ the abstract lattice isomorphic to 
$H^2(X,\mathbb{Z})$ for X equivalent by deformation to the Hilbert square on a K3 surface. We fix a copy of the hyperbolic plane 
$U \subset \Lambda$, with canonical basis 
$\lbrace e,f\rbrace$.
We take 
$h=e+f$ and we call 
$$\Lambda_h\subset \Lambda$$ the orthogonal complement of h. The group of isometries of Λ fixing h acts on 
$\Lambda_h$ as 
$\tilde{O}(\Lambda_h)=\lbrace \theta\in O(\Lambda_h) $ 
$|$ 
$\bar{\theta}=\operatorname{id}_{A_{\Lambda_h}} \rbrace$, cf. [Reference Gritsenko, Hulek and Sankaran34, proposition 3.12, Item i]. We denote 
$\mathcal{Q}_2=\lbrace [x]\in\mathbb{P}(\Lambda_h\otimes\mathbb{C})|x^2=0,(x,\bar{x}) \gt 0\rbrace$ and 
$\mathcal{P}_2= \mathcal{Q}_2/\tilde{O}(\Lambda_h)$.
Definition A.1.
We denote by 
$\mathcal{M}_2$ the coarse moduli space of 
$\langle 2 \rangle$-polarized hyper-Kähler manifolds equivalent by deformation to the Hilbert square on a K3 surface.
The moduli space 
$\mathcal{M}_2$ has dimension 20 and is irreducible; it comes with a period map 
$\mathcal{M}_2\to \mathcal{P}_2$ which is an open embedding, see [Reference Markman8, theorem 1.10].
There is an open embedding 
$\operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0// \operatorname{PGL}(6) \hookrightarrow \mathcal{M}_2$, cf. [Reference O’Grady20, proposition 1.2.1], which induces a period map 
$\operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0// \operatorname{PGL}(6)\to \mathcal{P}_2$.
We call 
$$g=e-f$$ the orthogonal complement of h in the fixed copy of U and 
$\ell$ a generator of the 
$\langle -2 \rangle$-part of Λ: we have 
$\Lambda_h=U^{\oplus 2}\oplus E_8(-1)^{\oplus 2}\oplus \mathbb{Z} g\oplus \mathbb{Z}\ell$.
Definition A.2.
We denote by 
$j\in O(\Lambda_h)$ the isometry exchanging g and 
$\ell$. For simplicity, we also call j the induced involution on 
$\mathcal{P}_2$.
The discriminant of 
$\Lambda_h$ is isomorphic to 
$\mathbb{Z}/2\mathbb{Z}\times \mathbb{Z}/2\mathbb{Z}$ as a group. We identify the class of 
$g/2$ as 
$(1,0)$ and the class of 
$\ell/2$ as 
$(0,1)$. Since 
$\bar{j}((1,0))=(0,1)$, by [Reference Nikulin32, corollary 1.5.2] the involution j does not extend to an isometry of Λ.
Since the period map is an open embedding, the involution j induces a birational involution on 
$\mathcal{M}_2$.
Theorem A.3. [Reference O’Grady17], theorem 1.1
The involution induced on 
$\mathcal{M}_2$ by j is the birational involution that sends 
$(X_A,D_A)$ to 
$(X_{A^\perp},D_{A^\perp})$.
The period map 
$(\operatorname{\mathbb{LG}(\bigwedge^3 V_6)}-\Sigma )\to \mathcal{P}_2$ commutes with j and the involution on 
$(\operatorname{\mathbb{LG}(\bigwedge^3 V_6)}-\Sigma )//\operatorname{PGL}(6)$ sending 
$[A]$ to 
$[A^\perp]$, see [Reference O’Grady16] and [Reference O’Grady19].
We consider now 
$A\in \Pi - (\Delta \cup \Sigma)$ and we fix 
$[V_5]\in Y^3_{A^\perp}$. Let S be the GM variety associated with 
$(V_6,V_5,A)$; we focus on it as the 
$\langle 10\rangle$-polarized 
$K3$ surface (S, H), with H the polarization associated with the embedding in 
$\mathbb{P}(W)$. The line bundle 
$H_2-2\delta$ is big and nef but not ample for a general choice of A.
The period map 
$\mathcal{M}_2\to \mathcal{P}_2$ extends to a surjective map 
$\mathcal{M}^\circ_2\to \mathcal{P}_2$, where 
$\mathcal{M}^\circ_2$ is the moduli space of hyper-Kähler manifolds with a big and nef divisor whose square is 2. Similarly, the period map on 
$\operatorname{\mathbb{LG}(\bigwedge^3 V_6)}^0// \operatorname{PGL}(6)$ extends to a period map 
$(\operatorname{\mathbb{LG}(\bigwedge^3 V_6)} - \Sigma)// \operatorname{PGL}(6)\to \mathcal{P}_2$. The period point of 
$[A]$ is the image of the period point of 
$(S^{[2]}, H_2-2\delta)$ via the involution j, cf. [Reference O’Grady19] (but beware, A is actually the dual of the Lagrangian subspace that O’Grady called A in his article).
As an easy consequence, it is possible to compute the Néron–Severi of XA from the one of S.
Lemma A.4.
The involution j induces an isometry 
$\operatorname{NS}(S^{[2]})\cap (H_2-2\delta)^\perp\cong \operatorname{NS}(X_A)\cap D_A^\perp$, in particular the Néron–Severi groups of XA and 
$S^{[2]}$ have same rank.
