1.1 Overview

It is usually hard to calculate with the virtual fundamental class $[M]^{\operatorname {vir}}$ of a moduli space $M$. In the rare situation that $M$ (and its perfect obstruction theory) is cut out of a smooth ambient space $\iota \colon M\hookrightarrow X$ by a section of a vector bundle $E$, the pushforward of the virtual cycle $\iota _*[M]^{\operatorname {vir}}=e(E)$ is the Euler class of $E$, making calculation possible.

In [Reference Gholampour and ThomasGT20] we described a generalisation to the case where $M$ is instead the deepest degeneracy locus of a map of vector bundles on a smooth ambient space. This we applied to nested Hilbert schemes of points on smooth surfaces, reproducing the virtual cycles of [Reference Gholampour, Sheshmani and YauGSY20a], which agree with those arising in Vafa–Witten theory [Reference Tanaka and ThomasTT20] or reduced local DT theory [Reference Gholampour, Sheshmani and YauGSY20b].

That technique does not extend to other degeneracy loci, but it does apply to their resolutions (1.2). We apply this to nested Hilbert schemes of points and curves on smooth surfaces with their virtual cycles [Reference Gholampour, Sheshmani and YauGSY20a] coming from Vafa–Witten theory [Reference Tanaka and ThomasTT20] or reduced DT theory [Reference Gholampour, Sheshmani and YauGSY20b]. The result is an effective new tool for computing sheaf-theoretic enumerative invariants of projective surfaces, such as those arising in Seiberg–Witten, Vafa–Witten, local DT and local PT theories.

1.2 Degeneracy loci

Let $X$ be a smooth complex quasi-projective variety and

(1.1)\begin{equation} \sigma\,\colon\ E_0\longrightarrow E_1 \end{equation}

a map of vector bundles of ranks $e_0$ and $e_1$ over $X$. For a positive integer $r\le e_0$, consider the $r$th degeneracy locus of $\sigma$,

\[ D_r(\sigma) \,:=\, \{ x\in X \colon\dim\ker(\sigma_x) \ge r \} \]

with scheme structure given by viewing it as the zero locus of

\[ \wedge^{e_0-r+1}\sigma\,\colon\ \wedge^{e_0-r+1} E_0\longrightarrow \wedge^{e_0-r+1} E_1. \]

That is, $D_r(\sigma )$ is the $(e_1-e_0+r-1)$th Fitting scheme of the first cohomology sheaf $h^1(E_{\bullet })=\operatorname {coker}\sigma$ of the complex $E_{\bullet }$ (1.1). In particular, it depends on $E_{\bullet }$ only up to quasi-isomorphism and we often denote it by $D_r(E_{\bullet })$.

For the nested Hilbert scheme applications in this paper we will take only $r=1$. On a first read-through of the abstract construction the reader may find it simpler to do the same, substituting projective bundles $\mathbb {P}(-)$ for the Grassmann bundles $\operatorname {Gr}(r,-)$ that follow.

1.3 Resolutions

Over points $x\in D_r$ with $\dim \ker(\sigma _x)=r$ there is a unique $r$-dimensional subspace of $E_0|_x$ on which $\sigma _x$ vanishes. Taking the set of all such $r$-dimensional subspaces for every point of $D_r$ defines a natural space $\widetilde {D}_r$ dominating $D_r$,

(1.2)

whose fibre over $x$ is $\operatorname {Gr}(r,\ker \sigma _x)$. Its scheme structureFootnote ¹ comes from viewing $\widetilde {D}_r(\sigma )\subset \operatorname {Gr}(r,E_0)\xrightarrow {\ p\ }X$ as the vanishing locus of the composition

(1.3)\begin{equation} \mathcal{U}\,{\hookrightarrow}\, p^*E_0 \xrightarrow{p^* \sigma} p^*E_1. \end{equation}

Here $\mathcal {U} \subset p^*E_0$ denotes the universal rank $r$ subbundle on $\operatorname {Gr}(r,E_0)$ (or its restriction to $\widetilde {D}_r$) whose fibre at $(x,U)$ (

) is $U$. If $\sigma$ is appropriately transverse, then each $D_r$ has codimension $r(e_1-e_0+r)$, singular locus $D_{r+1}$ and $\widetilde {D}_r\to D_r$ is a resolution of singularities. For arbitrary $\sigma$ we call $\widetilde {D}_r$ the virtual resolution of $D_r$ since it is virtually smooth.

Proposition 1 Expressing $\widetilde {D}_r(E_{\bullet })$ as the zero locus of the section (1.3) of the bundle $\mathcal {U}^*\otimes p^*E_1$ endows it with a perfect obstruction theory

\[ \{p^*T_X\longrightarrow\mathcal{U}^*\otimes\operatorname{Cone}(\mathcal{U}\to p^*E_\bullet)[1]\}^\vee\longrightarrow\mathbb{L}_{\widetilde{D}_r} \]

of virtual dimension $\operatorname {vd}:=\dim X-r(e_1-e_0+r)$. It depends only on the quasi-isomorphism class of the complex $E_\bullet$ (1.1). The resulting virtual cycle

\[ \big[\widetilde{D}_r(E_{\bullet})\big]^{\operatorname{vir}} \in A_{\operatorname{vd}}\big(\widetilde{D}_r(E_{\bullet})\big), \]

when pushed forward to $X$, has class given by the Thom–Porteous formulaFootnote ²

\[ \Delta_{r-\mathrm{rk}(E_{\bullet})}^r\big(c(-E_\bullet)\big) \in A_{\operatorname{vd}}(X). \]

This is a rewriting of parts of [Reference FultonFul98, Chapter 14] in the language of [Reference Behrend and FantechiBF97]. It is useful if we recognise a moduli space $M$ as having the same perfect obstruction theory as a $\widetilde {D}_r$. Then invariants defined by integration against $[M]^{\operatorname {vir}}$ can be calculated more easily on the smooth space $X$ if the integrand can be expressed as a pullback from $X$.

The drawback is that the map $\widetilde {D}_r\to X$ contractsFootnote ³ the exceptional locus of $\widetilde {D}_r\to D_r$, so most integrands (such as the one in (1.15) below) will not be pullbacks from $X$. We could instead work in $\operatorname {Gr}(r,E_0)$, to which $[\widetilde {D}_r]^{\operatorname {vir}}$ pushes forward to give $c_{re_1}(\mathcal {U}^*\otimes p^*E_1)$, but this description is not an invariant of the quasi-isomorphism class of $E_\bullet$.

However, in examples like ((1.10) and (1.12)) below each kernel $\ker \sigma _x=h^0(E_\bullet |_x)$ naturally embeds in (the fibre over $x$ of) a certain vector bundle $B\to X$,

(1.4)\begin{equation} h^0(E_\bullet|_x) \subseteq B_x \quad\forall x\in X. \end{equation}

The global condition is that $B^*$ surjects onto $\operatorname {coker}(\sigma ^*)=h^0(E_{\bullet }^\vee )$ over $D_r$ (2.18). Then $\widetilde {D}_r$ naturally embeds in $\operatorname {Gr}(r,B)/X$ and we can push forward the virtual class.

Theorem 1 Under the embedding $\iota \colon \,\widetilde {D}_r(E_{\bullet } ) \hookrightarrow \operatorname {Gr}(r,B)$, the pushforward of the virtual cycle to $A_{\operatorname {vd}}(\!\mathrm {Gr}(r,B))$ is given by

(1.5)\begin{equation} \iota_*\big[\widetilde{D}_r(E_\bullet)\big]^{\operatorname{vir}} = \Delta^{r}_{\mathrm{rk}(B)-\mathrm{rk}(E_{\bullet})}\big(c\,(\mathcal{Q}- q^*E_{\bullet})\big), \end{equation}

where $\mathcal {Q}$ is the universal quotient bundle over $q\colon \!\operatorname {Gr}(r,B)\to X$.

1.4 Nested Hilbert schemes

For simplicity consider first two-step Hilbert schemes of nested subschemes (of dimensions 0 and 1) of a fixed smooth projective surface $S$. Given $\beta \in H^2(S,\mathbb {Z})$ and integers $n_1,n_2\ge 0$, we set

(1.6)\begin{equation} S^{[n_1,n_2]}_\beta \ := \big\{I_1(-D)\subseteq I_2\subseteq\mathcal{O}_S \colon [D]=\beta,\ \mathrm{length}\,(\mathcal{O}_S/I_i)=n_i \big\}. \end{equation}

When $n_1=0=n_2$ we use $S_\beta =S_\beta ^{[0,0]}$ to denote the Hilbert scheme of divisors in class $\beta$. Conversely when $\beta =0$ we get the nested Hilbert scheme of points $S^{[n_1,n_2]}$ studied in [Reference Gholampour and ThomasGT20]; we review this briefly first.

1.5 Nested Hilbert schemes of points

For simplicity we set $H^{\ge 1}(\mathcal {O}_S)=0$ for now. For points $I_1,I_2\subset \mathcal {O}_S$ of $S^{[n_1]},\,S^{[n_2]}$, we have

(1.7)\begin{equation} \operatorname{Hom}(I_1,I_2) = \begin{cases} \mathbb{C}, & I_1\subseteq I_2, \\ 0, & I_1\not\subseteq I_2. \end{cases} \end{equation}

Therefore, $\iota \colon S^{[n_1,n_2]}\,{\hookrightarrow}\, S^{[n_1]}\times S^{[n_2]}$ is the degeneracy locus $D_1(E_{\bullet })$ of the two-term complexFootnote ⁴ of vector bundles

\[ E_{\bullet} = R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2)\quad\mathrm{over} \ S^{[n_1]}\times S^{[n_2]}, \]

which, when restricted to the point $(I_1,I_2)$, has 0th cohomology (1.7). Moreover, (1.7) shows that all higher degeneracy loci are empty, so we are in the situation of Footnote 3 with $r=1$. Thus, the Thom–Porteous formula gives

\[ \iota_*\big[S^{[n_1,n_2]}\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2)[1]\big)\cap\big[S^{[n_1]}\times S^{[n_2]}\big]. \]

In [Reference Gholampour and ThomasGT20] we also showed that the relevant perfect obstruction theory agrees (in K-theory at least) with that coming from Vafa–Witten theory. Therefore,Footnote ⁵ Vafa–Witten invariants can be written as integrals over the smooth ambient space $S^{[n_1]}\times S^{[n_2]}$, as exploited to great effect in [Reference LaarakkerLaa18, Reference LaarakkerLaa20] for instance.

1.6 Nested Hilbert schemes of curves and points

When $\beta \ne 0$ we consider

(1.8)\begin{equation} \operatorname{Hom}(I_1(-D),I_2) \quad\mathrm{for}\ D\text{ in class }\beta. \end{equation}

In contrast to (1.7), this can become arbitrarily big, with different elements corresponding (up to scale) to different nested subschemes of $S$. Therefore, the corresponding nested Hilbert scheme $S^{[n_1,n_2]}_\beta$ dominates the degeneracy locus $D_1(E_{\bullet })$ of the complex of vector bundles

(1.9)\begin{equation} E_{\bullet} = R{\mathscr{H}}\!om_\pi(\mathcal{I}_1(-\mathcal{D}),\mathcal{I}_2)\quad\mathrm{over} \ S^{[n_1]}\times S^{[n_2]}, \end{equation}

which, when restricted to the point $(I_1,I_2,\mathcal {O}(D))$, has 0th cohomology (1.8). Since a point of the nested Hilbert scheme $S^{[n_1,n_2]}_\beta$ is a one-dimensional subspace of the kernel (1.8) of the complex (1.9), we see that

\[ S^{[n_1,n_2]}_\beta = \widetilde{D}_1(E_{\bullet}) \]

is the virtual resolution of the degeneracy locus $D_1(E_{\bullet })$ with $r=1$. Since

(1.10)\begin{equation} \operatorname{Hom}(I_1(-D),I_2) \subseteq H^0(\mathcal{O}(D)), \end{equation}

the 0th cohomology of $E_{\bullet }$ embeds in $H^0(\mathcal {O}(D))$. So, we get an example of (1.4) with $r=1$ and $B$ the trivial vector bundle with fibre $H^0(\mathcal {O}(D))$. The embedding $\widetilde {D}_1(E_{\bullet })\hookrightarrow \mathbb {P}(B)$ of Theorem 1 then becomes the following.

Theorem 2 Suppose that $H^{\ge 1}(\mathcal {O}_S)=0$ for now. Then $S^{[n_1,n_2]}_\beta$ has a virtual cycle of dimension $n_1+n_2+h^0(\mathcal {O}(D))-1$, which, under the embedding

\[ \iota\,\colon\ S^{[n_1,n_2]}_\beta\,{\hookrightarrow}\, S^{[n_1]}\times S^{[n_2]}\times\mathbb{P}\big(H^0(\mathcal{O}(D))\big), \]

pushes forward to

\[ \iota_* \big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c^{}_{n_1+n_2}(\!-\mathcal{O}_{\mathbb{P}(H^0(\mathcal{O}(D)))}(-1)-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2(D))). \]

1.7 Reduced virtual cycle

For general $S$ with possibly nonzero $H^{\ge 1}(\mathcal {O}_S)$, we work instead with

(1.11)\begin{equation} E_{\bullet} = R{\mathscr{H}}\!om_\pi(\mathcal{I}_1(-\mathcal{L}_\beta),\mathcal{I}_2)\quad\mathrm{over} \ X:=S^{[n_1]}\times S^{[n_2]}\times \operatorname{Pic}_\beta(S), \end{equation}

which, when restricted to the point $(I_1,I_2,\mathcal {O}(D))$, has 0th cohomology (1.8). Here $\mathcal {L}_\beta$ is a Poincaré line bundle over $S\times \operatorname {Pic}_\beta (S)$, normalised (by tensoring by the pullback of $\mathcal {L}_\beta ^{-1}|_{\{x\}\times \operatorname {Pic}_\beta (S)}$ if necessary) so that $\mathcal {L}_\beta |_{\{x\}\times \operatorname {Pic}_\beta (S)}$ is trivial on some fixed basepoint $x\in S$.

Again, a point of $S^{[n_1,n_2]}_\beta$ is a one-dimensional subspace of the kernel (1.8) of the complex (1.11), so

\[ S^{[n_1,n_2]}_\beta = \widetilde{D}_1(E_{\bullet}) \]

is the virtual resolution of the degeneracy locus $D_1(E_{\bullet })$ with $r=1$. This gives a description of $S_\beta ^{[n_1,n_2]}$ as a projective cone over $S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$; see Proposition 4.7, generalising [Reference Dürr, Kabanov and OkonekDKO07, Lemma 2.15].

We can no longer take $B$ to have fibre $H^0(L)$ over $L\in \operatorname {Pic}_\beta (S)$ since this may jump in dimension. Instead we fix a sufficiently ample divisor $A\subset S$ and let $B$ be the bundle with fibre $H^0(L(A))$:

\[ B \,:=\, \pi_*\big(\mathcal{L}_\beta(A)\big) \quad\mathrm{over}\ X. \]

Therefore, the inclusions

(1.12)\begin{equation} \operatorname{Hom}(I_1(-L),I_2) \subseteq H^0(L) \subseteq H^0(L(A)) \end{equation}

give the required embeddings $h^0(E_{\bullet }|_x)\subset B_x$ of (1.4). Thus, $\widetilde {D}_1\subset \mathbb {P}(B)$ such that $\mathcal {O}_{\mathbb {P}(B)}(-1)$ restricts to $\mathcal {U}\to \widetilde {D}_1$ and Theorem 1 gives the following.

The $H^2(L)=0$ condition ensures that $E_{\bullet }$ is two-term. (We handle $H^2(L)\ne 0$ in the next section.) When $n_1=0$, this theorem was proven in [Reference Kool and ThomasKT14a, Theorem A.7]. Here, as in [Reference Kool and ThomasKT14a, Reference Gholampour, Sheshmani and YauGSY20a], we call the virtual cycle the reduced cycle $[S^{[n_1,n_2]}_\beta ]^{\mathrm {red}}$ since it differs from the one appearing in Vafa–Witten theoryFootnote ⁶ by an $H^2(\mathcal {O}_S)$ term in the obstruction theory.

The apparent difference between the formulae in Theorems 2 and 3 will be explained by the formula (2.25) in Theorem 2.24.

1.8 Nonreduced virtual cycle

In general, when $H^2(L)$ need not vanish, $E_{\bullet }$ (1.11) is a three-term complex with nonzero $h^2(E_{\bullet })$ cohomology sheaf. So, in order to apply our theory we use the splitting trick from [Reference Gholampour and ThomasGT20, § 6.1] to remove $R^2\pi _*\,\mathcal {O}[-2]$ from $E_{\bullet }(1)$ after pulling back to an affine bundle over $X$. (Since an affine bundle has the same Chow groups as its base, no information is lost from the virtual cycle.)

We only manage this on a neighbourhood of $S^{[n_1,n_2]}_\beta \subset \mathbb {P}(B)$, making $E_{\bullet }(1)$ a two-term complex there. By Proposition 1, this is enough to prove the following.

Theorem 4 This construction defines a virtual fundamental class

(1.13)\begin{equation} \big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} \in A_{n_1+n_2+\operatorname{vd}(S_\beta)}\big(S^{[n_1,n_2]}_\beta\big), \end{equation}

where $\operatorname {vd}(S_\beta )=\frac 12\beta (\beta -K_S)$. It agrees with the virtual cycle of [Reference Gholampour, Sheshmani and YauGSY20a].

1.9 Computing virtual cycles by comparison

Since the above neighbourhood need not be all of $\mathbb {P}(B)$, we get no useful pushforward formula for (1.13) in general.Footnote ⁷ This problem is down to the curve class $\beta$ alone; if $\beta =0$ it does not arise, whereas if $n_1=0=n_2$ the problem already arises for the Hilbert scheme $S_\beta$ of pure curves in class $\beta$. However,

\[ [S_\beta]^{\operatorname{vir}}\,\in\,A_{\operatorname{vd}_\beta}(S_\beta), \quad\operatorname{vd}_\beta:=\operatorname{vd}(S_\beta)=\frac12\beta(\beta-K_S), \]

is a well-studied class that has been understood by other methods [Reference Dürr, Kabanov and OkonekDKO07, Reference KoolKoo16] due to its importance in Seiberg–Witten theory. Separating out the parts of the obstruction theory governing the curve and the points, and applying the degeneracy locus technique to the latter only, we are able to prove the following comparison result.

Let $\mathcal {D}_\beta \subset S\times S_\beta$ denote the universal curve. We call

(1.14)\begin{equation} \mathsf{CO}_\beta^{[n_1,n_2]} := R\pi_*\,\mathcal{O}(\mathcal{D}_\beta)-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2(\mathcal{D}_\beta)) \end{equation}

the Carlsson–Okounkov K-theory class [Reference Carlsson and OkounkovCO12] on $S^{[n_1]}\times S^{[n_2]}\times S_\beta$.

Theorem 5 Pushing the virtual and reduced classes forwards along

\[ \iota \colon S^{[n_1,n_2]}_\beta \,{\hookrightarrow}\, S^{[n_1]}\times S^{[n_2]}\times S_\beta \]

gives

\[ \iota_*\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big) \cap\big[S^{[n_1]}\times S^{[n_2]}\big] \times \big[S_\beta\big]^{\operatorname{vir}} \]

and, if $H^2(L)=0$ for all effective $L\in \operatorname {Pic}_\beta (S)$,

\[ \iota_*\big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big) \cap \big[S^{[n_1]}\times S^{[n_2]}\big] \times \big[S_\beta\big]^{\mathrm{red}}. \]

When $n_1=0$ the first of these results was proved in [Reference KoolKoo16] and the second follows from [Reference Kool and ThomasKT14a, Appendix A].

1.10 $\ell$-step Hilbert schemes

The generalisation of Theorems 3 and 5 to $\ell$-step nested Hilbert schemes

\[ S^{[n_1,\ldots, n_\ell]}_{\beta_1,\ldots,\beta_{\ell-1}} \]

is straightforward. See § 5 for details.

1.11 Vafa–Witten invariants

Vafa–Witten invariants [Reference Tanaka and ThomasTT20] are made up of ‘instanton contributions’ and ‘monopole contributions’. The former are virtual Euler characteristics of moduli spaces of rank $r$ sheaves on $S$, as studied in [Reference Göttsche and KoolGK20] for instance. The latter are integrals over moduli spaces of chains of sheaves on $S$ of total rank $r$ with nonzero maps between them.

When $p_g(S)>0$ and $r$ is prime, a vanishing result [Reference ThomasTho20, Theorem 5.23] implies that the only nonzero monopole contributions come from moduli spaces of chains of rank 1 sheaves. After tensoring with a line bundle, these are nested Hilbert schemes. For instance, in rank 2 the relevant integrals take the form

(1.15)\begin{equation} \int_{[S^{[n_1,n_2]}_\beta]^{\operatorname{vir}}}\sum_i \rho^* \alpha_i \cup h^i = \sum_i\int_{\rho_* (h^i\cap [S^{[n_1,n_2]}_\beta]^{\operatorname{vir}})}\alpha_i, \end{equation}

where $\alpha _i\in H^*\big (S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)\big )$, $h:=c_1(\mathcal {O}_{\mathbb {P}(B)}(1))$ and

\[ \rho \colon S^{[n_1,n_2]}_\beta \longrightarrow S^{[n_1]}\times S^{[n_2]}\times \operatorname{Pic}_\beta(S). \]

Using Theorem 5, we can express this in terms of (a) integrals over the smooth space $S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$ and (b) integrals over $S_\beta$. The latter give the Seiberg–Witten invariants [Reference Dürr, Kabanov and OkonekDKO07, Reference Chang and KiemCK13],

\[ \mathsf{SW}_\beta\, :=\, \begin{cases} \deg\,[S_\beta]^{\operatorname{vir}}, & \operatorname{vd}_\beta=\frac{1}{2}\beta.(K_S-\beta)=0, \\ 0, & \mathrm{otherwise.}\end{cases} \]

Theorem 6 Suppose that $p_g(S)>0$. Then, in $H_* \big (S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)\big )$,

\begin{align*} \rho_* \big[S^{[n_1,n_2]}_\beta \big]^{\operatorname{vir}}& = \mathsf{SW}_\beta\cdot c_{n_1+n_2}\big(\!-\!R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes L)\big)\times[L],\\ \rho_*\big(\,\!h^i\cap\big [S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}}\big)& = 0 \quad\mathrm{for}\ i>0, \end{align*}

where $L\in \operatorname {Pic}_\beta (S)$, so $[L]$ is the generator of $H_0(\operatorname {Pic}_\beta (S),\mathbb {Z})$.

When $p_g(S)=0$, then $S$ may not be of ‘Seiberg–Witten simple type’, which means that we have to consider higher dimensional Seiberg–Witten moduli spaces. Letting $\mathsf {AJ}\colon S_\beta \to \operatorname {Pic}_\beta (S)$ denote the Abel–Jacobi map, the higher Seiberg–Witten invariants [Reference Dürr, Kabanov and OkonekDKO07] are

\[ \mathsf{SW}_\beta^{\,j}\, :=\, \mathsf{AJ}_*\big(h^j \,\cap \, [S_\beta]^{\operatorname{vir}}\big) \in H_{2(\operatorname{vd}_\beta-j)}(\operatorname{Pic}_\beta(S)) = \wedge^{2(\operatorname{vd}_\beta-j)}H^1(S). \]

Now set $\beta ^{\vee }:=K_S-\beta$, so that the condition that $H^2(L)\ne 0$ for some $L\in \operatorname {Pic}_\beta (S)$ is the condition that $\beta ^\vee$ is effective. Seiberg–Witten theory has a duality under $\beta \leftrightarrow \beta ^\vee$. This gives interesting dualities between invariants of nested Hilbert schemes under $\beta \leftrightarrow \beta ^\vee ,\ n_1\leftrightarrow n_2$. Define the map

\[ \rho^\vee\,\colon\ S^{[n_2,n_1]}_{\beta^\vee}\longrightarrow S^{[n_1]}\times S^{[n_2]}\times \operatorname{Pic}_\beta(S) \]

by replacing $\beta \leftrightarrow \beta ^\vee ,\ n_1\leftrightarrow n_2$ in $\rho \colon S^{[n_1,n_2]}_\beta \longrightarrow S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$ and then composing with $L\mapsto K_S\otimes L^{-1}\colon$ $\operatorname {Pic}_{\beta ^\vee }(S)\to \operatorname {Pic}_\beta (S)$.

When $p_g(S)=0$, this can fail because only the unordered pair $(\mathsf {SW}_\beta ,\mathsf {SW}_{\beta ^\vee })$ is invariant under oriented diffeomorphisms, rather than the individual invariant $\mathsf {SW}_\beta$.

1.12 Laarakker

In [Reference LaarakkerLaa18] (stable case) and [Reference LaarakkerLaa20] (general case), Laarakker used these results to compute Vafa–Witten invariants [Reference Tanaka and ThomasTT17, Reference Tanaka and ThomasTT20] and refined Vafa–Witten invariants [Reference ThomasTho20] on surfaces with $p_g>0$ (and $h^{0,1}=0$ for now).

By the comparison result for virtual cycles of Theorem 5, the contributions from points and curves split, with the curves contributing Seiberg–Witten invariants, i.e. well-understood integrals over linear systems of curves. Laarakker evaluated the contributions of Hilbert schemes of points via the method of [Reference Ellingsrud, Göttsche and LehnEGL01]. The result depends only on the curve class $\beta \in H_2(S,\mathbb {Z})$ and the cobordism class of the surface, and thus only on $c_1(S)^2,\,c_2(S)$ and $\beta ^2$. Therefore, these contributions can be calculated on K3 surfaces and toric surfaces (despite these not having $p_g > 0$!).

The results are very general, but the absolute simplest to state is for $S$ minimal of general type with $p_g(S)>0$ and $H_1(S,\mathbb {Z})=0$ such that $K_S$ is not divisible by 2.

Theorem 10 [Laa18] Let $S$ be as above. The monopole branch contributionsFootnote ⁸ of rank 2 Higgs pairs with $\det =K_S$ to the refined Vafa–Witten generating series $\sum _n\mathsf {VW}_{2,K_S,n}(t)\,q^n$ can be written

\[ A(t,q)^{\chi(\mathcal{O}_S)}B(t,q)^{c_1(S)^2}, \]

where

\[ A(t,q),\ B(t,q) \in \mathbb{Q}(t^{1/2})((q^{1/2})) \]

are universal functions, independent of $S$.

Furthermore, K3 calculations [Reference Göttsche and KoolGK20, Reference ThomasTho20] determine $A(t,q)$ completely in terms of modular forms. And, $B(t,q)$ can be predicted by modularity and the results of Göttsche–Kool [Reference Göttsche and KoolGK20] on the instanton contributions; they checked this prediction in low degree by toric computations.

1.13 Notation

Given a map $f\colon X\to Y$, we often use the same letter $f$ to denote its base change by any map $Z\to Y$, i.e. $f\colon X\times _YZ\to Z$. In particular, projections $S\times M\to M$ down the surface $S$ are all denoted by $\pi$. We often suppress pullback maps $f^*$ on sheaves when it should not cause confusion.

For a vector bundle $B$ over $X$, we denote by $\mathbb {P}(B)$ the projective bundle of lines in $B$. For a coherent sheaf $\mathcal {F}$ on $X$, we denote by $\mathbb {P}^*(\mathcal {F})=\operatorname {Proj}\,\operatorname {Sym}^\bullet \!\mathcal {F}$ the projective cone of quotient lines of $\mathcal {F}$. When $\mathcal {F}$ is locally free, then of course $\mathbb {P}^*(\mathcal {F})\cong \mathbb {P}(\mathcal {F}^*)$.

1.14 Splitting trick

Since we are ultimately interested in Chern class formulae, we often only care about the topological type of a given algebraic vector bundle. In particular, we will often wish to split nontrivial extensions of vector bundles, which does not change their topological type. One way to do this is by working with $C^\infty$ bundles; we prefer to stay within algebraic geometry and use the Jouanolou trick [Reference JouanolouJou73] instead.

Namely, if $Y$ is quasi-projective, there exists an affine bundle $\rho \colon \widetilde {Y} \to Y$ such that $\widetilde {Y}$ is an affine variety. Therefore, on pulling locally free sheaves back by $\rho ^*$, they become projective $\mathcal {O}_{\widetilde {Y}}$-modules and any extensions become split. But $\rho$ is a homotopy equivalence, inducing an isomorphism of Chow groups [Reference KreschKre99, Corollary 2.5.7]

(1.16)\begin{equation} \rho^* \colon A_{*} (Y) \xrightarrow{\sim} A_{*+\dim\rho\,}\big(\widetilde{Y}\big), \end{equation}

so formulae for cycles upstairs on $\widetilde {Y}$ induce similar formulae on $Y$.

A very explicit such affine bundle was used in [Reference Gholampour and ThomasGT20, § 6.1] and all the necessary pullbacks were laboriously documented. In this paper, for the sake of clarity and economy of notation, we will often just say ‘by the Jouanolou trick’ and give no further details. So, we will pretend we are working on $Y$, suppressing mention of the fact that we have replaced $Y$ by its homotopic affine model $\widetilde {Y}$ and then used the isomorphism (1.16) to recover results on $Y$ at the end.

2.1 Perfect obstruction theory

Let $I\subset \mathcal {O}_{\operatorname {Gr}(r,E_0)}$ denote the ideal of $\widetilde {D}_r$ generated by $\widetilde {\sigma }$ (2.7). Then $\widetilde {D}_r$ inherits the standard perfect obstruction theory as in the following diagram.

(2.11)

That is, the complex on the bottom row is a representative of the truncated cotangent complex $\mathbb {L}_{\widetilde {D}_r}$ and the top row is the virtual cotangent bundle of the obstruction theory.

Proposition 2.12 The virtual cotangent bundle $\mathbb {L}^{\operatorname {vir}}_{\widetilde {D}_r}$ is a cone

\[ \operatorname{Cone}\big\{\mathcal{U}\otimes p^*E_\bullet^\vee\big|_{\widetilde{D}_r}[-1]\longrightarrow\big(\mathcal{U}^*\!\otimes\mathcal{U}\,[-1]\ \oplus\ p^*\Omega_X\big)\big|_{\widetilde{D}_r}\big\} \]

of virtual dimension $\operatorname {vd}:=\dim X-r(e_1-e_0+r)$. It depends only on the quasi-isomorphism class of the complex $E_{\bullet }$ (1.1). The pushforward of the resulting virtual cycle

\[ \big[\widetilde{D}_r(E_\bullet)\big]^{\operatorname{vir}} \in A_{\operatorname{vd}}\big(\widetilde{D}_r(E_\bullet)\big) \]

to $X$ is given by the Thom–Porteous formula

\[ \Delta\,_{r-\mathrm{rk}(E_{\bullet})}^r\big(c(-E_\bullet)\big) \in A_{\operatorname{vd}}(X). \]

Proof. Modifying $\mathbb {L}^{\operatorname {vir}}_{\widetilde {D}_r}$ (2.11) by an acyclic complex, it is quasi-isomorphic to the total complex

(2.13)

where $\phi$ is the composition $\Omega _{\operatorname {Gr}}\cong \mathcal {U}\otimes (p^*E_0/\mathcal {U})^*\to \mathcal {U}\otimes p^*E_0^*$. This diagram was shown to commute in [Reference Gholampour and ThomasGT20, Claim 2, proof of Theorem 3.6].

Therefore, $\mathbb {L}^{\operatorname {vir}}_{\widetilde {D}_r}\cong \operatorname {Cone}(\mathcal {U}\otimes p^*E_{\bullet }^\vee |_{\widetilde {D}_r}\longrightarrow \operatorname {Cone}(\phi ))[-1]$. To determine $\operatorname {Cone}(\phi )$, we fit $\phi$ into the following commutative diagram, all of whose rows and columns are exact triangles.

(2.14)

We calculate the lower right hand arrow. By definition, it is the composition from the top left to the bottom right of the following diagram.

(2.15)

Here the upper row is the rightmost vertical exact triangle in (

2.14

), while the lower row is the exact triangle along the top row of (

2.14

). The first arrow in (

2.15

) is well known to be the relative (to $X$) Atiyah class of the bundle $\mathcal {U}$. So, using its absolute Atiyah class, we get the commutative diagram

which shows that the composition $\mathcal {U}\otimes \mathcal {U}^*\to p^*\Omega _X[2]$ is zero. Therefore, the bottom row of (

2.14

) gives $\operatorname {Cone}(\phi )\cong \mathcal {U}\otimes \mathcal {U}^*\oplus p^*\Omega _X|_{\widetilde {D}_r}[1]$, implying the claimed result.

Finally, since the perfect obstruction theory arises from the $\widetilde {D}_r$ description as the zero locus of a section of the bundle $\mathcal {U}^*\otimes p^*E_1\to \operatorname {Gr}$, the resulting virtual cycle is the localised top Chern class [Reference FultonFul98, § 14.1] of that bundle. Its pushforward to $\operatorname {Gr}$ is

(2.16)\begin{equation} c_{\,re_1}(\mathcal{U}^*\otimes p^*E_1); \end{equation}

further pushing down $p\colon \operatorname {Gr}\to X$ shows that the pushforward of $[\widetilde {D}_r]^{\operatorname {vir}}$ to $X$ is given by

\[ p_*\,c_{\,re_1}(\mathcal{U}^*\otimes p^*E_1) = \Delta\,_{e_1-e_0+r}^r\big(c(-E_\bullet)\big) \]

by [Reference FultonFul98, Theorem 14.4].

This proves Proposition 1 in the Introduction. But, in passing from $\operatorname {Gr}(r,E_0)$ down to $X$, we have contracted $\widetilde {D}_r$ back to $D_r$ and so lost information in general. (The exception is the case studied in [Reference Gholampour and ThomasGT20], where $r$ is the largest integer for which $D_r$ is nonempty; then $p|_{\widetilde {D}_r}$ is an embedding.)

We could work in $\operatorname {Gr}(r,E_0)$, in which the class of the virtual cycle is $c_{\,re_1}(\mathcal {U}^*\otimes p^*E_1)$, but this is not a quasi-isomorphism invariant of the complex $E_{\bullet }$. In examples such as (1.10) and (1.12), we can replace $E_0$ by a more canonical bundle $B$, so now we assume that we have chosen $B$ such that

(2.17)\begin{equation} h^0(E_{\bullet}|_x)\,{\hookrightarrow}\, B_x \quad\text{for each}\ x\in D_r. \end{equation}

For base-change reasons the precise condition is most easily stated via duals.

Choice. We choose a vector bundle $B$ on $X$ and a surjection

(2.18)\begin{equation} B^*\big|_{D_r}\longrightarrow h^0\big(E_{\bullet}^\vee\big)\big|_{D_r}\longrightarrow0. \end{equation}

For some purposes we will require that the surjection (2.18) extends to all of $X$,

(2.19)\begin{equation} B^*\longrightarrow h^0\big(E_{\bullet}^\vee\big)\longrightarrow0. \end{equation}

For instance, if $r=1$, this is immediate from (2.18).

Note that by restricting both exact sequences $E_1^*\to E_0^*\to h^0(E_{\bullet }^\vee )\to 0$ and (2.18) to $x\in D_r$ and dualising, we recover (2.17).

The surjection $B^*|_{D_r}\to (\operatorname {coker}\sigma ^*)|_{D_r}$ of (2.18) induces an embedding $\operatorname {Gr}(\operatorname {coker}\sigma ^*,r)\subset \operatorname {Gr}(B^*,r)$ since $\operatorname {Gr}(\operatorname {coker}\sigma ^*,r)$ lies over $D_r\subset X$. By Proposition 2.5, this is an embedding

(2.20)\begin{equation} \iota^{}_{B}\,\colon\ \widetilde{D}_r\,{\hookrightarrow}\,\operatorname{Gr}(r,B)\xrightarrow{\ q\ }X \end{equation}

such that $\iota _B^*\,\mathcal {U}_B\cong \iota ^*\mathcal {U}$. Here $\mathcal {U}_B$ and $\mathcal {Q}_B=q^*B/\mathcal {U}_B$ denote the universal sub- and quotient bundles on $\operatorname {Gr}(r,B)$, and $\iota \,\colon \,\widetilde {D}_r\hookrightarrow \operatorname {Gr}(r,E_0)$ is the embedding (2.6). We will compare these via a third embedding,

\[ I:=\iota\times^{}_X\iota^{}_{B} \colon \widetilde{D}_r\,{\hookrightarrow}\, \operatorname{Gr}(r,E_0)\times^{}_X\operatorname{Gr}(r,B), \]

fitting into the following Cartesian diagram.

(2.21)

Here the vertical maps are flat and the section $s:=\operatorname {id}_{\widetilde {D}_r}\!\times ^{}_{\!X\,}\iota ^{}_{B}$. As usual we suppress various pullback or base-change maps.

Lemma 2.22 The section $s$ in (2.21) is a regular embedding, with image cut out by the regular section of $\mathcal {U}^*\!\otimes \mathcal {Q}_B$ defined by the composition

(2.23)\begin{equation} \mathcal{U}\,{\hookrightarrow}\, B\longrightarrow\hspace{-5.5mm}\longrightarrow\mathcal{Q}_B \quad\mathrm{on}\quad \widetilde{D}_r\times^{}_X\operatorname{Gr}(r,B). \end{equation}

Proof. Since $s^*q^*\iota ^*\mathcal {U}=\iota ^*\mathcal {U}=s^*\mathcal {U}_B$ on $\widetilde {D}_r\times ^{}_X\operatorname {Gr}(r,B)$, we see that $s^*$ applied to (2.23) gives the same as $s^*$ applied instead to the composition

\[ \mathcal{U}_B\,{\hookrightarrow}\, B\longrightarrow\hspace{-5.5mm}\longrightarrow \mathcal{Q}_B \quad\mathrm{on}\quad \widetilde{D}_r\times^{}_X\operatorname{Gr}(r,B). \]

This is zero by the definition of $\mathcal {Q}_B$, so $s(\widetilde {D}_r)$ lies in the zero locus $Z$ of (2.23). We are left with showing that $Z\subseteq s(\widetilde {D}_r)$.

By the definition of $Z$, pulling back the composition (2.23) by $\iota ^{}_Z\colon Z\hookrightarrow \widetilde {D}_r\times ^{}_X\operatorname {Gr}(r,B)$ shows that the composition along the top row of the following diagram is zero. This uniquely fills in the dotted arrow $\phi$ to the lower short exact sequence in the following diagram.

Since $\phi$ is a vector bundle injection, it is an isomorphism. And, since $\mathcal {U}=\iota _B^*\,\mathcal {U}_B=q^*s^*\mathcal {U}_B$ is pulled back from $\widetilde {D}_r$, this gives the following diagram.

Thus, this subbundle is represented by both of the maps $\iota ^{}_Z$ and $s\circ q\circ \iota ^{}_Z$, but any subbundle is classified by a unique map $Z\to \widetilde {D}_r\times ^{}_X\operatorname {Gr}(r,B)$. So, $\iota ^{}_Z=s\circ q\circ \iota ^{}_Z$, which gives $Z\subseteq s(\widetilde {D}_r)$.

Since codim $(s(\widetilde {D}_r))=\dim \operatorname {Gr}(r,B)-\dim X=\operatorname {rank}\,(\mathcal {U}^*\!\otimes \!\mathcal {Q}_B)$, it follows that $s$ is a regular embedding.

Theorem 2.24 Let $b:=\operatorname {rank} B$. The pushforward of $[\widetilde {D}_r]^{\operatorname {vir}}$ to $\operatorname {Gr}(r,B)$ is given by

\[ \iota^{}_{B*}[\widetilde{D}_r]^{\operatorname{vir}} = \Delta^{r}_{b+e_1-e_0}\big(c\big(\mathcal{Q}_B-E_{\bullet}\big)\big). \]

If (2.19), holds this also equals

(2.25)\begin{equation} c^{}_{r(b-e_0+e_1)}\big(\mathcal{U}_B^*\,\otimes(B-E_{\bullet})\big). \end{equation}

Proof. By Proposition 2.5 and flat base change around the diagram (2.21),

\[ (\iota\times1)_*\,q^*[\widetilde{D}_r]^{\operatorname{vir}} = q_0^*\,\iota^{}_{*}[\widetilde{D}_r]^{\operatorname{vir}} = q_0^*\,c^{}_{re_1}(\mathcal{U}^*\otimes E_1). \]

Therefore, Lemma 2.22 gives

\begin{align*} I_*[\widetilde{D}_r]^{\operatorname{vir}} &= (\iota\times1)_*s_*[\widetilde{D}_r]^{\operatorname{vir}}\\ &= (\iota\times1)_*s_*s^*q^*[\widetilde{D}_r]^{\operatorname{vir}} \\ &=(\iota\times1)_*\big(c^{}_{r(b-r)}(q^*\mathcal{U}^*\otimes\mathcal{Q}_B)\cap q^*[\widetilde{D}_r]^{\operatorname{vir}}\big) \\ &= c^{}_{r(b-r)}(q_0^*\mathcal{U}^*\otimes\mathcal{Q}_B)\cap(\iota\times1)_*\,q^*[\widetilde{D}_r]^{\operatorname{vir}} \\ &= c^{}_{r(b-r)}(q_0^*\mathcal{U}^*\otimes\mathcal{Q}_B)\cap c^{}_{re_1}(q_0^*\mathcal{U}^*\otimes E_1) \\ &= c^{}_{r(b-r+e_1)}\big(q_0^*\mathcal{U}^*\otimes(\mathcal{Q}_B\oplus E_1)\big). \end{align*}

Pushing down $p\colon \operatorname {Gr}(r,E_0)\times ^{}_X\operatorname {Gr}(r,B)\to \operatorname {Gr}(r,B)$ gives

(2.26)\begin{equation} \iota^{}_{B*}[\widetilde{D}_r]^{\operatorname{vir}} = \Delta^{r}_{b+e_1-e_0}\big(c\big(\mathcal{Q}_B+E_1-E_0\big)\big) \end{equation}

by [Reference FultonFul98, Proposition 14.2.2].

To prove (2.25), we need to lift the surjection $B^*\to \operatorname {coker}(\sigma ^*)$ of (2.19) to a map $\psi \colon B^*\to E_0^*$; see (2.27) below for how to do this after using the Jouanolou trick of § 1.14. This then gives $h^0(E_{\bullet }|_x)\hookrightarrow B_x$ for all $x\in X$, which means that

\[ E_0\xrightarrow{\ (\sigma,\psi^*)\ }E_1\oplus B \]

is a subbundle. Therefore, its cokernel $C$ is a rank ($b+e_1-e_0$) vector bundle over $X$, equivalent in K-theory to $B-E_{\bullet }$. Thus, we can apply [Reference FultonFul98, Example 14.4.12] to rewrite (2.26) as

\begin{align*} \iota^{}_{B*}[\widetilde{D}_r]^{\operatorname{vir}} &= \Delta^{r}_{b+e_1-e_0}\big(c\big(C-\mathcal{U}_B\big)\big) \\ &= c^{}_{r(b+e_1-e_0)}\big(\mathcal{U}_B^*\,\otimes C\big) \\ &= c^{}_{r(b+e_1-e_0)}\big(\mathcal{U}_B^*\,\otimes(B-E_{\bullet})\big). \end{align*}

This formula should be compared with (2.16).

2.2 Digression: description as a deepest degeneracy locus

As usual fix a smooth quasi-projective variety $X$ carrying a two-term complex of bundles

\[ \sigma\colon E_0\longrightarrow E_1. \]

Almost by definition the virtual resolution $\widetilde {D}_r$ of the $r$th degeneracy locus can be seen as the deepest degeneracy locus of the complex $\mathcal {U}\to p^*E_1$ (1.3) up on $\operatorname {Gr}(r,E_0)$. Such deepest degeneracy loci (those for which $r$ is maximal, i.e. $D_{r+1}$ is empty) were studied in [Reference Gholampour and ThomasGT20].

Since $E_0$ is not a quasi-isomorphism invariant, we would again prefer to replace $\operatorname {Gr}(r,E_0)$ with (an affine bundle over) $\operatorname {Gr}(r,B)$. For this we require (2.19) to hold, giving the following diagram.

(2.27)

Using the Jouanolou trick described in § 1.14 we may, on replacing $X$ by an affine bundle over it (and pulling everything back to it), assume that $B^*$ is a projective $\mathcal {O}$-module. Then we may pick a lift $\psi \colon B^*\to E_0^*$.

Over $\operatorname {Gr}(r,B)$, with its universal quotient bundle $\pi \colon B\twoheadrightarrow \mathcal {Q}_B$ (suppressing some pullback maps as usual), we consider the composition

(2.28)

At a closed point $(x,V)\in \operatorname {Gr}(r,B)$ (i.e. a point $x\in X$ and an $r$-dimensional subspace $V\le B_x$), we have

\[ \ker\tau_x = \ker\sigma_x\cap\ker(\pi_x\circ \psi_x^*) = \ker\sigma_x\cap V, \]

where in the last expression we have used $\psi _x^*$ to identify $\ker \sigma _x$ as a subspace of $B_x$ by (2.17). In particular, $\ker \tau _x$ is at most $r$ dimensional, so $D_r(\tau )$ is a deepest degeneracy locus, and the locus of points where it is precisely $r$ dimensional is

(2.29)\begin{equation} D_r(\tau) = \big\{(x,U)\in\operatorname{Gr}(r,B) \colon U\subseteq\ker(\sigma_x)\big\} = \widetilde{D}_r(\sigma). \end{equation}

This bijection of sets generalises to $S$-points of $X$: given $f\colon S\to X$, we get a bijection between the sets

\begin{align*} &\big\{\mathrm{rank}\ r\ \mathrm{subbundles}\ U\subset f^*E_0 \colon U\,{\hookrightarrow}\,f^*E_0\xrightarrow{\ f^*\sigma\ }f^*E_1\ \text{is zero}\big\} \end{align*}

and

\begin{align*} & \big\{\mathrm{rank}\ r \ \mathrm{subbundles}\ (U\subset f^*E_0,\ V\subset f^*B) \colon\text{ the composition}\\ &\qquad \qquad \qquad\qquad \qquad U {\hookrightarrow}\!f^*E_0\xrightarrow{\ f^*\tau\!\ }f^*E_1\oplus f^*B/V\text{ is zero}\big\}. \end{align*}

The functors taking $f\colon S\to X$ to either of these two sets are therefore isomorphic. The first is (2.4) and is represented by $\widetilde {D}_r(\sigma )$ by Proposition 2.5. The second is represented by $D_r(\tau )\subset \operatorname {Gr}(r,B)/X$. Therefore, (2.29) is an isomorphism of schemes.

Furthermore, we can compare the perfect obstruction theory of Proposition 2.12 for $\widetilde {D}_r(\sigma )$ to the deepest degeneracy locus perfect obstruction theory of [Reference Gholampour and ThomasGT20, Theorem 3.6] for $D_r(\tau )$. In both cases the K-theory class of the virtual cotangent bundle is the restriction of

\[ \Omega_X+\mathcal{U}\otimes(E_{\bullet}^\vee-\mathcal{U}^*), \]

so their virtual cycles agree. Finally, the Thom–Porteous formula for $\widetilde {D}_r(\sigma )$ of Theorem 2.24 and the Thom–Porteous formula for $D_r(\tau )$ of [Reference Gholampour and ThomasGT20, Theorem 3.6] both give

\[ \Delta^{r}_{b+e_1-e_0}\big(c\big(\mathcal{Q}_B-E_{\bullet}\big)\big) \]

as the pushforward of the virtual cycle to $\operatorname {Gr}(r,B)$.

Theorem 2.30 Suppose that (2.19) holds. Then, after pulling back to an affine bundle, the embedding $\iota ^{}_{B}\colon \widetilde {D}_r(\sigma ) \hookrightarrow \operatorname {Gr}(r,B)$ of (2.20) is the deepest degeneracy locus of the map of vector bundles (2.28),

\[ \widetilde{D}_r(\sigma)\ \cong\ D_r(\tau). \]

The two resulting virtual cycles and Thom–Porteous formulae agree.

2.3 Generalised Carlsson–Okounkov vanishing

Proposition 2.31 Under the assumptions of Theorem 2.30,

\[ c_{\,b+e_1-e_0+i}\big(B+E_1-E_0\big) = 0\quad\forall\,i>0. \]

Proof. On an affine bundle over $X$, recall the map (2.28)

(2.32)\begin{equation} E_0\xrightarrow{\ (\sigma,\psi^*)\ }E_1\oplus B \end{equation}

of the last section. By (2.19), it is an injective map of bundles and so is quasi-isomorphic to a rank ($b-e_0+e_1$) vector bundle (its cokernel). Therefore, its higher Chern classes vanish on the affine bundle and so on $X$ by (1.16).

Taking $B=\mathcal {O}_X$ and $E_{\bullet }=R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2)$ (in the notation of § 4) over the product of Hilbert schemes of points $X=S^{[n_1]}\times S^{[n_2]}$, this recovers the original Carlsson–Okounkov vanishing of [Reference Carlsson and OkounkovCO12] and [Reference Gholampour and ThomasGT20, Corollary 8.1], at least when $H^{\ge 1}(\mathcal {O}_S)=0$ to ensure that $E_{\bullet }$ is two-term. The general surface $S$ can be handled by another splitting trick to remove $H^{\ge 1}(\mathcal {O}_S)$ from $E_{\bullet }$; see [Reference Gholampour and ThomasGT20, § 8].

4.1 Embedding and reduced virtual cycle

Throughout this section we require $\beta$ to be of Hodge type $(1,1)$. Until now everything has been true (but vacuous) when this assumption fails, and the results deformation invariant. The constructions of this section are only invariant under deformations of $S$ inside the Noether–Lefschetz locus in which $\beta$ is of type $(1,1)$.

As discussed in the Introduction, each fibre of $\widetilde {D}_1(\sigma )\to D_1(\sigma )$ admits a natural embedding

(4.10)\begin{equation} \mathbb{P}\big(\!\operatorname{Hom}(I_1,I_2\otimes L)\big) \subseteq \mathbb{P}\big(H^0(L)\big)\,\stackrel{s^{}_{\!A}}\,{\hookrightarrow}\,\,\mathbb{P}\big(H^0(L(A))\big). \end{equation}

Here we have fixed, once and for all, $s^{}_{\!A}\in H^0(\mathcal {O}_S(A))$ cutting out a divisor $A\subset S$ sufficiently positive that

\[ H^{\ge 1}(L(A))=0 \quad \quad \forall L\in \operatorname{Pic}_\beta(S). \]

(When $H^{\ge 1}(\mathcal {O}_S)=0$ and $L$ is effective, we may take $A=\emptyset$ and the proof of Theorem 3 below reduces to a proof of Theorem 2 instead.) So, we let $B$ be the vector bundle

(4.11)\begin{equation} B\,:=\,\pi_*(\mathcal{L}_\beta(A)) \quad\mathrm{over}\ X = S^{[n_1]}\times S^{[n_2]}\times\operatorname{Pic}_\beta(S). \end{equation}

Then (4.10) gives the embedding $h^0(E_{\bullet })\subset B_x$ of (1.4) at each point $x=(I_1,I_2,L)\in X$. The global version (2.19) is the composition of surjections

(4.12)

By (2.20), we get an embedding

(4.13)\begin{equation} \iota^{}_{B}\,\colon\ S_\beta^{[n_1,n_2]}\ \cong\ \widetilde{D}_1(E_{\bullet})\,{\hookrightarrow}\,\mathbb{P}(B)\xrightarrow{\ q\ }X. \end{equation}

Suppose now that, as in the statements of Theorems 2 and 3,

(4.14)\begin{equation} H^2(\mathcal{O}(D))=0 \quad\text{ for any effective divisor }D\text{ in class }\beta. \end{equation}

The resulting surjection

and base change show that $E_{\bullet }$ has $h^{\ge 2}=0$ over the image of $S^{[n_1,n_2]}_\beta$ in $X$ and therefore also over a neighbourhood $U$ thereof. Therefore, in (4.6), we can take $E_{\bullet }$ to be a two-term complex of vector bundles

(4.15)\begin{equation} R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\ \cong\ \big\{E_0 \xrightarrow{\sigma} E_1\big\} = :\,E_{\bullet} \quad\mathrm{over}\ U\subseteq X. \end{equation}

Proposition 4.7 then gives

\[ S^{[n_1,n_2]}_\beta\ \cong\ \widetilde{D}_1(E_{\bullet}) \]

and Proposition 2.12 endows it with a perfect obstruction theory, which we call the reduced obstruction theory.Footnote ¹²

Theorem 4.16 Suppose that $H^2(\mathcal {O}(D))=0$ for every effective divisor $D$ in class $\beta$. The above construction gives a reduced obstruction theory with virtual tangent bundle

(4.17)\begin{align} T^{\operatorname{vir}}_{S^{[n_1,n_2]}_\beta}& = -R{\mathscr{H}}\!om_{\pi}(\mathcal{I}_1,\mathcal{I}_1)^{}_0-R{\mathscr{H}}\!om_{\pi}(\mathcal{I}_2,\mathcal{I}_2)^{}_0 \nonumber\\ &\quad + R{\mathscr{H}}\!om_\pi\big(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta(1)\big)+R^1\pi_*\,\mathcal{O}-\mathcal{O} \end{align}

in K-theory and reduced virtual cycle

(4.18)\begin{equation} \big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} \in A_{n_1+n_2+\operatorname{vd}_\beta+p_g}\big(S^{[n_1,n_2]}_\beta\big), \end{equation}

where $\mathcal {O}(1):=\mathcal {O}_{\mathbb {P}(B)}(1)$ and $\operatorname {vd}_\beta :=\beta (\beta -K_S)/2$.

If $H^2(L)=0$ for all $L\in \operatorname {Pic}_\beta (S)$, then the pushforward of (4.18) to $\mathbb {P}(B)$ is

\[ \iota^{}_{B*}\big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} = c^{}_{n_1+n_2+d}\big(B(1)-R{\mathscr{H}}\!om_\pi\big(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta(1)\big)\big), \]

where $d=\frac {1}{2}A (2\beta +A-K_S)=\chi (L(A))-\chi (L)$ for any $L\in \operatorname {Pic}_\beta (S)$.

Proof. The virtual tangent bundle is described by the $r=1$ case of Proposition 2.12 with $\mathcal {U}=\mathcal {O}(-1)$, giving the restriction of

\[ T_X-\mathcal{O}_X-\mathcal{O}(-1)^*\otimes R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta) \]

to $S^{[n_1,n_2]}_\beta$. Since $X=S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$, we derive (4.17) from

\[ T_{\operatorname{Pic}_\beta(S)}\,\cong\,R^1\pi_*\,\mathcal{O}\quad\mathrm{and}\quad T_{S^{[n_i]}}\,\cong\,{\mathscr {E}}\!\,xt^1_\pi(\mathcal{I}_i,\mathcal{I}_i)^{}_0\,\cong\,R{\mathscr{H}}\!om_\pi(\mathcal{I}_i,\mathcal{I}_i)^{}_0[1]. \]

When $H^2(L)=0$ for all $L\in \operatorname {Pic}_\beta (S)$, then $E_{\bullet }$ is two-term over the whole of $X$, so we may apply Theorem 2.24 with $r=1$ to give the formula for the pushforward.

4.2 Comparison of reduced cycles

Next we will compare the reduced cycles on $S^{[n_1,n_2]}_\beta$ and $S_\beta$. Note that $S^{[n_1,n_2]}_\beta$ and $S_\beta \times S^{[n_1]}\times S^{[n_2]}$ are the virtual resolutions of the degeneracy loci of

\[ R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta) \quad\mathrm{and}\quad R\pi_*\,\mathcal{L}_\beta\quad\mathrm{over}\ X, \]

respectively. These are related by the obvious diagram

(4.19)

with the vertical column an exact triangle. Here $H:={\mathscr {E}}\!\,xt^2_\pi (\mathcal {O}/\mathcal {I}_1,\mathcal {L}_\beta )$, so

\[ H^*\ \cong\ \big(\mathcal{L}^*_\beta\otimes K_S\big)^{[n_1]} \]

by relative Serre duality down $\pi$. Thus, $H$ is a vector bundle.

So, we now use the Jouanolou trick of § 1.14, pulling back to an affine bundle over $U$, so that $H$ becomes a projective $\mathcal {O}$-module. In particular, its connecting homomorphism to $R\pi _*\,\mathcal {L}_\beta [2]$ in the vertical exact triangle (4.19) is zero over the open neighbourhood $U$ (4.15) over which it is supported in degrees $-2$ and $-1$. (Recall our assumption (4.14) that $H^2(\mathcal {O}_S(D))=0$ for every effective $D$ of class $\beta$.) So, over $U$, we may choose the splittings marked with dotted arrows in (4.19).

Proof. Taking $H^0$ of (4.19) at a closed point $x=(I_1,I_2,L)$ of $U$ gives the following diagram.

Since this diagram commutes, the diagonal arrow is an injection. Thus, by the Nakayama lemma, $\mathrm {Cone}(u)$, and any base change of it, has $h^{<0}=0$.

Taking $H^1$ instead gives part of the following diagram.

Again, this commutes and the rows are exact. Since $\operatorname {Ext}^1(I_1,I_2\otimes L)$ surjects onto $\ker(a)$, in which $H^1(L)$ lies, the diagonal arrow is surjective. Thus, $\operatorname {Cone}(u)$ has $h^{\ge 1}=0$ and is a vector bundle, as claimed.

So, it follows immediately from Theorem 3.5 and the Jouanolou trick that

\[ \big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} = c_{n_1+n_2}\big(\!\operatorname{Cone}(u)(1)\big)\cap\big[S_\beta\big]^{\mathrm{red}}\times\big[S^{[n_1]}\times S^{[n_2]}\big]. \]

On $S\times S_\beta$, the tautological composition $\mathcal {O}(-1)\to \pi ^*\pi _*\mathcal {L}_\beta \to \mathcal {L}_\beta$ gives the universal section

(4.21)\begin{equation} \mathcal{O}\xrightarrow{\ s_\beta\ }\mathcal{L}_\beta(1) \quad\text{cutting out the universal divisor}\ \mathcal{D}_\beta\subset S\times S_\beta. \end{equation}

In particular, $\mathcal {L}_\beta (1)\cong \mathcal {O}(\mathcal {D}_\beta )$ on $S\times S_\beta$. Thus, $\operatorname {Cone}(u)(1)$ represents the Carlsson–Okounkov K-theory class of (1.14):

\[ \operatorname{Cone}(u)(1) = \mathsf{CO}_\beta^{[n_1,n_2]} = R\pi_*\,\mathcal{O}(\mathcal{D}_\beta)-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2(\mathcal{D}_\beta)). \]

Hence, we have proved the second formula in Theorem 5:

Corollary 4.22 If $H^2(\mathcal {O}_S(D)) =0$ for all effective divisors in class $\beta$, then

\[ \big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big) \cap \big[S^{[n_1]}\times S^{[n_2]}\big] \times \big[S_\beta\big]^{\mathrm{red}}. \]

4.3 Comparison of virtual cycles

To recover the full perfect obstruction theory of [Reference Gholampour, Sheshmani and YauGSY20a] or Vafa–Witten theory on $S_\beta ^{[n_1,n_2]}$ from (the virtual resolution of) a degeneracy locus, we would have to modify the three-term complexes $R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta )$ and $R\pi _*\,\mathcal {L}_\beta$ by $H^2(\mathcal {O}_S)$ terms to make them two-term. This cannot be done over $X=S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$.Footnote ¹³

However, up on (the product of $S$ with) $S^{[n_1]}\times S^{[n_2]}\times S_\beta$, we can use the universal section $s_\beta$ (4.21) of $\mathcal {L}_\beta (1)$, notice the twist!, together with a choice of splitting $R\pi _*\,\mathcal {O}\cong \bigoplus _{i=0}^2R^i\pi _*\,\mathcal {O}[-i]$, to induce maps

The lower row is an exact triangle. We apply the Jouanolou trick of § 1.14, pulling back from $S^{[n_1]}\times S^{[n_2]}\times S_\beta$ to an affine bundle over it. Here $R^2\pi _*\,\mathcal {O}$ becomes a projective $\mathcal {O}$-module, so the right-hand diagonal arrow becomes zero. Thus, we can fill in the dotted arrow, giving the following commutative diagram.

(4.23)

Here the right-hand column is an exact triangle, with $H:={\mathscr {E}}\!\,xt^2_\pi (\mathcal {O}/\mathcal {I}_1,\mathcal {L}_\beta (1))$ a vector bundle Serre dual to

\[ H^*\ \cong\ \big(K_S\otimes\mathcal{L}_\beta^{-1}(-1)\big)^{[n_1]}. \]

On second cohomology sheaves $h^2$ we claim that the following diagram gives the surjections.

(4.24)

The claim for the maps $b,c,d,f$ follows from zeros in the obvious exact sequences in which each sits. It then follows from the diagram that $a$ and $e$ are also onto. So,if we use the notation

\[ \tau(\mathcal{F}) := \operatorname{Cone}\!\big(R^2\pi_*\,\mathcal{O}[-2]\longrightarrow\mathcal{F}\big) \]

for any of the maps $a,b,d\!\,\circ \,\!a$ or $e\!\,\circ \!\,b$ in (

), then each $\tau (\mathcal {F})$ has $h^{\ge 2}=0$ and can be represented by a two-term complex of vector bundles. The bottom row and right-hand column of (

) now give the diagram

(4.25)

which should be compared to (4.19). Again, the column is an exact triangle, and again $H$ is a projective $\mathcal {O}$-module on the affine bundle, so has vanishing connecting homomorphism to the complex $\tau (R\pi _*\,\mathcal {L}_\beta (1))[2]$ supported in degrees $-2$ and $-1$. Thus, we may split the vertical exact triangle and fill in the dotted arrows on $S^{[n_1]}\times S^{[n_2]}\times S_\beta$.

4.4 Extension to ambient space

Let $B=\pi _*(\mathcal {L}(A))$ be the bundle (4.11). Applying (4.13) when $n_1=0=n_2$ gives an embedding $S_\beta \subseteq \mathbb {P}(B)/\operatorname {Pic}_\beta (S)$ and so

\[ S^{[n_1]}\times S^{[n_2]}\times S_\beta \subseteq \mathbb{P}(B)/X, \]

where $X=S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$, as in (4.5).

We apply the Jouanolou trick to $\mathbb {P}(B)$, giving an affine bundle over it whose total space is an affine variety. The restriction of this affine bundle to $S^{[n_1]}\times S^{[n_2]}\times S_\beta$ is therefore also an affine variety, to which the constructions of the last § 4.3 apply. As usual we suppress pullbacks to these affine bundles.

All of the terms in (4.23) extend to (the affine bundle over) $\mathbb {P}(B)$, as do the horizontal and vertical arrows (canonically). As for the diagonal arrows, their targets are quasi-isomorphic to complexes of vector bundles $G_0\to G_1\to G_2$, and $R^2\pi _*\,\mathcal {O}$ is a projective $\mathcal {O}$-module on an affine variety, so the arrows can be represented by maps $R^2\pi _*\,\mathcal {O}\to G_2$. Again, since $R^2\pi _*\,\mathcal {O}$ is a projective $\mathcal {O}$-module, these maps can be extended to (the affine bundle over) $\mathbb {P}(B)\supset S^{[n_1]}\times S^{[n_2]}\times S_\beta$. So, we get an extension of the diagram (4.25) to (the affine bundle over) $\mathbb {P}(B)$.

Since the surjectivity (4.24) is an open condition, all the terms of this extended diagram can be represented by two-term complexes of vector bundles over a neighbourhood $U$ of (the affine bundle over) $S^{[n_1]}\times S^{[n_2]}\times S_\beta$ inside (the affine bundle over) $\mathbb {P}(B)$. By shrinking $U$ if necessary, we may assume that it is affine.Footnote ¹⁴

So, finally, the splitting map $H\to \tau (R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {L}_\beta (1)))$ of the vertical exact sequence in (4.25) extends from (the affine bundle over) $S^{[n_1]}\times S^{[n_2]}\times S_\beta$ to its affine neighbourhood $U$ since $H|_U$ is a projective $\mathcal {O}$-module (and hence has vanishing connecting homomorphism to the complex $\tau (R\pi _*\,\mathcal {L}_\beta (1))[2]$ supported in degrees $-2$ and $-1$). So, we can fill in the dotted arrows, giving

(4.26)

Lemma 4.20 now applies verbatim to the map $u$ of (4.25) to show that

(4.27)\begin{equation} \operatorname{Cone}(u)\text{ is a vector bundle of rank }n_1+n_2\text{ over }U. \end{equation}

NextFootnote ¹⁵ we describe a nice form of these complexes $\tau (\mathcal {F})$ on the open set $U$.

Lemma 4.28 Over $U$ each of the terms $\tau (\mathcal {F})$ in (4.26) has a two-term resolution by vector bundles

(4.29)\begin{equation} \tau(\mathcal{F})(-1)\ \simeq\,\big\{B\longrightarrow F\big\}, \end{equation}

which, on dualising, induces the surjection $B^*\to \hspace {-3mm}\to h^0((\tau (\mathcal {F})(-1))^\vee )$ of (4.12). Moreover, the arrows in (4.26) may be taken to be genuine maps of complexes between these resolutions.

Proof. The final claim follows from the fact that on any affine variety like $U$, maps in the derived category between complexes of locally free sheaves are quasi-isomorphic to genuine maps of complexes.

We start by working downstairs on $X$. The composition

(4.30)\begin{equation} R\pi_*\,\mathcal{L}_\beta\xrightarrow{\ s_A\ } R\pi_*\,\mathcal{L}_\beta(A) = \pi_*\,\mathcal{L}_\beta(A) = B \end{equation}

induces the inclusion $H^0(L)\hookrightarrow H^0(L(A))$ on restricting to any point $x=(I_1,I_2,L)\in X$ and taking $h^0$. Therefore, its cone has cohomology in degrees $1,2$ after base change to any point, so it can be represented by a two-term complex of vector bundles $G_1\to G_2$, meaning that we can write

\[ R\pi_*\,\mathcal{L}_\beta\ \cong\ \big\{B\longrightarrow G_1\longrightarrow G_2\big\}. \]

By construction, this resolution dualises to induce the surjection $B^*\to \hspace {-3mm}\to h^0((R\pi _*\,\mathcal {L}_\beta )^\vee )$ of (4.12).

Now pull back to (the affine bundle over) $\mathbb {P}(B)/X$, twist by $\mathcal {O}_{\mathbb {P}(B)}(1)$ and use the map $R^2\pi _*\,\mathcal {O}[-2]\to R\pi _*\,\mathcal {L}_\beta (1)$ of (4.23) (which as we already noted extends to the affine bundle over $\mathbb {P}(B)$). It can be represented by a genuine map of complexes, so we can represent its cone as

\[ \tau\big(R\pi_*\,\mathcal{L}_\beta(1)\big)\,\cong\,\big\{B(1)\to G(1)\big\},\ \quad G:=\ker\big[G_1\oplus(R^2\pi_*\,\mathcal{O})(-1)\to\hspace{-3mm}\to G_2\big] \]

over the affine open $U$ on which the second arrow is a surjection. This gives the result for $\mathcal {F}=R\pi _*\,\mathcal {L}_\beta (1)$.

For $\mathcal {F}=\tau (R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta (1)))$, we can now write the exact triangle defined by $u$ (4.26) as

\[ \tau\big(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta(1))\big)\xrightarrow{\ u\ }\big\{B(1)\to G(1)\big\}\longrightarrow\operatorname{Cone}(u) \]

with the last arrow a genuine map of complexes of vector bundles on $U$. Thus, the first term can be written as the two-term complex of vector bundles

\[ \tau\big(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta(1))\big)\ \cong\ \big\{ B(1)\longrightarrow\operatorname{Cone}(u)\oplus G(1)\big\}. \]

A similar argument applies to $\tau (R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {L}_\beta (1)))$.

So, we can now see (the affine bundle over) $S^{[n_1,n_2]}_\beta$ as the zeros in $U$ of the section of $F(1)$ defined by the composition

(4.31)\begin{equation} \mathcal{O}_{\mathbb{P}(B)}(-1)\,{\hookrightarrow}\, B\longrightarrow F, \end{equation}

where $\{B\to F\}$ is the representative (4.29) of $\tau (R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta ))$. For the confused reader, we summarise the construction.

4.5 Summary

By Proposition 4.7, $S^{[n_1,n_2]}_\beta$ is the virtual resolution $\widetilde {D}_1$ of the $r=1$ degeneracy locus of the complex $R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta )$ over $X$. Since this is three-term, $\{B\to G_1\to G_2\}$, in general this does not give it a natural virtual cycle, but it does express it as the zero locus in $q\colon \mathbb {P}(B)\to X$ of the compositionFootnote ¹⁶

(4.32)\begin{equation} \mathcal{O}_{\mathbb{P}(B)}(-1)\,{\hookrightarrow}\, q^*B\longrightarrow q^*G_1. \end{equation}

We cannot modify $R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta )$ by an $R^2\pi _*\,\mathcal {O}$ term on $X$ to get a two-term degeneracy locus construction directly. But, we can after pulling back to $\mathbb {P}(B)$ and twisting by $\mathcal {O}(1)$, replacing (4.32) by

(4.33)\begin{equation} \mathcal{O}_{\mathbb{P}(B)}(-1)\,{\hookrightarrow}\, q^*B\longrightarrow\ker\big[q^*G_1\oplus(R^2\pi_*\,\mathcal{O})(-1)\to\hspace{-3mm}\to q^*G_2\big] \end{equation}

on the open set $U$, where the last arrow is a surjection. This is (4.31).

Theorem 4.34 $S^{[n_1,n_2]}_\beta$ is the zero locus of (4.31). The virtual cycle of the resulting perfect obstruction theory agrees with the nonreduced virtual cycles of [Reference Gholampour, Sheshmani and YauGSY20a, Reference Tanaka and ThomasTT20] whenever they are defined.Footnote ¹⁷ It satisfies

\[ \iota_*\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big) \cap\big[S^{[n_1]}\times S^{[n_2]}\big] \times \big[S_\beta\big]^{\operatorname{vir}} \]

under pushforward by $\iota \colon S^{[n_1,n_2]}_\beta \,{\hookrightarrow}\, S^{[n_1]}\times S^{[n_2]}\times S_\beta$.

Proof. Since $\tau (R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta ))$ differs from $R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta )$ only in $h^{\ge 1}$ (even after any base change), $h^0$ is unaffected. In other words, the zero loci of (4.31) and (4.32) are the same and the latter is $S^{[n_1,n_2]}_\beta$ by Proposition 4.7.

The virtual tangent bundle differs from the one in Theorem 4.16 by only $R^2\pi _*\,\mathcal {O}[-1]$, so its K-theory class is

(4.35)\begin{equation} -R{\mathscr{H}}\!om_{\pi}(\mathcal{I}_1,\mathcal{I}_1)^{}_0-R{\mathscr{H}}\!om_{\pi}(\mathcal{I}_2,\mathcal{I}_2)^{}_0 + R{\mathscr{H}}\!om_\pi\big(\mathcal{I}_1,\mathcal{I}_2(\mathcal{D}_\beta)\big)-R\pi_*\,\mathcal{O}, \end{equation}

agreeing with the ones from [Reference Gholampour, Sheshmani and YauGSY20a, Reference Tanaka and ThomasTT20]. Since virtual classes depend only on the K-theory class of the virtual tangent bundle, this makes the virtual classes of [Reference Gholampour, Sheshmani and YauGSY20a, Reference Tanaka and ThomasTT20] equal to the localised top Chern class

(4.36)\begin{equation} \big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = s^!\,[0_{F(1)}] \in A_{n_1+n_2+\operatorname{vd}(S_\beta)}\big(S^{[n_1,n_2]}_\beta\big) \end{equation}

of the bundle $F(1)\to U$ and its section $s$ defined by (4.31).

Setting $n_1=0=n_2$, we get

\[ \big[S_\beta\big]^{\operatorname{vir}} = t^!\,[0_{G(1)}] \in A_{\operatorname{vd}(S_\beta)}\big(S_\beta\big), \]

where $B\to G$ is the resolution of $\tau (R\pi _*\,\mathcal {L}_\beta )$ on the open subset of (the affine bundle over) $\mathbb {P}(B\to \operatorname {Pic}_\beta (S))$, which is the image of $U$ (in the affine bundle over $\mathbb {P}(B\to X)$), and $t$ is the induced section of $G(1)$. Pulling back to $U$ gives

(4.37)\begin{equation} \big[S_\beta\big]^{\operatorname{vir}}\times S^{[n_1]}\times S^{[n_2]} = t^!\,[0_{G(1)}], \end{equation}

where as usual we have suppressed the pullback map on $G(1)$ and its section $t$. Now the map $u$ of (4.26) induces, via Lemma 4.28, a map

\[ \widetilde{u} \,\colon\,F(1)\longrightarrow G(1) \]

such that $\widetilde {u} \circ s=t$. By (4.27), $\widetilde {u}$ is a surjection with kernel the vector bundle $\operatorname {Cone}(u)$, so applying Lemma 3.4 and using (4.37) gives

\begin{align*} \big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} & = s^!\,[0_{F(1)}] \\ & = c_{n_1+n_2}(\operatorname{Cone}(u))\cap t^!\,[0_{G(1)}] \\ & = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big)\cap\big[S^{[n_1]}\times S^{[n_2]}\big]\times\big[S_\beta\big]^{\operatorname{vir}}. \end{align*}

Remark We have now proved Theorem 5 from the Introduction. When $p_g(S)>0$ and $H^2(\mathcal {O}_S(D))=0$ for all effective $D$ in class $\beta$, so that the reduced class is defined, the virtual cycle is automatically zero by the identity

(4.38)\begin{equation} \big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c^{}_{p_g}(\pi_*\,\mathcal{O})\cap\big[S^{[n_1,n_2]}_\beta\big]^{\mathrm{red}} = 0 \end{equation}

of Theorem 3.5.

6.1 Vafa–Witten invariants

On a polarised surface $(S,\mathcal {O}_S(1))$, Vafa–Witten theory is an enumerative sheaf-counting theory [Reference Tanaka and ThomasTT20]. The ‘instanton contributions’ compute virtual Euler numbers of moduli spaces of Gieseker stable sheaves of fixed rank $r$ and determinant $L$ on $S$, as studied by Göttsche–Kool [Reference Göttsche and KoolGK20].

This leaves the ‘monopole contributions’, which are integrals over moduli spaces of chains of sheaves (of total rank $r$) with nonzero maps between them. When the individual sheaves have rank 1, we get nested ideal sheaves (all tensored by a line bundle). And, when $r$ is prime and $p_g(S)>0$, a vanishing theorem [Reference ThomasTho20, § 5.2] implies that all nonzero contributions come from nested Hilbert schemes. A similar result applies in the semistable case [Reference Tanaka and ThomasTT17] and the refined version of both theories [Reference ThomasTho20].

The simplest case (for simplicity) is the monopole contribution to the Vafa–Witten invariants in rank 2 and fixed determinant when slope stability is the same as slope semistability. Here we get integrals over the virtual cycles of nested Hilbert schemes of pairs $I_1(-D)\subseteq I_2\subseteq \mathcal {O}_S$ for any $[D]=\beta$ satisfying the slope stabilityFootnote ¹⁸ condition $\deg (K_S(-D))>0$. We then sum over all $\beta ,\,n_1,\,n_2$ giving the same total Chern classes and then multiply by a factorFootnote ¹⁹ of $2^{2h^1(\mathcal {O}_S)}$.

The integrand is $1/e(N^{\operatorname {vir}})$, where $N^{\operatorname {vir}}$ is the moving part of the Vafa–Witten obstruction theory and $e$ is the $\mathbb {C}^*$-equivariant Euler class. So, by [Reference Tanaka and ThomasTT20, § 8], we want to compute the integral of

\[ \frac{e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1(-\mathcal{D}_\beta)K_S\,\mathfrak{t}))\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1(-\mathcal{D}_\beta),\mathcal{I}_2K_S^{-1}\mathfrak{t}^{-1}))} {e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_1K_S\mathfrak{t})^{}_0)\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_2K_S\mathfrak{t}))\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1(-\mathcal{D}_\beta)K_S^2\mathfrak{t}^2))} \]

over $[S_\beta ^{[n_1,n_2]}]^{\operatorname {vir}}$. Write $\mathcal {O}(\mathcal {D}_\beta )$ as $\mathcal {L}_\beta (1)$ as in (4.21). Let $h:=c_1(\mathcal {O}(1))$ on $\mathbb {P}(B)=\mathbb {P}(\pi _*\,\mathcal {L}_\beta (A))$ and its subschemes $S_\beta ^{[n_1,n_2]}$. Then each term $e(\,{\cdot }\,)$ takes the form

\[ e\big((E_0-E_1)(1)\big) = \frac{c_{r_0}(E_0(1))}{c_{r_1}(E_1(1))} = \frac{ \sum_{i=0}^{r_0}c_{r_0-i}(E_0)\cup h^i}{\sum_{i=0}^{r_1}c_{r_1-i}(E_1)\cup h^i}, \]

where the $E_i$ have only nonzero weight spaces (so the quotient makes sense in localised equivariant cohomology) and ranks $r_i$. Therefore, expanding in powers of $h^i$ with coefficients pulled back from $S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$, we get an expression

\[ \sum_i\int_{[S_\beta^{[n_1,n_2]}]^{\operatorname{vir}}}\rho^*\alpha_i\cup h^i = \sum_i\int_{\rho_*(h^i\cap[S_\beta^{[n_1,n_2]}]^{\operatorname{vir}})}\alpha_i, \]

where $\rho$ is part of the following commutative diagram.

(6.1)

Here $\mathsf {AJ}\colon S_\beta \to \operatorname {Pic}_\beta (S)$ is the Abel–Jacobi map. So, we should calculate the pushdown $\rho _*(h^i\cap [S_\beta ^{[n_1,n_2]}]^{\operatorname {vir}})$. This will prove Theorems 6–8 from the Introduction, and compute Vafa–Witten invariants in terms of integrals over the smooth spaces $X=S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$.

Let

\[ \mathsf{SW}_\beta := \begin{cases} \deg\,[S_\beta]^{\operatorname{vir}}, & \operatorname{vd}_\beta=\frac12\beta(K_S-\beta)=0, \\ 0, & \mathrm{otherwise}\end{cases} \]

denote the Seiberg–Witten invariant [Reference Dürr, Kabanov and OkonekDKO07, Reference Chang and KiemCK13] of $S$ in class $\beta \in H^2(S,\mathbb {Z})$. Things are easiest when $b^+(S)>1$, i.e. $p_g(S)>0$.

Theorem 6.2 Suppose that $p_g(S)>0$. Then, in $H_*\big (S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)\big )$,

\begin{align*} \rho_*\big [S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}}& = \mathsf{SW}_\beta\cdot c_{n_1+n_2}\big(\!-\!R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes L)\big)\times[L],\\ \rho_*\big(\,\!h^i\cap\big [S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}}\big)& = 0 \quad\mathrm{for}\ i>0, \end{align*}

where $L\in \operatorname {Pic}_\beta (S)$, so $[L]$ is the generator of $H_0(\operatorname {Pic}_\beta (S),\mathbb {Z})$.

Proof. If $\mathsf {AJ}_*[S_\beta ]^{\operatorname {vir}}=0$, then $\mathsf {SW}_\beta =0$, so by our comparison result of Theorem 4.34 (or Theorem 5 in the Introduction) both sides of the claimed identity are zero. So, we assume that $\mathsf {AJ}_*[S_\beta ]^{\operatorname {vir}}\ne 0$.

By [Reference Dürr, Kabanov and OkonekDKO07, Definition 3.19], this means that $\beta$ is a basic class. And, by [Reference Dürr, Kabanov and OkonekDKO07, Proposition 3.20] $S$ is of simple type, which implies that $\operatorname {vd}_\beta =0$.

Thus, $[S_\beta ]^{\operatorname {vir}}$ is a zero-dimensional class. Firstly, this implies that $h^i|_{[S_\beta ]^{\operatorname {vir}}}=0$ for $i>0$, and so the second claimed result. And, secondly, on restriction to

\[ S\times S^{[n_1]}\times S^{[n_2]}\times\big[S_\beta\big]^{\operatorname{vir}}, \]

the line bundle $\mathcal {L}_\beta (1)=\mathcal {O}(\mathcal {D}_\beta )$ is topologically equivalent to (the pullback of) $L$ for any $L\in \operatorname {Pic}_\beta (S)$. In particular, in the formula of Theorem 4.34,

(6.3)\begin{equation} \iota_*\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(\mathsf{CO}_\beta^{[n_1,n_2]}\big) \cap\big[S^{[n_1]}\times S^{[n_2]}\big] \times \big[S_\beta\big]^{\operatorname{vir}} \end{equation}

we may replace $\mathsf {CO}^{[n_1,n_2]}_\beta$ (1.14) by $R\pi _*\,L-R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes L)$. Applying $(\operatorname {id}\times \operatorname {id}\times \,\mathsf {AJ})_*$ then gives

\[ \rho_{*}\big [S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes L)\big)\times\deg\big [S_\beta\big]^{\operatorname{vir}}\,[\mathrm{point}], \]

where $[\,$point$\,]=[L]$ is the generator of $H_0(\operatorname {Pic}_\beta (S),\mathbb {Z})$.

Now we suppose that $b^+(S)=1$, i.e. $p_g(S)=0$. Then $S$ may not be of simple type, so we have to consider higher dimensional Seiberg–Witten moduli spaces and the higher Seiberg–Witten invariants [Reference Dürr, Kabanov and OkonekDKO07]

\[ \mathsf{SW}_\beta^{\,j} := \mathsf{AJ}_*\big(h^j \, \cap \, [S_\beta]^{\operatorname{vir}}\big) \in H_{2(\operatorname{vd}_\beta-j)}(\operatorname{Pic}_\beta(S)) = \wedge^{2(\operatorname{vd}_\beta-j)}H^1(S). \]

Theorem 6.4 If $p_g(S)=0$, then, in $H_{2(n_1+n_2+\operatorname {vd}_\beta -i)}\big (S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)\big )$,

\begin{align*} & \rho_* \big(h^i\cap\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}}\big) = \sum_{j=0}^{n_1+n_2}c_{n_1+n_2-j}\big(R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big)\cup\mathsf{SW}_\beta^{\,i+j}. \end{align*}

Proof. We again use our comparison formula (6.3). By [Reference ManivelMan16, Proposition 1],Footnote ²⁰

\begin{align*} c^{}_{n_1+n_2}\big(\mathsf{CO}^{[n_1,n_2]}_\beta\big)& = c^{}_{n_1+n_2}\big(R\pi_*\,\mathcal{L}_\beta(1)-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta(1))\big)\\ & = \sum_{j=0}^{n_1+n_2}c_{n_1+n_2-j}\big(R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big)\cup h^j. \end{align*}

So, $(\operatorname {id}\times \operatorname {id}\times \,\mathsf {AJ})_*(h^i\,\cap$ (6.3)$)=\rho _*(h^i\cap [S^{[n_1,n_2]}_\beta ]^{\operatorname {vir}})$ is

\[ \sum_{j=0}^{n_1+n_2}c_{n_1+n_2-j}\big(R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big)\cup\mathsf{AJ}_*\big(h^{i+j} \, \cap [S_\beta]^{\operatorname{vir}}\big). \]

We can get simpler formulae by splitting into the cases that $H^2(L)=0$ for all $L\in \operatorname {Pic}_\beta (S)$ or not. By Serre duality, this is the condition that $\beta ^\vee :=K_S-\beta$ is not effective or effective, respectively.

Proof. By the comparison result (6.3), both sides of the first identity vanish if $\operatorname {vd}_\beta <0$ or $\beta$ is not effective. So, we may assume that both $\beta ,\,\beta ^\vee$ are effective and $\operatorname {vd}_\beta \ge 0$. By [Reference Dürr, Kabanov and OkonekDKO07, Corollary 3.15], this implies that $\operatorname {vd}_\beta =0$ and $h^1(\mathcal {O}_S)=1$.

So, just as in (6.3), $\mathcal {O}(1)$ is trivial on the zero-dimensional $[S_\beta ]^{\operatorname {vir}}$ and we may replace $\mathsf {CO}^{[n_1,n_2]}_\beta$ in (6.3) by $R\pi _*\,L-R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes L)$ for one $L\in \operatorname {Pic}_\beta (S)$. So, applying $(\operatorname {id}\times \operatorname {id}\times \,\mathsf {AJ})_*$ to (6.3) gives

\[ \rho_*\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} = c_{n_1+n_2}\big(R\pi_*\,L-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes L)\big)\times\deg\,[S_\beta]^{\operatorname{vir}}\times[\mathrm{point}]. \]

This deals with $i=0$. When $i>0$, the left-hand side of the required identity vanishes by $\operatorname {vd}_\beta =0$ and (6.3), while the right-hand side vanishes by the generalised Carlsson–Okounkov vanishing of [Reference Gholampour and ThomasGT20, Theorem 3].

For the second result, we may assume that $\beta$ is of type $(1,1)$, otherwise $\operatorname {Pic}_\beta (S)$ is empty and both sides of the identity vanish. Since we also have $H^2(L)=0$ for all $L\in \operatorname {Pic}_\beta (S)$,the reduced cycle of § 4.1 is defined and, since $p_g(S)=0$, it equals the virtual cycle. Therefore, by Theorem 4.16 (or Theorem 3 of the Introduction), the pushforward of the virtual cycle to $\mathbb {P}(B)$ is given by

\begin{align*} &c^{}_{n_1+n_2+k}\big(B(1)-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta(1))\big)\\ &\quad = \sum_{j=0}^{n_1+n_2+k}c_{n_1+n_2+k-j}\big(B-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big)\cup c_1(\mathcal{O}(1))^j, \end{align*}

again by [Reference ManivelMan16, Proposition 1]. Here $k=\operatorname {rank}(B)-\operatorname {vd}_\beta -\chi (\mathcal {O}_S)$.

We now cup with $h^i=c_1(\mathcal {O}(1))^i$ and push down $q\colon \mathbb {P}(B)\to S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$ (which restricts to the map $\rho$ of (6.1)). Using the fact that $q_*\,h^{i+j}$ is the Segre class $s_{i+j-\chi (L(A))+1}(B)$ for $i+j\ge \chi (L(A))-1$, and zero otherwise, we find that $\rho _{*} (h^i \cap [S^{[n_1,n_2]}_\beta ]^{\operatorname {vir}})$ is

\begin{align*} & \sum_{j=\chi(L(A))-1-i}^{n_1+n_2+k}c_{n_1+n_2+k-j}\big(B-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big)\cup s_{i+j-\chi(L(A))+1}(B)\\ & \qquad =c_{n_1+n_2+i+k-\operatorname{rank}(B)+1}\big(-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes \mathcal{L}_\beta)\big). \end{align*}

Finally, we use the standard duality in Seiberg–Witten theory under $\beta \leftrightarrow \beta ^\vee$ to give interesting dualities between invariants of nested Hilbert schemes under $\beta \leftrightarrow \beta ^\vee$ and $n_1\leftrightarrow n_2$. Define the map

\[ \rho^\vee\,\colon\ S^{[n_2,n_1]}_{\beta^\vee}\longrightarrow S^{[n_1]}\times S^{[n_2]}\times \operatorname{Pic}_\beta(S) \]

by replacing $\beta \leftrightarrow \beta ^\vee ,\ n_1\leftrightarrow n_2$ in $\rho \colon S^{[n_1,n_2]}_\beta \longrightarrow S^{[n_1]}\times S^{[n_2]}\times \operatorname {Pic}_\beta (S)$ and then composing with $L\mapsto K_S\otimes L^{-1}\colon$ $\operatorname {Pic}_{\beta ^\vee }(S)\to \operatorname {Pic}_\beta (S)$.

Proof. By Theorem 6.2 with $\beta \leftrightarrow \beta ^\vee ,\ n_1\leftrightarrow n_2$ and $L\leftrightarrow K_S\otimes L^{-1}$,

\[ \rho^\vee_*\big[S^{[n_2,n_1]}_\beta\big]^{\operatorname{vir}} = \mathsf{SW}_{\beta^\vee}\cdot c_{n_1+n_2}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1\cdot K_S\cdot L^{-1})\big)\times[\mathrm{point}] \]

for a fixed $L\in \operatorname {Pic}_\beta (S)$. For $p_g(S)>0$, Seiberg–Witten invariants have the standard duality

\[ \mathsf{SW}_\beta = (-1)^{\chi(\mathcal{O}_S)}\,\mathsf{SW}_{\beta^\vee}. \]

(For instance, in [Reference Dürr, Kabanov and OkonekDKO07], this is part of Conjecture 0.1, and it is then shown that this would follow from proving that $\deg \,[S_{K_S}]^{\operatorname {vir}}=(-1)^{\chi (\mathcal {O}_S)}$ on all minimal general type surfaces. This latter identity was proved in [Reference Chang and KiemCK13].) And,

\[ c_{n_1+n_2}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1\cdot K_S\cdot L^{-1})\big)= (-1)^{n_1+n_2}c_{n_1+n_2}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\cdot L)\big) \]

by Serre duality down the fibres of $\pi$. This gives the first identity for $i=0$. For $i>0$, it is trivial by Theorem 6.2.

To prove the second identity, we distinguish three cases. If $\beta ^\vee$ is not effective, i.e. $H^2(L)=0$ for all $L\in \operatorname {Pic}_\beta (S)$, then the result is the last part of Theorem 6.5. Similarly, if $\beta$ is not effective, then we apply the last part of Theorem 6.5 with $\beta \leftrightarrow \beta ^\vee ,\,n_1\leftrightarrow n_2$. By $K_S\otimes \mathcal {L}_{\beta ^\vee }^{-1}=\mathcal {L}_\beta ,\ \operatorname {vd}_\beta =\operatorname {vd}_{\beta ^\vee }$ and Serre duality down the fibres of $\pi$,

\[ c_{d+i}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1\otimes\mathcal{L}_{\beta^\vee})\big) = (-1)^{d+i}c_{d+i}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_{\beta})\big), \]

which gives the required result.

Finally, we consider $\beta$ and $\beta ^\vee$ both effective. If $\operatorname {vd}_\beta <0$, then the left-hand side of the identity is zero by the comparison result of Theorem 5. On the right-hand side, we have

\begin{align*} & c_{n_1+n_2+h^1(\mathcal{O}_S)-\operatorname{vd}_\beta}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_{\beta})\big)\\ &\quad = \sum_{i\ge0}c_i(-R\pi_*\,\mathcal{L}_\beta)\cdot c_{n_1+n_2+h^1(\mathcal{O}_S)-\operatorname{vd}_\beta-i}\big(R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_{\beta})\big). \end{align*}

The first term of the sum vanishes for $i>\dim \operatorname {Pic}_\beta (S)=h^1(\mathcal {O}_S)$, while the second term vanishes for $i < h^1(\mathcal {O}_S)-\operatorname {vd}_\beta$ by the generalised Carlsson–Okounkov vanishing of [Reference Gholampour and ThomasGT20, Theorem 3]. When $\operatorname {vd}_\beta <0$, this makes the whole sum vanish.

So, we may assume that $\operatorname {vd}_\beta \ge 0$ and $\beta ,\,\beta ^\vee$ are both effective. This implies that $\operatorname {vd}_\beta =0$ and $h^1(\mathcal {O}_S)=1$ by [Reference Dürr, Kabanov and OkonekDKO07, Corollary 3.15].

So, just as in (6.3), $\mathcal {O}(1)$ is trivial on the zero-dimensional $[S_\beta ]^{\operatorname {vir}}$ and we may replace $\mathsf {CO}^{[n_1,n_2]}_\beta$ in (6.3) by the pullback of $R\pi _*\,\mathcal {L}_\beta -R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta )$. Thus, pushing (6.3) down by $(\operatorname {id}\times \operatorname {id}\times \,\mathsf {AJ}_\beta )_*$ gives $\rho _*[S^{[n_1,n_2]}_\beta ]^{\operatorname {vir}}$ as

(6.7)\begin{equation} c_{n_1+n_2}\big(R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\big)\cap\big[S^{[n_1]}\times S^{[n_2]}\big]\times\mathsf{AJ}_{\beta*}\,[S_\beta]^{\operatorname{vir}}. \end{equation}

Swapping $\beta \leftrightarrow \beta ^\vee$ and $n_1\leftrightarrow n_2$ shows that $\rho ^\vee _*[S^{[n_2,n_1]}_{\beta ^\vee }]^{\operatorname {vir}}$ is

\[ c_{n_1+n_2}\big(R\pi_*\,\mathcal{L}_{\beta^\vee}\,-\,R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1\otimes\mathcal{L}_{\beta^\vee})\big)\cap\big[S^{[n_1]}\times S^{[n_2]}\big]\times\mathsf{AJ}_{\beta^\vee*}\,[S_{\beta^\vee}]^{\operatorname{vir}}. \]

By Serre duality down $\pi$ and the identity $c_i(E)=(-1)^ic_i(E^\vee )$, this is

(6.8)\begin{align} & (-1)^{n_1+n_2}c_{n_1+n_2}\big(\,R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\big) \cap \big[S^{[n_1]}\times S^{[n_2]}\big] \times\mathsf{AJ}_{\beta^\vee*}\, [S_{\beta^\vee}]^{\operatorname{vir}}. \end{align}

We have noted that $h^1(\mathcal {O}_S)=1$, so $\chi (\mathcal {O}_S)=0$. Thus, identifying $\operatorname {Pic}_\beta (S)\cong \operatorname {Pic}_{\beta ^\vee }(S)$ via $L\mapsto K_S\otimes L^{-1}$, [Reference Dürr, Kabanov and OkonekDKO07, Theorem 3.16] gives

\[ \mathsf{AJ}_{\beta*}\,[S_\beta]^{\operatorname{vir}} = \mathsf{AJ}_{\beta^\vee*}\,[S_{\beta^\vee}]^{\operatorname{vir}}+c_1(-R\pi_*\,\mathcal{L}_\beta). \]

Since $\operatorname {Pic}_\beta (S)$ is one dimensional, $c^{}_{\ge 2}(-R\pi _*\,\mathcal {L}_\beta )=0$. And, $c_{\ge n_1+n_2+1}(R\pi _*\,\mathcal {L}_\beta -R{\mathscr{H}}\!om_\pi (\mathcal {I}_1,\mathcal {I}_2\otimes \mathcal {L}_\beta ))=0$ by the generalised Carlsson–Okounkov vanishing of [Reference Gholampour and ThomasGT20, Theorem 3]. Therefore,

\begin{align*} & c_1(-R\pi_*\,\mathcal{L}_\beta)\,c_{n_1+n_2}\big(\!\,R\pi_*\,\mathcal{L}_\beta-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\big)\\ &\qquad = c_{n_1+n_2+1}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\big). \end{align*}

Substituting this into (6.8) and comparing with (6.7) yields

\begin{align*} \rho^\vee_*\big[S^{[n_2,n_1]}_{\beta^\vee}\big]^{\operatorname{vir}} & = (-1)^{n_1+n_2}\rho_*\big[S^{[n_1,n_2]}_\beta\big]^{\operatorname{vir}} \\ & \quad -(-1)^{n_1+n_2}c_{n_1+n_2+1}\big(\!-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2\otimes\mathcal{L}_\beta)\big)\,\cap \big[S^{[n_1]}\times S^{[n_2]}\big]. \end{align*}

Multiplying by $(-1)^{n_1+n_2}$ and noting that $d=n_1+n_2+1$ then gives the required result for $i=0$.

For $i>0$, the restrictions of $h^i$ to both $[S_\beta ]^{\operatorname {vir}}$ and $[S^{[n_1,n_2]}_\beta ]^{\operatorname {vir}}$ are trivial (the former by $\operatorname {vd}_\beta =0$, the latter by the comparison result Theorem 5).

Proof. We sketch the argument for two-step nested Hilbert schemes; the general case is no more complicated. Handling stability as in Footnote 18, we express the invariants as a sum over $\beta$ with $\deg \beta <\deg K_S$ of terms

\[ \mathsf{VW}_{2,\beta,n} = \sum_{n=n_1+n_2} \int_{[S^{[n_1,n_2]}_\beta]^{\operatorname{vir}}}\frac{1}{e(N^{\operatorname{vir}})}. \]

Let $L$ be any line bundle with $c_1(L)=\beta$. Applying Theorem 6.2 gives

(6.10)\begin{equation} \mathsf{VW}_{2,\beta,n} = \mathsf{SW}_\beta \sum_{n=n_1+n_2} \int_{S^{[n_1]}\times S^{[n_2]}} A(\mathcal{I}_1,\mathcal{I}_2,L), \end{equation}

where the integrand $A(\mathcal {I}_1,\mathcal {I}_2,L)$ is

\[ \frac{c_n(-R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2 L))\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2L,\mathcal{I}_1K_S\,\mathfrak{t}))\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_2LK_S^{-1}\mathfrak{t}^{-1}))} {e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_1,\mathcal{I}_1K_S\mathfrak{t})^{}_0)\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_2K_S\mathfrak{t}))\ e(R{\mathscr{H}}\!om_\pi(\mathcal{I}_2,\mathcal{I}_1L^{-1}K_S^2\mathfrak{t}^2))}\,. \]

The coefficient $\mathsf {SW}_\beta$ is proved in [Reference Dürr, Kabanov and OkonekDKO07, Reference Chang and KiemCK13] to be a Seiberg–Witten invariant, which is an oriented diffeomorphism invariant when $p_g(S)>0$.Footnote ²¹

We would now like to apply the inductive method of [Reference Ellingsrud, Göttsche and LehnEGL01], which deals with integrals over single Hilbert schemes $S^{[n_1]}$. This is adapted in [Reference Göttsche, Nakajima and YoshiokaGNY08, § 5]Footnote ²² to deal with products of Hilbert schemes $S^{[n_1]}\times S^{[n_2]}$, essentially by applying [Reference Ellingsrud, Göttsche and LehnEGL01] to the disjoint union $T:=S\sqcup S$ and its Hilbert schemes

\[ T^{[n]} = \bigsqcup_{n=n_1+n_2} S^{[n_1]}\times S^{[n_2]}. \]

Let $\mathfrak {I}_1$ and $\mathfrak {I}_2$ denote the ideal sheaves on $S\times T^{[n]}$ whose restrictions to $S\times S^{[n_1]} \times S^{[n_2]}$ are $\mathcal {I}_1$ and $\mathcal {I}_2$, respectively. Then (6.10) can be rewritten

\[ \mathsf{VW}_{2,\beta,n} = \mathsf{SW}_\beta \int_{T^{[n]}} A(\mathfrak{I}_1,\mathfrak{I}_2,L). \]

The induction writes this as an integral over $T^{[n-1]}\times S$, then $T^{[n-2]}\times S^2$ and so on, finally giving an integral over $S^n$. The first step pulls back along the generically finite map $\psi$, divides by $\deg \psi$ and then pushes down $\sigma$, in the following diagram.

(6.11)

We split $T^{[n-1,n]}$ into two (unions of) connected components

\[ \biggl(\bigsqcup_{n_1+n_2=n,\ n_1\ge1}S^{[n_1-1,n_1]}\times S^{[n_2]}\biggr)\ \sqcup\ \biggl(\bigsqcup_{n_1+n_2=n,\ n_2\ge1}S^{[n_1]}\times S^{[n_2-1,n_2]}\biggr) \]

and restrict the diagram to each in turn; we describe the first. On each of its components $\sigma$ is the projection down the cone $\mathbb {P}^*(I_Z)\to S^{[n_1-1]}\times S$, where $Z=Z_{n_1-1}\subset S^{[n_1-1]}\times S$ is the universal subscheme, multiplied by the identity on $S^{[n_2]}$. As such it carries a line bundle $\mathcal {O}(1)$; it is the kernel of $\mathcal {O}^{[n_1]}\to \hspace {-3mm}\to \mathcal {O}^{[n_1-1]}$ in the obvious notation.

By [Reference Göttsche, Nakajima and YoshiokaGNY08, (5.4)] and [Reference Ellingsrud, Göttsche and LehnEGL01, (10)], the integrand $A(\mathfrak {I}_1,\mathfrak {I}_2,L)$ pulls back to a rational function in $\mathfrak {t}$ and the Chern classes of:

1. (1) $R{\mathscr{H}}\!om_\pi (\mathfrak {I}_i,\mathfrak {I}_j\cdot \xi )$ pulled back from $T^{[n-1]}$, where $\xi$ is a tensor product of powers of $L$ and $K_S$;

2. (2) $\mathfrak {I}_1, \mathfrak {I}_2$ pulled back from $T^{[n-1]}\times S$;

3. (3) $L$ and $T_S$ pulled back from $S$; and

4. (4) the line bundle $\mathcal {O}(1)$.

Taking the coefficient of $\mathfrak {t}^0$ gives a polynomial in these terms. Since (1)–(3) are pulled back from $T^{[n-1]}\times S$, we can push down by $\sigma _*$ using the projection formula and [Reference Ellingsrud, Göttsche and LehnEGL01, Lemma 1.1] to replace powers of $c_1(\mathcal {O}(1))$ by polynomials in the Chern classes of $\mathfrak {I}_1$ and $\mathfrak {I}_2$. This completes the first step.

The second (and later) steps are similar, with (6.11) replaced by the following diagram.

The pullback and pushdown handle classes pulled back from either factor of $T^{[n-1]}\times S$, just as before. The new issue is to deal with Chern classes of $\mathfrak {I}_1$ and $\mathfrak {I}_2$ on $T^{[n-1]}\times S$. Here we use [Reference Göttsche, Nakajima and YoshiokaGNY08, (5.2)] to express them as pullbacks from $T^{[n-2]}\times S\times S$. So, the induction continues, just as in [Reference Ellingsrud, Göttsche and LehnEGL01, Proposition 3.1].

The final result is an integral over $S^n$, which can be written as a polynomial in the numbers $c_1(S)^2,\,c_2(S),\,\beta ^2$ and $c_1(S).\beta$. See also [Reference Göttsche, Nakajima and YoshiokaGNY08, Lemma 5.5] and [Reference LaarakkerLaa18, Proposition 7.2], or [Reference Kool and ThomasKT14b, § 3] when $n_1=0$.

Since $e(S),\,p_1(S)$ and the intersection form are oriented diffeomorphism invariants of $S$, so are $c_1(S)^2,\,c_2(S),\,\beta ^2$. This leaves $c_1(S).\beta$, but, as an observant referee pointed out, we may assume that this equals $-\beta ^2$; otherwise $\operatorname {vd}_\beta \ne 0$, so $\mathsf {SW}_\beta =0$ and the integral (6.10) vanishes.