OPERATOR CALCULUS FOR Q̃-POLYNOMIALS AND SCHUBERT POLYNOMIALS

Alain Lascoux and Piotr Pragacz

dédié à Maria

Contents:

Introduction

1. Divided differences associated with the hyperoctahedral groups.

2. Reproducing kernels and a vanishing property.

3. Action of ▽ on the basis {𝔖_{μ} · Q̃_{I}} - an inductive approach.

4. Action of ▽ on the basis {𝔖_{μ} · Q̃_{I}} via the vanishing property.

5. Key formula and vertex operators.

Appendix A (written in collaboration with J. Ratajski):

Symplectic Schubert polynomials à la polonaise. Appendix B: Three geometric applications. References

Typeset by AMS-TEX

Introduction

The goal of the present paper is to develop convenient algebraic tools to compute in the cohomology ring of Lagrangian flag manifolds as well as with Lagrangian degeneracy loci. There is quite a long series of papers devoted to this goal overlapping among others [B-G-G], [B-H], [B], [D1,2], [F-K], [F1,2], [H-B] and [P-R1,2]. The present paper will certainly not end this list. The cohomological formulas related to a symplectic form are extremely useful for computing in the cohomology (or Chow) rings of various moduli spaces including the moduli spaces of curves and the moduli spaces of abelian varieties in characteristic p. This is, perhaps, a raison d’être for the present paper.

Our basic tool here is a certain basis which is orthonormal with respect to the scalar product defined by the top symplectic divided difference. This basis has a natural geometric explanation. Consider a vector bundle V of rank 2n, equipped with a nondegenerate symplectic form. Then the Lagrangian flag bundle parametrizing all complete Lagrangian flags in V fibers over the Lagrangian Grassmann bundle parametrizing Lagrangian n-bundles with a fiber being the “usual” flag variety of all complete flags in an n-dimensional vector space. The basis is a natural “product-basis” associated with this fibration. It consists of products of Q̃-polynomials from [P-R2] (which provide a basis for the cohomology of the base of the fibration) times Schubert polynomials (which provide a basis for the cohomology of the fiber). We extensively investigate the action of different operators (notably divided differences) on this basis. It appears that such tools as a certain vanishing property (which is much in spirit of interpolation theory) and vertex operators (which have their origin in the theory of infinite-dimensional Lie algebras) are extremely useful to achieve this goal. The main technical result of the present paper is the “key formula” (Theorem 5.1). Many formulas reduce to the computation of the image of products of the type x_{1}^{k}Q̃_{J} under divided differences. The main algebraical upshot of this work is a study of the structure of a polynomial ring as a free module over the ring of polynomials symmetrical in the squares of the variables.

Even if the elements of the basis in question do not represent the Schubert classes in the Lagrangian flag manifold, they are very useful in the study of such classes. Recall that the polynomial representatives for those classes were defined already in [B-G-G] and [D1,2]. Recently, some algebro-combinatorial constructions of such representatives were studied in [B-H] and [F-K] and some other appeared implicitly in [F1,2]. Following [P-R2], we study in this paper still other polynomial representatives of Lagrangian Schubert classes in Appendix A. They seem to be well suited for the purposes of algebraic geometry. They are defined with the help of divided differences starting from some very simple “top” polynomial, and for maximal Grassmannian elements they are equal to certain Q̃-polynomials - exactly those predicted for them by the Giambelli-type formula from [P, Sect.6]. Also, as shown in the present paper, these polynomials behave well w.r.t. different operators which geometrically often correspond to Gysin maps. Symplectic Schubert polynomials à la polonaise are not positive in general (though involve much smaller quantities of negative monomials than the original polynomials of [B-G-G] and [D1,2]). In general, these polynomials involve much fewer monomials than the other known families of Lagrangian Schubert classes representatives; hence they are also better suited for numerical computations.

This paper contains also some simple proofs of previously known results: a new operator proof of the Giambelli-type formula for maximal Lagrangian Schubert classes from [P, Sect.6], and a purely algebraic proof of the congruence modulo the corresponding ideal of two (competing) expressions for diagonals from [F1,2] and [P-R2].

The article is organized as follows.

In Section 1 we collect needed information on the hyperoctahedral groups and divided differences associated with them. Moreover, we recall two families of polynomials crucial for this paper: Schubert polynomials and Q̃-polynomials. Several scalar products on polynomial rings are also discussed together with their orthogonality properties.

In Section 2 we study two reproducing kernels for a certain scalar product defined on the ring of symmetric polynomials with values in the ring of symmetric polynomials in the squares of the variables. These kernels stem from [F1,2] and [P-R2] and are motivated by geometry. More precisely, they are polynomial representatives of the classes of relative diagonals in Lagrangian Grassmann bundles. For a recent geometric development of this idea, we refer the reader to [G]. We prove that these kernels satisfy a certain vanishing property that characterizes them almost completely (see [L-S] for the type A_{n}).

In Section 3 we derive in an elementary way (invoking only simple properties of divided differences and the linearity formula for Q̃-polynomials) how a certain key operator ▽ acts on our product-basis.

In Section 4 we achieve the samy goal using the vanishing property. The proof is much shorter, but it relies on the operator characterization of Q̃-polynomials.

Section 5 is the heart of the paper. The key formula is proven there with the help of vertex operators associated with Schur’s P -functions and Q′-functions from [L-L-T2]. The starting point of our approach is a presentation of the “exceptional” divided difference ∂_{0} as an (infinite) differential operator. This observation is, in fact, the main motivation of using the vertex operators in various calculations in this section.

Appendix A contains the proofs of two crucial properties of symplectic Schubert polynomials à la polonaise i.e. stability and the maximal Grassmannian property. Using the key formula, we establish in a purely algebraic way the above mentioned operator characterization of Q̃-polynomials. The table of the polynomials for n = 3 is also included.

In Appendix B we apply the above calculations to give a new operator proof of the Lagrangian Giambelli-type formula, to derive a new Gysin formula as well as to provide a new formula for Lagrangian flagged degeneracy loci.

In the present paper we prove, among others, the results announced in [P-R2, Appendix B]. We plan to present the orthogonal variants of this theory in a separate publication.

Acknowledgements. The authors started to work on this project during their stay in the Max-Planck-Institut für Mathematik in Bonn in November 1995. They gratefully acknowledge the hospitality and stimulating atmosphere of the MPIfM.

The second named author thanks the KBN grant No. 2PO3A 05112 for supporting this research.

1. Divided differences associated with the hyperoctahedral groups.

Recall first the definition of the hyperoctahedral group i.e. the Weyl group of the symplectic group. Let n be a positive integer. Let S_{n} be the symmetric group (of all bijections of {1, …, n} ) generated by the simple transpositions s_{i} = [1, …, i − 1, i + 1, i, i + 2, …, n], i = 1, …, n − 1. The hyperoctahedral group W = W_{n} is an extension of S_{n} by an element s_{0} such that

(1) s_{0}^{2} = 1, s_{0} · s_{1} · s_{0} · s_{1} = s_{1} · s_{0} · s_{1} · s_{0}, s_{0} · s_{i} = s_{i} · s_{0} for i ≥ 2.

More precisely, W = S_{n} ⋉ ℤ_{2}^{n} where S_{n} acts on ℤ_{2}^{n} by permutations. Writing a typical element of W as w = (σ, τ) with σ ∈ S_{n} and τ ∈ ℤ_{2}^{n}, we have for w′ = (σ′, τ′),

w · w′ = (σσ′, δ),

where δ_{i} = τ_{σ′(i)} · τ′_{i}. To represent elements of W we use the standard “barredpermutation” notation, writing them as permutations with bars in those places (numbered with “i”) where τ_{i} = −1. We have s_{0} = [\overline{1}, 2, …, n], and the right multiplication by s_{i} is given by

[w_{1}, w_{2}, …, w_{n}] · s_{0} = [\overline{w}_{1}, w_{2}, …, w_{n}]
and [w_{1}, w_{2}, …, w_{n}] · s_{i} = [w_{1}, …, w_{i−1}, w_{i+1}, w_{i}, w_{i+2}, …, w_{n}].

Let ω = ω^{(n)}, w_{0} = w_{0}^{(n)} denote respectively the elements of maximal length of S_{n} and W. In addition, we set

v_{0} = v_{0}^{(n)} = w_{0} · ω = ω · w_{0}.

In the above barred permutation notation,

ω = [n, n − 1, …, 1], w_{0} = [\overline{1}, …, \overline{n}], v_{0} = [\overline{n}, …, \overline{1}].

Let X = X_{n} = (x_{1}, …, x_{n}) be a sequence of commuting independent variables. The group algebra of W acts on the ring ℤ[X] of polynomials in X: s_{i} exchanges x_{i} and x_{i+1}, 1 ≤ i ≤ n − 1, and s_{0} replaces x_{1} by −x_{1} (all remaining variables being unchanged).

In the present paper we shall make an extensive use of the algebra of divided differences, with generators ∂_{0}, ∂_{1}, …, ∂_{n−1} acting on the ring ℤ[X] according to

∂_{i}(f) = (f − s_{i}f)/(x_{i} − x_{i+1}), i ≥ 1

and ∂_{0} = (f − s_{0}f)/2x_{1}.

These generators satisfy the standard braid relations for the symmetric group S_{n} and the braid relations (1). Consequently, for every w ∈ W, there exists a welldefined divided difference ∂_{w} = ∂_{i_{1}} ∘…∘∂_{i_{l}} which can be obtained from any reduced decomposition s_{i_{1}} · . . . · s_{i_{l}} of w (see [B-G-G] and [D1,2]). In particular,

(2)  ∂_{v_{0}} = ∂_{0}(∂_{1}∂_{0}) … (∂_{n−1} … ∂_{1}∂_{0}) = (∂_{0}∂_{1} … ∂_{n−1}) … (∂_{0}∂_{1})∂_{0}

and

(3)  ∂_{w_{0}} = ∂_{v_{0}} ∘ ∂_{ω} = ∂_{ω} ∘ ∂_{v_{0}}.

Recall (see e.g. [Mcd2]) that the operator ∂_{ω} coincides with the Jacobi symmetrizer

f ↦ ∑_{μ∈S_{n}} μ[f/ △ (X)],

where △ (X) = ∏_{i<j≤n}(x_{i} − x_{j}).

On the other hand, the operator ∂_{w_{0}} (that appears in the Weyl character formula for type C_{n}) coincides with

f ↦ (−1)^{n(n−1)/2} ∑_{w∈W} w[f/□(X)],

where

□(X) =△ (X) ∏_{i≤j≤n}(x_{i} + x_{j}) = 2^{n}x_{1} · . . . · x_{n} ∏_{i<j≤n}(x_{i}^{2} − x_{j}^{2}).

This follows, for example, from [P-R2, Proposition 5.5] by noting that s_{0} appears n times in the reduced decomposition

w_{0} = s_{0} · (s_{1} · s_{0} · s_{1}) · . . . · (s_{n−1} · . . . · s_{2} · s_{1} · s_{0} · s_{1} · s_{2} · . . . · s_{n−1}) .

Observe that these expressions of ∂_{ω} and ∂_{w_{0}} imply that for every symmetric (in X) polynomial f, the operator ∂_{v_{0}} acts as

(4)  f ↦ (−1)^{n(n−1)/2} ∑_{w∈W/S_{n}} w[f/ ∏_{i≤j≤n}(x_{i} + x_{j})].

(To see this, recall that the Jacobi symmetrizer is a surjection onto the ring of symmetric polynomials in X.)

Let now T_{i}, i = 1, …, n, be the operator on ℤ[X] acting only on x_{i} by T_{i}(x_{i}) = −x_{i}. Note that T_{i} is conjugate to s_{0}:

T_{i} = s_{i−1} ∘ … ∘ s_{1} ∘ s_{0} ∘ s_{1} ∘ … ∘ s_{i−1}.

With the help of the T_{i}’s, the operator ∂_{v_{0}} has a third expression which respects the symmetry between the variables.

Lemma 1.1. One has the following equalities of operators on ℤ[X]:

\begin{aligned}∑_{w∈W}(−1)^{l(w)}w & = ∑_{μ∈S_{n}}(−1)^{l(μ)}μ (1 − T_{n}) … (1 − T_{1}) \\ & = (1 − T_{n}) … (1 − T_{1}) ∑_{μ∈S_{n}}(−1)^{l(μ)}μ. \end{aligned}

Indeed, these equalities follow immediately from W = S_{n} ⋉ ℤ_{2}^{n} invoking the following formula for the length l(w) of w = (σ, τ) ∈ W:

(5)  l(w) = ∑_{i=1}^{n} a_{i} + ∑_{j,τ_{j}=−1}(2b_{j} + 1),

where a_{i} = card{p | p > i & σ_{p} < σ_{i}} and b_{j} = card{p | p < j & σ_{p} < σ_{j}}. The equality (5) can be proved by an easy induction on l(w).

To simplify signs in many formulas of the present paper, we shall also use ∂′_{i} = −∂_{i} (1 ≤ i ≤ n − 1). For w ∈ W, by ∂′_{w} we will denote the divided difference operator where we use the ∂′_{i}’s instead of the ∂_{i}’s. Finally, instead of ∂_{v_{0}}, we will use (6) ▽ := ∂_{0}(∂′_{1}∂_{0}) … (∂′_{n−1} … ∂′_{1}∂_{0}) = (∂_{0}∂′_{1} … ∂′_{n−1}) … (∂_{0}∂′_{1})∂_{0} = (−1)^{n(n−1)/2}∂_{v_{0}}.

With the help of the T_{i}’s, the restriction of ∂_{v_{0}} (or ▽) to symmetric polynomials has a simple expression:

Lemma 1.2. For any symmetric polynomial f in X,

\begin{aligned}∂_{v_{0}}(f) & = \frac{1}{□(X)}(1 − T_{n}) … (1 − T_{1})(f · △(X)) \\ & = \frac{1}{2x_{n}} (1 − T_{n}) … \frac{1}{2x_{1}} (1 − T_{1})(f/ ∏_{i<j}(x_{i} + x_{j})). \end{aligned}

Indeed, let us first record

(7)  ( ∑_{μ∈S_{n}}(−1)^{l(μ)}μ)△(X) = n!△(X).

Since f is symmetric, one has by (7)

f = \frac{1}{n!}∂_{ω}(△(X) · f),

and thus

\begin{aligned}∂_{v_{0}}(f) & = \frac{1}{n!}∂_{w_{0}}(△(X) · f) \\ & = \frac{1}{□(X)}(1 − T_{n}) … (1 − T_{1})(f · △(X)) \end{aligned}

by Lemma 1.1 and the equality (7) again.

The second equality is a restatement of the first one because x_{i}^{2} are scalars w.r.t. T_{j}.

Let SP(X) denote the ring of symmetric polynomials in X. It follows from Lemma 1.2 (e.g. by remarking that (1 − T_{i})^{2} = 0) that the operator ▽ sends SP(X) to SP(X^{2}) where X^{2} := (x_{1}^{2}, …, x_{n}^{2}). More explicitly, the image of the ℤ-linear basis of Schur polynomials is given by the next proposition (see [P-R, Theorem 5.13]).

We shall use the following notation. Given a sequence α = (α_{1}, …, α_{n}) ∈ ℕ^{n}, we will denote by x^{α} the monomial x_{1}^{α_{1}} · . . . · x_{n}^{α_{n}}. Given two sequences α, β ∈ ℕ^{n}, we shall denote by α + β the sequence (α_{1} + β_{1}, …, α_{n} + β_{n}). Moreover, set ρ_{k} = (k, k − 1, …, 2, 1).

Proposition 1.3. The image of an S-polynomial under ▽, ▽(s_{I}(X)), is nonzero only if the partition I is of the form ρ_{n} + 2J for some partition J. In this case,

▽(s_{ρ_{n}+2J}(X)) = ψ_{2}(s_{J}(X)) ,

where ψ_{2} : ℤ[X] → ℤ[X^{2}] is the ring homomorphism defined by ψ_{2}(x_{i}) = x_{i}^{2}, i = 1, …, n .

Proof. Since

s_{I}(X) = ∂_{ω}(x^{I+ρ_{n−1}}),

we have

▽(s_{I}(X)) = ∂_{w_{0}}(x^{I+ρ_{n−1}}).

Using the first equality in Lemma 1.1, we see that ▽(s_{I}(X)) ≠ 0 only if all the parts of I + ρ_{n−1} are odd. This holds iff l(I) = n and I is strict, or equivalently, iff I = I′ + ρ_{n} for some partition I′. But I′ + ρ_{n} + ρ_{n−1} has all the parts odd iff I′ = 2J for some partition J.

Since we have

(1 − T_{n}) … (1 − T_{1})(x^{2J+ρ_{n}+ρ_{n−1}}) = 2^{n}x^{2J+ρ_{n}+ρ_{n−1}},

we infer finally that

▽(s_{2J+ρ_{n}}(X)) = ∂_{w_{0}}(x^{2J+ρ_{n}+ρ_{n−1}})

= 2^{n} ∑_{μ∈S_{n}} μ[x^{2J+ρ_{n}+ρ_{n−1}}/□(X)]

= ∑_{μ∈S_{n}} μ[x^{2J+2ρ_{n−1}}/ △ (X^{2})]

= ψ_{2}(s_{J}(X)) ,

as asserted. □

It is, perhaps, in order to mention that D.E. Littlewood used in [Li2] the notions of 2-quotient, 2-core and 2-sign (whose definitions can be found e.g. in loc.cit., [Mcd1] and [D-L-T]) to evaluate ψ_{2}(s_{J}(X)). In fact, he computed more generally ψ_{p}(s_{J}(X)), where, for any integer p ≥ 2, ψ_{p}(x_{i}) = x_{i}^{p} , i = 1, 2, … For another computation of ψ_{p}(s_{J}(X)) as a quotient of two S-polynomials see [P-R2, Proposition 5.11].

The expression of ▽ as a sum, with rational-function coefficients, of elements of W, is not totally straightforward, but we shall only need the leading term described by the following lemma.

Lemma 1.4. Write ▽ = ∑_{w∈W} a_{w}w, where a_{w} are rational functions in X. Then

there is a unique w = v_{0} of maximal length such that a_{w} ≠ 0. Moreover, a_{v_{0}} = (−1)^{n(n+1)/2} ∏_{i≤j≤n}(x_{i} + x_{j})^{−1}.

Indeed, we have, using (6), the following presentation of ▽:

\begin{aligned}▽ = \frac{1}{2x_{1}} (1 − s_{0})\frac{1}{x_{2} − x_{1}} & (1 − s_{1}) … \frac{1}{x_{n} − x_{n−1}} (1 − s_{n−1}) … \\ & … \frac{1}{2x_{1}} (1 − s_{0})\frac{1}{x_{2} − x_{1}} (1 − s_{1})\frac{1}{2x_{1}} (1 − s_{0}) \end{aligned}

which implies, by an easy induction on n, that its term of maximal length equals

(−1)^{n(n+1)/2} \frac{1}{2x_{1}} s_{0} \frac{1}{x_{2} − x_{1}} s_{1} … \frac{1}{x_{n} − x_{n−1}} s_{n−1} …

… \frac{1}{2x_{1}} s_{0} \frac{1}{x_{2} − x_{1}} s_{1} \frac{1}{2x_{1}} s_{0}

= (−1)^{n(n+1)/2} \frac{1}{2^{n}x_{1} · . . . · x_{n} ∏_{i<j≤n}(x_{i} + x_{j})} v_{0},

as asserted.

Corollary 1.5. With the notation of Lemma 1.4, the unique w = (σ, τ) for which card{j | τ_{j} = −1} = n and a_{w} ≠ 0 is w = v_{0}.

Indeed, this follows directly from the previous lemma once we observe that, by (5), v_{0} is the element of the smallest length among the barred permutations with n bars.

The following “commutation property” will be frequently used in this article.

Lemma 1.6. For any μ ∈ S_{n},

▽ ∘ ∂_{μ} = ∂_{ωμω} ∘ ▽ .

Indeed, recalling that the symmetric group S_{n} acts on the set of barred permutations from the \underline{right} by permuting the places, and, from the \underline{left} by permuting the values, we get the equality

[\overline{n}, \overline{n − 1}, …, \overline{2}, \overline{1}] · s_{i} = s_{n−i} · [\overline{n}, \overline{n − 1}, …, \overline{2}, \overline{1}] ,

whence

▽ ∘ ∂_{i} = ∂_{n−i} ∘ ▽

which immediately implies the assertion.

All the operators introduced above will be used to describe the ring ℤ[X] as an SP(X^{2})-module. First we consider ℤ[X] as an SP(X)-module.

The ring ℤ[X] is a free SP(X)-module with a basis provided by Schubert polynomials 𝔖_{μ}, μ ∈ S_{n}, defined in the following way. For a permutation μ ∈ S_{n} we set

𝔖_{μ} := 𝔖_{μ}(X) := ∂_{μ^{−1}ω}(x_{1}^{n−1}x_{2}^{n−2} … x_{n−1})

- the Schubert polynomial associated with μ. Observe that, equivalently, the polynomials 𝔖_{μ} are defined inductively by the equation

∂_{i}𝔖_{μ} = 𝔖_{μs_{i}}

if μ(i) > μ(i + 1) , the “top” polynomial being 𝔖_{ω} = x^{ρ_{n−1}} . Recall that the Schubert polynomials have the following stability property. Let S_{n} ↪ S_{m} be the embedding of the symmetric groups via the first n components (n < m). Then we have for μ ∈ S_{n} ⊂ S_{m},

𝔖_{μ}(X_{m})|_{x_{n+1}=...=x_{m}=0} = 𝔖_{μ}(X_{n}) .

It follows from this equality that 𝔖_{μ} is a well-defined polynomial for each permutation μ ∈ S_{∞} = ⋃_{n} S_{n}.

On ℤ[X] there is a scalar product

( , ) : ℤ[X] × ℤ[X] → SP(X),

defined for f, g ∈ ℤ[X] by

(f, g) = ∂_{ω}(f · g) .

Schubert polynomials are almost orthonormal basis w.r.t. ( , ) . Given w ∈ W, we set X^{w} = (w(x_{1}), …, w(x_{n})) and \overline{X} = X^{w_{0}} = (−x_{1}, …, −x_{n}). In particular, X^{ω} = (x_{n}, …, x_{2}, x_{1}) and \overline{X}^{ω} = (−x_{n}, …, −x_{1}).

One has for any μ, ν ∈ S_{n},

(𝔖_{μ}(X), 𝔖_{νω}(\overline{X}^{ω})) = δ_{μ,ν}

(the Kronecker delta). For more on Schubert polynomials see [Mcd2] and the references therein.

Now we will describe the ring SP(X) as an SP(X^{2})–module. As is well-known, SP(X) ⊗ ℚ is a free SP(X^{2}) ⊗ ℚ–module with a basis provided by Schur’s Qpolynomials {Q_{I}(X)}, where strict I ⊂ ρ_{n}. In this article, we want a basis over ℤ, and moreover geometry imposes the need to take a basis better adapted to the scalar product ⟨ , ⟩ defined below. Therefore, we shall use the following basis of Q̃-polynomials modelled on Schur’s Q-polynomials. The reader can find in [P-R2], where these polynomials were extensively studied, another geometric motivation of their choice. We set Q̃_{i} := Q̃_{i}(X) := e_{i}(X) - the i-th elementary symmetric polynomial in X. Given two nonnegative integers i, j, we define

Q̃_{i,j} := Q̃_{i,j}(X) = Q̃_{i} · Q̃_{j} + 2 ∑_{p=1}^{j}(−1)^{p}Q̃_{i+p} · Q̃_{j−p} .

Finally, for any partition I = (i_{1} ≥ i_{2} ≥ … ≥ i_{l(I)} > 0) , the polynomial Q̃_{I} = Q̃_{I}(X) is defined recurrently on l = l(I) by putting for odd l,

Q̃_{I} := Q̃_{I}(X) = ∑_{j=1}^{l}(−1)^{j−1}Q̃_{i_{j}} · Q̃_{(i_{1},... ,i_{j−1},i_{j+1},... ,i_{l})}

and for even l,

Q̃_{I} := Q̃_{I}(X) = ∑_{j=2}^{l}(−1)^{j}Q̃_{i_{1},i_{j}} · Q̃_{(i_{2},... ,i_{j−1},i_{j+1},... ,i_{l})} .

The ring SP(X) is a free SP(X^{2})-module with a basis provided by Q̃-polynomials Q̃_{I}(X) , where strict I ⊂ ρ_{n}. The polynomial Q̃_{I}(X_{n}) can be defined in another recursive way as follows. For any strict partition I one has

(8)  Q̃_{I}(X_{n}) = ∑_{j=0}^{l(I)} x_{n}^{j}( ∑_{|I|−|J|=j} Q̃_{J}(X_{n−1})),

where the sum is over all (i.e. not necessary strict) partitions J ⊂ I such that I/J has at most one box in every row (see [P-R2, Proposition 4.1]). Moreover, given a strict partition I′ = (i_{1}, …, j, j, …, i_{k}) and denoting I = (i_{1}, …, i_{k}), one has the factorization property

(9)  Q̃_{I′} = Q̃_{j,j} · Q̃_{I}.

The following scalar product ⟨ , ⟩ : SP(X) × SP(X) → SP(X^{2}) defined by ⟨f, g⟩ = ▽(f · g), will be of particular importance for the present paper. We have, for strict I, J ⊂ ρ_{n},

(10)  ⟨Q̃_{I}(X), Q̃_{ρ_{n}∖J}(X)⟩ = δ_{I,J} ,

where ρ_{n} ∖ J denotes the strict partition whose parts complement the parts of J in {n, n − 1, …, 2, 1} (see [P-R, Theorem 5.23]).

Denote by

⟨( , )⟩ : ℤ[X] × ℤ[X] → SP(X^{2})

the scalar product defined for f, g ∈ ℤ[X] by

⟨(f, g)⟩ = (−1)^{n(n−1)/2}∂_{w_{0}}(f · g) .

The SP(X^{2})-basis {𝔖_{μ} · Q̃_{I}} of ℤ[X], where μ ∈ S_{n} and I ⊂ ρ_{n} are strict, is especially convenient with regard to the scalar product ⟨( , )⟩ because obviously

⟨(𝔖_{μ}(X) · Q̃_{I} , 𝔖_{νω}(\overline{X}^{ω}) · Q̃_{ρ_{n}∖J})⟩ = δ_{μ,ν}δ_{I,J} .

Note, however, that the elements of this basis are not representatives of Schubert cycles in the manifold of complete Lagrangian flags, but form a natural “productbasis” associated with a fibration of this variety over the Lagrangian Grassmannian of maximal isotropic subspaces, with the “usual” complete flag variety as its fiber (for more about that, see [P-R2, Sect.5]).

The main theme of the present paper is to study the action of different operators on this basis. Among the operators in question the following divided difference operators will be of particular importance:

(11)  ▽_{k} = (∂′_{n−k} … ∂′_{1}∂_{0}) … (∂′_{n−2} … ∂′_{1}∂_{0})(∂′_{n−1} … ∂′_{1}∂_{0})

(thus ▽ = ▽_{n}).

As a consequence of the main results of the present paper, we will obtain a purely algebraic proof of the next proposition not invoking the characteristic map and the result from [P] quoted below. For details, see Corollary A.8 in Appendix A. Since, however, we will need this result in Section 4, we record it now with a proof which uses proprieties available in already published literature.

Proposition 1.7. For each strict partition I ⊂ ρ_{n} there exists a divided difference operator (in the ∂′_{i}’s and ∂_{0}) ∂^{I} : ℤ[X] → ℤ[X] such that ∂^{I}(Q̃_{I}) = 1 and such that, for any strict J ≠ I with |J| = |I|, ∂^{I}(Q̃_{J}) = 0 .

(Note that from the first assertion it follows that ∂^{I} = ∂′_{w} ∘ ∂_{0} for some w.)

Proof. This proof uses the Borel characteristic map [B] and the Giambelli-type formula for Lagrangian Grassmannians from [P]. Recall (see [P]) that Schubert classes σ(I) in the Lagrangian Grassmannian G of isotropic n-subspaces in ℂ^{2n} equipped with a nondegenerate symplectic form, are indexed by strict partitions I ⊂ ρ_{n}. There exists a ring homomorphism (the characteristic map)

c : SP(X) ⟶ A(G)

such that for a homogeneous symmetric polynomial f,

c(f) = ∑_{I,|I|=deg f} ∂′_{w_{I}} (f)σ(I),

where w_{I} ∈ W is the barred permutation

(12)  w_{I} = [\overline{i_{1}} > … > \overline{i_{l}}, j_{1} < … < j_{n−l}]

(for details, see [B], [B-G-G] and [D1,2]). Now [P, Theorem 6.17] says that

c(Q̃_{I}) = σ(I) .

This implies that ∂^{I} := ∂′_{w_{I}} fulfills the assertion of the proposition. □

2. Reproducing kernels and a vanishing property.

Let X = (x_{1}, …, x_{n}), Y = (y_{1}, …, y_{n}) be two sequences of independent variables (we fix n).

In the following, by ≡ we will understand the congruence in the ring SP(X) ⊗ SP(Y) modulo the ideal generated by all f(X^{2}) − f(Y^{2}), where f symmetrical and X^{2} stands for (x_{1}^{2}, …, x_{n}^{2}). We define the following polynomials in SP(X)⊗SP(Y) (the definition of F (X, Y) is inspired by [F1,2] and the one of Q̃(X, Y) - by [P-R2]).

\begin{aligned}F (X, Y) & := |e_{n+1+j−2i}X + e_{n+1+j−2i}Y |_{1≤i,j≤n} \\ & = | \begin{matrix}e_{n}X + e_{n}Y & 0 & … \\ e_{n−2}X + e_{n−2}Y & e_{n−1}X + e_{n−1}Y & … \\ ⋮ & ⋮ & ⋮ \\ & & 2 & e_{1}X + e_{1}Y \end{matrix} | , \end{aligned}

and

(13)  Q̃(X, Y) := ∑ Q̃_{I}X · Q̃_{ρ_{n}∖I}Y,

where the summation is over all strict partitions I ⊂ ρ_{n}.

The following results will give us some reproducing kernels for the scalar product

⟨ , ⟩ : ℤ[X] × ℤ[X] ⟶ SP(X^{2})

defined in the previous section.

Proposition 2.1. For any f ∈ SP(X) ,

⟨f(X), Q̃(X, Y)⟩ ≡ f(Y) .

Proof. Since we are working modulo ≡ which allow us to exchange SP(X^{2})-coeffi-cients to the corresponding SP(Y^{2})-coefficients, it suffices to check the equality :

⟨Q̃_{I}(X), Q̃(X, Y)⟩ = Q̃_{I}(Y) ,

where I ⊂ ρ_{n} is a strict partition. This equality follows immediately from the orthogonality of the Q̃_{I}’s w.r.t. ⟨ , ⟩ recalled in (10). □

To prove a similar property for F (X, Y), we need the following result.

Proposition 2.2. (i) For w ∈ S_{n} ⊂ W_{n},

F (X, X^{w}) = F (X, X) = 2^{n}s_{ρ_{n}}(X) .

(ii) For w ∈ W_{n} ∖ S_{n},

F (X, X^{w}) = 0 .

Proof. (i) This is obvious from a familiar expression for s_{ρ_{n}} as a determinant in elementary symmetric functions. (Note that we use that the degree-zero-entries in the matrix defining F are equal to 2.) (ii) Let w ∈ W_{n} ∖S_{n}. If w has odd number of bars, then the (1,1)-entry e_{n}X +e_{n}Y becomes zero for Y = X^{w} and thus F (X, X^{w}) = 0 . Hence we can assume that the number of bars is 2k where k > 0. Since, without loss of generality, we can renumber the variables, we may assume that

w = [1, 2, …, n − 2k, \overline{n − 2k + 1}, …, \overline{n}].

Let X′ = (x_{1}, …, x_{n−2k}) and X′′ = (x_{n−2k+1}, x_{n−2k+2}, …, x_{n}) . Then specializing Y = X^{w}, e_{i}X + e_{i}Y becomes

2 ∑_{j≥0} e_{i−2j}(X′) · e_{2j}(X′′) .

We claim that if we replace the entries of the matrix of F (X, Y) by putting ∑_{j≥0} e_{i−2j}(X′)e_{2j}(X′′) instead of e_{i}X + e_{i}Y, then the determinant of the soobtained matrix M is zero. This can be seen, e.g., as follows. Consider the n × nmatrix [e_{n+1+j−2i}], where e_{i} are indeterminates. If we substitute e_{i} := e_{i}(X′) , then the determinant of this specialized matrix N vanishes; indeed, s_{ρ_{n}}(X′) = 0 because card(X′) < n. But using some elementary operations on rows of N, we arrive at M . More precisely, this is done as follows:

• add e_{2l−2}(X′′) times the l-th row to the top row (all l ≥ 1); this changes the top row of N to the top row of M .

• add e_{2l−4}(X′′) times the l-th row to the second row (all l ≥ 2); this changes the second row of N to the second row of M .

Continuing down the rows, N is changed to M . The assertion follows. □

We will refer to the assertions of the lemma as a “vanishing property”.

Proposition 2.3. For any f ∈ SP(X) ,

⟨f(X), F (X, Y)⟩ ≡ f(Y) .

Proof. Observe that ⟨f(X), F (X, Y)⟩ is a combination of symmetric polynomials in Y with coefficients from SP(X^{2}):

⟨f(X), F (X, Y)⟩ = ∑_{I} c_{I}(X^{2}) · s_{I}(Y),

where I runs over partitions and c_{I}(X^{2}) ∈ SP(X^{2}). By (4) we have

⟨f(X), F (X, Y)⟩ = ∑_{w∈W_{n}/S_{n}} f(X^{w}) · F (X^{w}, Y)/2^{n}s_{ρ_{n}}(X^{w}) .

The assertion to be proved says: by replacing in ⟨f(X), F (X, Y)⟩ the coefficients c_{I}(X^{2}) by the polynomials c_{I}(Y^{2}) from SP(Y^{2}) , we get f(Y) . Hence, it suffices to check that the following specialized polynomial:

⟨f(X), F (X, Y)⟩|_{Y =X} = ∑_{w∈W_{n}/S_{n}} f(X^{w}) · F (X^{w}, X)/2^{n}s_{ρ_{n}}(X^{w})

is equal to f(X). But we know by Proposition 2.2 that only one summand on the right-hand side survives, giving us f(X) , as desired. □

The content of the next proposition provides an algebraic link between the formulas of [F1,2] and [P-R2].

Proposition 2.4. : F (X, Y) ≡ Q̃(X, Y) .

Proof. We can write

F (X, Y) ≡ ∑ f_{I}(Y)Q̃_{I}(X) ,

where the sum is over strict partitions I ⊂ ρ_{n} and f_{I}(Y) ∈ SP(Y) . By Proposition 2.3 and the orthogonality (10) of the Q̃_{I}(X)’s, we get for strict I ⊂ ρ_{n}

Q̃_{I}(Y) ≡ ⟨Q̃_{I}(X), F (X, Y)⟩
≡ ⟨Q̃_{I}(X), ∑ f_{J}(Y)Q̃_{J}(X)⟩
= f_{ρ_{n}∖J}(Y).

This implies f_{I}(Y) ≡ Q̃_{ρ_{n}∖I}(Y). Consequently,

F (X, Y) ≡ ∑_{strict I⊂ρ_{n}} Q̃_{ρ_{n}∖I}(Y)Q̃_{I}(X) = Q̃(X, Y) ,

as asserted. □

Remark 2.5. This result was known earlier from comparison of geometric formulas of [F1,2] and [P-R2]. Here, we have given a purely algebraic proof of it.

Example 2.6. For n = 2, 3 F (X, Y) = Q̃(X, Y) . For n = 4 ,

Q̃(X, Y) − F (X, Y) = (x_{1}^{2} + x_{2}^{2} + x_{3}^{2} + x_{4}^{2} − y_{1}^{2} − y_{2}^{2} − y_{3}^{2} − y_{4}^{2})(x_{1}^{2}x_{2}^{2}x_{3}^{2}x_{4}^{2} − y_{1}^{2}y_{2}^{2}y_{3}^{2}y_{4}^{2}) .

Finally, observe that Proposition 2.4 (combined with Proposition 2.2) implies that Q̃(X, Y) has also the vanishing property. The following variant of it will be used in Section 4.

Lemma 2.7. (i) For any w ∈ S_{n} (and w_{0} = [\overline{1}, \overline{2}, …, \overline{n}] ),

Q̃(X^{w_{0}w}, \overline{X}) = 2^{n}s_{ρ_{n}}(\overline{X}) .

(ii) For w ∈ W_{n} ∖ w_{0}S_{n},

Q̃(X^{w}, \overline{X}) = 0 .

OPERATOR CALCULUS FOR Q̃-POLYNOMIALS AND SCHUBERT POLYNOMIALS 15

3. Action of ▽ on the basis {𝔖_{μ} · Q̃_{I}} - an inductive approach.

In this section we give an elementary inductive proof of the following Theorem 3.1. (i) For μ ∈ S_{n}, writing X = X_{n} and ω = ω^{(n)}, one has

▽(𝔖_{μ}(X) · Q̃_{ρ_{n}}(X)) = (−1)^{l(μ)}𝔖_{μ}(X^{ω}) = 𝔖_{μ}(\overline{X}^{ω}).

(ii) For strict I ⊊ ρ_{n}, μ ∈ S_{n}, ▽(𝔖_{μ}(X) · Q̃_{I}(X)) = 0.

We give first an outline of the proof. For the need of induction we write

▽^{(k)} = (∂_{0}∂′_{1} … ∂′_{k−1})(∂_{0}∂′_{1} … ∂′_{k−2}) … (∂_{0}∂′_{1})∂_{0} .

We have

(14)  ▽ = ▽^{(n)} = ∂_{0}∂′_{1} … ∂′_{n−1}▽^{(n−1)}.

We first show that it suffices to prove assertions (i) and (ii) for μ = ω. Then, we prove (i) and (ii) for μ = ω by a simultaneous induction on n using (14). In the course of the proof of (i), we use the linearity formula (8) for Q̃_{I} and

∂_{1}∂_{2} … ∂_{n−1}(x_{1}x_{2}^{2} … x_{n−1}^{n−1}) = x_{3}x_{4}^{2} … x_{n}^{n−2}.

In the course of the proof of (ii), we use the linearity formula for Q̃_{I} and the vanishing property:

∂_{0}∂_{1} … ∂_{n−1}(x_{n}^{j}x_{n−1}^{n−2}x_{n−2}^{n−3} … x_{2}) = 0

for 0 ≤ j ≤ n − 1.

We now pass to a detailed proof. Lemma 3.2. It suffices to verify assertions (i) and (ii) of the theorem for μ = ω. Proof. Denote v = v_{0}; so that ▽ = ∂′_{v} and 𝔖_{μ} = 𝔖_{μ}(X_{n}), Q̃_{I} = Q̃_{I}(X_{n}) for brevity. Since Q̃_{I} is symmetric in X, we have

▽(𝔖_{μ}Q̃_{I}) = (−1)^{n(n−1)/2}∂_{v}(𝔖_{μ}Q̃_{I})
= ±∂_{vμ^{−1}ω}(x^{ρ_{n−1}} Q̃_{I})
= ±∂_{vων^{−1}ωω}(x^{ρ_{n−1}} Q̃_{I})  (we substitute μ = ωνω)
= ±∂_{ν^{−1}ωv}(x^{ρ_{n−1}} Q̃_{I})  (vω = ωv is the element of biggest length

in W and commutes with any element of it)

= ∂_{ν^{−1}ω}▽(x^{ρ_{n−1}} Q̃_{I})

= {(−1)^{n(n−1)/2}∂_{ν^{−1}ω}(ω(x^{ρ_{n−1}}))  for I = ρ_{n} by the case μ = ω 0  for I ≠ ρ_{n} by the case μ = ω

= \{{(−1)^{l(μ)}𝔖_{μ}(X^{ω})  for I = ρ_{n} \atop 0  for I ≠ ρ_{n}.}

□

In the proof of the next proposition we will need the following lemma.

Lemma 3.3. : ∂_{1}∂_{2} … ∂_{n−1}(x_{1}x_{2}^{2} … x_{n−1}^{n−1}) = x_{3}x_{4}^{2} … x_{n}^{n−2}.

Proof. We show by descending induction on i (i = n, n − 1, …, 1) that after applying ∂_{i} and before applying ∂_{i−1} the resulting sum has the properties:

1) it contains a unique summand without x_{i}, namely

x_{1}x_{2}^{2} … x_{i−1}^{i−1}x_{i+1}^{i−1}x_{i+2}^{i} … x_{n}^{n−2}

2) all but one summands are monomials of the form

x_{1}x_{2}^{2} … x_{i−1}^{i−1}x_{i}^{a}x_{i+1}^{•} …,

where 0 < a ≤ i − 1.

The beginning of the induction is trivial: before applying ∂_{n−1}, we have the unique summand x_{1}x_{2}^{2} … x_{n−1}^{n−1}. All monomials in the image of 2) under ∂_{i−1} contain x_{i−1}, so the unique summand without x_{i−1} in the image of ∂_{i−1} stems by applying ∂_{i−1} to 1) and equals

x_{1}x_{2}^{2} … x_{i−2}^{i−2}x_{i}^{i−2}x_{i+1}^{i−1} … x_{n}^{n−2}.

All other monomials in the image of 1) are of the form:

x_{1}x_{2}^{2} … x_{i−2}^{i−2}x_{i−1}^{a}x_{i}^{•} …,

where 0 < a ≤ i−2. This is also the form of all monomials in the image of 2) under ∂_{i−1}, and the inductive step follows. Consequently, after applying ∂_{2} and before applying ∂_{1}, all but one summands are of the form: x_{1}x_{2}x_{3}^{•} …, and they go to zero under ∂_{1}. The unique summand not containing x_{2} is

x_{1}x_{3}x_{4}^{2} … x_{n}^{n−2} ,

and it goes under ∂_{1} to x_{3}x_{4}^{2} … x_{n}^{n−2}. This proves the lemma. □

Proposition 3.4. : ▽(x^{ρ_{n−1}} · Q̃_{ρ_{n}}(X_{n})) = (−1)^{n(n−1)/2}ω(x^{ρ_{n−1}}).

Proof. We have

▽(x^{ρ_{n−1}} Q̃_{ρ_{n}}(X_{n})) = ▽^{(n)}(x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n})x_{1}^{2} … x_{n−1}^{2}x_{n})
= ∂_{0}∂′_{1} … ∂′_{n−1}▽^{(n−1)}(x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n−1} + x_{n})x_{1}^{2} … x_{n−1}^{2}x_{n}) .

We apply the linearity formula (8) to Q̃_{ρ_{n−1}}(X_{n−1} + x_{n}). All summands different from Q̃_{ρ_{n−1}}(X_{n−1}) are of the form Q̃_{J}(X_{n−1})x_{n}^{•} where J ⊊ ρ_{n−1}. After applying to J the factorization property (9) (i.e. factoring out Q̃_{pp}(X_{n−1}) corresponding to all pairs of equal rows), as well as factoring out x_{n}^{•}, we apply ▽^{(n−1)} to x^{ρ_{n−2}} Q̃_{J′}(X_{n−1}) where J′ ⊊ ρ_{n−1} is the resulting strict partition. We get the vanishing by the induction assumption (ii) for n − 1. Thus the above expression is equal to

∂_{0}∂′_{1} … ∂′_{n−1}▽^{(n−1)}(x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n−1})x_{1}^{2} … x_{n−1}^{2}x_{n})
= ∂_{0}∂′_{1} … ∂′_{n−1}(x_{1}^{2} … x_{n−1}^{2}x_{n}▽^{(n−1)}(x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n−1})))
= ∂_{0}∂′_{1} … ∂′_{n−1}(x_{1}^{2} … x_{n−1}^{2}x_{n}(x_{2}x_{3}^{2} … x_{n−1}^{n−2} · x_{1} … x_{n−1}))
= (−1)^{(n−1)(n−2)/2}∂_{0}(x_{1} … x_{n} ∂′_{1} … ∂′_{n−1}(x_{1}x_{2}^{2} … x_{n−1}^{n−1}))

by the induction assumption (i) for n − 1.

Now, using Lemma 3.3, the above expression is rewritten as (−1)^{n(n−1)/2} times

∂_{0}(x_{1}x_{2}x_{3}^{2} … x_{n}^{n−1}) = x_{2}x_{3}^{2} … x_{n}^{n−1} = ω(x^{ρ_{n−1}}),

and the proof of the proposition is complete. □

Proposition 3.5. : ▽(x^{ρ_{n−1}} · Q̃_{I}(X_{n})) = 0 for strict I ⊂ ρ_{n}.

Proof. We apply the linearity formula (8) to Q̃_{I}(X_{n}) = Q̃_{I}(X_{n−1} +x_{n}). Consider a typical summand Q̃_{J}(X_{n−1}) · x_{n}^{•}. Recall that J is a partition (not necessary strict) obtained from I by subtracting boxes from I, but no more then two from one row. Moreover, all parts of J are less than or equal to n − 1. By dividing x^{ρ_{n−1}} by x_{1} · . . . · x_{n−1} and multiplying Q̃_{J}(X_{n−1}) · x_{n}^{•} by x_{1} · . . . · x_{n−1}, we get the summand

x^{ρ_{n−2}} Q̃_{(n−1)J}(X_{n−1}) · x_{n}^{•} ,

whose factor depending on X_{n−1} is an element of the basis associated with X_{n−1}. If J = (n − 1)J′ then J′ ⊊ ρ_{n−1} (J′ = ρ_{n−1} implies I = ρ_{n}, which is impossible). Then using the factorization property (9) to Q̃_{(n−1)J}(X_{n−1}) and factoring out all possible Q̃_{p,p}(X_{n−1}) for p = n − 1, …, 1 , we apply ▽^{(n−1)} to x^{ρ_{n−2}} Q̃_{J′′}(X_{n−1}) where J′′ ⊊ ρ_{n−1} is strict, thus getting the vanishing by the induction assumption (ii) for n − 1. Hence we can assume j_{1} ≤ n − 2. Observe that in J no three consecutive rows are equal. Therefore it is impossible to subtract from J a pair of equal rows to get ρ_{n−2}. Hence the only J which after a possible subtraction of pairs of equal parts and adding the part (n − 1), gives us ρ_{n−1} is the partition ρ_{n−2} itself. The unique possibilities for the initial partition I are then

I = (n − 1, n − 2, …, j + 2, j, j − 1, …, 1) j = 1, …, n − 1.

(Pictorially:

•
•  •
•  •  •
•  •  •  •  ∗
•  •  •  •  •  ∗
•  •  •  •  •  •  ∗

where the total number of rows is n − 2 and the number of rows without “*” equals j. In the above figure, we have n = 8 and j = 3.) This brings us to the summand

x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n−1})x_{n}^{j} 1 ≤ j ≤ n − 1.

We have, by the induction assumption (i) for n − 1,

\begin{aligned}▽(x^{ρ_{n−1}} Q̃_{ρ_{n−1}}(X_{n−1})x_{n}^{j}) & = ∂_{0}∂′_{1} … ∂′_{n−1}(x_{n}^{j}▽^{(n−1)}(x^{ρ_{n−2}} Q̃_{ρ_{n−1}}(X_{n−1}))) \\ & = ±∂_{0}∂_{1} … ∂_{n−1}(x_{n}^{j}x_{n−1}^{n−2}x_{n−2}^{n−3} … x_{2}). \end{aligned}

We show that this expression is zero for 1 ≤ j ≤ n − 1 by induction on n. If j = n or j = n − 1, then

∂_{0}∂_{1} … ∂_{n}(x_{n+1}^{j}x_{n}^{n−1}x_{n−1}^{n−2} … x_{2}) = 0

by direct calculation. Assume 1 ≤ j ≤ n − 1. Then ∂_{n} sends x_{n+1}^{j}x_{n}^{n−1}x_{n−1}^{n−2} … to

x_{n+1}^{j}x_{n}^{j}(x_{n}^{n−1−j−1} + x_{n}^{n−1−j−2}x_{n+1} + …)x_{n−1}^{n−2} …

(for j = n, the third factor is empty). In every monomial, the variable x_{n} appears with exponent ≤ j + n − 1 − j − 1 = n − 2. Hence, by the induction assumption, ∂_{0}∂_{1} … ∂_{n−1} annihilates this expression. The proposition has been proved. □

This completes the proof of Theorem 3.1.

Corollary 3.6. Im(▽) is a free SP(X^{2})-module with a basis {𝔖_{μ}(X^{ω})}_{μ∈S_{n}}.

We end this section with the following, related formula.

Proposition 3.7. For any μ ∈ S_{n}, ▽(𝔖_{μ} ·s_{I}(X)) ≠ 0 iff I is of the form ρ_{n} +2J. In this case,

▽(𝔖_{μ}(X) · s_{I}(X)) = (−1)^{l(μ)}𝔖_{μ}(X^{ω}) · ψ_{2}(s_{J}(X))

(where ψ_{2} is the ring homomorphism sending x_{i} to x_{i}^{2}, i = 1, …, n).

Proof. It follows from Theorem 3.1 (e.g. by developing s_{I}(X) in the SP(X^{2})-basis of Q̃-polynomials) that

▽(𝔖_{μ}(X) · s_{I}(X)) = c · 𝔖_{μ}(X^{ω})

for some c ∈ SP(X^{2}). The coefficient c is computed in the following way, using Lemma 1.6:

\begin{aligned}c & = ∂′_{ωμω}▽(𝔖_{μ}(X) · s_{I}(X)) = ▽∂′_{μ}(𝔖_{μ}(X) · s_{I}(X)) \\ & = (−1)^{l(μ)}▽(s_{I}(X)), \end{aligned}

and the assertion follows from the formula recalled in Proposition 1.3. □

4. Action of ▽ on the basis {𝔖_{μ} · Q̃_{I}} via the vanishing property.

The goal of this section is to derive the formula of Theorem 3.1 via the vanishing property of Section 2. The starting point of the approach of this section is the determination of the image of ▽ : ℤ[X] → ℤ[X].

Proposition 4.1. Im(▽) is a free SP(X^{2})-module with a basis {𝔖_{μ}(X^{ω})}_{μ∈S_{n}}.

Proof. We write for brevity: Q̃_{I} = Q̃_{I}(X), SP = SP(X) and SP^{2} = SP(X^{2}). Knowing that ℤ[X] is an SP-module with a basis {𝔖_{μ}(X^{ω})}_{μ∈S_{n}} and SP is an SP^{2}-module with a basis {Q̃_{I}}_{strict I⊂ρ_{n}}, we can write any element in Im(▽) as a unique expression

(15)  ∑ c_{μ,I} · 𝔖_{μ}(X^{ω}) · Q̃_{I} ,

where c_{μ,I} ∈ SP^{2} and μ, I are as above. Suppose that there exist ν, J with |J| ≥ 1 such that c_{ν,J} ≠ 0. Let μ be a permutation of biggest length among such ν’s. The operator ∂′_{ωμω} sends 𝔖_{μ}(X^{ω}) to 1 and all 𝔖_{ν}(X^{ω}), where ν ≠ μ and l(ν) ≤ l(μ), to 0. Applying this operator to (15), we get

(16)  ∑ c_{μ,J} · Q̃_{J} + (SP^{2}-combination of the 𝔖_{ν}(X^{ω})’s).

Let I be a partition of biggest weight among those J such that c_{μ,J} ≠ 0. Let ∂^{I} be the divided difference operator (in the ∂′_{i}’s and ∂_{0}) s.t. ∂^{I}(Q̃_{I}) = 1 (hence ∂^{I} = ∂′_{w} ∘ ∂_{0} for some w), and ∂^{I}(Q̃_{J}) = 0 for all strict J ≠ I, |J| ≤ |I| (see Proposition 1.7). Then the image of (16) under ∂^{I} is c_{μ,I} ≠ 0. (Observe that ∂^{I}(𝔖_{ν}(X^{ω}) = 0 because the 𝔖_{ν}(X^{ω})’s do not depend on x_{1}.)

On the other hand, using ▽∂′_{ωμω} = ∂′_{μ}▽ (Lemma 1.6), we infer that (16) belongs to Im(▽). Since ∂_{0}▽ = 0, we have ∂^{I}▽ = 0, and we get a contradiction with the above calculation.

The proposition has been proved. □

We now give

Another proof of Theorem 3.1.

By the preceding proposition we know that

▽(𝔖_{μ}(X) · Q̃_{ρ_{n}∖I}(X)) = ∑ c_{I,ν} · 𝔖_{ν}(X^{ω}) ,

where c_{I,ν} ∈ SP(X^{2}). Invoking the definition (13) of Q̃(X, Y), we thus have, by linearity,

\begin{aligned}▽(𝔖_{μ}(X) · Q̃(X, \overline{Y})) & = ▽(𝔖_{μ}(X) · ∑_{strict I⊂ρ_{n}}(−1)^{|I|}Q̃_{ρ_{n}∖I}(X) · Q̃_{I}(Y)) \\ & = ∑_{I,ν}(−1)^{|I|}c_{I,ν} · 𝔖_{ν}(X^{ω}) · Q̃_{I}(Y). \end{aligned}

Specialize now Y in X (that is, we put y_{1} = x_{1}, …, y_{n} = x_{n}). Since the family \{𝔖_{μ}(X^{ω})·Q̃_{I}(X)\} is an SP(X^{2})-basis, the value of ▽(𝔖_{μ}(X)·Q̃(X, \overline{Y})) under this specialization still determines the coefficients c_{I,ν}.

Write ▽ = ∑_{w∈W} a_{w}w, where a_{w} are rational functions in X. Then

▽(𝔖_{μ}(X) · Q̃(X, \overline{Y})) = ∑_{w∈W} a_{w}𝔖_{μ}(X^{w}) · Q̃(X^{w}, \overline{Y}) ,

where W acts only on X-variables. It follows, from Corollary 1.5 and Lemma 2.7, that among those w, for which a_{w} ≠ 0 there is only one w = v_{0} = [\overline{n}, \overline{n − 1}, …, \overline{1}] = w_{0}ω, for which Q̃(X^{w}, \overline{X}) ≠ 0; in fact, we have

Q̃(X^{v_{0}}, \overline{X}) = 2^{n}s_{ρ_{n}}(\overline{X}).

Since a_{v_{0}} = (2^{n}s_{ρ_{n}}(\overline{X}))^{−1} by Lemma 1.4, we get finally

\begin{aligned}▽(𝔖_{μ}(X) · Q̃(X, \overline{Y})) |_{X=Y} & = 𝔖_{μ}(X^{v_{0}}) \\ & = (−1)^{l(μ)}𝔖_{μ}(X^{ω}). \end{aligned}

This implies that the coefficients c_{I,ν} are as asserted, and the proof is complete. □

5. Key formula and vertex operators

In this section we prove the following identity.

Theorem 5.1 (Key formula). Suppose n ≥ k > 0. Let IkJ be a strict partition and H a strict partition not containing k. Then

(−1)^{n−1}∂_{n−1}∂_{n−2} … ∂_{1}∂_{0}(x_{1}^{n−k}Q̃_{IkJ}(X_{n})) = (−1)^{l(I)}Q̃_{IJ}(X_{n})

and

∂_{n−1}∂_{n−2} … ∂_{1}∂_{0}(x_{1}^{n−k}Q̃_{H}(X_{n})) = 0.

Our proof of the theorem will use some differential operators (called usually “vertex operators”). The vertex operators that we shall use appeared already in the literature (see e.g. [D-K-M], [D-J-K-M1] and [D-J-K-M2]), but in the context different from the one in the present paper. Our notation and conventions for vertex operators follow rather the exposition of [C-T].

Besides Q̃-polynomials we will need Schur’s P -functions and Q′-functions from [L-L-T2]. (We use the term “function” in the sense of [Mcd1] i.e., a function depends on an infinite sequence of variables (x_{1}, x_{2}, …).) Given a symmetric function f, we denote

f(X_{n}) = f_{|x_{n+1}=x_{n+2}=...=0},

the corresponding symmetric polynomial in the first n variables.

For the purposes of this paper it suffices to work with Schur P -functions indexed by sequences of positive integers (like in Schur’s original paper [S]). Recall that for J ∈ ℕ^{∗m} with odd m,

P_{J} = P_{j_{1}} · P_{j_{2},... ,j_{m}} − P_{j_{2}} · P_{j_{1},j_{3},... ,j_{m}} + … + P_{j_{m}} · P_{j_{1},... ,j_{m−1}}

and with even m,

P_{J} = P_{j_{1},j_{2}} · P_{j_{3},... ,j_{m}} − P_{j_{1},j_{3}} · P_{j_{2},j_{4},... ,j_{m}} + … + P_{j_{1},j_{m}} · P_{j_{2},... ,j_{m−1}} .

Here, P_{j} = ∑ s_{L}, sum over all hook partitions L of j, and

P_{i,j} = P_{i} · P_{j} + 2 ∑_{k=1}^{j−1}(−1)^{k}P_{i+k}P_{j−k} + (−1)^{j}P_{i+j}.

For another expression of P_{J} in the form of a quadratic polynomial in the Sfunctions, see [L-L-T1].

If we restrict ourselves to a finite set of indeterminates X_{n}, where n > |J|, then P_{J}(X_{n}) is given by the following symmetrization:

(17)  P_{J}(X_{n}) = π(x^{J} ∏_{{i<j≤n \atop i≤m}}(1 + \frac{x_{j}}{x_{i}} )),

where π is the symmetrizer defined for f ∈ ℤ[X_{n}] by

π(f) = ∑_{w∈S_{n}} w[f x^{ρ_{n−1}}/△(X)] = ∂_{ω}(f · x^{ρ_{n−1}}).

Recall that P_{J} is an antisymmetric function of the components of J; in particular, P_{J} = 0 if two components of J are equal.

Given a symmetric function f, let D_{f} be the adjoint operator to the multiplication by f w.r.t. the standard scalar product on the ring of symmetric functions (see [Mcd1]).

Proposition 5.2. Let k > 0. Then

P_{kJ} = ∑_{j≥0}(−1)^{j}P_{k+j} · D_{e_{j}}(P_{J}).

Proof. Suppose that n >> 0 (in fact, n ≥ k + |J| will do the job). Using (17) and π^{2} = π, we get

P_{kJ}(X_{n}) = π(x_{1}^{k}(1 + \frac{x_{2}}{x_{1}} ) … (1 + \frac{x_{n}}{x_{1}} )P_{J}(x_{2}, …, x_{n})).

Passing to the λ-ring setup, we have

P_{kJ}(X_{n}) = \frac{1}{2}π(x_{1}^{k}(1 + \frac{x_{1}}{x_{1}} )(1 + \frac{x_{2}}{x_{1}} ) … (1 + \frac{x_{n}}{x_{1}} )P_{J}(X_{n} − x_{1}))

= \frac{1}{2}π((∑_{i=0}^{n} x_{1}^{k−i}e_{i}(X_{n}))(∑_{j≥0}(−x_{1})^{j} · (D_{e_{j}}P_{J})(X_{n})))

= \frac{1}{2} ∑_{i} ∑_{j}(−1)^{j}s_{k+j−i}(X_{n}) · e_{i}(X_{n}) · (D_{e_{j}}P_{J})(X_{n})

= ∑_{j≥0}(−1)^{j}P_{k+j}(X_{n}) · (D_{e_{j}}P_{J})(X_{n}).

Above, we have used the equalities π(x_{1}^{i}) = s_{i}(X_{n}) for i ∈ ℕ, and

(∑ e_{i}) · (∑ s_{j}) = 1 + 2 ∑ P_{i}.

The proposition has been proved. □

By introducing the following differential operator on the ring of symmetric functions

V_{k}^{e} := P_{k} − P_{k+1}D_{e_{1}} + P_{k+2}D_{e_{2}} − …,

we can restate the formula in Proposition 5.2 as the equation

V_{k}^{e}(P_{J}) = P_{kJ} .

We also need Q′-functions: for a partition I, Q′_{I} is the image of Q̃_{I} under the involution which exchanges e_{i} and s_{i}, i ∈ ℕ. In the following remark, for the reader’s convenience, we collect some properties of these functions.

Remark 5.3. (i) Let Q′_{I}(q) be the Hall-Littlewood polynomial where the initial alphabet X is replaced by X/(1 − q) (in the sense of λ-rings). Then Q′_{I} equals Q′_{I}(q) when q is specialized to −1.

(ii) Equivalently, using raising operators R_{ij} (see e.g. [Mcd1]), one has

Q′_{I} = ∏_{i<j} \frac{1 − R_{ij}}{1 + R_{ij}} h_{I} ,

where h_{I} is the product of complete homogeneous symmetric functions associated with the parts of I.

(iii) Q′-functions can be defined also using differential operators. In fact, this approach extends their definition to all finite sequences of integers. For k ∈ ℤ one has

Q′_{kJ} = (s_{k} − 2s_{k+1}D_{P_{1}} + 2s_{k+2}D_{P_{2}} − …)(Q′_{J}) .

(iv) Littlewood [Li1] showed that the P and Q-Hall-Littlewood bases are adjoint w.r.t. a certain scalar product. Going back to the standard scalar product on the ring of symmetric functions, his result says that {P_{I}(q)} is adjoint to {Q′_{I}(q)}, where I runs over all partitions. Specializing q to −1, one gets that {Q′_{J}} and {P_{J}}, where J runs over the set of strict partitions, are two adjoint bases of some free ℤ-modules A and B of symmetric functions w.r.t. the standard scalar product.

We refer the reader to [Li1], [Mcd1], [L-L-T2] and [D-L-T] for details and precise references to the results described in this remark.

Introduce now two other differential operators on the ring of symmetric functions:

U_{k}^{s} := D_{P_{k}} − s_{1}D_{P_{k+1}} + s_{2}D_{P_{k+2}} − …

and

U_{k}^{e} := D_{P_{k}} − e_{1}D_{P_{k+1}} + e_{2}D_{P_{k+2}} − ….

In particular, by Remark 5.3(iii), we have for k > 0

U_{k}^{s}(Q′_{J}) = (−1)^{k} \frac{1}{2}Q′_{(−k)J} .

Our goal now is to investigate the action of the operator U_{k}^{s} on Q̃-functions. We compute first the action of U_{k}^{e} on Q′-functions.

Proposition 5.4. Suppose k > 0. Let IkJ be a strict partition and H- a strict partition not containing k. Then

U_{k}^{e}(Q′_{IkJ}) = (−1)^{l(I)}Q′_{IJ}

and

U_{k}^{e}(Q′_{H}) = 0 .

Proof. It follows from Proposition 5.2 that V_{k}^{e}(B) ⊂ B, and hence U_{k}^{e}(A) ⊂ A. Denote by [ , ] the standard scalar product on the ring of symmetric functions. For strict partitions K, L we must evaluate

[U_{k}^{e}(Q′_{K}), P_{L}] = [Q′_{K}, V_{k}^{e}(P_{L})] = [Q′_{K}, P_{kL}] .

This immediately yields both assertions. □

Applying the involution on the ring of symmetric polynomials, that exchanges e_{i} with s_{i} for i ∈ ℕ, we get

Proposition 5.5. Suppose k > 0. Let IkJ be a strict partition and H- a strict partition not containing k. Then

U_{k}^{s}(Q̃_{IkJ}) = (−1)^{l(I)}Q̃_{IJ}

and

U_{k}^{s}(Q̃_{H}) = 0 .

We now want to link the above differential operators with divided difference operators. To this end we give first the following presentation of the operator induced by s_{0} on the ring of symmetric functions.

Lemma 5.6. For every symmetric function f,

s_{0}(f) = f − 2x_{1}D_{P_{1}}(f) + 2x_{1}^{2}D_{P_{2}}(f) − 2x_{1}^{3}D_{P_{3}}(f) + ….

Proof. We will use the λ-ring setup. For any symmetric function f and three alphabets X, B, C, one has

f(X − B + C) = ∑ s_{I}(−B + C)D_{s_{I}} (f)(X),

where C − B denotes the formal difference of the alphabets C and B.

Take now B = x_{1}, C = \overline{x}_{1}; then X − B + C is obtained from X by changing x_{1} to \overline{x}_{1}. On the other hand, the only nonzero s_{I}(−B + C) for those special B and C, are those for which I is a hook I = (j + 1, 1^{i}).

In that case, for I ≠ 0, one has the explicit value

s_{I}(−B + C) = (\overline{x}_{1} − x_{1})\overline{x}_{1}^{j}(−x_{1})^{i}

which specializes to 2(−x_{1})^{|I|} under \overline{x}_{1} ↦ −x_{1}. Since the sum of all hook Schur S-functions of weight k is equal to P_{k}, collecting all hooks of the same weight, one finally gets

f((−x_{1}), x_{2}, x_{3}, …) = f − 2x_{1}D_{P_{1}}(f) + 2x_{1}^{2}D_{P_{2}}(f) − …,

the right-hand side being evaluated in x_{1}, x_{2}, …. □

Corollary 5.7. For every symmetric function f ,

∂_{0}(f) = D_{P_{1}}(f) − x_{1}D_{P_{2}}(f) + x_{1}^{2}D_{P_{3}}(f) − ….

Let us define the following operator D_{h}^{(n)} on the ring of symmetric functions by

D_{h}^{(n)}(f) := ∂_{n−1} … ∂_{1}∂_{0}(x_{1}^{h} · f) .

Proposition 5.8. Fix n ∈ ℕ. Let 0 ≤ h < n. If f is a symmetric function, then one has the equality

((−1)^{n−1}D_{h}^{(n)}f)(X_{n}) = (U_{n−h}^{s}f)(X_{n}) ;

in other words,

((−1)^{n−1}D_{h}^{(n)}f)(X_{n}) = (D_{P_{n−h}}f − s_{1} · D_{P_{n−h+1}}f + s_{2} · D_{P_{n−h+2}}f − …)(X_{n}).

Proof. We have

∂_{0}(x_{1}^{2j+1} · f) = (−x_{1})^{2j+1}∂_{0}(f) + x_{1}^{2j} · f

and

∂_{0}(x_{1}^{2j} · f) = x_{1}^{2j}∂_{0}(f).

If 2j + 1 < n, then ∂_{n−1} … ∂_{1}(x_{1}^{2j}f) = 0 . The assertion follows from Corollary 5.7 by using the well-known formula ∂_{n−1} … ∂_{1}(x_{1}^{p}) = s_{p−n+1}(X_{n}) for p ∈ ℕ. □

This completes the proof of Theorem 5.1. (Note that the case k = n was proved earlier in [P-R2] by another method.)

We end this section by showing how the key formula explicates the action of the operators ▽_{k} defined by (11) on the functions specified in the following

Proposition 5.9. Let 0 < k ≤ n and n − 1 ≥ α_{1} > α_{2} > … > α_{k} ≥ 0 . Then the image under ▽_{k} of

∂′_{ω^{(k)}}(x_{1}^{α_{1}}x_{2}^{α_{2}} … x_{k}^{α_{k}} · Q̃_{I}(X_{n})) = ∂′_{ω^{(k)}}(x_{1}^{α_{1}}x_{2}^{α_{2}} … x_{k}^{α_{k}}) · Q̃_{I}(X_{n})

is nonzero only if n − α_{1}, n − α_{2}, …, n − α_{k} are parts of I. In this case, the image is

(−1)^{∑ p_{j}−k}Q̃_{J}(X_{n}) ,

where J is the strict partition with parts {i_{1}, i_{2}, … } ∖ {n − α_{1}, …, n − α_{k}} and p_{j} is such that i_{p_{j}} = n − α_{j} for j = 1, …, k.

We need some properties of the powers of ▽_{1}.

Lemma 5.10. Let α_{i} ≤ n − i for i = 1, …, k. Then for a symmetric function f ,

▽_{1}^{k}(x_{1}^{α_{1}} · . . . · x_{k}^{α_{k}} · f) = ▽_{1}(x_{1}^{α_{k}} · . . . · ▽_{1}(x_{1}^{α_{2}} · ▽_{1}(x_{1}^{α_{1}} · f)) …) .

In other words, by identifying a monomial with the operator of multiplication by this monomial, the lemma asserts the following equality of operators on the ring of symmetric functions:

▽_{1}^{k} ∘ x_{1}^{α_{1}} … x_{k}^{α_{k}} = ▽_{1} ∘ x_{1}^{α_{k}} ∘ … ∘ ▽_{1} ∘ x_{1}^{α_{2}} ∘ ▽_{1} ∘ x_{1}^{α_{1}} .

Proof. The proof is by induction on k. For simplicity, we only prove the case k = 2 and show how the case k = 2 implies the case k = 3 . This is sufficient to see the general pattern of the inductive proof. For k = 2 we have, using the Leibniz rule,

▽_{1}^{2}(x_{1}^{α}x_{2}^{β} · f) =
= ▽_{1}∂′_{n−1} … ∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f)
= ▽_{1}∂′_{n−1} … ∂′_{1}[∂_{0}(x_{1}^{α} · f) · x_{2}^{β}]
= ▽_{1}∂′_{n−1} … ∂′_{2}[∂′_{1}∂_{0}(x_{1}^{α} · f) · x_{1}^{β} + ∂_{0}(x_{1}^{α} · f) · ∂′_{1}(x_{2}^{β})].

Observe that

∂′_{n−1} … ∂′_{2}(∂_{0}(x_{1}^{α} · f) · ∂′_{1}(x_{2}^{β})) = 0

because ∂_{0}(x_{1}^{α} · f) is a scalar w.r.t. ∂′_{n−1} … ∂′_{3}∂′_{2} and we have β ≤ n − 2 . We conclude that

▽_{1}^{2}(x_{1}^{α}x_{2}^{β} · f) = ▽_{1}(x_{1}^{β} · ▽_{1}(x_{1}^{α} · f)) ,

as asserted because x_{1}^{β} is a scalar w.r.t. ∂′_{n−1} … ∂′_{2}.

For k = 3 we have by the Leibniz rule

▽_{1}^{3}(x_{1}^{α}x_{2}^{β}x_{3}^{γ} · f) =
= ▽_{1}^{2}∂′_{n−1} … ∂′_{1}[∂_{0}(x_{1}^{α}x_{2}^{β} · f) · x_{3}^{γ}]
= ▽_{1}^{2}∂′_{n−1} … ∂′_{2}[∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f) · x_{3}^{γ}]
= ▽_{1}^{2}∂′_{n−1} … ∂′_{3}[∂′_{2}∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f) · x_{2}^{γ} + ∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f) · ∂′_{2}(x_{3}^{γ})] .

Observe that

∂′_{n−1} … ∂′_{3}[∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f) · ∂′_{2}(x_{3}^{γ})] = 0

because ∂′_{1}∂_{0}(x_{1}^{α}x_{2}^{β} · f) is a scalar w.r.t. ∂′_{n−1} … ∂′_{3} and γ ≤ n − 3 . Since x_{2}^{γ} is a scalar w.r.t. ∂′_{n−1} … ∂′_{3} , the expression in question can be rewritten as

▽_{1}^{2}(x_{2}^{γ}·▽_{1}(x_{1}^{α}x_{2}^{β} · f))
= ▽_{1}∂′_{n−1} … ∂′_{1}(x_{2}^{γ} · ∂_{0}▽_{1}(x_{1}^{α}x_{2}^{β} · f))
= ▽_{1}∂′_{n−1} … ∂′_{2}(∂′_{1}∂_{0}▽_{1}(x_{1}^{α}x_{2}^{β} · f) · x_{1}^{γ} + ∂_{0}▽_{1}(x_{1}^{α}x_{2}^{β} · f) · ∂′_{1}(x_{2}^{γ}))

by the Leibniz rule. But ∂_{0}▽_{1}(x_{1}^{α}x_{2}^{β} · f) = ∂_{0}(x_{1}^{β} · ▽_{1}(x_{1}^{α} · f)) is a scalar w.r.t. ∂′_{n−1} … ∂′_{2} and γ ≤ n − 3 ; this implies

∂′_{n−1} … ∂′_{2}[∂_{0}▽_{1}(x_{1}^{α}x_{2}^{β} · f) · ∂′_{1}(x_{2}^{γ})] = 0.

Since x_{1}^{γ} is a scalar w.r.t. ∂′_{n−1} … ∂′_{2}, the expression in question equals

▽_{1}(x_{1}^{γ} · ▽_{1}^{2}(x_{1}^{α}x_{2}^{β} · f)) = ▽_{1}(x_{1}^{γ} · ▽_{1}(x_{1}^{β} · ▽_{1}(x_{1}^{α} · f)))

by the case k = 2 proved above. □

Proof of Proposition 5.9: The assertion follows directly from Lemma 5.10, the equality ▽_{k} ∘ ∂′_{ω^{(k)}} = ▽_{1}^{k} and the key formula (Theorem 5.1). □

Example 5.11. n = 6, k = 3, Q̃_{I} = Q̃_{I}(X_{6}) .

▽_{3}(∂′_{ω^{(3)}}(x_{1}^{4}x_{2}^{1}x_{3}^{0}) · Q̃_{654321})
= ±▽_{1}(x_{1}^{0} · ▽_{1}(x_{1}^{1} · ▽_{1}(x_{1}^{4} · Q̃_{654321})))
= ±▽_{1}(x_{1}^{0} · ▽_{1}(x_{1}^{1} · Q̃_{65431}))
= ±▽_{1}(x_{1}^{0} · Q̃_{6431})
= Q̃_{431} .

We give now other useful consequences of the key formula. Given two barred permutations w and v, we shall say that “ u transforms w to v ” (or “ we pass from w to v using u ”) if w · u^{−1} = v and l(w) − l(u) = l(v).

Proposition 5.12. Let I = (i_{1}, …, i_{l}) ⊂ ρ_{n} be a strict partition. Then

∂′_{i_{1}−1} … ∂′_{1}∂_{0}(Q̃_{I}(X_{n})) = Q̃_{(i_{2},... ,i_{l})}(X_{n}) .

Proof. Suppose that {i_{1}, …, i_{l}, j_{1} < j_{2} < … < j_{k}} = {1, …, n}. Let w, v, u and μ be elements of W_{n}, which transform the barred permutations as indicated in the following square

[\overline{j_{1}}, …, \overline{j_{k}}, \overline{i_{1}}, \overline{i_{2}}, …, \overline{i_{l}}] ⟵^{μ} [\overline{j_{1}} < … < \overline{j_{p−1}} < \overline{i_{1}} < \overline{j_{p}} < … < \overline{j_{k}}, \overline{i_{2}}, …, \overline{i_{l}}]
↓^{v}  ↓^{w}
[\overline{i_{1}}, \overline{i_{2}}, …, \overline{i_{l}}, j_{1}, …, j_{k}] ⟶^{u} [\overline{i_{2}}, …, \overline{i_{l}}, j_{1} < … < j_{p−1} < i_{1} < j_{p} < … < j_{k}]

for some p. Note that μ = s_{k} · . . . · s_{p+1} · s_{p}.

We have w = u · v · μ and l(w) = l(u) + l(v) + l(μ). Hence ∂′_{w} = ∂′_{u} ∘ ∂′_{v} ∘ ∂′_{μ}. Observe that

v = (s_{l} … s_{1}s_{0}) … (s_{n−2} … s_{1}s_{0})(s_{n−1} … s_{1}s_{0}) · ω^{(k)} .

Therefore by Proposition 5.9, we have

\begin{aligned}∂′_{v}(x_{1}^{n−j_{1}}x_{2}^{n−j_{2}}· & . . . · x_{k}^{n−j_{k}} Q̃_{ρ_{n}}(X_{n})) = \\ & = (−1)^{n−j_{1}+n−j_{2}+...+n−j_{k}} Q̃_{(i_{1},... ,i_{l})}(X_{n}) . \end{aligned}

In a similar way, we infer

\begin{aligned}∂′_{w}(x_{1}^{n−j_{1}}· & . . . · x_{p}^{n−i_{1}} · . . . · x_{k+1}^{n−j_{k}} Q̃_{ρ_{n}}(X_{n})) = \\ & = (−1)^{n−j_{1}+...+n−i_{1}+...+n−j_{k}} Q̃_{(i_{2},... ,i_{l})}(X_{n}) . \end{aligned}

Since ∂′_{i_{1}−1} … ∂′_{1}∂_{0} = ∂′_{u}, to prove the assertion it suffices to show that

∂′_{μ}(x_{1}^{n−j_{1}} · . . . · x_{p}^{n−i_{1}} · . . . · x_{k+1}^{n−j_{k}}) = (−1)^{n−i_{1}}x_{1}^{n−j_{1}} · . . . · x_{k}^{n−j_{k}} .

But this is an obvious equality for dominant Schubert polynomials:

\begin{aligned}∂_{k} … ∂_{p+1}∂_{p} & 𝔖_{[n+1−j_{1},... ,n+1−i_{1},... ,n+1−j_{k},n+1−i_{2},... ,n+1−i_{l}]} = \\ & = 𝔖_{[n+1−j_{1},... ,n+1−j_{k},n+1−i_{1},... ,n+1−i_{l}]} , \end{aligned}

where n + 1 − i_{1} is placed in the p-th place, in the first expression.

Thus the proposition has been proved. □ Example 5.13. n = 9, I = (6, 4, 3, 2) . The elements w, v, u and μ of W_{9} are defined by the following diagram:

[\overline{1}, \overline{5}, \overline{7}, \overline{8}, \overline{9}, \overline{6}, \overline{4}, \overline{3}, \overline{2}] ⟵^{μ} [\overline{1}, \overline{5}, \overline{6}, \overline{7}, \overline{8}, \overline{9}, \overline{4}, \overline{3}, \overline{2}]
↓^{v} ↓^{w}
[\overline{6}, \overline{4}, \overline{3}, \overline{2}, 1, 5, 7, 8, 9] ⟶^{u} [\overline{4}, \overline{3}, \overline{2}, 1, 5, 6, 7, 8, 9]

We have, with Q̃_{I} = Q̃_{I}(X_{9}),

\begin{aligned}∂′_{v}(x_{1}^{8}x_{2}^{4}x_{3}^{2}x_{4}^{1}x_{5}^{0} · Q̃_{ρ_{9}}) & = ±Q̃_{6432} , \\ ∂′_{w}(x_{1}^{8}x_{2}^{4}x_{3}^{3}x_{4}^{2}x_{5}^{1}x_{6}^{0} · Q̃_{ρ_{9}}) & = ±Q̃_{432} and \\ ∂_{5}∂_{4}∂_{3}(x_{1}^{8}x_{2}^{4}x_{3}^{3}x_{4}^{2}x_{5}^{1}x_{6}^{0}) & = x_{1}^{8}x_{2}^{4}x_{3}^{2}x_{4}^{1}x_{5}^{0} . \end{aligned}

Corollary 5.14. Let I = (i_{1}, …, i_{k}, …, i_{l}) be a strict partition. Then

∂′_{i_{k}−1} … ∂′_{1}∂_{0} … ∂′_{i_{1}−1} … ∂′_{1}∂_{0}(Q̃_{I}(X_{n})) = Q̃_{(i_{k+1},i_{k+2},... ,i_{l})}(X_{n}) . In particular ( invoking the notation of (12) ), we get

∂′_{w_{I}} (Q̃_{I}(X_{n})) = 1 .

Corollary 5.15. Let I = (i_{1}, …, i_{l}) be a strict partition. Then for j > i_{1} ,

∂_{j−1} … ∂_{1}∂_{0}(Q̃_{I}(X_{n})) = 0 .

Indeed, we have

∂_{j−1} … ∂_{i_{1}}∂_{i_{1}−1} … ∂_{1}∂_{0}(Q̃_{I}(X_{n})) = ∂_{j−1} … ∂_{i_{1}} Q̃_{(i_{2},...)}(X_{n}) = 0 .

Observe that this implies that for any strict partition J ≠ I which precedes I in the reverse lexicographic order (see [Mcd1]),

∂_{w_{J}} (Q̃_{I}(X_{n})) = 0.

Remark 5.16. Some of the above results for “trapezoidal” partitions I = (n, n − 1, …, k + 1) were originally proved in [P-R2] by different methods.

Appendix A (written in collaboration with J. Ratajski): Symplectic Schubert polynomials à la polonaise.

Billey and Haiman [B-H] and Fomin and Kirillov [F-K] have defined various Schubert polynomials for type C_{n}. Following [P-R2, Appendix B], we propose here other candidates to represent the Schubert classes in the variety of complete Lagrangian flags.

Let (x_{1}, x_{2}, …) be a sequence of independent variables. Let w_{0} = w_{0}^{(n)} be the longest element in the Weyl group W_{n} of type C_{n}. Define

C_{w_{0}} = C_{w_{0}}(X_{n}) := (−1)^{n(n−1)/2}x^{ρ_{n−1}} Q̃_{ρ_{n}}(X_{n}) ,

and for an arbitrary w ∈ W_{n},

C_{w} := C_{w}(X_{n}) = ∂′_{w^{−1}w_{0}}(C_{w_{0}})

- the symplectic Schubert polynomial associated with w.

Example A.1. Symplectic Schubert polynomials for n = 2 are:

\begin{matrix}C_{[\overline{1},\overline{2}]} = & − x_{1}^{3}x_{2} − x_{1}^{2}x_{2}^{2}, \\ C_{[1,\overline{2}]} = −x_{1}^{2}x_{2} & ,  C_{[\overline{2},\overline{1}]} = x_{1}^{2}x_{2} + x_{1}x_{2}^{2}, \\ C_{[\overline{2},1]} = x_{1}x_{2} & ,  C_{[2,\overline{1}]} = x_{2}^{2}, \\ C_{[2,1]} = x_{2} & ,  C_{[\overline{1},2]} = x_{1} + x_{2}, \\ C_{[1,2]} & = 1 . \end{matrix}

Their expression in terms of the 𝔖Q̃-basis is:

\begin{matrix}C_{[\overline{1},\overline{2}]} = & − 𝔖_{[2,1]}Q̃_{21} , \\ C_{[1,\overline{2}]} = −𝔖_{[2,1]}Q̃_{2} & ,  C_{[\overline{2},\overline{1}]} = Q̃_{21} , \\ C_{[\overline{2},1]} = Q̃_{2} & ,  C_{[2,\overline{1}]} = −𝔖_{[2,1]}Q̃_{1} + (x_{1}^{2} + x_{2}^{2}) + Q̃_{2} , \\ C_{[2,1]} = −𝔖_{[2,1]} + Q̃_{1} & ,  C_{[\overline{1},2]} = Q̃_{1} , \\ C_{[1,2]} & = 1 . \end{matrix}

Theorem A.2. (Stability) Suppose that m > n. Let W_{n} ↪ W_{m} be the embedding via the first n components. Then for any w ∈ W_{n} ⊂ W_{m}, the following equality holds:

C_{w}(X_{m})|_{x_{n+1}=...=x_{m}=0} = C_{w}(X_{n}) .

Proof. To show the assertion we must prove that the above definition is consistent with the divided difference recurrences. Taking into account the equality

w_{0}^{(n)} = w_{0}^{(n−1)}s_{n−1}s_{n−2} … s_{1}s_{0}s_{1} … s_{n−2}s_{n−1} ,

and our sign conventions, the wanted assertion is an immediate consequence of the equality:

∂′_{n−1} … ∂′_{2}∂′_{1}∂_{0}∂_{1}∂_{2} … ∂_{n−1}(f_{n})|_{x_{n}=0} = f_{n−1} ,

where f_{n} = x^{ρ_{n−1}} Q̃_{ρ_{n}}(X_{n}) .

We need a simple

Lemma A.3. For any polynomial f = f(X_{n}) ,

∂_{n−1} … ∂_{1}∂_{0}(x_{1}^{2}x_{2} … x_{n} · f)|_{x_{n}=0} = 0 .

Indeed, after applying ∂_{0} we get a (possibly zero) sum of monomials divisible by x_{1} · . . . · x_{n}. Since x_{1} · . . . · x_{n} is a scalar w.r.t. ∂_{n−1} … ∂_{1} , the final substitution x_{n} = 0 gives the vanishing.

Suppose now that

(18)  ∂_{1}∂_{2} … ∂_{n−1}(x^{ρ_{n−1}}) = ∑_{J} a_{J}x^{J} ,

where a_{J} ∈ ℤ, so that

∂_{1}∂_{2} … ∂_{n−1}(f_{n}) = (∑ a_{J}x^{J})Q̃_{ρ_{n}}(X_{n}) .

We claim that if for J = (j_{1}, …) with a_{J} ≠ 0 we have j_{1} ≥ 1 , then

∂_{n−1} … ∂_{1}∂_{0}(x^{J} Q̃_{ρ_{n}}(X_{n})) |_{x_{n}=0} = 0 .

This follows from the lemma since Q̃_{ρ_{n}}(X_{n}) = x_{1} · . . . · x_{n} · Q̃_{ρ_{n−1}}(X_{n}) . We next observe Lemma A.4. With the notation of (18), if a_{J} ≠ 0 and j_{1} = 0 , then J = (0, n − 2, n − 3, …, 2, 1, 0) .

Indeed, let M be a monomial appearing in the decomposition of

∂_{2} … ∂_{n−1}(x_{2}^{n−2}x_{3}^{n−3} … x_{n−1})

as a sum of monomials. To get in ∂_{1}(x_{1}^{n−1}M) a monomial without x_{1} , the monomial M must be independent of x_{2} (x_{1}x_{2} is a scalar w.r.t. ∂_{1}). By induction we can assume M = x_{3}^{n−3} … x_{n−1} . But then, the unique monomial in ∂_{1}(x_{1}^{n−1}x_{3}^{n−3} … x_{n−1}) that does not depend on x_{1}, is x_{2}^{n−2}x_{3}^{n−3} … x_{n−1}, as claimed.

Hence we must compute:

∂′_{n−1} … ∂′_{1}∂_{0}(x_{2}^{n−2} · x_{3}^{n−3} · . . . · x_{n−1} · Q̃_{ρ_{n}}(X_{n})) |_{x_{n}=0}
= ∂′_{n−1} … ∂′_{1}∂_{0}(x_{2}^{n−2} · . . . · x_{n−1} · x_{1} · . . . · x_{n} · Q̃_{ρ_{n−1}}(X_{n})) |_{x_{n}=0}

= ∂′_{n−1} … ∂′_{1}∂_{0}(x_{2}^{n−2} · . . . · x_{n−1} · x_{1} · . . . · x_{n} · Q̃_{ρ_{n−1}}(x_{2}, …, x_{n})) |_{x_{n}=0}

by Lemma A.3. The last expression can be rewritten as

(19)  ∂′_{n−1} … ∂′_{1}(x_{2}^{n−1} … x_{n−1}^{2}x_{n} · Q̃_{ρ_{n−1}}(x_{2}, …, x_{n})) |_{x_{n}=0}.

Now we need

Lemma A.5. Let x, y be indeterminates and f = f(x, y) a polynomial function depending on x, y and perhaps on some other variables. Then denoting by ∂_{xy} the divided difference f ↦ \frac{f(x,y)−f(y,x)}{x−y}, we have

∂_{xy}(y · f(x, y)) |_{y=0} = ∂_{xy}(y · f(0, y)) |_{y=0} .

Indeed, y · f(x, y) − y · f(0, y) = xy · g for some polynomial g. Then

∂_{xy}(y · f(x, y) − yf(0, y)) = ∂_{xy}(xy · g) = xy∂_{xy}(g)

vanishes after the substitution y = 0 .

Coming back to our expression (19), we see that before applying ∂′_{n−1} we can substitute x_{n−1} = 0 , so before applying ∂′_{n−2} we can substitute x_{n−2} = 0 etc. In other words, in the course of computing (19) we can after applying ∂′_{1} substitute x_{2} = 0 , after applying ∂′_{2} substitute x_{3} = 0, …, after applying ∂′_{n−2} substitute x_{n−1} = 0 , and at the end substitute x_{n} = 0 .

Using the Leibniz rule, we get

∂′_{1}(x_{2}^{n−1} … x_{n−1}^{2}x_{n} · Q̃_{ρ_{n−1}}(x_{2}, …, x_{n})) |_{x_{2}=0}
= [∂′_{1}(x_{2}^{n−1} … x_{n}) · s_{1}Q̃_{ρ_{n−1}}(x_{2}, …, x_{n})] |_{x_{2}=0}
+ [x_{2}^{n−1} … x_{n} · ∂′_{1}(Q̃_{ρ_{n−1}}(x_{2}, …, x_{n}))] |_{x_{2}=0}
= x_{1}^{n−2}x_{3}^{n−2} … x_{n} · Q̃_{ρ_{n−1}}(x_{1}, x_{3}, …, x_{n}) + 0.

Similarly, applying ∂′_{2} to the obtained polynomial and substituting x_{3} = 0 , we get

x_{1}^{n−2}x_{2}^{n−3}x_{4}^{n−3}x_{5}^{n−4} … x_{n} · Q̃_{ρ_{n−1}}(x_{1}, x_{2}, x_{4}, …, x_{n}) .

Continuing in this way, we get after applying ∂′_{n−1}

x_{1}^{n−2}x_{2}^{n−3} … x_{n−2} · Q̃_{ρ_{n−1}}(x_{1}, …, x_{n−1}) = f_{n−1} .

This completes the proof of the stability property. □

Theorem A.6. (the maximal Grassmannian property) Let I = (i_{1} > … > i_{l} > 0) be a strict partition contained in ρ_{n}. Then, invoking the notation of (12), one has the equality

C_{w_{I}} (X_{n}) = Q̃_{I}(X_{n}) .

Proof. The proof is very similar to the one of Proposition 5.12, but we give all details for the reader’s convenience. We first explain how, in the course of the proof, we pass from w_{0} to w_{I}.^{1} Write k = n − l. First, by using an appropriate permutation σ, we pass from w_{0} = [\overline{1}, \overline{2}, …, \overline{n}] to

[\overline{j_{1}}, \overline{j_{2}}, …, \overline{j_{k}}, \overline{i_{1}}, …, \overline{i_{l}}] ,

where {i_{1}, …, i_{l}, j_{1}, …, j_{k}} = {1, …, n} and j_{1} < … < j_{k}. Secondly, using ω^{(k)}, we pass to

[\overline{j_{k}}, \overline{j_{k−1}}, …, \overline{j_{1}}, \overline{i_{1}}, …, \overline{i_{l}}] .

Thirdly, using

(s_{n−k} … s_{1}s_{0}) … (s_{n−2} … s_{1}s_{0})(s_{n−1} … s_{1}s_{0}) ,

we pass to

[\overline{i_{1}}, …, \overline{i_{l}}, j_{1}, j_{2}, …, j_{k}] = w_{I} .

Observe that after applying to C_{w_{0}}(X_{n}) the divided difference corresponding to the first permutation, we get

\begin{aligned}C & [\overline{j_{1}},\overline{j_{2}},... ,\overline{j_{k}},\overline{i_{1}},... ,\overline{i_{l}}] \\ & = (−1)^{n(n−1)/2+l(σ)}𝔖_{[n+1−j_{1},... ,n+1−j_{k},n+1−i_{1},... ,n+1−i_{l}]} · Q̃_{ρ_{n}}(X_{n}) \end{aligned}

because σ transforms ω^{(n)} to

[n + 1 − j_{1}, …, n + 1 − j_{k}, n + 1 − i_{1}, …, n + 1 − i_{l}].

We have

𝔖_{[n+1−j_{1},... ,n+1−j_{k},n+1−i_{1},... ,n+1−i_{l}]} = x_{1}^{n−j_{1}}x_{2}^{n−j_{2}} … x_{k}^{n−j_{k}},

1Recall that we pass from w to v using u (or u transforms w to v) if w · u^{−1} = v and l(w) − l(u) = l(v). We then have ∂_{u}C_{w} = C_{v}. Similarly, if for μ, ν, σ ∈ S_{n}, σ transforms μ to ν, then ∂_{σ}𝔖_{μ} = 𝔖_{ν}.

see [Mcd2]. The operator corresponding to the composition of the second permutation and the third barred permutation is ▽_{k} ∘ ∂′_{ω^{(k)}} , and using Proposition 5.9 we get

▽_{k} ∘∂′_{ω^{(k)}}(x_{1}^{n−j_{1}}x_{2}^{n−j_{2}} … x_{k}^{n−j_{k}} Q̃_{ρ_{n}}(X_{n})) = (−1)^{n−j_{1}+n−j_{2}+...+n−j_{k}} Q̃_{(i_{1},... ,i_{l})}(X_{n}) .

Since n − j_{1} + … + n − j_{k} is the length of the permutation passing from [n, n − 1, …, 2, 1] to [j_{1}, j_{2}, …, j_{k}, i_{1}, …, i_{l}] and σ passes from [1, 2, …, n] to [j_{1}, j_{2}, …, j_{k}, i_{1}, …, i_{l}] , we see that the above signs fit to give us

(▽_{k} ∘ ∂′_{ω^{(k)}} ∘ ∂′_{σ})((−1)^{n(n−1)/2}x^{ρ_{n−1}} Q̃_{ρ_{n}}(X_{n})) = Q̃_{I}(X_{n}) ,

as asserted. □

Example A.7. n = 6 , I = (4, 3, 1) . We compute, with Q̃_{I} = Q̃_{I}(X_{6}),

\begin{aligned}C_{[\overline{1},\overline{2},\overline{3},\overline{4},\overline{5},\overline{6}]} ⟶^{∂_{σ}} & C_{[\overline{2},\overline{5},\overline{6},\overline{4},\overline{3},\overline{1}]} \\ & = ±𝔖_{[5,2,1,3,4,6]} · Q̃_{ρ_{6}} = ±x_{1}^{4}x_{2}^{1}x_{3}^{0} · Q̃_{ρ_{6}} . \end{aligned}

Then to compute the image under ▽_{3} ∘ ∂′_{ω^{(3)}} of the last element, we follow Example 5.11 to get finally Q̃_{(4,3,1)} .

From the maximal Grassmannian property it follows the following operator characterization of Q̃-polynomials.

Corollary A.8. For any strict partition I, w = w_{I} is the unique element of W such that l(w) = |I| and ∂_{w}(Q̃_{I}(X)) ≠ 0; in fact, ∂_{w_{I}} (Q̃_{I}(X)) = 1 .

We end this Appendix with the table of symplectic Schubert polynomials for n = 3 computed with the help of ACE (see [V1,2]). In this table 𝔖_{i,j,k} := 𝔖_{[i,j,k]}(x_{1}, x_{2}, x_{3}) and E_{i} := e_{i}(x_{1}^{2}, x_{2}^{2}, x_{3}^{2}).

C_{[1,2,3]} = 1 C_{[1,3,2]} = x_{3} = −𝔖_{1,3,2} + Q̃_{1} C_{[2,1,3]} = x_{2} + x_{3} = −𝔖_{2,1,3} + Q̃_{1} C_{[\overline{1},2,3]} = x_{1} + x_{2} + x_{3} = Q̃_{1} C_{[3,1,2]} = x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{1} + 𝔖_{3,1,2} + Q̃_{2} C_{[2,3,1]} = x_{3}^{2} = 𝔖_{2,3,1} − 𝔖_{1,3,2}Q̃_{1} + E_{1} + Q̃_{2} C_{[\overline{2},1,3]} = x_{1}x_{2} + x_{1}x_{3} + x_{2}x_{3} = Q̃_{2} C_{[\overline{1},3,2]} = (x_{1} + x_{2} + x_{3})x_{3} = −𝔖_{1,3,2}Q̃_{1} + E_{1} + 2Q̃_{2} C_{[2,\overline{1},3]} = x_{2}^{2} + x_{2}x_{3} + x_{3}^{2} = −𝔖_{2,1,3}Q̃_{1} + E_{1} + Q̃_{2} C_{[\overline{3},1,2]} = x_{1}x_{2}x_{3} = Q̃_{3} C_{[3,2,1]} = x_{2}x_{3}^{2} = −𝔖_{3,2,1} + 𝔖_{2,3,1}Q̃_{1} − E_{1}𝔖_{2,1,3} − 𝔖_{2,1,3}Q̃_{2} + 𝔖_{3,1,2}Q̃_{1} − 𝔖_{1,3,2}Q̃_{2} + Q̃_{3} + Q̃_{21} C_{[\overline{2},3,1]} = (x_{1} + x_{2})x_{3}^{2} = −𝔖_{1,3,2}Q̃_{2} + Q̃_{3} + Q̃_{21} C_{[1,\overline{2},3]} = −x_{1}^{2}(x_{2} + x_{3}) = −𝔖_{2,1,3}Q̃_{2} + Q̃_{3} C_{[3,\overline{1},2]} = (x_{2} + x_{3})x_{2}x_{3} = −E_{1}𝔖_{2,1,3} − 2𝔖_{2,1,3}Q̃_{2} + 𝔖_{3,1,2}Q̃_{1} + Q̃_{3} + Q̃_{21} C_{[2,3,\overline{1}]} = x_{3}^{3} = 𝔖_{2,3,1}Q̃_{1} − E_{1}𝔖_{1,3,2} − 𝔖_{1,3,2}Q̃_{2} + E_{1}Q̃_{1} + Q̃_{3} C_{[\overline{2},\overline{1},3]} = x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{1}x_{2}x_{3} + x_{1}x_{3}^{2} + x_{2}^{2}x_{3} + x_{2}x_{3}^{2} = Q̃_{21} C_{[1,\overline{3},2]} = −x_{1}^{2}x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{3} C_{[\overline{3},2,1]} = x_{1}x_{2}x_{3}^{2} = −𝔖_{1,3,2}Q̃_{3} + Q̃_{31} C_{[3,\overline{2},1]} = x_{2}^{2}x_{3}^{2} = −𝔖_{2,1,3}Q̃_{3} − 𝔖_{2,1,3}Q̃_{21} + 𝔖_{3,1,2}Q̃_{2} + E_{2} + Q̃_{31} C_{[1,3,\overline{2}]} = −(x_{1}^{2} + x_{2}^{2})x_{3}^{2} = 𝔖_{2,3,1}Q̃_{2} − 𝔖_{1,3,2}Q̃_{3} − E_{2} C_{[\overline{3},\overline{1},2]} = (x_{1} + x_{2} + x_{3})x_{1}x_{2}x_{3} = Q̃_{31} C_{[3,2,\overline{1}]} = x_{2}x_{3}^{3} = −𝔖_{3,2,1}Q̃_{1} + E_{1}𝔖_{2,3,1} + 2𝔖_{2,3,1}Q̃_{2} − E_{1}𝔖_{2,1,3}Q̃_{1} − 𝔖_{2,1,3}Q̃_{3} + E_{1}𝔖_{3,1,2} + 𝔖_{3,1,2}Q̃_{2} − 𝔖_{1,3,2}Q̃_{3} − 𝔖_{1,3,2}Q̃_{21} + E_{2} + E_{1}Q̃_{2} + 𝔖_{1,2,3}Q̃_{31} C_{[\overline{2},3,\overline{1}]} = (x_{1}^{2} + x_{1}x_{3} + x_{2}^{2} + x_{2}x_{3})x_{3}^{2} = −𝔖_{1,3,2}Q̃_{21} + 2E_{2} + E_{1}Q̃_{2} + Q̃_{31} C_{[\overline{1},\overline{2},3]} = −(x_{1}x_{2} + x_{1}x_{3} + x_{2}^{2} + x_{3}^{2})x_{1}^{2} = −𝔖_{2,1,3}Q̃_{21} + Q̃_{31} C_{[1,2,\overline{3}]} = x_{1}^{2}x_{2}^{2}x_{3} = 𝔖_{2,3,1}Q̃_{3} C_{[2,\overline{3},1]} = −(x_{2} + x_{3})x_{1}^{2}x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{31} + 𝔖_{3,1,2}Q̃_{3} C_{[\overline{3},\overline{2},1]} = (x_{1}x_{2} + x_{1}x_{3} + x_{2}x_{3})x_{1}x_{2}x_{3} = Q̃_{32} C_{[3,1,\overline{2}]} = −(x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{2}^{2}x_{3})x_{2}x_{3} = −𝔖_{3,2,1}Q̃_{2} + 𝔖_{2,3,1}Q̃_{3} + 𝔖_{2,3,1}Q̃_{21} + E_{2}𝔖_{2,1,3} + 𝔖_{3,1,2}Q̃_{3} − E_{2}𝔖_{1,3,2} − 𝔖_{1,3,2}Q̃_{31} C_{[\overline{1},\overline{3},2]} = −(x_{1} + x_{2} + x_{3})x_{1}^{2}x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{31} C_{[\overline{3},2,\overline{1}]} = −(x_{1}x_{2} − x_{3}^{2})x_{1}x_{2}x_{3} = −𝔖_{1,3,2}Q̃_{31} + E_{1}Q̃_{3} + Q̃_{32} C_{[3,\overline{2},\overline{1}]} = (x_{1}^{2} + x_{2}x_{3})(x_{2} + x_{3})x_{2}x_{3} = −2E_{2}𝔖_{2,1,3} − E_{1}𝔖_{2,1,3}Q̃_{2} − 𝔖_{2,1,3}Q̃_{31} + 𝔖_{3,1,2}Q̃_{21} + E_{2}Q̃_{1} + E_{1}Q̃_{3} + Q̃_{32} C_{[\overline{1},3,\overline{2}]} = −(x_{1}^{3}x_{3} +2x_{1}^{2}x_{2}^{2} +x_{1}^{2}x_{2}x_{3} +x_{1}^{2}x_{3}^{2} +x_{1}x_{2}^{2}x_{3} +x_{2}^{3}x_{3} +x_{2}^{2}x_{3}^{2})x_{3} = 𝔖_{2,3,1}Q̃_{21} − 𝔖_{1,3,2}Q̃_{31} − E_{2}Q̃_{1} C_{[2,1,\overline{3}]} = x_{1}^{2}x_{2}^{3}x_{3} = −𝔖_{3,2,1}Q̃_{3} + 𝔖_{2,3,1}Q̃_{31} − E_{3} C_{[\overline{2},\overline{3},1]} = −(x_{1}x_{2} + x_{1}x_{3} + x_{2}x_{3})x_{1}^{2}x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{32} C_{[\overline{3},1,\overline{2}]} = −(x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{2}^{2}x_{3})x_{1}x_{2}x_{3} = −𝔖_{1,3,2}Q̃_{32} + 2E_{3} C_{[\overline{1},2,\overline{3}]} = (x_{1} + x_{2} + x_{3})x_{1}^{2}x_{2}^{2}x_{3} = 𝔖_{2,3,1}Q̃_{31} C_{[2,\overline{3},\overline{1}]} = −(x_{2}^{2} + x_{2}x_{3} + x_{3}^{2})x_{1}^{2}x_{2}x_{3} = −E_{1}𝔖_{2,1,3}Q̃_{3} − 𝔖_{2,1,3}Q̃_{32} + 𝔖_{3,1,2}Q̃_{31} C_{[\overline{3},\overline{2},\overline{1}]} = (x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{1}x_{2}x_{3} + x_{1}x_{3}^{2} + x_{2}^{2}x_{3} + x_{2}x_{3}^{2})x_{1}x_{2}x_{3} = Q̃_{321} C_{[3,\overline{1},\overline{2}]} = −(2x_{1}^{2}x_{2}^{2} +2x_{1}^{2}x_{2}x_{3} +x_{1}^{2}x_{3}^{2} +x_{2}^{3}x_{3} +x_{2}^{2}x_{3}^{2})x_{2}x_{3} = −𝔖_{3,2,1}Q̃_{21} +2E_{2}𝔖_{2,3,1} + E_{1}𝔖_{2,3,1}Q̃_{2} + 𝔖_{2,3,1}Q̃_{31} + E_{2}𝔖_{2,1,3}Q̃_{1} + 𝔖_{3,1,2}Q̃_{31} − E_{2}𝔖_{1,3,2}Q̃_{1} − E_{1}𝔖_{1,3,2}Q̃_{3} − 𝔖_{1,3,2}Q̃_{32} C_{[\overline{2},1,\overline{3}]} = (x_{1}x_{2} − x_{3}^{2})x_{1}^{2}x_{2}^{2}x_{3} = 𝔖_{2,3,1}Q̃_{32} − E_{3}Q̃_{1} C_{[1,\overline{3},\overline{2}]} = x_{1}^{2}x_{2}x_{3}(x_{1}^{2} + x_{2}x_{3})(x_{2} + x_{3}) = 2E_{3}𝔖_{2,1,3} + 𝔖_{3,1,2}Q̃_{32} + E_{3}Q̃_{1} C_{[2,\overline{1},\overline{3}]} = (x_{2}^{2} + x_{2}x_{3} + x_{3}^{2})x_{1}^{2}x_{2}^{2}x_{3} = −𝔖_{3,2,1}Q̃_{31} + E_{1}𝔖_{2,3,1}Q̃_{3} + 𝔖_{2,3,1}Q̃_{32} C_{[\overline{2},\overline{3},\overline{1}]} = −(x_{1}^{2}x_{2}+x_{1}^{2}x_{3}+x_{1}x_{2}^{2}+x_{1}x_{2}x_{3}+x_{1}x_{3}^{2}+x_{2}^{2}x_{3}+x_{2}x_{3}^{2})x_{1}^{2}x_{2}x_{3} = −𝔖_{2,1,3}Q̃_{321} C_{[\overline{3},\overline{1},\overline{2}]} = −(x_{1} + x_{2})(x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{1}x_{2}x_{3} + x_{1}x_{3}^{2} + x_{2}^{2}x_{3} + x_{2}x_{3}^{2})x_{1}x_{2}x_{3} = −𝔖_{1,3,2}Q̃_{321} C_{[1,\overline{2},\overline{3}]} = −x_{1}^{2}x_{2}^{2}x_{3}(x_{1}^{2} + x_{2}x_{3})(x_{2} + x_{3}) = −𝔖_{3,2,1}Q̃_{32} + 2E_{3}𝔖_{2,3,1} + E_{3}𝔖_{2,1,3}Q̃_{1} − E_{3}𝔖_{1,3,2}Q̃_{1} C_{[\overline{2},\overline{1},\overline{3}]} = (x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{1}x_{2}x_{3} + x_{1}x_{3}^{2} + x_{2}^{2}x_{3} + x_{2}x_{3}^{2})x_{1}^{2}x_{2}^{2}x_{3} = 𝔖_{2,3,1}Q̃_{321} C_{[\overline{1},\overline{3},\overline{2}]} = (x_{1}^{2}x_{2} + x_{1}^{2}x_{3} + x_{1}x_{2}^{2} + x_{1}x_{2}x_{3} + x_{1}x_{3}^{2} + x_{2}^{2}x_{3} + x_{2}x_{3}^{2})x_{1}^{3}x_{2}x_{3} = 𝔖_{3,1,2}Q̃_{321} C_{[\overline{1},\overline{2},\overline{3}]} = −(x_{1}^{2}x_{2}+x_{1}^{2}x_{3}+x_{1}x_{2}^{2}+x_{1}x_{2}x_{3}+x_{1}x_{3}^{2}+x_{2}^{2}x_{3}+x_{2}x_{3}^{2})x_{1}^{3}x_{2}^{2}x_{3} = −𝔖_{3,2,1}Q̃_{321}.

Appendix B: Three geometric applications

1. In Corollary A.8 we have obtained a purely algebraic proof (based on the key formula) of the following result. For any strict partition J with |J| = |I|, we have ∂_{w_{J}} (Q̃_{I}) ≠ 0 only if I = J , and, in this case, we get ∂′_{w_{I}} (Q̃_{I}) = 1 . Apply this to the characteristic map

c : SP(X) ⟶ A^{∗}(G)

given for a homogeneous symmetric polynomial f by

c(f) = ∑_{I,|I|=deg f} ∂′_{w_{I}} (f)σ(I)

(the notation as in Section 1). We infer

(20)  c(Q̃_{I}) = σ(I)

which is the Giambelli-type formula for the Lagrangian Grassmannian, originally obtained in [P, Sect.6] by a different method. Note that this purely algebraic operator proof of (20) gives us, after invoking the Pieri-type formula for Hall-Littlewood functions from [Mo], a Pieri-type formula for Lagrangian Schubert classes obtained originally, by other methods, in [H-B] (see also [P-R1] for a divided difference approach).

2. The key formula admits the following interpretation in terms of Gysin maps. Let X be a scheme and V a rank 2n vector bundle on X equipped with a nondegenerate symplectic form. Let V_{n} be a rank n isotropic subbundle of V . By p : F → X we denote the flag bundle parametrizing pairs A ⊂ B of subbundles of V such that rank A = n − 1 , rank B = n, A ⊂ V_{n} and B is Lagrangian. Let C ⊂ D be the tautological bundles of respective ranks n − 1 and n, on F . Set S = D/C.

Suppose n ≥ k > 0. Let IkJ be a strict partition and H a strict partition not containing k. Then the key formula says:

(21)  p_{∗}(c_{1}(S^{∨})^{n−k} · Q̃_{IkJ}(D^{∨})) = (−1)^{l(I)}Q̃_{IJ}(V_{n}^{∨})

and

p_{∗}(c_{1}(S^{∨})^{n−k} · Q̃_{H}(D^{∨})) = 0 .

For k = n, this result agrees with Proposition 3.1 in [P-R2], which was proved using geometric arguments. Does it exist a geometric proof for the key formula in general? One more comment may be here in order. Note that the arguments in both sides of (21) are different whereas in the key formula, they are the same. As a matter of fact, in the above translation of the key formula, we have used a result from [P-R2, Sect.8] allowing us to make an appropriate exchange of the sequences of the Chern roots of the bundles involved under the action of p_{∗}.

3. The results of [P-R2] and the present paper combined with some results of [F1,2] allow us to generalize the formulas for the Schubert classes in Lagrangian Grassmann bundles from [P-R2] to the case of arbitrary Lagrangian flag bundles and flagged Lagrangian degeneracy loci. Here, for simplicity, we describe just the difference between our formulas and the ones obtained by Fulton in [F1,2]. Let X_{w} (w ∈ W_{n}) be the degeneracy locus defined as in [F1, p.243] with the help of a nondegenerate symplectic form on a rank 2n vector bundle V which is equipped with two complete filtrations:

E_{1} ⊂ E_{2} ⊂ … ⊂ E_{n} ⊂ V,
F_{1} ⊂ F_{2} ⊂ … ⊂ F_{n} ⊂ V

by isotropic subbundles. Then, it follows from Proposition 2.4 of the present paper (or its geometric counterpart in [P-R2]), and the results of [F1,2], that the fundamental class of X_{w} is evaluated by the expression given in [F1, Theorem (C_{n}) on p.243], where instead of F (X, Y) (denoted by △ in loc.cit.) one uses Q̃(X, Y). More precisely, if the ambient variety is Cohen-Macaulay and X_{w} is of pure codimension n^{2} − l(w), then

[X_{w}] = ∂_{w^{−1}}( ∏_{i+j≤n}(x_{i} − y_{j})Q̃(X, Y)) ∩ [X],

where we specialize x_{i} := −c_{1}(E_{i}/E_{i−1}) and y_{i} := −c_{1}(F_{n+1−i}/F_{n−i}) for i = 1, …, n.

We point out this difference between the approach of [F1,2] and ours because it appears that for numerical computations (e.g. with the help of ACE of [V1,2]) the reproducing kernel Q̃(X, Y) is more economical than F (X, Y).

References

[B-G-G] I.M. Bernstein, I.M. Gel’fand, S.I. Gel’fand, Schubert cells and cohomology of the

spaces G/P, Russian Math. Surveys 28 (1973), 1–26.

[B-H] S. Billey, M. Haiman, Schubert polynomials for the classical groups, Journal of the

Amer. Math. Soc. 8 (1995), 443–482.

[B] A. Borel, Sur la cohomologie des espaces fibres principaux et des espaces homogènes de groupes de Lie compacts, Ann. of Math. 57 (1953), 115–207.

[C-T] Ch. Carré, J.-Y. Thibon, Plethysm and vertex operators, Adv. Appl. Math. 13 (1992), 390–403.

[D-K-M] E. Date, M. Kashiwara, T. Miwa, Vertex operators and τ functions, Proc. Japan Acad., Ser. A 57 (1981), 387–392.

[D-J-K-M1] E. Date, M. Jimbo, M. Kashiwara, T. Miwa, A new hierarchy of soliton equations of KP-type, Physica D 4 (1982), 343–365.

[D-J-K-M2] E. Date, M. Jimbo, M. Kashiwara, T. Miwa, Transformation groups for soliton equations in: “Non-linear Integrable Systems – Classical Theory and Quantum Theory”, (M. Jimbo, T. Miwa eds.), World Scientific (1983), 41-119.

[D1] M. Demazure, Invariants symétriques entiers des groupes de Weyl et torsion, Inv. Math. 21 (1973), 287–301.

[D2] M. Demazure, Désingularisation des variétés de Schubert generalisées, Ann. Scient. Éc. Norm. Sup. 7 (1974), 53–88.

[D-L-T] J. Désarménien, B. Leclerc, J.-Y. Thibon, Hall-Littlewood functions and Kostka-Foulkes polynomials in representation theory, Preprint – Institut Gaspard Monge 94-13 (1994).

[F-K] S. Fomin, A.N. Kirillov, Combinatorial B_{n}-analogs of Schubert polynomials, Trans. AMS 348 (1996), 3591–3620.

[F1] W. Fulton, Schubert varieties in flag bundles for the classical groups, Proceedings of the Hirzebruch 65 Conference on Algebraic Geometry, IMCP 9 (1996), 241–262.

[F2] W. Fulton, Determinantal Formulas for Orthogonal and Symplectic Degeneracy Loci, J. Diff. Geometry 43 (1996), 276-290.

[G] W. Graham, The class of the diagonal of flag bundles, J. Differential Geometry 45 (1997), 471–487.

[H-B] H. Hiller, B. Boe, Pieri formula for SO_{2n+1}/U_{n} and Sp_{n}/U_{n}, Adv. in Math. 62 (1986), 49–67.

[L-L-T1] A. Lascoux, B. Leclerc, J.-Y. Thibon, Une nouvelle expression des fonctions P de Schur, C. R. Acad. Sci. Paris t. 316 (1993), 221–224.

[L-L-T2] A. Lascoux, B. Leclerc, J.-Y. Thibon, Fonctions de Hall-Littlewood et polynômes de Kostka-Foulkes aux racines de l’unité, C. R. Acad. Sci. Paris t. 316 (1993), 1–6.

[L-S] A. Lascoux, M.P. Schützenberger, Algèbre des différences divisées, Discrete Math. 99 (1992), 165–179.

[Li1] D.E. Littlewood, On certain symmetric functions, Proc. London Math. Soc. (3) (1961), 485-498.

[Li2] D.E. Littlewood, Modular representations of symmetric groups, Proceedings of the Royal Society A 209 (1951), 333-353.

[Mcd1] I.G. Macdonald, Symmetric functions and Hall polynomials, 2nd edition, Oxford Univ. Press, 1995.

[Mcd2] I.G. Macdonald, Notes on Schubert polynomials, Publ. LACIM-UQUAM, Montréal, 1991.

[Mo] A.O. Morris, A note on the multiplication of Hall-Littlewood polynomials, Journal London Math. Soc. 39 (1964), 481–488.

[P] P. Pragacz, Algebro-geometric applications of Schur S- and Q-polynomials, Séminare d’Algèbre Dubreil–Malliavin 1989–1990 (M.-P. Malliavin - ed.), Springer Lecture Notes in Math. 1478 (1991), 130–191.

[P-R1] P. Pragacz, J. Ratajski, Pieri type formula for isotropic Grassmannians; the operator approach, Manuscripta Math. 79 (1993), 127–151.

[P-R2] P. Pragacz, J. Ratajski, Formulas for Lagrangian and orthogonal degeneracy loci; Q̃-polynomial approach, Compositio Math. 107 (1997), 11–87.

[S] I. Schur, Über die Darstellung der symmetrischen und der alternierenden Gruppe durch gebrochene lineare Substitutionen, J. reine angew. Math. 139 (1911), 155-250.

38 ALAIN LASCOUX AND PIOTR PRAGACZ

[V1] S. Veigneau, ACE, an Algebraic Combinatorics Environment for the computer algebra system MAPLE: User’s reference manual, Version 2.0 , http://phalanstere.univmlv.fr/∼sv/ACE, IGM 97-05 (1996).

[V2] S. Veigneau, Calcul Symbolique et Calcul Distribué en Combinatoire Algébrique, Université de Marne-la-Vallée , Thesis, IGM 96-39 (1996).

A.L. : Institut Gaspard Monge, Université de Marne-la-Vallée, 2 rue de la Butte-Verte, 93166 Noisy-le-Grand cedex, France P.P. : Mathematical Institute of Polish Academy of Sciences, Chopina 12, 87-100 Toruń, Poland J.R. : Mathematical Institute of Polish Academy of Sciences, Śniadeckich 8, 00-950 Warszawa, Poland