arXiv++ Combinatorics

Serrano (2010) introduced the shifted plactic monoid, governing Haiman's (1989) mixed insertion algorithm, as a type B analogue of the classical plactic monoid that connects jeu de taquin of Young tableaux with the Robinson-Schensted-Knuth insertion algorithm. Serrano proposed a corresponding definition of skew shifted plactic Schur functions. Cho (2013) disproved Serrano's conjecture regarding this definition, by showing that the functions do not live in the desired ring and hence cannot provide an algebraic interpretation of tableau rectification or of the corresponding structure coefficients. Cho asked for a new definition with particular properties. We introduce such a definition and prove that it behaves as desired. We also introduce a new jeu de taquin theory that computes mixed insertion.

We give a new Littlewood-Richardson rule for the Schubert structure coefficients of isotropic Grassmannians, equivalently for the multiplication of $P$-Schur functions. Serrano (2010) previously gave a formula in terms of classes in his shifted plactic monoid. However, this formula is challenging to use because of the difficulty of characterizing shifted plactic classes. We give the first algebraic proof of this formula. We then use it to obtain a new rule that is easy to implement. Our rule is based on identifying a subtle analogue of Yamanouchi tableaux, which we characterize. We show that for some families of structure coefficients, our rule leads to an algorithm with exponentially better time complexity than the original rule of Stembridge (1989).

The plactic monoid $\mathbf{P}$ of Lascoux and Schützenberger (1981) plays an important role in proofs of the Littlewood-Richardson rule for computing multiplicities in the linear representation theory of the symmetric group $\mathfrak{S}_n$ and the cohomology of Grassmannians. Commonly, $\mathbf{P}$ is defined as a quotient of a free monoid by relations derived from a careful analysis of Schensted's insertion algorithm and the jeu de taquin algorithm on semistandard Young tableaux. However, Lascoux and Schützenberger also gave an intrinsic characterization of $\mathbf{P}$ via a universal property. Serrano's (2010) shifted plactic monoid $\mathbf{S}$ is an analogue of $\mathbf{P}$ that governs instead the projective representation theory of $\mathfrak{S}_n$ and the cohomology of isotropic Grassmannians. We provide a universal property for $\mathbf{S}$, analogous to the Lascoux-Schützenberger characterization of $\mathbf{P}$.