arXiv++ Combinatorics

We introduce three families of vectors $|\underlineλ^{so}\rangle$, $|\underlineλ^{sp}\rangle$ and $|\underlineλ^{o}\rangle$ parametrized by partitions in the Fock space by using products of adjoint vertex operators. We show that the quotient space of the dual vacuum vector is spanned by the partition vectors indexed by a special family of partitions. The partition-indexed vectors also help us to derive the dual Littlewood identities of types B, C, and D in a new manner associated to the special family of partitions. As an application, we obtain a new free fermionic construction to show that the measures related to dual Littlewood identities introduced by Rains \cite[Section 7]{Rai2000} and Betea \cite[Section 3]{Be2020} are determinantal with repect to some explicit correlation kernels. Furthermore we establish a number of generalized Littlewood identities summed over certain restricted partitions by computing the inner products with elements indexed by one-column partitions {or generalized partitions $(0^m)$} in the complete dual Fock space. {In particular, for each positive integer $n$, we obtain generalized Littlewood identities for $(-n)$-asymmetric partitions. We show that these generalized Littlewood identities contain several well-known Littlewood-type identities as special cases. Consequently we also give a new proof of the generalized Littlewood identity \cite[(5.25)]{LSV2008} for Lie superalgebras. } %We also produce infinite generalized Littlewood identities by calculating the inner products between these elements and some elements indexed by one-column partitions in the complete dual Fock space.

We introduce two vertex operators to realize skew odd orthogonal characters $so_{λ/μ}(x^{\pm})$ and derive the Cauchy identity for the skew characters via Toeplitz-Hankel-type determinant similar to the Schur functions. The method also gives new proofs of the Jacobi--Trudi identity and Gelfand--Tsetlin patterns for $so_{λ/μ}(x^{\pm})$. Moreover, combining the vertex operators related to characters of types $C,D$ (\cite{Ba1996,JN2015}) and the new vertex operators related to $B$-type characters, we obtain three families of symmetric polynomials that interpolate among characters of $SO_{2n+1}(\mathbb{C})$, $SO_{2n}(\mathbb{C})$ and $Sp_{2n}(\mathbb{C})$, Their transition formulas are also explicitly given among symplectic and/or orthogonal characters and odd orthogonal characters.

In this paper, we first introduce a family of universal symplectic functions $sp_λ(\mathbf{x}^{\pm};\mathbf{z})$ that include symplectic Schur functions $sp_λ(\mathbf{x}^{\pm})$, odd symplectic characters $sp_λ(\mathbf{x}^{\pm};z)$, universal symplectic characters $sp_λ(\mathbf{z})$ and intermediate symplectic characters as subfamilies. We then realize the universal symplectic functions by vertex operators, which naturally lead to their skew versions, and show that $sp_λ(\mathbf{x}^{\pm};\mathbf{z})$ obey the general branching rules. This also gives the Gelfand-Tsetlin representations of odd symplectic characters and a transition formula between odd symplectic characters and symplectic Schur functions. Secondly we introduce a family of universal orthogonal functions $o_λ(\mathbf{x}^{\pm};\mathbf{z})$ and their skew versions in a similar manner, and we provide their vertex operator realizations and obtain transition formulas and the branching rule. The universal orthogonal functions $o_λ(\mathbf{x}^{\pm};\mathbf{z})$ generalize orthogonal Schur functions $o_λ(\mathbf{x}^{\pm})$, odd orthogonal Schur functions $so_λ(\mathbf{x}^{\pm})$, universal orthogonal characters $o_λ(\mathbf{z})$ as well as intermediate orthogonal characters. Thirdly, we give vertex operator realizations for the $CB$-interpolating Schur functions $s^{CB}_λ(x;β)$ introduced by Bisi and Zygouras (Adv. Math., 2022) and the $DB$-interpolating Schur functions $s^{DB}_λ(x;β)$ interpolating between characters of type $D$ and $B$. As an application, we show $s^{CB}_λ(x;β)$ are equal to the orthosymplectic Schur polynomials $spo_λ(x/β)$, thus give a short proof of the generalization of the Brent-Krattenthaler-Warnaar identity obtained by Kumari (arXiv:2401.01723).

Using the vertex operator representations for symplectic and orthogonal Schur functions, we define two families of symmetric functions and show thatthey are the skew symplectic and skew orthogonal Schur polynomials defined implicitly by Koike and Terada and satisfy the general branching rules. Furthermore, we derive the Jacobi-Trudi identities and Gelfand-Tsetlin patterns for these symmetric functions. Additionally, the vertex operator method yields their Cauchy-type identities. This demonstrates that vertex operator representations serve not only as a tool for studying symmetric functions but also offers unified realizations for skew Schur functions of types A, C, and D.

We use the correlation functions of vertex operators to give a proof of Cauchy's formula \begin{align*} \prod^K_{i=1}\prod^N_{j=1}(1-x_iy_j)=\sum_{μ\subseteq [K\times N]}(-1)^{|μ|}s_μ\{x\}s_{μ'}\{y\}. \end{align*} As an application of the interpretation, we obtain an expansion of $\prod^\infty_{i=1}(1-q^i)^{i-1}$ in terms of half plane partitions.