arXiv++ Combinatorics

We introduce two vertex operators to realize skew odd orthogonal characters $so_{λ/μ}(x^{\pm})$ and derive the Cauchy identity for the skew characters via Toeplitz-Hankel-type determinant similar to the Schur functions. The method also gives new proofs of the Jacobi--Trudi identity and Gelfand--Tsetlin patterns for $so_{λ/μ}(x^{\pm})$. Moreover, combining the vertex operators related to characters of types $C,D$ (\cite{Ba1996,JN2015}) and the new vertex operators related to $B$-type characters, we obtain three families of symmetric polynomials that interpolate among characters of $SO_{2n+1}(\mathbb{C})$, $SO_{2n}(\mathbb{C})$ and $Sp_{2n}(\mathbb{C})$, Their transition formulas are also explicitly given among symplectic and/or orthogonal characters and odd orthogonal characters.

In this paper, we first introduce a family of universal symplectic functions $sp_λ(\mathbf{x}^{\pm};\mathbf{z})$ that include symplectic Schur functions $sp_λ(\mathbf{x}^{\pm})$, odd symplectic characters $sp_λ(\mathbf{x}^{\pm};z)$, universal symplectic characters $sp_λ(\mathbf{z})$ and intermediate symplectic characters as subfamilies. We then realize the universal symplectic functions by vertex operators, which naturally lead to their skew versions, and show that $sp_λ(\mathbf{x}^{\pm};\mathbf{z})$ obey the general branching rules. This also gives the Gelfand-Tsetlin representations of odd symplectic characters and a transition formula between odd symplectic characters and symplectic Schur functions. Secondly we introduce a family of universal orthogonal functions $o_λ(\mathbf{x}^{\pm};\mathbf{z})$ and their skew versions in a similar manner, and we provide their vertex operator realizations and obtain transition formulas and the branching rule. The universal orthogonal functions $o_λ(\mathbf{x}^{\pm};\mathbf{z})$ generalize orthogonal Schur functions $o_λ(\mathbf{x}^{\pm})$, odd orthogonal Schur functions $so_λ(\mathbf{x}^{\pm})$, universal orthogonal characters $o_λ(\mathbf{z})$ as well as intermediate orthogonal characters. Thirdly, we give vertex operator realizations for the $CB$-interpolating Schur functions $s^{CB}_λ(x;β)$ introduced by Bisi and Zygouras (Adv. Math., 2022) and the $DB$-interpolating Schur functions $s^{DB}_λ(x;β)$ interpolating between characters of type $D$ and $B$. As an application, we show $s^{CB}_λ(x;β)$ are equal to the orthosymplectic Schur polynomials $spo_λ(x/β)$, thus give a short proof of the generalization of the Brent-Krattenthaler-Warnaar identity obtained by Kumari (arXiv:2401.01723).

Using the vertex operator representations for symplectic and orthogonal Schur functions, we define two families of symmetric functions and show thatthey are the skew symplectic and skew orthogonal Schur polynomials defined implicitly by Koike and Terada and satisfy the general branching rules. Furthermore, we derive the Jacobi-Trudi identities and Gelfand-Tsetlin patterns for these symmetric functions. Additionally, the vertex operator method yields their Cauchy-type identities. This demonstrates that vertex operator representations serve not only as a tool for studying symmetric functions but also offers unified realizations for skew Schur functions of types A, C, and D.