arXiv++ Combinatorics

Why does the low dimensionality of representations, typically $d\approx 1000$, not prevent modern embedding-based retrieval models from scaling to billions, or even trillions, of data points? To answer this question, we study maximal-margin embeddings in the following retrieval model, classically studied in communication complexity [PS86] and more recently in embedding-based retrieval [WBNL26]. Let $A\in \{0,1\}^{N\times n}$ be a matrix indicating whether each of $N$ queries is relevant to each of $n$ documents. We are interested in the largest margin $m>0,$ denoted by $\mathsf{m}^{\mathsf{rd}}(d, A),$ for which there exist unit norm embeddings of the queries and documents $\{U_j\}_{j = 1}^N, \{V_i\}_{i = 1}^n$ with the following property. $\langle U_j, V_i\rangle \ge m$ whenever $A_{ji} = 1$ and $\langle U_j, V_i\rangle \le -m$ otherwise. A large margin is a key proxy for representation quality: it controls both robustness to perturbations and compositional generalization across queries. Our main theorem establishes that the best possible margin without a restriction on the dimension, $\mathsf{m}^{\mathsf{rd}}(+\infty, A),$ can be nearly achieved in dimension $d = O(\mathsf{m}^{\mathsf{rd}}(+\infty, A)^{-2}\log n)$ which improves a theorem of [BDES02]. Together with a matching lower bound in Theorem 1.5, we conclude that when $A\in \{0,1\}^{\binom{n}{k}\times n}$ is the matrix containing all possible $k$-sparse rows once, dimension $d = O(k\log (n/k))$ is necessary and sufficient for the maximal possible margin $\mathsf{m}^{\mathsf{rd}}(+\infty, A) = Θ(k^{-1/2})$ in this setting. This fully resolves the setup of [WBNL26]. We also give several constructions for large margins when $d = o(k\log (n/k)).$ Finally, we empirically test the InfoNCE and sigmoid losses for producing large margin embeddings and demonstrate a clear advantage of the sigmoid loss.

Consider a $d$-uniform random hypergraph on $n$ vertices in which hyperedges are included iid so that the average degree is $n^δ$. The projection of a hypergraph is a graph on the same $n$ vertices where an edge connects two vertices if and only if they belong to some hyperedge. The goal is to reconstruct the hypergraph given its projection. An earlier work of Bresler, Guo, and Polyanskiy (COLT 2024) showed that exact recovery for $d=3$ is possible if and only if $δ< 2/5$. This work completely resolves the question for all values of $d$ for both exact and partial recovery and for both cases of whether multiplicity information about each edge is available or not. In addition, we show that the reconstruction fidelity undergoes an all-or-nothing transition at a threshold. In particular, this resolves all conjectures from Bresler, Guo, and Polyanskiy (COLT 2024).

Consider two sequences of $n$ independent and identically distributed fair coin tosses, $X=(X_1,\ldots,X_n)$ and $Y=(Y_1,\ldots,Y_n)$, which are $ρ$-correlated for each $j$, i.e. $\mathbb{P}[X_j=Y_j] = {1+ρ\over 2}$. We study the question of how large (small) the probability $\mathbb{P}[X \in A, Y\in B]$ can be among all sets $A,B\subset\{0,1\}^n$ of a given cardinality. For sets $|A|,|B| = Θ(2^n)$ it is well known that the largest (smallest) probability is approximately attained by concentric (anti-concentric) Hamming balls, and this can be proved via the hypercontractive inequality (reverse hypercontractivity). Here we consider the case of $|A|,|B| = 2^{Θ(n)}$. By applying a recent extension of the hypercontractive inequality of Polyanskiy-Samorodnitsky (J. Functional Analysis, 2019), we show that Hamming balls of the same size approximately maximize $\mathbb{P}[X \in A, Y\in B]$ in the regime of $ρ\to 1$. We also prove a similar tight lower bound, i.e. show that for $ρ\to 0$ the pair of opposite Hamming balls approximately minimizes the probability $\mathbb{P}[X \in A, Y\in B]$.

We consider list-decoding in the zero-rate regime for two cases: the binary alphabet and the spherical codes in Euclidean space. Specifically, we study the maximal $τ\in [0,1]$ for which there exists an arrangement of $M$ balls of relative Hamming radius $τ$ in the binary hypercube (of arbitrary dimension) with the property that no point of the latter is covered by $L$ or more of them. As $M\to \infty$ the maximal $τ$ decreases to a well-known critical value $τ_L$. In this work, we prove several results on the rate of this convergence. For the binary case, we show that the rate is $Θ(M^{-1})$ when $L$ is even, thus extending the classical results of Plotkin and Levenshtein for $L=2$. For $L=3$ the rate is shown to be $Θ(M^{-\tfrac{2}{3}})$. For the similar question about spherical codes, we prove the rate is $Ω(M^{-1})$ and $O(M^{-\tfrac{2L}{L^2-L+2}})$.

We give new bounds on the reliability function of a typewriter channel with 5 inputs and crossover probability $1/2$. The lower bound is more of theoretical than practical importance; it improves very marginally the expurgated bound, providing a counterexample to a conjecture on its tightness by Shannon, Gallager and Berlekamp which does not need the construction of algebraic-geometric codes previously used by Katsman, Tsfasman and Vlăduţ. The upper bound is derived by using an adaptation of the linear programming bound and it is essentially useful as a low-rate anchor for the straight line bound.

Log-Sobolev inequalities (LSIs) upper-bound entropy via a multiple of the Dirichlet form (i.e. norm of a gradient). In this paper we prove a family of entropy-energy inequalities for the binary hypercube which provide a non-linear comparison between the entropy and the Dirichlet form and improve on the usual LSIs for functions with small support. These non-linear LSIs, in turn, imply a new version of the hypercontractivity for such functions. As another consequence, we derive a sharp form of the uncertainty principle for the hypercube: a function whose energy is concentrated on a set of small size, and whose Fourier energy is concentrated on a small Hamming ball must be zero. The tradeoff between the sizes that we derive is asymptotically optimal. This new uncertainty principle implies a new estimate on the size of Fourier coefficients of sparse Boolean functions. We observe that an analogous (asymptotically optimal) uncertainty principle in the Euclidean space follows from the sharp form of Young's inequality due to Beckner. This hints that non-linear LSIs augment Young's inequality (which itself is sharp for finite groups).

The capacity of a graph is defined as the rate of exponential grow of independent sets in the strong powers of the graph. In strong power, an edge connects two sequences if at each position letters are equal or adjacent. We consider a variation of the problem where edges in the power graphs are removed among sequences which differ in more than a fraction $δ$ of coordinates. For odd cycles, we derive an upper bound on the corresponding rate which combines Lovász' bound on the capacity with Delsarte's linear programming bounds on the minimum distance of codes in Hamming spaces. For the pentagon, this shows that for $δ\ge {1-{1\over\sqrt{5}}}$ the Lovász rate is the best possible, while we prove by a Gilbert-Varshamov-type bound that a higher rate is achievable for $δ< {2\over 5}$. Communication interpretation of this question is the problem of sending quinary symbols subject to $\pm 1\mod 5$ disturbance. The maximal communication rate subject to the zero undetected-error equals capacity of a pentagon. The question addressed here is how much this rate can be increased if only a fraction $δ$ of symbols is allowed to be disturbed

A mapping of $k$-bit strings into $n$-bit strings is called an $(α,β)$-map if $k$-bit strings which are more than $αk$ apart are mapped to $n$-bit strings that are more than $βn$ apart. This is a relaxation of the classical problem of constructing error-correcting codes, which corresponds to $α=0$. Existence of an $(α,β)$-map is equivalent to existence of a graph homomorphism $\bar H(k,αk)\to \bar H(n,βn)$, where $H(n,d)$ is a Hamming graph with vertex set $\{0,1\}^n$ and edges connecting vertices differing in $d$ or fewer entries. This paper proves impossibility results on achievable parameters $(α,β)$ in the regime of $n,k\to\infty$ with a fixed ratio ${n\over k}= ρ$. This is done by developing a general criterion for existence of graph-homomorphism based on the semi-definite relaxation of the independence number of a graph (known as the Schrijver's $θ$-function). The criterion is then evaluated using some known and some new results from coding theory concerning the $θ$-function of Hamming graphs. As an example, it is shown that if $β>1/2$ and $n\over k$ -- integer, the ${n\over k}$-fold repetition map achieving $α=β$ is asymptotically optimal. Finally, constraints on configurations of points and hyperplanes in projective spaces over $\mathbb{F}_2$ are derived.

Consider the problem of packing Hamming balls of a given relative radius subject to the constraint that they cover any point of the ambient Hamming space with multiplicity at most $L$. For odd $L\ge 3$ an asymptotic upper bound on the rate of any such packing is proven. Resulting bound improves the best known bound (due to Blinovsky'1986) for rates below a certain threshold. Method is a superposition of the linear-programming idea of Ashikhmin, Barg and Litsyn (that was used previously to improve the estimates of Blinovsky for $L=2$) and a Ramsey-theoretic technique of Blinovsky. As an application it is shown that for all odd $L$ the slope of the rate-radius tradeoff is zero at zero rate.

Consider the linear space of functions on the binary hypercube and the linear operator $S_δ$ acting by averaging a function over a Hamming sphere of radius $δn$ around every point. It is shown that this operator has a dimension-independent bound on the norm $L_p \to L_2$ with $p = 1+(1-2δ)^2$. This result evidently parallels a classical estimate of Bonami and Gross for $L_p \to L_q$ norms for the operator of convolution with a Bernoulli noise. The estimate for $S_δ$ is harder to obtain since the latter is neither a part of a semigroup, nor a tensor power. The result is shown by a detailed study of the eigenvalues of $S_δ$ and $L_p\to L_2$ norms of the Fourier multiplier operators $Π_a$ with symbol equal to a characteristic function of the Hamming sphere of radius $a$ (in the notation common in boolean analysis $Π_a f=f^{=a}$, where $f^{=a}$ is a degree-$a$ component of function $f$). A sample application of the result is given: Any set $A\subset \FF_2^n$ with the property that $A+A$ contains a large portion of some Hamming sphere (counted with multiplicity) must have cardinality a constant multiple of $2^n$.