Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms
Ziwei Su, Junyu Ren, Victor Veitch
stat.ML, cs.AI, cs.LG, math.OC
Submitted: 2026-06-29
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes.
Terminology
Abstract
Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes. However, surprisingly, empirical studies reveal that despite this, these "discarded" norms seem to correlate with semantic properties such as concept specificity, token frequency, and human uncertainty. In this work, we provide a formal theoretical framework explaining this phenomenon. By analyzing the optimization dynamics, we derive an analytic formula demonstrating that embedding length naturally encodes this information as a byproduct of the training process. We also show how this gives rise to signals that can serve as "free" calibration tools in specific models and retrieval tasks, providing a grounded explanation for a previously heuristic observation.
Sources
- On the Importance of Embedding Norms in Self-Supervised Learning
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- The Double-Ellipsoid Geometry of CLIP
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- jina-embeddings-v3: Multilingual Embeddings With Task LoRA
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- NormFace: L2 Hypersphere Embedding for Face Verification
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Demystifying CLIP Data
- Qwen3 Technical Report
- Heated-Up Softmax Embedding
Related papers
- Behavior of prediction performance metrics with rare events
- Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
- A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
- Scalable Posterior Uncertainty for Flexible Density-Based Clustering
- A Transport Geometry of Benign Overfitting
- Efficient quadratic entropy with distance sketches