How Far Do Simple Transformations Translate Across Text Embedding Models?
Orange Research · Orange Research · Orange Research · Orange Research
cs.LG
Submitted: 2026-08-06
Updated: 2026-09-02
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: We investigate whether simple transformations can translate representations across heterogeneous text embedding models.
Terminology
Abstract
We investigate whether simple transformations can translate representations across heterogeneous text embedding models. Understanding how independently trained models organize semantic information is an enabler for AI-to-AI latent communication without decoding into human-readable text. Focusing on lightweight translators such as linear mappings, we test the literature hypothesis of latent universality in a realistic text setting beyond simplified benchmarks. Across nine embedding models differing in architecture, pooling strategy, and training objective, we evaluate compatibility using CKA, downstream transfer, fidelity, and retrieval. Simple translators recover meaningful shared structure and support transfer for some compatible pairs, but fail sharply for others. Compatibility depends jointly on architecture, training objective, pooling, and data distribution. Overall, the results show that heterogeneous embedding spaces are not universally related by simple mappings as often suggested in some literature.
Sources
- Revisiting Model Stitching to Compare Neural Representations
- Transferring Linear Features Across Language Models With Model Stitching
- The Platonic Representation Hypothesis
- Latent Collaboration in Multi-Agent Systems
- Enabling Agents to Communicate Entirely in Latent Space
- A Survey on Latent Reasoning
- What makes ImageNet good for transfer learning?
- Offline bilingual word vectors, orthogonal transformations and the inverted softmax
- Relative representations enable zero-shot latent space communication
- Ventriloquist LLMs: Linear Alignment of Late-Stage Representations
- Latent Space Translation via Inverse Relative Projection
- From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication
- Efficient Estimation of Word Representations in Vector Space
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks