Mapping Similarity Spaces across Embedding Models with Synthetic Query Probing
Marcin Rozmus, Peter van der Putten
cs.CL
Submitted: 2026-08-06
Comments: Accepted for 29th International Conference on Discovery Science, October 5-9, 2026, Mainz, Germany
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Retrieval-Augmented Generation systems rely on similarity scores to retrieve relevant content, yet scores are not directly comparable across embedding models due to differing geometric properties,
Terminology
Abstract
Retrieval-Augmented Generation systems rely on similarity scores to retrieve relevant content, yet scores are not directly comparable across embedding models due to differing geometric properties, complicating model migration and limiting threshold reuse. We study how similarity scores can be related by learning mappings between score distributions rather than embeddings. We introduce Synthetic Query Probing, generating queries from documents to create controlled query-chunk pairs, enabling large-scale, reference-free analysis of cross-model similarity behavior. We evaluate the approach on multiple embedding configurations and learn score conversion functions using linear, isotonic, and quantile mappings. Experiments on SciFact and a proprietary corpus show that while models largely agree on rankings, their absolute scores exhibit systematic distortions. Learned mappings partially align these spaces and improve threshold portability, with isotonic regression performing best. Our results highlight the need for cross-model calibration and position Synthetic Query Probing as a scalable framework for analyzing embedding comparability.
Sources
- The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Calibrated Similarity for Reliable Geometric Analysis of Embedding Spaces
- Semantics at an Angle: When Cosine Similarity Works Until It Doesn't
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering