SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval
cs.IR, cs.AI
Submitted: 2026-06-17
Updated: 2026-08-27
Comments: EMNLP 2026 Findings
Code: https://github.com/huggingface/sentence-transformers
License: http://creativecommons.org/licenses/by/4.0/
The gist: With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access.
Terminology
Abstract
With the rapid expansion of massive multilingual corpora, Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access. MLIR enables users to retrieve semantically relevant documents from multilingual text collections using a single-language query. However, recent multilingual dense retrieval models often exhibit a strong preference for documents in the same language as the query. This leads to severe language bias, where top-ranked results are dominated by documents of specific languages, even when documents in other languages contain more semantically relevant information. To address this issue, we propose SHIFT, a training-free method applicable in the indexing stage. Specifically, SHIFT utilizes parallel translation pairs to estimate a relative language vector for each target language with respect to a source language. Subsequently, SHIFT corrects the language-specific offset by subtracting this relative language vector from document embeddings during indexing. Our comprehensive evaluation across four MLIR benchmarks and diverse dense retrieval models confirms that SHIFT can effectively mitigate language bias and enhance MLIR performance.
Sources
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- MMTEB: Massive Multilingual Text Embedding Benchmark
- NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval
- Gemini Embedding: Generalizable Embeddings from Gemini
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Overview of the TREC 2024 NeuCLIR Track
- Investigating Language Preference of Multilingual RAG Systems
- EmbeddingGemma: Powerful and Lightweight Text Representations
- Multilingual E5 Text Embeddings: A Technical Report
- HLTCOE at TREC 2024 NeuCLIR Track
- Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?
- MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
- From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction