Do Multilingual Encoders Produce Language-Consistent Semantic IDs?

arXiv:2610.01139 · cs.IR, cs.CL · Submitted 2026-10-01 · Read on arXiv

cs.IR, cs.CL

Submitted: 2026-10-01

Updated: 2026-10-01

Comments: 7 pages, 8 tables. Accepted as a short paper at WiNLP 2026, co-located with EMNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Embeddings — 11× in this paper · explained in 2 episodes
• Consistency — 7× in this paper · explained in 3 episodes
• Quantization — 6× in this paper · explained in 12 episodes
• Alignment — 5× in this paper · explained in 1 episode
• Accuracy — 4× in this paper · explained in 1 episode
• Transformer — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Cross-lingual Transfer — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Robustness — 1× in this paper · explained in 5 episodes

Sources

• Parallel Tokenizers: Rethinking Vocabulary Design in Cross-Lingual Transfer of Low-Resource Languages
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search
• Multilingual E5 Text Embeddings: A Technical Report
• Decoupled Residual Quantization for Robust Semantic IDs in Recommendation
• Qwen2.5 Technical Report

Related papers

• The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
• SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
• MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
• How Do LLM-Generated Texts Impact Term-Based Retrieval Models?
• MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
• Training with Missed Targets in Generative Recommendation: Separating Supervision from Probability Competition