Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
cs.CL
Submitted: 2026-08-24
Updated: 2026-08-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving.
Terminology
Abstract
We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving. Its largest member is a sparse 10B-parameter Mixture-of-Experts encoder with approximately 1.8B active parameters per token. Across English, Russian, multilingual, and code MTEB benchmarks, this model achieves the strongest aggregate performance within the family on all four evaluated suites. In our vLLM benchmark with 1024-token inputs, it processes 114.5k tokens per second, providing 25 percent higher throughput than the dense 3B model and 1.56-2.65x the throughput of the evaluated external systems. The family also includes a dense 3B encoder and a distilled 480M encoder for tighter compute and memory budgets. We train the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions. The resulting 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42 percent fewer parameters. We release all three model checkpoints.
Sources
- jina-embeddings-v5-text: Task-Targeted Embedding Distillation
- MMTEB: Massive Multilingual Text Embedding Benchmark
- Towards General Text Embeddings with Multi-stage Contrastive Learning
- Distilling the Knowledge in a Neural Network
- Nomic Embed: Training a Reproducible Long Context Text Embedder
- Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
- Mixtral of Experts
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Gecko: Versatile Text Embeddings Distilled from Large Language Models
- Gemini Embedding: Generalizable Embeddings from Gemini
- GUIDE: Guided Initialization and Distillation of Embeddings
- Representation Learning with Contrastive Predictive Coding
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Improving Text Embeddings with Large Language Models
- C-Pack: Packed Resources For General Chinese Embeddings
- One Student Knows All Experts Know: From Sparse to Dense
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering