NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
cs.IR, cs.AI, cs.CV
Submitted: 2026-08-31
Updated: 2026-08-31
Code: https://github.com/erikkaum/maxsim
Project page: https://hcompai.github.io/late-interaction-kernels/how-it-works.html
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models.
Terminology
Abstract
Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.
Sources
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- Flamingo: a Visual Language Model for Few-Shot Learning
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
- Longformer: The Long-Document Transformer
- PaliGemma: A versatile 3B VLM for transfer
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- EuroBERT: Scaling Multilingual Encoders for European Languages
- BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- MaskGIT: Masked Generative Image Transformer
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
- Cautious Weight Decay
- SOLO: A Single Transformer for Scalable Vision-Language Modeling
- M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
- Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling
- Scaling Vision Transformers to 22 Billion Parameters
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings
- Unveiling Encoder-Free Vision-Language Models
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- How Do LLM-Generated Texts Impact Term-Based Retrieval Models?
- MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
- Training with Missed Targets in Generative Recommendation: Separating Supervision from Probability Competition