Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

arXiv:2608.30725 · cs.CL · Submitted 2026-08-31 · Read on arXiv

cs.CL

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/dnotitia/geometric-bottleneck

License: http://creativecommons.org/licenses/by/4.0/

The gist: Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g.

Terminology

Abstract

Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by 30+,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from >99% to near chance and iterated INLP to 37 -- 50% while LRL retrieval moves within plus or minus 1.5,pp and all tier means within 2.2,pp, tracking random controls. The linear bias is a symptom, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from 22.1% to 69.1% on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering +9.6 / +17.1,pp on LRL XM3600 retrieval (1, 000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

Sources

Related papers