Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation

The paper introduces Task-Conditional Flow Matching (TCFM), a multilingual embedding adaptation framework that addresses the limitations of applying a single contrastive objective across heterogeneous tasks. The authors argue that multilingual text embedding models are commonly adapted using a single training objective across diverse tasks, despite different tasks requiring fundamentally different optimization strategies. Their framework selectively applies Flow Matching to translation tasks while optimizing retrieval, classification, and pair-classification tasks with objectives better aligned to their learning dynamics.

The paper identifies key weaknesses in contrastive learning for multilingual adaptation: "it struggles in diverse multi-task or multilingual settings as it treats all unpaired instances in a batch as strict negatives, they often suffer from the false-negative problem, where semantically similar sentences are incorrectly pushed apart. Furthermore, this rigid separation can fragment the continuous embedding space, disrupting the smooth geometry needed to accurately model cross-lingual transformations. The authors note that optimizing a single objective across heterogeneous task families often produces conflicting training signals, where improvements on some tasks come at the expense of others."

Flow Matching is proposed as well-suited for multilingual embedding adaptation because it learns smooth transformations between continuous representations by modeling a vector field rather than directly aligning embeddings, and translation pairs provide naturally corresponding representations. However, since retrieval, classification, and pair-classification capture fundamentally different semantic relationships, making a uniform transport objective sub-optimal across these heterogeneous tasks, the framework applies Flow Matching conditionally.

The framework operates on sentence embeddings fθ(x) produced by the encoder, with a frozen pretrained teacher encoder f0(·) providing stable reference representations.

Four task families are curated:

• Translation: parallel sentences and semantically equivalent pairs which provide explicit source-target correspondence, making it well suited for learning continuous semantic transformations across languages

• Classification: samples sharing the same label, although these samples belong to the same semantic class, they are not necessarily paraphrases

• Pair Classification: labeled sentence pairs that describe the semantic relationship between two texts, such as whether they are semantically similar or belong to different relationship categories

• Retrieval and Re-ranking: query and document play distinct semantic roles and are therefore not interchangeable, making the relationship inherently asymmetric

Multi-Positive Contrastive Loss: A multi-positive InfoNCE objective is applied symmetrically for translation... and asymmetrically for retrieval (query to document only). False negatives are mitigated by excluding highly similar non-positives from the denominator using a similarity threshold, though this filtering is disabled for pair-classification where similar negatives provide essential learning signals.

Flow Matching: Activated only for translation task families, since parallel translation pairs represent semantically equivalent sentences across languages and therefore admit meaningful continuous transformations. The velocity prediction network is conditioned on the task instruction embedding extracted from the frozen teacher encoder. The Flow Matching objective minimizes the discrepancy between the predicted and target transport directions. A bounded transport objective is also introduced where the transported representation is then encouraged to align with the target embedding, which allows the learned vector field to influence the embedding space while preventing large representation shifts.

Teacher Preservation: Both pointwise and relational objectives are used to prevent drift from the pretrained encoder's semantic structure. The pointwise loss encourages each example to remain close to its original embedding direction, while the relational objective matches the pairwise similarity structure of each mini-batch.

Hard-Negative Margin Repair: A cosine-margin objective ensures that an anchor x remains closer to its positive sample y than to a hard negative n by at least a predefined margin, activated only for task families that contain explicit hard negatives.

Monolingual Consistency: A SimCSE-style self-consistency objective regularizes local neighborhoods and ensures the monolingual integrity of the representations remains robust as cross-lingual clusters are merged.

• Stage 1: Cross-Lingual Alignment — focuses exclusively on translation supervision with greater emphasis on Flow Matching and teacher-preservation objectives, enabling the model to establish stable multilingual representations before introducing more heterogeneous supervision.

• Stage 2: Multi-Task Semantic Adaptation — introduces classification and NLI datasets while maintaining a replay buffer of 40k translation pairs (approximately 23% of the training mixture) to effectively mitigate the catastrophic forgetting of the cross-lingual alignment.

• Stage 3: Retrieval Adaptation and Monolingual Regularization — incorporates asymmetric query-document retrieval with language-identity supervision, retaining the 40,000-sample replay buffer at approximately 41% of total Stage 3 data.

Training data includes local wide parallel (self-curated via translating English Wikipedia sentences into 16 Indic languages), Samanantar, MASSIVE intent datasets, IndicSentiment, IndicXNLI, IndicMSMARCO, and Bhasha. To prevent gradient imbalance caused by massive parallel corpora overpowering low-resource tasks, the authors employ stratified subsampling across languages and cap maximum dataset sizes.

Evaluation uses the Indic MTEB v3 benchmark, spanning seven task families: Bitext Mining, Classification, Clustering, Pair Classification, Retrieval, Reranking, and STS. Models evaluated include google/embeddinggemma-300m, microsoft/harrier-oss-v1-270m, microsoft/harrier-oss-v1-0.6B, Qwen/Qwen3-Embedding-4B, and Qwen/Qwen3-Embedding-8B (the latter two using LoRA).

TCFM establishes a new state-of-the-art on Indic MTEB:

• Harrier-0.6B: +3.59 points overall, with the largest improvement observed on the Clustering benchmark, where TCFM increases performance by more than 21 points. Cross-lingual alignment also improves, with IN22ConvBitextMining improving by +3.30 points and IndicCrosslingualSTS improving by +2.24 points.

• Qwen3-Embedding-8B: +1.97 points absolute gain, with improvements across six of seven benchmark categories; Retrieval exhibits a minor decrease (-0.28).

• EmbeddingGemma-300M: +2.34 points overall, with notable gains in Bitext Mining (+4.48) and STS (+2.53).

• Harrier-270M: +4.40 points overall, the largest relative improvement among the extended models.

• Qwen3-Embedding-4B: +1.65 points overall, with +11.88 points jump in Clustering.

Small performance decreases were noted on a few tasks including SanskritShlokasClassification (-5.31), IN22GenBitextMining (-0.16), NepaliNewsClassification (-0.14), and XQuADRetrieval (-0.23), indicating that multilingual embedding adaptation remains a challenging multi-objective optimization problem.

Component Analysis: Flow Matching combined with teacher preservation induces gains on Retrieval (+1.06) and Clustering (+21.15), while contrastive learning with teacher preservation yields strong improvements on Pair Classification (+6.27), Clustering (+21.32), and STS (+0.46), demonstrating complementary strengths across different downstream task families. Removing teacher preservation substantially reduces Retrieval (-12.48) and STS (-8.99).

Curriculum Effectiveness: Stage 1 establishes cross-lingual alignment; Stage 2 introduces semantic discrimination but can cause task interference (e.g., Qwen3-Embedding-8B experiences a regression in clustering (-3.36)); Stage 3 resolves multi-task optimization friction via monolingual regularization.

Task-Conditional Flow Matching: Restricting Flow Matching exclusively to translation tasks yields the highest overall Indic MTEB score (+1.67 vs. +1.63) compared to uniform application across all tasks, validating the task-conditional design.

The paper concludes that applying a single contrastive objective across all training data is suboptimal for adapting multilingual embeddings and that task-aware optimization matching the loss function to the underlying task semantics is a highly effective strategy for multilingual representation learning.

The authors acknowledge: evaluation limited to 22 Indian languages in Indic MTEB; the claim that Flow Matching improves embedding space uniformity is currently... empirical without formal geometric analysis; unexplored repulsive flow dynamics for hard negatives; and additional computational overhead requiring a frozen teacher model, velocity prediction network, multi-stage curriculum, and high quality parallel translation data, which may limit its application in zero resource languages where such parallel corpora are unavailable.

Improvements to AI Systems

1. Task-aware loss selection instead of a single contrastive objective.

Use Flow Matching for translation/paraphrase tasks, contrastive loss for retrieval, classification-specific losses for labeled semantic classes, and pair-classification objectives for NLI-style relationships.

What it can do: Multilingual embedding models avoid conflicting training signals, improving overall performance across diverse task families rather than sacrificing certain tasks.

2. Conditional Flow Matching for cross-lingual alignment.

Train a velocity-prediction network conditioned on task instructions to model smooth continuous transformations between translation-pair embeddings, combined with a bounded transport objective.

What it can do: Produce a smoother, less fragmented embedding space across languages, yielding large gains in bitext mining, cross-lingual STS, and clustering (e.g., +21 points on clustering).

3. Frozen-teacher preservation regularization.

Add pointwise (embedding direction) and relational (pairwise similarity structure) losses against a frozen pretrained teacher encoder, keeping each mini-batch aligned with the original semantic structure.

What it can do: Prevent catastrophic drift during multilingual adaptation, maintaining retrieval and STS performance that would otherwise drop substantially (e.g., -12.48 retrieval, -8.99 STS if removed).

4. Multi-stage curriculum with translation replay buffer.

Stage 1: pure cross-lingual alignment with Flow Matching. Stage 2: introduce classification/NLI with a replay buffer of ~23% translation pairs. Stage 3: add asymmetric retrieval and monolingual regularization with ~41% translation replay.

What it can do: Resolve multi-task optimization friction, reduce task interference (e.g., avoid clustering regression in Stage 2), and remember cross-lingual alignment while adapting to heterogeneous tasks.

5. Task-specific negative management.

For translation/retrieval, filter out highly similar false negatives from the contrastive denominator. For pair-classification, disable filtering so similar negatives provide essential discriminative signals. Add a hard-negative margin repair loss for task families with explicit hard negatives.

What it can do: Improve embedding discrimination where needed, avoid pushing semantically similar sentences apart erroneously, and boost pair-classification and retrieval ranking accuracy.

6. Monolingual self-consistency regularization (SimCSE-style).

Enforce that each sentence embedding remains consistent with its own perturbed version, in addition to cross-lingual cluster merging.

What it can do: Preserve monolingual representational integrity and local neighborhood structure, preventing cross-lingual alignment from distorting within-language semantics.

7. Stratified subsampling and dataset capping.

Limit over-representation from massive parallel corpora and ensure low-resource tasks/languages appear in proportions that prevent gradient imbalance.

What it can do: Deliver stable, balanced performance across multiple languages and tasks, including low-resource Indic languages, avoiding overfitting to high-resource translation data.

8. Task-conditional velocity prediction with teacher instructions.

Extract the task-identity embedding from the frozen teacher encoder and feed it into the Flow Matching velocity network so the transport field adapts per task.

What it can do: Enable a single model to learn flexible, task-conditioned transformation geometry, outperforming uniform Flow Matching across all tasks (+1.67 vs +1.63 on Indic MTEB) and supporting flexible deployment on new task families.

Multilingual text embedding models are commonly adapted using a single training objective across diverse tasks, despite different tasks requiring fundamentally different optimization strategies. We introduce Task-Conditional Flow Matching (TCFM), a multilingual embedding adaptation framework that selectively applies Flow Matching to translation tasks while optimizing retrieval, classification, and pair-classification tasks with objectives better aligned to their learning dynamics. TCFM further combines teacher-guided representation preservation with a three-stage curriculum to enable stable adaptation. Evaluated on the Indic Massive Text Embedding Benchmark, TCFM establishes a new state-of-the-art, consistently improving embedding quality across a diverse set of multilingual tasks and generalizing across embedding model families. We will publicly release the codebase and datasets upon acceptance of the paper.