Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning

The gist The core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real–synthetic separation, and Sinkhornbased OT ensures balanced fine-grained patch correspondence; this synergy improves grounding, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScoreF1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real–synthetic centroid gap by 41%

Architecture Overview

The pipeline employs a compute-aware Bengali captioning pipeline where a frozen MaxViT [29] backbone supplies stable visual patches, a Bengalinative mBART-50 [19] decoder generates text, and a lightweight bridge links the modalities The frozen MaxViT yields stable visual patches, a Bengalinative mBART-50 decodes, and a lightweight bridge links the modalities The architecture is designed for efficiency by freezing the vision tower and training only the decoder, enabling easy transfer to other low-resource languages

Data Construction and Preprocessing

The training corpus is constructed by translating and verifying EN–BN caption pairs using LaBSE [6] and rendering paired synthetic images via bilingual prompts (“A photo of: EN. In Bengali: BN”) using Kandinsky 2.1 [27] This process involves a three-step pipeline: Translation → Verification → Synthesis The semantic verification step computes crosslingual sentence embeddings with LaBSE and measures cosine similarity si = cos φ(y en i), φ(y bn i) A pair is accepted iff si ≥ τ with τ = 0.55; otherwise it is discarded For each verified pair (y en i, y bn i) we synthesize an image x˜i using Kandinsky-2.1 under a bilingual prompt: A photo of: y en i. In Bengali: y bn i Because x˜ is conditioned on the same y, the label remains consistent while styles/layouts vary, regularizing caption grounding

The Tri-Loss Objective

The training optimizes a joint objective that couples caption learning on real images with caption-conditioned alignment between real–synthetic pairs: L = LCE(x, y) + λPAL LPAL(x, x, y ˜) + αLInfoNCE + β LOT The three components work synergistically to improve grounding under scarce Bengali supervision

1. Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, where PAL reuses decoder crossattention to form text-conditioned weights over patches and align pooled real and synthetic descriptors with a cosine loss

2. Information Noise-Contrastive Estimation (InfoNCE) enforces global discrimination between real and synthetic pooled features

3. Sinkhorn-based Optimal Transport (OT) ensures balanced fine-grained patch correspondence by enforcing balanced patch-wise matching under the same PAL attention

Key Training Mechanics

The training protocol involves computing LCE(x, y) on real images, and when a matched synthetic x˜ is present, applying LPAL, LInfoNCE, and LOT using the same caption y Gradients update only the bridge and decoder; the vision encoder remains frozen The decoder consumes projected visual tokens via cross-attention and autoregressively predicts Bengali tokens under teacher forcing The cross-attention maps from the last few decoder layers provide token-level saliency over patches, which are reused to form weights for the PatchAlignment Loss

Results and Insights

The combined PAL+InfoNCE+OT objective outperforms a CE-only baseline on MSCOCO-1k and Flickr30k-1k PAL produces the largest jump in performance compared to CE+InfoNCE+OT, with BLEU-4 rising from 7.52 to 10.19 and METEOR from 26.22 to 27.29 PAL anchors alignment to caption-relevant regions, while InfoNCE and OT serve as light regularizers that refine it The embedding view shows that real and synthetic patch descriptors move closer in the regions the decoder actually attends to, with centroid distance shrinking from 0.361 to 0.213 This indicates that PAL teaches the model where to align

Limitations

The study is compute-aware (single Kaggle P100) and generated 110k synthetic images but trained on 80k, and kept the vision tower frozen Limitations include data scale/diversity, model capacity, attention as lens constraints, and MT reliance & synthetic bias The authors note that PAL relies on decoder cross-attention; when attention is diffuse (counting/relations), alignment may under- or mis-constrain patches Semantic reasoning could be enhanced by adding lightweight LLM critics for reranking or enforcing caption→QA→caption consistency The framework can serve as a foundation for several future directions, such as incorporating region cues (e.g., SAM masks) or word–patch optimal transport

Conclusion

This synergy improves grounding and boosts BLEU-n, METEOR, and BERTScore on MSCOCO1k and Flickr30k-1k, showing that targeted alignment plus lightweight regularization yields accurate and generalizable captions while remaining compute-light and transferable to other low-resource languages This framework can serve as a foundation for several future directions, such as incorporating region cues (e.g., SAM masks) or word–patch optimal transport The synergy improves grounding and boosts BLEU-n, METEOR, and BERTScore on MSCOCO1k and Flickr30k-1k, showing that targeted alignment plus lightweight regularization yields accurate and generalizable captions while remaining compute-light and transferable to other low-resource languages This framework can serve as a foundation for several future directions, such as incorporating region cues (e.g., SAM masks) or word–patch optimal transport This synergy improves grounding and boosts BLEU-n, METEOR, and BERTScore on MSCOCO1k and Flickr30k-1k, showing that targeted alignment plus lightweight regularization yields accurate and generalizable captions while remaining compute-light and transferable to other low-resource languages This framework can serve as a foundation for several future directions, such as incorporating region cues (e.g., SAM masks) or word–patch optimal transport The synergy improves grounding and boosts BLEU-n, METEOR, and BERTScore on MSCOCO1k and Flickr30k-1k, showing that targeted alignment plus lightweight regularization yields accurate and generalizable captions while remaining compute-light and transferable to other low-resource languages This framework can serve as a foundation for several future directions, such as incorporating region cues (e.g., SAM masks) or word–patch optimal transport The synergy improves grounding and boosts BLEU-n, METEOR, and BERTScore on MSCOCO1k and Flickr30k-1k, showing that targeted alignment plus lightweight regularization yields accurate and generalizable captions while remaining compute-light and transferable to other low-resource languages <ref:2509.

1. The tri-loss objective (PAL+InfoNCE+OT) creates a mutually reinforcing cycle that neither achieves alone, which enables cross-modal grounding directly on Bengali tokens. This allows the improved system to generate captions that are both fluent (via CE loss), contextually accurate for the image (via PAL), and robust against background noise or spurious matches (via InfoNCE and OT).

2. The use of a frozen MaxViT backbone combined with only fine-tuning the decoder enables efficient training on a single GPU and easy transfer to other low-resource languages. This allows for rapid adaptation of the Bengali captioner to new, unseen low-resource languages with minimal task-specific fine-tuning, preserving strong ImageNetrained prior.

3. The pipeline constructs a Bengali captioning corpus and synthesis pipeline by verifying EN–BN caption pairs and generating large-scale bilingual-prompted synthetic images to form real–synthetic training triplets, which addresses the scarcity of paired data. This allows the system to effectively learn from abundant, albeit synthetically generated, data without requiring scarce real Bengali supervision for every caption.

4. Patch-Alignment Loss (PAL) reuses decoder crossattention to form text-conditioned weights over patches and align pooled real and synthetic descriptors, which steers learning to caption-relevant regions. This means the system focuses its alignment efforts precisely where the Bengali text indicates important visual details, improving grounding compared to methods that use global alignment.

5. Entropic Optimal Transport (OT) enforces balanced patch-wise matching under the same PAL attention and pushes alignment down to patch-to-patch transport. This capability allows the system to capture multi-instance or small objects that a single pooled vector may blur, leading to finer, more precise localization of caption elements.

6. The framework provides consistent gains in BLEU, METEOR, and BERTScore under limited Bengali supervision on MSCOCO-1k and Flickr30k-1k. This demonstrates the improved system's ability to achieve stronger caption faithfulness even when supervision is scarce, making it highly reliable for real-world low-resource applications.