Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning
The research addresses poor performance of vision-language models in low-resource languages like Bengali by creating a compute-aware captioning pipeline. It introduces a novel tri-loss objective combining Patch-Alignment Loss (PAL), InfoNCE, and Sinkhorn Optimal Transport. This synergy aligns real and synthetic visual patches based on decoder attention, significantly improving grounding and achieving state-of-the-art results on MSCOCO and Flickr30k datasets.