GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views

GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views

This paper proposes the GRASP (Granularity-Aware Region Alignment and Semantic Prototype) learning framework to address fine-grained cross-modal understanding in drone views, which is essential for aerial vision-language navigation. The authors identify two dual challenges inherent in drone scenarios: at the macro level, overwhelming background clutter in visual representations leads to Cross-Modal Focus Misalignment, where the model prioritizes global environmental similarities over specific object details; at the micro level, Visual Isomorphism creates ambiguity, where candidates share similar geometric structures yet differ only in subtle attributes.

To address these challenges, the framework introduces two synergistic strategies. First, Region-Focused Alignment (RFA) is designed to promote object-centric cross-modal alignment while suppressing background interference. RFA introduces region-level image-text contrastive learning and a cross-modal ranking loss to recalibrate alignment across both full-image and object-centric levels. Specifically, RFA includes global representation learning via symmetric InfoNCE objective, region-level representation learning using a learnable region query vector that dynamically aggregates visual patch features within the target bounding box, and hard negative mining with margin constraint that enforces a margin-based ranking constraint separating positive pairs from the hardest in-batch negatives.

Second, Semantic Perturbation Enhanced Matching (SPEM) is proposed to tackle visual isomorphism. SPEM actively constructs semantically perturbed hard negatives via Textual Semantic Perturbation (TSP) and Visual Semantic Perturbation (VSP). TSP deterministically substitutes fine-grained attribute words to generate hard textual negatives while preserving the exact sentence context, using a hierarchical dictionary-based substitution with three priority levels: primary attributes (color and orientation words), antonym substitution, and random perturbation. VSP establishes a foreground-purified Semantic Prototype Codebook (SPC) and leverages it to synthesize visual negatives strictly bounded within plausible attribute distributions, effectively averting out-of-distribution noise. The SPC is constructed by clustering attention-selected foreground patches via K-Means, and perturbations are generated by interpolating region embeddings with top-k neighbor prototypes. The codebook is dynamically updated via Exponential Moving Average.

The framework employs cross-modal matching with synthesized negatives, where the fusion encoder receives five types of image-text pairs: positive pairs, in-batch hard image negatives, in-batch hard text negatives, TSP-generated text negatives, and VSP-generated visual negatives. The model is optimized via binary cross-entropy loss, which effectively supervises dense, token-to-patch interactions, explicitly penalizing the model for localized semantic mismatches. A grounding loss is also retained for region-text alignment supervision.

Notably, RFA and SPEM are employed exclusively during the training phase, achieving significant performance gains with absolute zero additional computational overhead during inference. The training uses a two-stage procedure: the first stage performs warm-up training for 1 epoch using all five loss functions without TSP/VSP-generated hard negatives, after which foreground features are extracted to initialize SPC; the second stage introduces SPEM and trains for 2 epochs with TSP and VSP-generated hard negatives.

Experiments on the GeoText-1652 benchmark demonstrate that GRASP achieves competitive performance, improving R@1 by +3.3% for Image Query and +0.9% for Text Query compared to the GeoText-1652 baseline, with mR improvement of +1.99%. Ablation studies confirm the synergy between RFA and SPEM, with the full model achieving the best mR of 38.04%. On the unseen ERA dataset, GRASP achieves mR of 39.47% in zero-shot evaluation, outperforming the best fine-tuned method VCSR by +0.51% and improving over the GeoText-1652 baseline by +1.47% mR. Visualizations of activation maps demonstrate that GRASP transitions from background-dominated global alignment to precise, attribute-aware semantic grounding, and retrieval case studies show GRASP retrieving correct images at Rank 1 where baselines fail.

Based on the paper, here are the specific improvements I can make to AI systems:

1. Implement region-focused contrastive learning with background suppression: I can add a learnable region query vector that dynamically aggregates visual patch features within target bounding boxes, combined with a cross-modal ranking loss that enforces a margin between positive pairs and the hardest in-batch negatives. This improves object-centric alignment in cluttered scenes, enabling the AI to focus on specific objects rather than global environmental similarities.

2. Add hierarchical semantic perturbation for hard negative generation: I can build a dictionary-based substitution system with three priority levels—primary attributes (color, orientation), antonym substitution, and random perturbation—to deterministically generate hard textual negatives that preserve sentence context. This trains the AI to distinguish subtle attribute differences in text, improving fine-grained retrieval and matching.

3. Construct a foreground-purified semantic prototype codebook for visual perturbation: I can cluster attention-selected foreground patches via K-Means to build a codebook, then generate visual negatives by interpolating region embeddings with top-k neighbor prototypes, strictly bounded within plausible attribute distributions. This avoids out-of-distribution noise and enhances the AI's ability to handle visual isomorphism where candidates share similar geometry but differ in subtle attributes.

4. Use binary cross-entropy loss over dense token-to-patch interactions: Instead of only global alignment, I can supervise token-to-patch correspondences explicitly, penalizing localized semantic mismatches. This improves the AI's precision in grounding specific text tokens to specific visual regions, reducing ambiguity in fine-grained cross-modal tasks.

5. Apply a two-stage training procedure with warm-up and dynamic codebook updates: First, train for one epoch with all losses but without synthesized hard negatives, then extract foreground features to initialize the semantic prototype codebook. Second, introduce semantic perturbation and train for two epochs, updating the codebook via exponential moving average. This stabilizes training and ensures the codebook reflects current feature distributions, improving robustness in zero-shot settings.

6. Achieve zero additional inference overhead: Since RFA and SPEM are used only during training, I can deploy the improved model without any extra computational cost at inference time. This allows the AI to run efficiently in real-time applications like drone navigation while retaining the fine-grained understanding learned during training.

The improved AI system can: perform fine-grained cross-modal retrieval and matching in drone views, accurately distinguishing objects with subtle attribute differences (e.g., color, orientation) despite background clutter, achieve higher recall at rank 1 for both image and text queries, generalize better to unseen datasets in zero-shot settings, and operate with no additional latency or compute during deployment.

Fine-grained cross-modal understanding in drone views is essential for aerial vision-language navigation. However, the inherent wide field of view and overhead perspective of drone scenarios impose dual challenges on vision-language understanding. At the macro level, overwhelming background clutter in visual representations leads to Cross-Modal Focus Misalignment, where the model prioritizes global environmental similarities over specific object details. At the micro level, Visual Isomorphism creates ambiguity, where candidates share similar geometric structures yet differ only in subtle attributes. To address these challenges, we propose the Granularity-Aware Region Alignment and Semantic Prototype (GRASP) learning framework, enhancing discriminative capability through two synergistic strategies. Specifically, we introduce Region-Focused Alignment (RFA) to promote object-centric cross-modal alignment while suppressing background interference. Concurrently, to tackle visual isomorphism, we propose Semantic Perturbation Enhanced Matching (SPEM), which leverages a foreground-purified Semantic Prototype Codebook (SPC) to construct semantically perturbed negatives for fine-grained semantic discrimination. Extensive experiments on the GeoText-1652 benchmark and the unseen ERA dataset demonstrate that GRASP achieves competitive performance in drone-view fine-grained image-text retrieval, validating its effectiveness for cross-modal understanding in aerial scenarios. Our code implementation is available at https://github.com/UCAS-JC/GRASP.