Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation

arXiv:2506.03583 · cs.CV · Submitted 2025-06-04 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation".

Tom: The gist The proposed MRSNet achieves state-of-the-art performance across multiple evaluation metrics on the NWPU-Refer dataset by integrating intra-scale and hierarchical feature interaction modules.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: We saw that the paper "Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation" introduces MRSNet, focusing on intra-scale and cross-scale feature interactions.

Jane: They frame this as a solution to the challenges existing RRSIS datasets have with resolution, scene diversity, and category coverage.

Tom: They propose a new dataset called NWPU—Refer, which is described as the largest and most diverse RRSIS dataset to date.

Lu: That dataset comprises fifteen thousand three high-resolution images with resolutions ranging from one thousand twenty-four to two thousand forty-eight pixels.

Meng: And it covers scenes from more than thirty countries across five continents, which is a big jump in diversity for this kind of data.

Tom: On top of that, the dataset has forty-nine thousand seven hundred forty-five annotated targets covering single-object, multi-object, and no-target scenarios.

Jane: The core idea they present is how MRSNet integrates two key modules: the IFIM and the HFIM to ensure robust feature extraction.

Lu: Specifically, each encoder stage uses an Intra-scale Feature Interaction Module, or IFIM to capture fine details within that specific scale.

Meng: And then there's the Hierarchical Feature Integration Module, or HFIM, which is designed to aggregate information across different scales while keeping the spatial structure intact.

Lalam: This combination aims to ensure robust feature extraction and aggregation for referring remote sensing segmentation tasks.

Conclusion: Tom: We've been looking at "Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation" and MRSNet.

Jane: This paper really focuses on how to make those visual models understand natural language instructions when they are looking at remote sensing pictures.

Tom: Right, the authors introduce MRSNet, which centers on combining intra-scale and cross-scale feature interactions to handle that complexity.

Jane: They did this using the NWPU—Refer dataset, which is huge for remote sensing because it has fifteen thousand three high-resolution images from over thirty countries.

Lu: What’s really interesting is how they specifically designed the IFIM to manage those fine details inside each encoder stage.

Meng: I was wondering about that integration part again, how they keep those different scales talking without messing up the spatial arrangement of things.

Lalam: The HFIM handles that by making sure information from different scales fuses together while preserving the spatial structure.

Tom: They are essentially ensuring the model sees those small details and understands how they fit into the bigger picture contextually.

Jane: That means when you give it a command, like "find all trucks on this highway," it understands that relationship across different zoom levels in the image.

Lu: The results show MRSNet gets state-of-the-art performance on metrics like IoU and precision at thresholds, especially with higher IoU requirements.

Meng: From an engineering standpoint, getting that level of performance on such diverse remote sensing data is pretty impressive; it shows the model handles real imagery noise well.

Tom: And what this means is we’re moving toward AI systems that can interpret those complex visual commands in satellite data with much greater accuracy than before.

Jane: This opens up possibilities for automated monitoring or identifying specific patterns across vast geographic areas just from simple text descriptions.

Lalam: For culture, it suggests future language models will be able to understand visual context and relationships with a lot more nuance than we have now.

Member, IEEE

cs.CV

Submitted: 2025-06-04

Updated: 2026-10-09

DOI: 10.1109/TGRS.2026.3704617

Code: https://github.com/CVer-Yang/NWPU-Refer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The proposed MRSNet achieves state-of-the-art performance across multiple evaluation metrics on the NWPU-Refer dataset by integrating intra-scale and hierarchical feature interaction modules.

Key concepts

Referring Remote Sensing Image Segmentation (RRSIS): This is a difficult computer vision task where a user describes an object using natural language, and the system must precisely locate and segment that object within a remote sensing image. It requires linking textual descriptions to specific pixel-level regions in aerial or satellite imagery.
Intra-scale Feature Interaction Module (IFIM): This module refines visual features at each stage of the network by capturing multi-scale information. It uses techniques like Fourier Transforms and graph convolutions to model how different scales and spatial relationships interact, leading to more discriminative features for target identification.
Hierarchical Feature Integration Module (HFIM): This module ensures that feature information is well-integrated across different scales. It combines local features, global features from self-attention, and frequency domain information through operations like PixelShuffle to create a unified feature map that preserves both fine details and broad context.

Terminology

Generalization — 1× in this paper · explained in 13 episodes

Summary

The gist The proposed MRSNet achieves state-of-the-art performance across multiple evaluation metrics on the NWPU-Refer dataset by integrating intra-scale and hierarchical feature interaction modules.

Dataset and Problem Context

Referring Remote Sensing Image Segmentation (RRSIS) is a complex task that integrates computer vision and natural language processing, which is distinguished by its interactivity and flexibility, enabling models to interpret user intent and generate personalized segmentation outcomes<ref:2506.03583#pg2> Existing datasets for RRSIS suffer from critical limitations in resolution, scene diversity, and category coverage, which hinders the generalization and real-world applicability of refer segmentation models<ref:2506.03583#pg2> To facilitate the development of this field, we introduce NWPU—Refer, the largest and most diverse RRSIS dataset to date, comprising 15,003 high-resolution images (1024-2048px) spanning 30+ countries with 49,745 annotated targets supporting single-object, multi-object, and non-object segmentation scenarios<ref:2506.03583#pg2> The NWPU-Refer dataset is a comprehensive resource for remote sensing image segmentation, featuring a diverse collection of images with resolutions ranging from 0.12m to 0.5m<ref:2506.03583#pg5> It includes 49,745 manually generated annotations, covering no-target and multi-target scenarios, ensuring precision and reliability<ref:2506.03583#pg5>.

Model Architecture

The architecture of our proposed MRSNet is depicted in Fig. 4, comprising three core components: the Intra-scale Feature Interaction Module (IFIM), the Hierarchical Feature Integration Module (HFIM), and the decoder<ref:2506.03583#pg5>. We utilize the Swin-Transformer [43] as the backbone for visual feature extraction and the BERT network [44] as the backbone for linguistic feature extraction<ref:2506.03583#pg5>. At each encoder stage, an intra-scale feature interaction module is employed to capture features of targets across varying scales and model their interrelationships<ref:2506.03583#pg5>. Moreover, a hierarchical feature interaction module is implemented to facilitate efficient cross-scale feature aggregation while preserving spatial integrity<ref:2506.03583#pg5>.

Intra-Scale Feature Interaction Module (IFIM)

The IFIM is designed to perform three key functions: capturing multi-scale visual features, modeling contextual relationships between targets, and achieving visual-linguistic alignment<ref:2506.03583#pg5>. This module enhances feature completeness and discriminability, significantly advancing complex scene referring segmentation capabilities<ref:2506.03583#pg5>.

a) Pyramidal Spatial-Spectral Refinement Submodule: This submodule refines the visual feature map in both spatial and frequency domains across multiple scales<ref:2506.03583#pg5>. It involves applying a Fourier Fast Transform to extract the frequency domain magnitude M and phase P, which are then concatenated and processed through a convolution layer followed by a softmax activation to generate frequency weights Wf<ref:2506.03583#pg5>. The spatial features Xs and frequency domain features Xf are then fused through concatenation followed by a convolution operation to produce the refined feature map<ref:2506.03583#pg5>.

b) Context-Aware Spatial Relation Modeling Submodule: This submodule is designed to extract interactions between targets<ref:2506.03583#pg5>. It constructs a spatial adjacency matrix A to represent the connectivity between each pixel and its surrounding neighbors<ref:2506.03583#pg5>. Contextual features Ccontext are processed through a graph convolution network to capture pixel-wise relationships while maintaining spatial and semantic consistency<ref:2506.03583#pg5>.

c) Cross Model Align Submodule: This submodule is designed to selectively align visual features with textual features<ref:2506.03583#pg5>. It computes the similarity between visual queries Q and linguistic keys K, resulting in an attention map S, which is then used to weight the linguistic values V, producing the enhanced visual features VL that are aligned with the linguistic features<ref:2506.03583#pg5>.

Hierarchical Feature Integration Module (HFIM)

The HFIM ensures the integrity of feature information by redistributing channel information into spatial dimensions through a PixelShuffle operation<ref:2506.03583#pg5>. The module leverages depthwise separable convolutions, spatial self-attention, and frequency self-attention to achieve comprehensive multi-scale feature fusion and efficient encoding across spatial and frequency domains<ref:2506.03583#pg5>.

a) Spatial domain feature Integration: Local features Flocal are captured through depthwise separable convolutions, batch normalization (BN), and ReLU activation to efficiently capture local neighborhood information<ref:2506.03583#pg5>. Global features Fglobal are derived using a multi-head self-attention mechanism where attention scores measure the similarity between queries and keys<ref:2506.03583#pg5>.

b) Frequency Domain Feature Integration: Normalized similarities between the query Qf and key Kf are computed in the frequency domain, and the value Vf are weighted to generate frequency-domain features Ffreq<ref:2506.03583#pg5>. These features are then transformed back to the spatial domain through an inverse Fourier transform, yielding Ffreq<ref:2506.03583#pg5>.

c) Feature Fusion: The Flocal, Flocal,Ffreq are concatenated along the channel dimension to form the unified feature Ffuse<ref:2506.03583#pg5>. A 1 × 1 convolution is then applied to reduce the channel dimensionality, resulting in the final fused feature<ref:2506.03583#pg5>.

Experimental Results

Extensive experiments on the proposed NWPU-Refer dataset demonstrate that MRSNet achieves state-of-the-art performance across multiple evaluation metrics, validating its effectiveness<ref:2506.03583#pg2>. MRSNet consistently outperforms these baselines, achieving state-of-the-art results in precision at thresholds (P@0.7, P@0.8, P@0.9), oIoU, and mIoU<ref:2506.03583#pg5>. The significant performance gains of MRSNet can be attributed to its architectural innovations<ref:2506.03583#pg5>.

Ablation Study

The introduction of PSR brings about significant improvements in precision, especially at higher IoU thresholds, while the incorporation of CSR further refines the target relationships<ref:2506.03583#pg5>. The combined effect of both modules demonstrates a synergistic enhancement, yielding the highest performance across all evaluated metrics<ref:2506.03583#pg5>.

Conclusion

In this paper, we introduce a large-scale RRSIS dataset, named NWPU-Refer, which offers significant advantages in data scale and scene diversity compared to existing RRSIS datasets<ref:2506.03583#pg2>. We also present the Multi-scale Referring Segmentation Network (MRSNet), a sophisticated model designed to address the challenges of RRSIS<ref:2506.03583#pg5>. MRSNet effectively extracts object features and their interrelationships across different scales through the use of an Intra-scale Feature Interaction Module at each encoder stage<ref:2506.03583#pg5>. Furthermore, a Hierarchical Feature Integration Module is designed to ensure comprehensive integration of global semantic information and local detailed features, enabling the model to handle complex spatial variations and intricate object relationships<ref:2506.03583#pg5>. Extensive experiments on the proposed NWPU-Refer dataset demonstrate that MRSNet achieves state-of-the-art performance, setting a new benchmark for future research in this field<ref:2506.03583#pg2>.

References

[1] Z. Wei, L. Chen, Y. Jin, X. Ma, T. Liu, P. Ling, B. Wang, H. Chen, and J. Zheng, “Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 28 619–28 630

[43] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

[44] J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

[6]

Improvements for AI systems

The introduction of NWPU-Refer dataset allows for a large-scale RRSIS dataset to date comprising 15,003 high-resolution images spanning 30+ countries, enabling models to generalize across urban, rural, forested, desert, and oceanic landscapes. This improves model generalization by training on real-world complexities beyond simplified scenarios.
The Multi-scale Referring Segmentation Network (MRSNet) incorporates the Intra-scale Feature Interaction Module (IFIM) to capture fine-grained details within each encoder stage and the Hierarchical Feature Interaction Module (HFIM) to enable seamless cross-scale feature fusion, preserving spatial integrity while enhancing discriminative power. This allows MRSNet to handle both multi-scale objects and complex spatial relationships simultaneously.
The Cross Model Align Submodule utilizes a mechanism where the attention map S is then used to weight the linguistic values V, producing the enhanced visual features VL that are aligned with the linguistic features, ensuring cross-modal relationships are captured. This enables better alignment between text and image features, improving segmentation accuracy when dealing with complex or ambiguous language descriptions.
The Hierarchical Feature Integration Module (HFIM) ensures the integrity of feature information by redistributing channel information into spatial dimensions through a PixelShuffle operation, which allows the model to leverage fine-grained details while maintaining a global context. This capability is crucial for accurately segmenting objects of varying scales, as demonstrated by its ability to handle both local and global features.