Understand Before Detect: Vision--Language Learning for Omni-Domain Infrared Small Target Detection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Understand Before Detect".
Jane: Here's the gist of the paper. Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics.
Tom: The paper introduces a new paradigm called “understand before detect”, which formulates omnidomain IRST detection as an understanding-driven process where holistic infrared target understanding precedes precise detection.
Jane: Building on this paradigm, the authors propose JinSight, which first develops holistic IRST understanding through language supervision and then transfers the learned cross-domain representations to precise small-target detection.
Tom: The Core Problem Addressed Existing deep learning methods follow a task-specific supervised learning paradigm that simplifies the full-scene infrared observations to sparse target supervision, discarding the semantics that remain invariant across heterogeneous domains.
Paper summary: Tom: Basically, the paper is tackling this issue of domain shifts in infrared surveillance where things look totally different depending on whether you're on a ground or airborne platform. Jane The authors say that current deep learning methods often follow a task-specific approach, which means they throw away the scene content and target context when they only focus on sparse target supervision. Tom That leads to three specific problems: first, domain heterogeneity collapse where different background structures get lumped together, second, scene information loss because you compress the whole picture into just binary labels, and third, target supervision scarcity because those tiny targets give very weak signals with binary masks.
Lu: And language supervision is presented as a natural way to recover the semantics that gets lost when you only have those binary masks. Tom Exactly. They point out that using simple vision-language models like CLIP favors isolated concepts, which isn't good for understanding how a few-pixel target relates to its whole surrounding scene. Jane So, they move away from that by proposing this new vision-language paradigm where you first instruction tune a generative visual language model on these understanding tasks and then transfer those learned visual representations for dense prediction.
Meng: From an engineering standpoint, the focus on learning domain-invariant semantics across all these different infrared domains is ambitious. Tom It really is. The paper proposes JinSight, which does this in two stages: Stage one uses instruction tuning to learn IRST-aware representations across those heterogeneous domains, and Stage two introduces Latent Semantic Interaction to efficiently swap the global language-aligned semantics with the fine-grained spatial features for detection.
Lalam: I've reviewed the literature on vision models, and this focus on compositional supervision through autoregressive objectives in Stage one is a clever way to capture those scene-level cues beyond just binary classification. Jane So, they are using these instruction tasks—like detailed captioning or physical visual question answering—to build a representation that is aware of the infrared context before it ever tries to do the actual detection task.
Conclusion: Tom: The title "Understand Before Detect" really captures the core message of this work from Haoyang Yuan and his team. Jane It means the method shifts the focus away from just pixel-wise prediction and towards a holistic infrared understanding that happens first, before any actual detection is even attempted. Tom So, for someone who only listens to the show, what this means is that instead of building a detector trained on one specific type of infrared image you'll get an AI that learns the general rules of what makes up an infrared scene across many different platforms and wavebands.
Lu: It’s about making the model learn what a target *is* in context, not just where it *is*. Jane And they show that this approach helps with generalization because the backbone learns these domain-invariant semantics before it tackles detection. Tom They found that when you use both scene-oriented and target-centric instructions, like the OmniIRST-VL dataset they built—which has over thirty-nine thousand annotations across six different tasks—the model gets the best performance <ref:2608.07015#pg1>.
Meng: The results are compelling because they show consistent transfer across three different datasets: NUDT-SIRST, NUAASIRST, and IRSTD-1K <ref:2608.07015#pg1>. Jane That consistency under domain shifts is what really matters for real-world surveillance where you don't always know what kind of image you're looking at. Tom The main contribution is this two-stage JinSight framework where the instruction tuning builds the understanding, and then the latent semantic interaction efficiently transfers that global knowledge to a precise small target detector.
Lalam: From my perspective, this paper shows how grounding infrared representations in language semantics allows a single model to handle massive heterogeneity across different observation platforms and spectral bands. Jane So, it suggests that for difficult problems like omni-domain detection, using language to structure the understanding phase is a very strong way to improve performance over older methods.
Tom: So, looking ahead, this work sets up a new way of thinking about IRST detection—moving from a simple binary prediction task to an instruction-driven understanding process that generalizes across domains.
Haoyang Yuan, Boyang Li, Yingqian Wang, Yimian Dai, Nuo Chen, Xinfei Huang, Shuqi Yi, Zaiping Lin, Weidong Sheng, Wei An
National University of Defense Technology Changsha China College of Computer Science Nankai University Peking University
cs.CV
Submitted: 2026-08-07
Updated: 2026-11-02
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics.
Key concepts
- Omni-domain IRST Detection
- This refers to detecting small targets using infrared cameras across many different imaging platforms (like airborne or ground-based) and spectral bands. The challenge is that the scenes and targets look very different depending on the platform, making it hard for a single model to work everywhere.
- Understand Before Detect Paradigm
- Instead of immediately trying to find a target with a simple detection task, this approach focuses first on building a deep understanding of the entire infrared scene and its targets using language. This holistic understanding is learned before the final precise detection step.
- JinSight Framework
- JinSight is a two-stage vision-language system. Stage one uses instruction tuning on a large dataset to learn domain-invariant representations (understanding). Stage two uses Latent Semantic Interaction (LSI) to efficiently combine these learned global semantics with fine spatial details for accurate detection.
- OmniIRST-VL Dataset
- This is the first large vision-language dataset specifically designed for omni-domain IRST. It includes over 39,000 image-instruction annotations covering six different tasks, allowing the model to learn scene understanding and target reasoning across diverse infrared domains.
Terminology
Summary
Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics. p. 0 The paper introduces a new paradigm called “understand before detect”, which formulates omnidomain IRST detection as an understanding-driven process where holistic infrared target understanding precedes precise detection. p. 0 Building on this paradigm, the authors propose JinSight, which first develops holistic IRST understanding through language supervision and then transfers the learned cross-domain representations to precise small-target detection. p. 0
The Core Problem Addressed
Existing deep learning methods follow a task-specific supervised learning paradigm that simplifies the full-scene infrared observations to sparse target supervision, discarding the semantics that remain invariant across heterogeneous domains. This approach suffers substantially under domain shifts because it discards scene content and target–context relations. p. 0 Specifically, binary abstractions obscure the inherent information richness of omni-domain infrared data by collapsing diverse background structures into a single category, leading to domain heterogeneity collapse and scene information loss. p. 1 Furthermore, few-pixel targets yield extremely sparse positive learning signals due to the nature of binary masks. p. 1
The Proposed Framework
JinSight is a two-stage vision–language framework centered on a shared ViT backbone. Stage I involves instruction tuning a generative VLM on OmniIRST-VL to learn IRST-aware representations across heterogeneous domains. p. 1 This stage uses an autoregressive objective to provide compositional supervision, preserving rich scene-level information beyond binary classification and capturing domain-invariant semantics. p. 1 Stage II introduces Latent Semantic Interaction (LSI), which exchanges language-aligned global semantics with fine-grained spatial features in a compact lowrank space for precise small target detection. p. 0
Key Components and Data Construction
The framework relies on the OmniIRST-VL dataset, the first large-scale vision–language dataset for omni-domain IRST detection, comprising over 39k image–instruction–response annotations. This dataset covers six complementary instruction tasks spanning scene-level understanding and target-centric reasoning, including Detailed Captioning, Physical VQA, Visual Grounding (Image and Region Level), and Counting (Image and Region Level). p. 3 The data annotation process involves metadata extraction, controlled generation via language augmentation, automatic quality filtering by infrared experts, and disagreement adjudication to ensure semantic reliability. p. 2
Performance and Contributions
Extensive experiments demonstrate that JinSight consistently outperforms existing state-of-the-art approaches. The method achieves over 14% IoU improvement on the WideIRSTD benchmark, validating its effectiveness for omni-domain IRST detection. p. 0 The main contributions include proposing a novel paradigm that reformulates detection from pixel-wise binary prediction into holistic infrared understanding followed by detection. p. 1 JinSight is proposed as a two-stage framework where instruction tuning learns IRST-aware representations and LSI efficiently transfers global semantics to fine-grained spatial features. p. 1 The authors also show that using both scene-oriented and target-centric instructions provides the best performance, demonstrating the complementary benefits of holistic understanding prior to detection. p. 6
Ablation and Efficiency Analysis
An ablation study confirms the contribution of both components, showing that adding IRST-oriented visual instruction tuning yields relative improvements in IoU and Pd. Similarly, LSI alone produces gains in IoU and Pd by exchanging global semantics with fine-grained spatial features. p. 6 The optimal performance is achieved when both components are used together, increasing IoU by 5. p. 645% and Pd by 4.94% over the baseline. p. 6 Furthermore, the study investigates the effect of the Latent Dimension of Latent Semantic Interaction (LSI) on efficiency and detection performance, finding that a latent dimension of r = 256 yields optimal performance while retaining lower latency and memory consumption. p. 6 The best decoder for dense prediction across representative methods is UperNet, achieving the highest IoU and lowest Fa. p. 5
Conclusion on Generalization
The results on cross-dataset generalization, conducted using leave-one-dataset-out evaluation on NUDT-SIRST, NUAASIRST, and IRSTD-1K, demonstrate that JinSight achieves the highest Pd and IoU under all three shifts. This consistent transfer across heterogeneous domains is attributed to the “understand before detect” paradigm enabling the backbone to learn domain-invariant scene and target semantics before detection. p. 6 The method successfully enables a single model to generalize across heterogeneous infrared domains by grounding infrared representations in language semantics. p. 0 The final conclusion is that JinSight consistently outperforms state-of-the-art methods, achieving over 14% improvement in IoU compared with existing methods. p. 6 >
--- Page 1 ---
Understand Before Detect: Vision–Language Learning for Omni-Domain Infrared Small Target Detection
Haoyang Yuan, Boyang Li, Yingqian Wang, Yimian Dai, Nuo Chen, Xinfei Huang, Shuqi Yi, Zaiping Lin, Weidong Sheng, Wei An
Abstract
Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics. Previous deep learning-based methods have been developed for visual-only paradigms and achieved promising performance on domain-specific tasks. p. 0 However, existing methods follow the task-specific supervised learning paradigm. p. 0 This paradigm simplifies the full-scene infrared observations to sparse target supervision, discarding the semantics that remain invariant across heterogeneous domains. p. 0 Consequently, detection performance suffers substantially under domain shifts. p. 0 To handle this issue, we introduce “understand before detect”, a paradigm that formulates omnidomain IRST detection as an understanding-driven process, where holistic infrared target understanding precedes precise detection. p. 0 Building on this paradigm, we propose JinSight, which first develops holistic IRST understanding through language supervision and then transfers the learned cross-domain representations to precise small-target detection. p. 0 By grounding infrared representations in language semantics, JinSight enables a single model to generalize across heterogeneous infrared domains. p. 0 We then introduce Latent Semantic Interaction (LSI), which exchanges language-aligned global semantics with fine-grained spatial features in a compact lowrank space. p. 0 To address the lack of multimodal omni-domain IRST benchmarks, we build OmniIRST-VL, the first largescale, highly diverse vision–language dataset for omni-domain IRST detection. p. 0 It comprises over 39k annotations across six complementary instruction tasks covering both scene-level understanding and target-centric reasoning. p. 0 Extensive experiments demonstrate that JinSight consistently outperforms existing state-of-the-art approaches and achieves over 14% IoU improvement on the WideIRSTD benchmark, validating its effectiveness for omni-domain IRST detection. p. 0 p. 0
--- Page 2 ---
Following the “understand before detect” paradigm, JinSight first learns a unified IRST-aware backbone from omnidomain instructions through a generative VLM (Chen et al. 2024b) and then adapts it to detection and understanding tasks. p. 0 Different from task-specific supervised learning, this new paradigm employs an autoregressive objective to provide compositional supervision. p. 1 As a result, it not only preserves rich scene-level information beyond binary classification but also captures inherent domain-invariant semantics across heterogeneous infrared domains. p. 1 p. 1
--- Page 3 ---
borne, airborne, and ground-based platforms cover different spectral bands (e.g., NIR, SWIR, LWIR) and observe scenes ranging from open skies to sea surfaces. p. 0 This motivates omni-domain IRST (Omni-IRST) detection (Shi et al. p. 0 2025), which seeks to detect targets across heterogeneous infrared domains with a unified detector. p. 0 However, variations across these domains induce dramatic shifts in scene context, background characteristics, and target appearance. p. 0 Such shifts, compounded by the weak responses and sparse spatial footprints of infrared small targets pose substantial challenges to learning shared representations that preserve target–background discriminability. p. 0
--- Page 4 ---
Existing IRST methods have mainly improved visual perception within specific domains. CNN-based models exploit local inductive biases, multi-scale feature fusion, and dense cross-layer connections to preserve small target responses. p. 0 Transformer-based or hybrid models introduce long-range contextual modeling to suppress background clutter. p. 0 However, these methods still follow the conventional target detection paradigm, casting IRST detection as pixel-wise binary target–background prediction. p. 1 This formulation encounters three critical limitations in OmniIRST detection: (1) Domain heterogeneity collapse: diverse background structures are indiscriminately collapsed into a single background category, leaving their variations unexplored. p. 1 (2) Scene information loss: rich full-scene observations are compressed into coarse binary labels, discarding scene content and target–context relations. p. 1 (3) Target supervision scarcity: binary masks provide only pixel-wise supervision, and few-pixel targets therefore yield extremely sparse positive learning signals. p. 1
--- Page 5 ---
Language supervision provides a natural means to recover the semantics omitted by binary masks. Accordingly, several methods have explored vision–language models for IRST detection by leveraging CLIP-based representations. p. 1 However, CLIP’s bag-ofwords behavior favors isolated concepts over compositional relations, limiting fine-grained understanding of how few-pixel targets relate to their surrounding scenes. p. 1 To address these challenges, the authors propose a new vision–language paradigm for Omni-IRST detection, termed “understand before detect”. p. 1 Specifically, it first instruction-tunes a generative vision–language model (VLM) on IRST understanding tasks and then transfers the learned visual representations to downstream dense prediction tasks. p. 1
--- Page 6 ---
Different from task-specific supervised learning, this new paradigm employs an autoregressive objective to provide compositional supervision. As a result, it not only preserves rich scene-level information beyond binary classification but also captures inherent domain-invariant semantics across heterogeneous infrared domains.
--- Page 7 ---
To address these challenges, we propose JinSight (Joint Vision–Language Learning for Infrared Sight), which establishes a holistic understanding of infrared scenes and targets, then transfers the acquired semantics to robust small target detection. In the first stage, visual instruction tuning employs language responses to learn IRST-aware representations across heterogeneous domains, producing instruction-tuned semantics for dense prediction. p. 1 In the second stage, we introduce low-rank Latent Semantic Interaction (LSI), which efficiently exchanges instruction-tuned global semantics with multi-scale spatial features in a compact latent space for precise small target detection. p. 1
--- Page 8 ---
A low-rank FFN subsequently refines the latent semantic tokens. Following ViT-Adapter, the resulting features are organized into a multi-scale pyramid and passed to a task-specific head. p. 5 The authors construct OmniIRST-VL, the first large-scale OmniIRST vision–language dataset for omni-domain IRST detection. p. 1 Extensive experiments demonstrate that JinSight consistently outperforms existing state-of-the-art approaches and achieves over 14% IoU improvement on the WideIRSTD benchmark, validating its effectiveness for omni-domain IRST detection.
--- Page 9 ---
Figure 1: Comparison between conventional domain-specific IRST learning (top) and JinSight (bottom) in terms of (a) data construction, (b) learning paradigm, and (c) task scope. Following the “understand before detect” paradigm, JinSight first learns a unified IRST-aware backbone from omnidomain instructions through a generative VLM and then adapts it to detection and understanding tasks. p. 0
--- Page 10 ---
Table 1: Comparison of different datasets across supported tasks, imaging systems, and wavebands. OmniIRST-VL is the only dataset spanning all three observation platforms and spectral bands and contains the largest number of targets no larger than 10 pixels (7,421). p. 2
--- Page 11 ---
Figure 3: JinSight architecture. (a) Inputs for Stage I instruction tuning and Stage II dense prediction. p. 4 p. 4 p. 4 The “understand before detect” framework: Stage I learns an IRST-aware ViT via instruction supervision; Stage II uses LSI to fuse global semantics with fine-grained spatial features in a compact latent space. p. 4 p. 4
--- Page 12 ---
JinSight is a two-stage framework centered on a shared ViT backbone. In Stage I, we fine-tune a generative VLM on OmniIRST-VL with IRST-specific instructions to obtain an IRST-aware visual encoder that captures both scene context and target–centric cues. p. 4 p. 4 p. 4 In Stage II, we transfer its adapted ViT to dense prediction, remove the language projector and decoder, and introduce LSI modules to exchange global semantic representations with multi-scale spatial features. p. 4 p. 4 p. 4
--- Page 13 ---
We propose JinSight, a two-stage vision–language framework: visual instruction tuning first learns IRST-aware representations across heterogeneous domains, and LSI then exchanges the instruction-tuned global semantics with fine-grained multi-scale features in a compact lowrank space. OmniIRST-VL provides complementary scene-oriented and target-centric instructions across diverse infrared domains, enabling the backbone to learn domain-invariant scene and target semantics before detection. p. 5 p. 5 >
--- Page 14 ---
Figure 3: JinSight architecture. (a) Inputs for Stage I instruction tuning and Stage II dense prediction. The “understand before detect” framework: Stage I learns an IRST-aware ViT via instruction supervision; Stage II uses LSI to fuse global semantics with fine-grained spatial features in a compact latent space. >
--- Page 15 ---
JinSight is a two-stage framework centered on a shared ViT backbone. In Stage I, we fine-tune a generative VLM on OmniIRST-VL with IRST-specific instructions to obtain an IRST-aware visual encoder that captures both scene context and target–centric cues. In Stage II, we transfer its adapted ViT to dense prediction, remove the language projector and decoder, and introduce LSI modules to exchange global semantic representations with multi-scale spatial features. >
--- Page 16 ---
JinSight is a two-stage framework centered on a shared ViT backbone. In Stage I, we fine-tune a generative VLM on OmniIRST-VL with IRST-specific instructions to obtain an IRST-aware visual encoder that captures both scene context and target–centric cues. In Stage II, we transfer its adapted ViT to dense prediction, remove the language projector and decoder, and introduce LSI modules to exchange global semantic representations with multi-scale spatial features. >
--- Page 17 ---
We propose JinSight, a two-stage vision–language framework: visual instruction tuning first learns IRST-aware representations across heterogeneous domains, and LSI then exchanges the instruction-tuned global semantics with fine-grained multi-scale features in a compact lowrank space. OmniIRST-VL provides complementary scene-oriented and target-centric instructions across diverse infrared domains, enabling the backbone to learn domain-invariant scene and target semantics before detection. >
--- Page 18 ---
Figure 3: JinSight architecture. (a) Inputs for Stage I instruction tuning and Stage II dense prediction. The “understand before detect” framework: Stage I learns an IRST-aware ViT via instruction supervision; Stage II uses LSI to fuse global semantics with fine-grained spatial features in a compac… p. 4
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Understand before detect paradigm allows JinSight to first learn a unified IRST-aware backbone from omnidomain instructions through a generative VLM, which captures both scene context and target-centric cues. This enables a single model to generalize across heterogeneous infrared domains rather than following the task-specific supervised learning paradigm.
-
JinSight's two-stage framework uses instruction tuning to learn
IRST-aware representations across heterogeneous domains,
which preserves full-scene context, followed by Latent Semantic Interaction (LSI) toexchange instruction-tuned global semantics with fine-grained multi-scale spatial features in a compact lowrank space.
This allows precise small target detection by fusing holistic understanding with spatial details. -
The OmniIRST-VL dataset, containing over 39k image-instruction-response annotations across six complementary tasks, provides supervision from
holistic background understanding to fine-grained target reasoning,
addressing the limitations of binary masks by offering scene-oriented and target-centric instructions. This enriches supervision for infrared small targets and learns transferable semantics across domains. -
The LSI module efficiently transfers global IRST semantics with spatial features, improving detection accuracy by 5.45% over the baseline while reducing false alarm rate relative to single-component settings, indicating that
instruction tuning provides IRST-aware semantic knowledge, while LSI effectively transfers these semantics into multiscale spatial features.
-
The model can achieve superior cross-domain generalization when tested on domain shifts like synthetic multi-scene imagery or real variable resolution observations, demonstrating
consistent transfer across heterogeneous domains
and outperforming existing state-of-the-art methods.
Sources
- Qwen3-VL Technical Report
- Vision Transformer Adapter for Dense Predictions
- The First Competition on Resource-Limited Infrared Small Target Detection Challenge: Methods and Results
- Probing Deep into Temporal Profile Makes the Infrared Small Target Detector Much Better
- Decoupled Weight Decay Regularization
- Leveraging Language Prior for Infrared Small Target Detection
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs