CogVis: Must Open-Vocabulary Change Detection Perceive the Scene Anew for Every Query?

The paper addresses Open-Vocabulary Change Detection (OVCD) in remote sensing, which requires detecting bi-temporal changes under arbitrary, text-specified semantic categories. The authors note the core limitation of existing approaches: existing methods often entangle temporal perception, semantic discrimination, and region verification, causing unstable results and redundant computation.

The paper identifies two dominant pipelines in prior work:

• M-C-I (Mask-Compare-Identify): shares category-independent proposals, but downstream recognition inherits their localization errors. Missed candidates cannot be recovered, whereas spurious candidates may still receive plausible labels.

• I-M-C (Identify-Mask-Compare): injects category priors before temporal comparison, enabling category-aware reasoning but repeatedly parses the image pair for every query.

The authors argue that both designs couple temporal evidence, semantic uncertainty, and regional reliability within a fixed cascade, which leads to the central research question posed in the title: must OVCD perceive the scene anew for every query? Their answer is no, and they propose a paradigm that separates query-shared temporal perception from query-specific semantic and regional decisions.

Inspired by human visual change perception, the authors reformulate OVCD. They explain the cognitive analogy: "humans first form a global perception of scene-level changes, then leverage semantic memory to infer plausible categories, and finally verify each candidate region to reject non-semantic disturbances such as shadows, texture variations, seasonal shifts, and misregistration."

Accordingly, they propose CogVis, described as a cognitive memory-guided framework that reformulates OVCD as a perception-memory-verification paradigm. The main contributions claimed are:

1. Reforming OVCD into a perception-memory-verification paradigm that disentangles the decisions coupled within the fixed M-C-I and I-M-C cascades.

2. Instantiating the paradigm as CogVis, where SCP, SMC, and ARF jointly address semantic-temporal-structural entanglement, image-query score shifts, and unreliable region predictions.

3. Validating the factorization on seven benchmarks, showing consistent cross-task gains and reduced multi-query temporal computation.

The framework uses a frozen SAM3 visual encoder and CLIP text embeddings. Given bi-temporal images I1, I2 and an open vocabulary C = c1,..., c K, CogVis predicts a change mask for each query. The three decision scopes are handled by three modules:

SCP extracts query-shared temporal evidence once per image pair. Using the frozen feature pyramid, it constructs a paired descriptor:

X = Catch(F1, F2, F2 − F1, F1 ⊙ F2)

A Change Adapter maps this to a residual Δ, gated so its magnitude is capped at 25% of the frozen-feature magnitude. The change head predicts the category-agnostic prior B = σ(Up H chg(X, Δ)). Crucially, the original feature pyramid is preserved for semantic decoding, so temporal perception is decoupled from semantic decisions and shared across all queries.

SMC addresses the fact that open-vocabulary responses vary across scenes and categories, making a fixed decision boundary unreliable. It constructs a query-specific semantic-change response Rc and calibrates its threshold from memory.

Response construction: For query c, prompt competition suppresses ambiguous activation via:

Q̃ c t = Q c t ⊙ (Q c t / (Q c t + C c t + ε)) ρ

The competition-adjusted responses produce four cues: Dc captures semantic transitions, Gc enforces support from the change prior B, Sc suppresses persistent semantics in stable regions, and pc measures the change in query presence. Their agreement defines Rc = Π[0,1](Dc ⊙ Gc ⊙ Sc · pc).

Memory construction and retrieval: Each source image-query pair stores a normalized 1556-dimensional key and its oracle threshold. The key includes scene context, temporal discrepancy, query embeddings, and 20 statistics of Rc. The oracle threshold maximizes IoU over a grid. At inference, top-16 entries are retrieved with cosine similarity and temperature-weighted (Tm = 0.07) to form an anchor τ̄c. A Score Adapter predicts a bounded correction Δτc, yielding:

τc = Π[τmin,τmax](max τ̄c + Δτc, 0.75τc 0)

This design directly compensates for scene- and category-dependent score shifts without modifying the semantic response map.

Since pixelwise thresholding may retain fragmented or weak responses, ARF verifies connected components. Each component ωj is encoded by a 19-dimensional descriptor φj covering geometry, semantic confidence, temporal support, calibrated margin, and semantic-temporal agreement. A Gate Adapter predicts reliability uj = σ(A gate(φj)). The retention rule is:

Mc = ∪ ωj ∈ Ωc aj ≥ ηa ∨ uj ≥ τr

with large-region protection threshold ηa = 0.04 and Gate threshold τr = 0.60. This preserves large components directly, while smaller candidates must pass learned semantic-temporal-structural verification.

The visual encoder, semantic decoder, and CLIP encoders remain frozen. The Change, Score, and Gate Adapters are optimized sequentially as separate checkpoints using balanced BCE + soft Dice for SCP, Smooth-L1 residual regression for SMC, and BCE for ARF, trained only on the CA-CDD dataset.

CogVis was evaluated on seven benchmarks spanning semantic CD (SECOND, SCSCD), binary CD (CLCD, DSIFN, LEVIR-CD, WHU-CD), and building-damage assessment (xBD).

Quantitative results: CogVis achieves the best metric on all seven benchmarks, surpassing the strongest competitor by 0.05–8.17 percentage points. Specifically:

• SECOND: 32.48% mIoU (+0.81 over prior best)

• SCSCD: 25.36% mIoU (+2.14)

• CLCD: 33.67% IoU (+2.19)

• DSIFN: 48.50% IoU (+0.95)

• LEVIR-CD: 72.55% IoU (+0.05)

• WHU-CD: 75.86% IoU (+0.14)

• xBD: 32.68% mIoU (+8.17, the largest gain)

The authors observe: The larger gains on CLCD and DSIFN are consistent with the framework's ability to suppress appearance-induced temporal nuisances, while the xBD improvement is attributed to temporal support and region verification for severe damage states.

Efficiency: CogVis reaches 0.230 FPS with 4.82 GiB peak memory, providing 28.50% higher throughput than the next-fastest method. This gain arises from shared temporal computation rather than parameter reduction. Query-scaling analysis shows at K = 10 queries, CogVis requires 10.03 s, compared with 12.56–956.60 s for category-wise baselines.

Component ablation: The full model outperforms Vanilla by 13.12 mIoU on SECOND, 21.57 IoU on CLCD, and 4.25 mIoU on xBD. SCP delivers the largest gains on SECOND and CLCD (+2.95 and +8.13 points), while SMC contributes most on xBD (+1.85). ARF only consistently improves results when both SCP and SMC are enabled, improving SECOND, CLCD, and xBD by 4.44, 4.76, and 1.82 points, respectively, confirming ARF as a final verification stage for temporally stabilized and semantically calibrated candidates rather than a generic post-processing module.

Memory-bank ablation: The authors test whether gains depend on the CA-CDD subset. Memory-only exclusion reduces SECOND/CLCD/xBD by 0.38/0.35/0.02 points, while excluding the subset from training and memory reduces them by 1.06/0.50/0.22. Critically, disabling the remaining cross-source memory causes further losses of 8.03, 7.47, and 1.60 points, indicating that the gains derive from transferable response-threshold knowledge rather than same-dataset memorization. A 10% bank retains 94.9%/97.7%/95.6% of full-memory gains while reducing storage from 525.2 to 52.6 MiB.

SCP analysis: SCP increases ROC AUC from 0.79 with raw RGB differences to 0.96 with the learned change prior, demonstrating suppression of appearance-only differences.

Sensitivity: Varying K mem from 8 to 32 or T m from 0.05 to 0.10 causes only marginal changes across all benchmarks. Region-level parameters (τr, ηa) have larger effects, particularly on CLCD.

The supplement provides class-resolved results showing CogVis ranks first for Building, Tree, Low vegetation, and Surface on SECOND and leads on Water and Vegetation on SCSCD. On xBD, CogVis obtains the strongest F1 for no damage, major damage, and destroyed buildings. The supplement also includes a large-scene application mapping the January 2025 Palisades and Eaton wildfires in Los Angeles County using overlap-aware tiled inference, which preserve[s] roof- and block-level spatial continuity without obvious seams.

The paper concludes: CogVis separates query-shared temporal perception from query-dependent semantic calibration and region-level verification, allowing each source of uncertainty to be addressed at the appropriate stage. Through the three modules, it reuses scene-level change evidence, transfers response-threshold knowledge across domains, and suppresses unreliable regions using complementary semantic, temporal, and structural cues. The experiments demonstrate consistent state-of-the-art performance, strong cross-task transferability, and a 28.50% improvement in inference throughput, showing that open-vocabulary change understanding need not reinterpret the scene for every query.

The code is publicly available at https://github.com/KotlinWang/CogVis.

Improvements to AI systems:

1. Decouple shared perception from per-query reasoning. Instead of re-encoding or re-parsing the entire input for each query, the system computes a shared, category-agnostic temporal or scene representation once. A lightweight adapter then handles query-specific decisions. This reduces redundant computation, especially when the number of queries is large (e.g., 10+ semantic categories).

2. Use a memory bank of calibrated thresholds for open-vocabulary scoring. Store scene-context features, query embeddings, and response statistics from past examples along with their optimal decision thresholds. At inference, retrieve similar entries and combine them with a learned correction to produce a context-dependent threshold. This improves calibration and reduces false positives/negatives caused by score shifts across scenes and categories.

3. Gate the magnitude of learned feature adaptations. When adding a task-specific residual to a frozen encoder, cap the residual’s magnitude (e.g., at 25% of frozen-feature norm). This preserves the generalization of the pretrained representation while allowing task-specific tuning, preventing catastrophic forgetting and improving cross-domain robustness.

4. Verify predictions at the region level, not just pixel level. Instead of thresholding per-pixel scores, group responses into connected components and classify each component as reliable or spurious using a compact descriptor (geometry, semantic confidence, temporal support, calibrated margin, agreement between semantic and temporal cues). Large components are retained directly; smaller ones pass through a learned gate. This suppresses fragmented, weak, or non-semantic changes (shadows, seasons, misregistration).

5. Combine multiple complementary cues into a single response map. Aggregate semantic transition strength, support from a category-agnostic change prior, suppression of persistent semantics in stable regions, and query-presence change. This agreement-based fusion improves detection of subtle changes while filtering appearance-only differences.

What the improved AI system can do:

• Handles open-vocabulary change detection with higher accuracy across multiple benchmarks (e.g., +0.05 to +8.17 IoU/mIoU) without retraining on each target dataset.

• Maintains near-constant inference time as the number of queries grows, because temporal perception is computed once. At 10 queries, it can be ~10x faster than category-wise baselines and 28.5% higher throughput than the next-fastest method.

• Transfers threshold calibration knowledge across domains – even using a memory bank trained on one dataset (CA-CDD) improves performance on other datasets like SECOND, CLCD, and xBD, indicating learned response-threshold patterns generalize beyond the source domain.

• Suppresses nuisance changes (shadows, seasonal variation, registration errors) via a category-agnostic change prior learned from a frozen feature pyramid and a region-level verification step, leading to large gains on datasets with appearance-driven changes (e.g., CLCD +2.19, DSIFN +0.95).

• Performs robust building damage assessment (xBD +8.17 mIoU) by leveraging temporal support and region verification to disambiguate severe damage states.

• Processes large scenes in tiles without seams (e.g., wildfire damage mapping) through overlap-aware tiled inference while preserving spatial continuity.

• Uses frozen visual and text encoders with only small adapters, making it memory-efficient (4.82 GiB peak) and easy to deploy on limited hardware.