ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentation

The gist The ContourVLA introduces a vision-language-action policy that recasts Generalized Referring Expression Segmentation (GRES) as a closed-loop perception-action process where editable contours serve as explicit policy states conditioning multimodal perception and evolving through geometric actions.

ContourVLA Architecture

ContourVLA formulates GRES as a text-conditioned visuomotor decision process where an editable contour serves as an explicit geometric state that regulates multimodal perception and evolves through pointwise geometric actions It consists of a Vision-Language Interaction Module (VLIM) that acts as a perceptual expert grounding text-specified referents and providing multimodal observations for state-conditioned perception, and a Geometric Action Decoder (GAD) that predicts geometric action chunks from these observations Executing each chunk updates the contour state and triggers the next perception–action cycle, progressively refining the contour from coarse localization to precise boundaries

Evolution-Aware Semantic Scheduling (EASS)

EASS adapts dual-axis multimodal perception through the evolving contour state On the spatial axis, Spatial Visual Sampling uses contour geometry as a prior to sample bidirectional boundary cues over an adaptive receptive field On the semantic axis, Multimodal Semantic Routing encodes instance geometry, evolution progress, contour motion, and decoding depth as queries to select the most relevant VLIM representations This process involves constructing state-conditioned observations along spatial and semantic axes

Dustbin-Augmented Entropic Credit Transport GRPO (DECT-GRPO)

DECT-GRPO couples module-specific likelihood optimization with dustbin-augmented entropic transport to derive instance-level credits for grounding tokens and contour actions while accounting for false positives and missed targets It uses a pairwise coverage matrix Qg augmented with a null-match score b∅ to establish soft prediction–target correspondences through entropy-regularized optimal transport, yielding the globally consistent soft assignment P⋆g The resulting transport-based reward Rg is calculated from transport-aware precision and recall under P⋆g, jointly capturing contour accuracy, target coverage, and prediction–target correspondence

Training Strategy

ContourVLA adopts a two-stage training strategy: Supervised Fine-Tuning (SFT) establishes grounding and contour-trajectory priors, followed by DECT-GRPO post-training with self-rollout feedback Stage I involves Grounding and Detection pretraining on LocateAnything-Data to train VLIM to produce language-conditioned grounding outputs Stage II performs Supervised Contour Fine-Tuning jointly on the four datasets to initialize the contour-based perception–action policy Stage III applies DECT-GRPO post-training to jointly optimize the grounding and contour actions from final segmentation feedback

Key Contributions and Results

The main contributions include introducing ContourVLA, the first to recast GRES as an active perception–action contour policy in which geometric states bridge multimodal perception and contour actions, proposing EASS, a state-conditioned perceptual scheduler that replaces static readouts with spatial visual sampling for adaptive boundary-cue acquisition and multimodal semantic routing for evolution-aware selection of multilevel multimodal features, and DECT-GRPO, which couples module-specific likelihood optimization with dustbin-augmented entropic transport to derive instance-level credits for grounding tokens and contour actions while accounting for false positives and missed targets ContourVLA achieves 83.5, 80.3, and 74.4 gIoU on val, testA, and testB respectively on gRefCOCO

Ablation Studies

Table 2 shows that the combination of Spatial Visual Sampling (SVS) or Semantic Routing (MSR) with re-observation improves both metrics compared to the baseline, raising gIoU from 71.1 to 74.2 and cIoU from 60.8 to 63.5 Full EASS (D) raises gIoU from 74.2 to 77.6 and cIoU from 63.5 to 67.4 Table 3 demonstrates that full DECT-GRPO gains a further 3.0 gIoU and 5.1 instance F1@0.5 points over shared advantage GRPO, with dustbin-free credit yielding lower metrics

Conclusion

ContourVLA achieves the highest gIoU on all three gRefCOCO splits (by 2.7–8.7 points) and mIoU on all eight RES splits, demonstrating consistent gains in expression-averaged quality with competitive cumulative pixel overlap A single resulting checkpoint is evaluated across all benchmark splits without further benchmark-specific fine-tuning, supporting the applicability of one policy to both task formulations under the joint-training protocol The final contours are rasterized into segmentation masks and evaluated by dustbin-augmented entropic transport The resulting transport plan provides both a transport-based rollout reward and instance-level credit maps The final score is reported as a percentage for no-target accuracy, where an empty prediction receives a score of one when the target is empty The final boundary evaluation operates on expression-level union masks and does not require instance matching The cumulative contour latency increases from 226.96 to 233.63 ms for Full EASS compared to frozen observations at matched action budgets

A Additional Implementation and Experimental Details

The final contours are rasterized into segmentation masks and evaluated by dustbin-augmented entropic transport The predicted contour remains subject to the bounded transitions in equation 22–equation 23, so these references are not guaranteed to be reachable within every chunk The resulting policy loss is LDECT = 1/G X G g=1 [−1 Lg X Lg n=1 Sclip(ρ y g,n, Ay g,n) − λa Dg X ξ∈Dg Sclip(ρ a g,ξ, Aa g,ξ) + βKLKg] The training data supervision and objective for Stage III is the four-dataset mixture Final segmentation feedback; joint optimization of the grounding and contour-action policies The action loss is additionally omitted when Mg = 0, N = 0, or Dg = 0 The final policy evaluation uses the shared group-relative advantage Ag to modulate the shared rollout feedback at corresponding grounding-token spans and contour-action positions The mechanism therefore differentiates the strength of shared rollout feedback across instances without assigning independent advantage signs This scoped design isolates geometric correction and keeps inference stable, as an initially missed referent cannot be recovered through later contour updates The fixed ordered-vertex state provides efficient, interpretable boundary updates, but retains the initialized contour topology and point budget The final training data supervision and objective for Stage III is the four-dataset mixture Final segmentation feedback; joint optimization of the grounding and contour-action policies The action loss is additionally omitted when Mg = 0, N = 0, or Dg = 0 The final policy evaluation uses the shared group-relative advantage Ag to modulate the shared rollout feedback at corresponding grounding-token spans and contour-action positions

Representative Transport and Credit Cases

Ambiguous matching shows that both predictions receive the same credit, e = (0.6837, 0.

1. Bold header: Active Perception-Action Closed Loop

This system recasts Generalized Referring Expression Segmentation (GRES) as a closed-loop perception–action process in which editable contours guide multimodal perception and evolve through geometric actions. This allows the system to iteratively refine boundaries from coarse localization to precise boundaries by executing geometric action chunks, effectively correcting initial errors through explicit geometric feedback.

2. Bold header: Adaptive Multimodal Feature Selection

The Evolution-Aware Semantic Scheduling (EASS) enables the system to adapt perception based on state changes, using Spatial Visual Sampling for boundary cues and Multimodal Semantic Routing to select features conditioned on evolving state queries, ensuring adaptive multimodal perception under evolving contour states.

3. Bold header: Instance-Level Credit Optimization

The Dustbin-Augmented Entropic Credit Transport GRPO (DECT-GRPO) optimizes the joint policy by deriving instance-level credits for grounding tokens and contour actions using transport-based rewards, which enables targeted optimization by assigning credit based on soft prediction–target correspondences that account for false positives and missed targets.

4. Bold header: Robust Performance Across GRES Splits

The system achieves state-of-the-art results across all GRES benchmarks by demonstrating consistent gains in expression-averaged quality with competitive cumulative pixel overlap, exceeding baselines by up to 8.7 points on gRefCOCO val, testA, and testB.

5. Bold header: Unified Policy Application

The single resulting checkpoint trained jointly on four dataset splits performs well across both GRES and conventional referring segmentation tasks, supporting the applicability of one policy to both task formulations under the joint-training protocol.