Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detection
cs.CV, cs.AI
Submitted: 2025-09-29
Updated: 2026-08-31
Comments: Accepted by EMNLP 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision-language models (VLMs) have advanced multimodal perception, demonstrated by open-vocabulary object detection with simple language queries.
Terminology
Abstract
Vision-language models (VLMs) have advanced multimodal perception, demonstrated by open-vocabulary object detection with simple language queries. State-of-the-art VLMs still struggle to handle complex queries involving descriptive attributes and relational clauses. To address this problem, we propose restructuring linguistic representations according to the hierarchical relations within sentences for language-based object detection. A key insight is that textual tokens should be disentangled into core components-objects, attributes, and relations-and aggregated into hierarchically structured sentence-level representations. Building on this principle, we introduce the TaSe (Talk in Pieces, See in Whole) framework with three main contributions: (1) a hierarchical synthetic captioning dataset spanning three tiers from category names to descriptive sentences; (2) the three-component disentanglement module guided by a novel disentanglement loss function, transforms text embeddings into subspace compositions; and (3) aggregating disentangled components into hierarchically structured embeddings guided by the proposed hierarchical objectives. Experimental results under the OmniLabel benchmark show a 24% performance improvement, demonstrating the importance of linguistic compositionality.
Sources
- The Llama 3 Herd of Models
- Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
- LoRA: Low-Rank Adaptation of Large Language Models
- GPT-4o System Card
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Decoupled Weight Decay Regularization
- DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Representation Learning with Contrastive Predictive Coding
- Compositional Entailment Learning for Hyperbolic Vision-Language Models
- DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding
- When and why vision-language models behave like bags-of-words, and what to do about it?
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models