Multimodal Cross-Condition Learning for Temporal Sentence Grounding in Videos
cs.CV
Submitted: 2026-10-06
Updated: 2026-10-06
Comments: 9 pages, 3 figures, 4 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Temporal sentence grounding requires a video representation that preserves query-relevant evidence and a sentence representation that reflects the visual context in which it is interpreted.
Terminology
Abstract
Temporal sentence grounding requires a video representation that preserves query-relevant evidence and a sentence representation that reflects the visual context in which it is interpreted. Prior end-to-end grounding addresses the first requirement through sentence-conditioned visual adaptation, but leaves the language representation independent of the video. This work proposes Multimodal Cross-Condition Learning (MCCL). Its Video Prefix Adapter (VPA) converts pooled visual context into a sequence of continuous tokens that condition a frozen DistilBERT encoder. Combined with the Sentence Conditioned Adapter (SCADA), VPA supports an ordered exchange from vision to language and back to vision. A comprehensive study examines two video backbones, LSTM and Mamba temporal modules, and four conditioning modes on two datasets. Bidirectional conditioning exceeds the stronger one-way mode by 1.6--2.3 mIoU points in all four Charades-STA settings. ActivityNet Captions exhibits a different pattern: bidirectionality helps I3D--Mamba, whereas video-to-text conditioning is stronger with C3D. The study extends one-way visual adaptation into a configurable multimodal framework and reports descriptive gains over the corresponding baselines.
Sources
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- ClipCap: CLIP Prefix for Image Captioning
- Towards Debiasing Temporal Sentence Grounding in Video
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs