Multimodal Cross-Condition Learning for Temporal Sentence Grounding in Videos

arXiv:2611.00033 · cs.CV · Submitted 2026-10-06 · Read on arXiv

cs.CV

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 9 pages, 3 figures, 4 tables

License: http://creativecommons.org/licenses/by/4.0/

The gist: Temporal sentence grounding requires a video representation that preserves query-relevant evidence and a sentence representation that reflects the visual context in which it is interpreted.

Terminology

Abstract

Temporal sentence grounding requires a video representation that preserves query-relevant evidence and a sentence representation that reflects the visual context in which it is interpreted. Prior end-to-end grounding addresses the first requirement through sentence-conditioned visual adaptation, but leaves the language representation independent of the video. This work proposes Multimodal Cross-Condition Learning (MCCL). Its Video Prefix Adapter (VPA) converts pooled visual context into a sequence of continuous tokens that condition a frozen DistilBERT encoder. Combined with the Sentence Conditioned Adapter (SCADA), VPA supports an ordered exchange from vision to language and back to vision. A comprehensive study examines two video backbones, LSTM and Mamba temporal modules, and four conditioning modes on two datasets. Bidirectional conditioning exceeds the stronger one-way mode by 1.6--2.3 mIoU points in all four Charades-STA settings. ActivityNet Captions exhibits a different pattern: bidirectionality helps I3D--Mamba, whereas video-to-text conditioning is stronger with C3D. The study extends one-way visual adaptation into a configurable multimodal framework and reports descriptive gains over the corresponding baselines.

Sources

Related papers