Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding

arXiv:2609.31135 · cs.CV, cs.AI, cs.MM · Submitted 2026-09-25 · Read on arXiv

cs.CV, cs.AI, cs.MM

Submitted: 2026-09-25

Updated: 2026-09-25

Comments: 14 pages total. 8 pages main manuscript, 3 pages references, 3 pages additional material

License: http://creativecommons.org/licenses/by/4.0/

The gist: Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query.

Terminology

Abstract

Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.

Sources

Related papers