Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition".
Jane: Here's the gist of the paper. Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues The proposed Similarity Volume Aggregation (SimVA) framework constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities.
Tom: Framework Overview SimVA is a novel framework designed to model motion dynamics through spatio-temporal similarity aggregation between video and text.
Jane: It constructs a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time.
Tom: This volume is formed by computing cosine similarities between patch-level visual tokens and text embeddings to produce spatial matching maps of size H × W × NC, which are then extended along the temporal axis of T frames to form a dense volume of size T × H × W × NC.
Paper summary: Lu: To wrap up, the paper "Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition" introduces SimVA as a new way to model action recognition by building a dense four dimensional similarity volume directly from patch level visual text similarities.
Tom: The core of the thesis is that this volume preserves fine grained spatio temporal cues, unlike prior methods that aggregate everything into one global representation first.
Jane: It claims they achieve this through a structured pipeline involving spatial aggregation to refine intra frame context, motion aware modulation to inject inter frame variation cues, and Mamba based temporal aggregation to model the evolution across time.
Meng: The practical implication is that it gives us a framework that is designed for action recognition tasks where precision in movement details matters, even when dealing with open vocabulary.
Lalam: They also address efficiency by introducing a class sampling strategy that selects a subset of relevant classes based on global visual alignment, making the aggregation scalable to large vocabularies.
Jane: So simply put, they're creating a detailed similarity map over time and space that allows for much finer discrimination of actions than previous methods allowed.
Tom: The authors mention their limitation is tied to their class sampling budget because it’s fixed, meaning they suggest adaptive strategies would be the next step for better scalability.
Lu: That's the main caveat—the fixed nature of the class sampling strategy is what limits its immediate use across truly massive vocabularies without further adaptation.
Jane: It points toward a future where we might develop smarter ways to select which actions to focus on based on how well they align with the video input.
Conclusion: Tom: So we've been digging into SimVA, this paper by Lu and Tom, which basically builds this whole dense four dimensional similarity volume from just patch level visual text similarities to recognize actions open vocabulary.
Jane: Right, so instead of one big blob of information for an action, they are creating a detailed map that shows how things look and move across space and time simultaneously.
Lu: The core idea is moving away from those old methods where you just squeeze everything into one global representation first, which usually blurs out the fine details about when something happens or exactly what’s happening spatially.
Meng: But building that whole volume sounds computationally heavy, especially for open vocabulary where there are tons of classes to consider. How do they manage that?
Lalam: They use a clever class sampling strategy based on a global video representation, averaging those key tokens over time to pick the most relevant actions first, which makes the final volume size manageable.
Tom: So it's about making sure you only compute similarity for the actions that actually matter based on how well they align globally across the whole video. That seems like a smart way to cut down on processing load.
Jane: And then they refine that volume using spatial aggregation, motion aware modulation to track how things move between frames, and a Mamba block to model that evolution over time. It’s a really structured approach to learning motion dynamics directly from the similarity itself.
Lu: The results show it’s competitive with other methods in zero shot and few shot settings, which is important because it means this detailed volume isn't just theoretical; it works on real action recognition benchmarks.
Meng: I'm looking at those numbers now, and if they maintain a budget similar to CLIP based methods, that’s actually pretty impressive for a system that has this much spatio temporal detail packed into it.
Tom: It means we can potentially get much finer discrimination between subtle actions—like the exact nuance of a hand gesture—because the volume keeps those local patches preserved instead of washing them out.
Jane: So what does this mean for people who just want to use AI to understand videos? It suggests we might finally get systems that aren't just good at identifying "walking" but can really tell the difference between a brisk walk and a jog, based on how the movement unfolds over time.
Lalam: Exactly. This framework shows that modeling spatio temporal similarity directly is a powerful way to capture action dynamics without losing those crucial local cues.
Tom: It’s definitely about getting the mechanics right—building this specific structure allows for that high fidelity in action recognition. But now we need to see how robust this thing stays when we try to scale it up beyond just the current benchmark set.
Ewha Womans University
cs.CV
Submitted: 2026-05-22
Updated: 2026-11-03
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 77/100
The gist: Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment…
Key concepts
- Similarity Volume
- A dense 4D structure (T × H × W × NC) built from patch-level visual-text similarities. It maps how specific image patches relate to text descriptions across both space (H×W) and time (T), preserving detailed interaction cues rather than just global features.
- Class Sampling Strategy
- A method to reduce computational load by selecting a small, relevant subset of classes. It averages video tokens over time to create a global representation and then picks the top-M classes based on their similarity score against text embeddings, making the volume tractable.
- Motion-aware Modulation
- A mechanism using a 3x3 convolution to estimate offsets between adjacent frames. This injects information about inter-frame motion directly into the similarity representations, helping the model understand how visual similarities evolve over time.
- Mamba Block
- A specialized temporal aggregation module used to model the evolution of spatial similarities across time. It is chosen because it effectively captures sequential state-space modeling, allowing the framework to learn complex trajectories of patch-level similarity.
Terminology
Summary
Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues The proposed Similarity Volume Aggregation (SimVA) framework constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities. p. 0 kr
Abstract
Recent Open-Vocabulary Action Recognition (OVAR) methods typically aggregate
visual features into a global representation before computing text alignment, a
process that obscures local patch information and fine-grained spatio-temporal
cues. We propose Similarity Volume Aggregation (SimVA), a framework that
constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities.Where the paper says it
Framework Overview
SimVA is a novel framework designed to model motion dynamics through spatio-temporal similarity aggregation between video and text. It constructs a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time. p. 1 By performing this alignment within the similarity space, SimVA
naturally extends the powerful image-text alignment capabilities of CLIP to the spatio-temporal
domain, explicitly modeling how localized visual patterns evolve in interaction with textual semantics. Specifically, we construct a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time. Specifically, we construct a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time. Cosine similarities between patch-level visual
tokens and text embeddings produce spatial matching maps of size H × W × NC , which are then
extended along the temporal axis of T frames to form a dense volume of size T × H × W × NC .Where the paper says it
Where the paper says it
Class Sampling Strategy
To address the computational cost associated with constructing the full similarity volume over all classes, SimVA introduces a class sampling strategy that selects a subset of relevant classes based on global visual alignment. This strategy involves computing a global video representation V¯ by averaging the frame-level [CLS] tokens v t over time and then selecting the top-M classes based on prior similarity scores Sglobal = V¯⊤Ftext. p. 3 Specifically, we compute a global video representation V¯ ∈ R
D by averaging the frame-level
[CLS] tokens v
t ∈ R
D over time, followed by ℓ2 normalization:
V¯ =
V
∥V ∥
, V =
1
T
X
T
t=1
v
t
. (2)
We then compute prior similarity scores for all classes as Sglobal = V¯ ⊤Ftext, and select a subset of
the top-M classes IM. Finally, the text embeddings of this selected subset, denoted as F
′
text ∈ RM×D, replace the original
Ftext in Eq. This strategy significantly reduces the computational burden, shrinking the class
dimension of the similarity volume to a tractable size M, i.Where the paper says it
Where the paper says it
Structured Aggregation Pipeline
The class-conditioned similarity volume Z0 is refined through a structured aggregation pipeline consisting of spatial aggregation, motion-aware modulation, and temporal aggregation.
-
Spatial Aggregation uses Swin Transformer blocks to refine intra-frame similarity patterns by enforcing local consistency and reducing matching noise. p. 1
Where the paper says it
Based on this formulation, we perform a structured aggregation process directly in the similarity space. Spatial aggregation using Swin Transformer blocks [22] first refines intra-frame similarity patterns by enforcing local consistency and reducing matching noise.
-
Motion-aware Modulation estimates inter-frame offsets using a 3 × 3 convolution to inject adjacent-frame variation cues into the similarity representations. p. 4
Where the paper says it
Prior to temporal aggregation, we apply motion-aware modulation, which estimates inter-frame offsets from consecutive CLIP image features to incorporate local motion cues into the similarity representations. Specifically, given consecutive frame features F t img and F t+1 img , we concatenate them and estimate an offset using a 3 × 3 convolution ψ as u t = tanh(ψ3×3([F t img, Ft+1 img ])), where u t ∈ R 2×H×W denotes the estimated offset, and [·, ·] represents channel-wise concatenation.
-
Temporal Aggregation employs a Mamba block to model the evolution of these spatially aligned similarities across time. p. 1
Where the paper says it
After spatial aggregation, we introduce a motion modulation that adapts the spatially refined similarity representations based on motion vectors derived from adjacent frames, producing motion-aware similarity representations for temporal modeling rather than relying solely on static appearance-based matching. Temporal aggregation with Mamba [9] then models the evolution of these spatially aligned similarities across time.
Performance and Efficiency
SimVA achieves competitive performance across zero-shot, few-shot, and base-to-novel benchmarks. The framework maintains a computational budget comparable to recent CLIP-based methods with only 124. p. 6 Despite
constructing a dense spatio-temporal similarity volume, SimVA requires only 124. 6M parameters
and 315 GFLOPs, maintaining a computational budget comparable to recent CLIP-based methods. Despite
constructing a dense spatio-temporal similarity volume, SimVA requires only 124. 6M parameters
and 315 GFLOPs, maintaining a computational budget comparable to recent CLIP-based methods. Adding motion-aware modulation (G) yields additional mean gains, with the clearest improvement on
HMDB and modest gains on UCF and K600, suggesting that local inter-frame variations can further
complement the spatially refined similarity representations before temporal aggregation. Overall,
SimVA (G) achieves the highest mean performance across all datasets, supporting the effectiveness of
combining similarity-space spatial refinement, motion-aware modulation, and temporal aggregation
for spatio-temporal action recognition. While the Transformer variant achieves competitive results, Mamba (G) consistently performs better
across datasets. This suggests that sequential state-space modeling is more suitable for aggregating
patch-level similarity trajectories, where motion-relevant cues need to be selectively propagated over
time.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
The SimVA framework employs a fixed class-sampling budget to ensure scalable similarity aggregation. The authors note that developing adaptive class-sampling strategies remains a promising direction for further improving scalability. p. 8 However, a manually specified budget may not be optimal across diverse recognition
vocabularies or varying computational constraints. Developing adaptive class-sampling strategies
remains a promising direction for further improving scalability. 4 Visualization
Fig. 3 qualitatively demonstrates that our motion-aware modulation module effectively highlights
action-relevant regions while suppressing the static background.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Move from global feature aggregation to dense similarity aggregation to preserve local patch information, as proposed by SimVA, enabling
fine-grained spatio-temporal cues.
This allows for precise discrimination of fine-grained actions that depend on subtle temporal variations, which is superior to methods thatpredominantly rely on extracting frame-wise global tokens (e.g., [CLS]).
-
Implement a class sampling strategy based on global visual alignment to reduce computational cost while maintaining coverage, as described in the paper. This allows the system to scale effectively for large vocabularies by reducing the class dimension of the similarity volume from NC to M, making
the construction of the similarity volume over all classes computationally prohibitive
manageable. -
Integrate a structured aggregation pipeline consisting of spatial aggregation using Swin Transformer blocks to enforce intra-frame consistency and motion-aware modulation that
inject[s] inter-frame variation cues.
This sequential refinement allows the model to model bothintra-frame contextual information and inter-frame motion dynamics,
leading to better action recognition. -
Utilize Mamba for temporal aggregation instead of a Transformer, as Mamba's forward scanning mechanism is suitable for
selectively state-space temporal operator that performs input-dependent state updates along the temporal sequence.
This design allows the model toefficiently refine temporal similarity patterns
by integrating locally enhanced similarity patterns into coherent action-level dynamics. -
Employ a memory-efficient embedding dimension of df = 64 for the projected similarity feature space, as it achieved
the best average performance
and provided acompact and effective similarity projection.
This suggests that using df=64 over larger dimensions like 128 provides the optimal trade-off between refinement capability and computational efficiency.
Abstract
Recent Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues. We propose Similarity Volume Aggregation (SimVA), a framework that constructs a dense 4D spatio-temporal similarity volume from patch-level visual-text similarities. SimVA constructs a spatio-temporal similarity volume over local video tokens and action classes, and employs class sampling to ensure similarity aggregation scalable to large vocabularies. The similarity volume is refined by spatial aggregation, which contextualizes local similarity patterns to improve intra-frame consistency. Motion-aware modulation further injects inter-frame variation cues, highlighting dynamically changing regions. Mamba-based temporal aggregation then models the evolution of class-conditioned similarity patterns across frames. By maintaining dense visual-text correspondence, SimVA effectively transfers CLIP to video action recognition, achieving competitive performance across zero-shot, few-shot, and base-to-novel benchmarks.
Sources
- A Short Note about Kinetics-600
- CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo
- FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition
- The Kinetics Human Action Video Dataset
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- ActionCLIP: A New Paradigm for Video Action Recognition
- AIM: Adapting Image Models for Efficient Video Action Recognition
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs