Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition

arXiv:2605.23288 · cs.CV · Submitted 2026-05-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition".

Jane: Here's the gist of the paper. Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues The proposed Similarity Volume Aggregation (SimVA) framework constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities.

Tom: Framework Overview SimVA is a novel framework designed to model motion dynamics through spatio-temporal similarity aggregation between video and text.

Jane: It constructs a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time.

Tom: This volume is formed by computing cosine similarities between patch-level visual tokens and text embeddings to produce spatial matching maps of size H × W × NC, which are then extended along the temporal axis of T frames to form a dense volume of size T × H × W × NC.

Paper summary: Lu: To wrap up, the paper "Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition" introduces SimVA as a new way to model action recognition by building a dense four dimensional similarity volume directly from patch level visual text similarities.

Tom: The core of the thesis is that this volume preserves fine grained spatio temporal cues, unlike prior methods that aggregate everything into one global representation first.

Jane: It claims they achieve this through a structured pipeline involving spatial aggregation to refine intra frame context, motion aware modulation to inject inter frame variation cues, and Mamba based temporal aggregation to model the evolution across time.

Meng: The practical implication is that it gives us a framework that is designed for action recognition tasks where precision in movement details matters, even when dealing with open vocabulary.

Lalam: They also address efficiency by introducing a class sampling strategy that selects a subset of relevant classes based on global visual alignment, making the aggregation scalable to large vocabularies.

Jane: So simply put, they're creating a detailed similarity map over time and space that allows for much finer discrimination of actions than previous methods allowed.

Tom: The authors mention their limitation is tied to their class sampling budget because it’s fixed, meaning they suggest adaptive strategies would be the next step for better scalability.

Lu: That's the main caveat—the fixed nature of the class sampling strategy is what limits its immediate use across truly massive vocabularies without further adaptation.

Jane: It points toward a future where we might develop smarter ways to select which actions to focus on based on how well they align with the video input.

Conclusion: Tom: So we've been digging into SimVA, this paper by Lu and Tom, which basically builds this whole dense four dimensional similarity volume from just patch level visual text similarities to recognize actions open vocabulary.

Jane: Right, so instead of one big blob of information for an action, they are creating a detailed map that shows how things look and move across space and time simultaneously.

Lu: The core idea is moving away from those old methods where you just squeeze everything into one global representation first, which usually blurs out the fine details about when something happens or exactly what’s happening spatially.

Meng: But building that whole volume sounds computationally heavy, especially for open vocabulary where there are tons of classes to consider. How do they manage that?

Lalam: They use a clever class sampling strategy based on a global video representation, averaging those key tokens over time to pick the most relevant actions first, which makes the final volume size manageable.

Tom: So it's about making sure you only compute similarity for the actions that actually matter based on how well they align globally across the whole video. That seems like a smart way to cut down on processing load.

Jane: And then they refine that volume using spatial aggregation, motion aware modulation to track how things move between frames, and a Mamba block to model that evolution over time. It’s a really structured approach to learning motion dynamics directly from the similarity itself.

Lu: The results show it’s competitive with other methods in zero shot and few shot settings, which is important because it means this detailed volume isn't just theoretical; it works on real action recognition benchmarks.

Meng: I'm looking at those numbers now, and if they maintain a budget similar to CLIP based methods, that’s actually pretty impressive for a system that has this much spatio temporal detail packed into it.

Tom: It means we can potentially get much finer discrimination between subtle actions—like the exact nuance of a hand gesture—because the volume keeps those local patches preserved instead of washing them out.

Jane: So what does this mean for people who just want to use AI to understand videos? It suggests we might finally get systems that aren't just good at identifying "walking" but can really tell the difference between a brisk walk and a jog, based on how the movement unfolds over time.

Lalam: Exactly. This framework shows that modeling spatio temporal similarity directly is a powerful way to capture action dynamics without losing those crucial local cues.

Tom: It’s definitely about getting the mechanics right—building this specific structure allows for that high fidelity in action recognition. But now we need to see how robust this thing stays when we try to scale it up beyond just the current benchmark set.

Ewha Womans University

cs.CV

Submitted: 2026-05-22

Updated: 2026-11-03

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment…

Key concepts

Similarity Volume
A dense 4D structure (T × H × W × NC) built from patch-level visual-text similarities. It maps how specific image patches relate to text descriptions across both space (H×W) and time (T), preserving detailed interaction cues rather than just global features.
Class Sampling Strategy
A method to reduce computational load by selecting a small, relevant subset of classes. It averages video tokens over time to create a global representation and then picks the top-M classes based on their similarity score against text embeddings, making the volume tractable.
Motion-aware Modulation
A mechanism using a 3x3 convolution to estimate offsets between adjacent frames. This injects information about inter-frame motion directly into the similarity representations, helping the model understand how visual similarities evolve over time.
Mamba Block
A specialized temporal aggregation module used to model the evolution of spatial similarities across time. It is chosen because it effectively captures sequential state-space modeling, allowing the framework to learn complex trajectories of patch-level similarity.

Terminology

Summary

Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues The proposed Similarity Volume Aggregation (SimVA) framework constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities. p. 0

Where the paper says it

kr Abstract Recent Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues. We propose Similarity Volume Aggregation (SimVA), a framework that constructs a dense 4D spatio-temporal similarity volume from patch-level visualtext similarities.

Framework Overview

SimVA is a novel framework designed to model motion dynamics through spatio-temporal similarity aggregation between video and text. It constructs a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time. p. 1

Where the paper says it

By performing this alignment within the similarity space, SimVA naturally extends the powerful image-text alignment capabilities of CLIP to the spatio-temporal domain, explicitly modeling how localized visual patterns evolve in interaction with textual semantics. Specifically, we construct a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time.

This volume is formed by computing cosine similarities between patch-level visual tokens and text embeddings to produce spatial matching maps of size H × W × NC, which are then extended along the temporal axis of T frames to form a dense volume of size T × H × W × NC. p. 1
Where the paper says it

Specifically, we construct a 4D spatio-temporal similarity volume that explicitly preserves finegrained visual–text interactions across space and time. Cosine similarities between patch-level visual tokens and text embeddings produce spatial matching maps of size H × W × NC , which are then extended along the temporal axis of T frames to form a dense volume of size T × H × W × NC .

Class Sampling Strategy

To address the computational cost associated with constructing the full similarity volume over all classes, SimVA introduces a class sampling strategy that selects a subset of relevant classes based on global visual alignment. This strategy involves computing a global video representation V¯ by averaging the frame-level [CLS] tokens v t over time and then selecting the top-M classes based on prior similarity scores Sglobal = V¯⊤Ftext. p. 3

Where the paper says it

Specifically, we compute a global video representation V¯ ∈ R D by averaging the frame-level [CLS] tokens v t ∈ R D over time, followed by ℓ2 normalization: V¯ = V ∥V ∥ , V = 1 T X T t=1 v t . (2) We then compute prior similarity scores for all classes as Sglobal = V¯ ⊤Ftext, and select a subset of the top-M classes IM.

This significantly reduces the class dimension of the similarity volume to a tractable size M. p. 3
Where the paper says it

Finally, the text embeddings of this selected subset, denoted as F ′ text ∈ RM×D, replace the original Ftext in Eq. This strategy significantly reduces the computational burden, shrinking the class dimension of the similarity volume to a tractable size M, i.

Structured Aggregation Pipeline

The class-conditioned similarity volume Z0 is refined through a structured aggregation pipeline consisting of spatial aggregation, motion-aware modulation, and temporal aggregation.

  1. Spatial Aggregation uses Swin Transformer blocks to refine intra-frame similarity patterns by enforcing local consistency and reducing matching noise. p. 1

    Where the paper says it

    Based on this formulation, we perform a structured aggregation process directly in the similarity space. Spatial aggregation using Swin Transformer blocks [22] first refines intra-frame similarity patterns by enforcing local consistency and reducing matching noise.

  2. Motion-aware Modulation estimates inter-frame offsets using a 3 × 3 convolution to inject adjacent-frame variation cues into the similarity representations. p. 4

    Where the paper says it

    Prior to temporal aggregation, we apply motion-aware modulation, which estimates inter-frame offsets from consecutive CLIP image features to incorporate local motion cues into the similarity representations. Specifically, given consecutive frame features F t img and F t+1 img , we concatenate them and estimate an offset using a 3 × 3 convolution ψ as u t = tanh(ψ3×3([F t img, Ft+1 img ])), where u t ∈ R 2×H×W denotes the estimated offset, and [·, ·] represents channel-wise concatenation.

  3. Temporal Aggregation employs a Mamba block to model the evolution of these spatially aligned similarities across time. p. 1

    Where the paper says it

    After spatial aggregation, we introduce a motion modulation that adapts the spatially refined similarity representations based on motion vectors derived from adjacent frames, producing motion-aware similarity representations for temporal modeling rather than relying solely on static appearance-based matching. Temporal aggregation with Mamba [9] then models the evolution of these spatially aligned similarities across time.

Performance and Efficiency

SimVA achieves competitive performance across zero-shot, few-shot, and base-to-novel benchmarks. The framework maintains a computational budget comparable to recent CLIP-based methods with only 124. p. 6

Where the paper says it

Despite constructing a dense spatio-temporal similarity volume, SimVA requires only 124. 6M parameters and 315 GFLOPs, maintaining a computational budget comparable to recent CLIP-based methods.

6M parameters and 315 GFLOPs. p. 6
Where the paper says it

Despite constructing a dense spatio-temporal similarity volume, SimVA requires only 124. 6M parameters and 315 GFLOPs, maintaining a computational budget comparable to recent CLIP-based methods.

Ablation studies confirm that combining spatial and temporal aggregation with motion-aware modulation yields the highest mean performance across all datasets. p. 6
Where the paper says it

Adding motion-aware modulation (G) yields additional mean gains, with the clearest improvement on HMDB and modest gains on UCF and K600, suggesting that local inter-frame variations can further complement the spatially refined similarity representations before temporal aggregation. Overall, SimVA (G) achieves the highest mean performance across all datasets, supporting the effectiveness of combining similarity-space spatial refinement, motion-aware modulation, and temporal aggregation for spatio-temporal action recognition.

The Mamba-based temporal module is shown to perform better than a Transformer-based alternative in capturing sequential state-space modeling for aggregating patch-level similarity trajectories. p. 7
Where the paper says it

While the Transformer variant achieves competitive results, Mamba (G) consistently performs better across datasets. This suggests that sequential state-space modeling is more suitable for aggregating patch-level similarity trajectories, where motion-relevant cues need to be selectively propagated over time.

Limitations and Future Work

The SimVA framework employs a fixed class-sampling budget to ensure scalable similarity aggregation. The authors note that developing adaptive class-sampling strategies remains a promising direction for further improving scalability. p. 8

Where the paper says it

However, a manually specified budget may not be optimal across diverse recognition vocabularies or varying computational constraints. Developing adaptive class-sampling strategies remains a promising direction for further improving scalability.

The visualization of the motion-aware modulation module qualitatively demonstrates that it effectively highlights action-relevant regions while suppressing the static background. p. 7
Where the paper says it

4 Visualization Fig. 3 qualitatively demonstrates that our motion-aware modulation module effectively highlights action-relevant regions while suppressing the static background.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Move from global feature aggregation to dense similarity aggregation to preserve local patch information, as proposed by SimVA, enabling fine-grained spatio-temporal cues. This allows for precise discrimination of fine-grained actions that depend on subtle temporal variations, which is superior to methods that predominantly rely on extracting frame-wise global tokens (e.g., [CLS]).

  2. Implement a class sampling strategy based on global visual alignment to reduce computational cost while maintaining coverage, as described in the paper. This allows the system to scale effectively for large vocabularies by reducing the class dimension of the similarity volume from NC to M, making the construction of the similarity volume over all classes computationally prohibitive manageable.

  3. Integrate a structured aggregation pipeline consisting of spatial aggregation using Swin Transformer blocks to enforce intra-frame consistency and motion-aware modulation that inject[s] inter-frame variation cues. This sequential refinement allows the model to model both intra-frame contextual information and inter-frame motion dynamics, leading to better action recognition.

  4. Utilize Mamba for temporal aggregation instead of a Transformer, as Mamba's forward scanning mechanism is suitable for selectively state-space temporal operator that performs input-dependent state updates along the temporal sequence. This design allows the model to efficiently refine temporal similarity patterns by integrating locally enhanced similarity patterns into coherent action-level dynamics.

  5. Employ a memory-efficient embedding dimension of df = 64 for the projected similarity feature space, as it achieved the best average performance and provided a compact and effective similarity projection. This suggests that using df=64 over larger dimensions like 128 provides the optimal trade-off between refinement capability and computational efficiency.

Abstract

Recent Open-Vocabulary Action Recognition (OVAR) methods typically aggregate visual features into a global representation before computing text alignment, a process that obscures local patch information and fine-grained spatio-temporal cues. We propose Similarity Volume Aggregation (SimVA), a framework that constructs a dense 4D spatio-temporal similarity volume from patch-level visual-text similarities. SimVA constructs a spatio-temporal similarity volume over local video tokens and action classes, and employs class sampling to ensure similarity aggregation scalable to large vocabularies. The similarity volume is refined by spatial aggregation, which contextualizes local similarity patterns to improve intra-frame consistency. Motion-aware modulation further injects inter-frame variation cues, highlighting dynamically changing regions. Mamba-based temporal aggregation then models the evolution of class-conditioned similarity patterns across frames. By maintaining dense visual-text correspondence, SimVA effectively transfers CLIP to video action recognition, achieving competitive performance across zero-shot, few-shot, and base-to-novel benchmarks.

Sources

Related papers