Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax

summary

The gist

Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal information density The paper presents Quicksviewer…

In short

Quicksviewer is a Large Multimodal Model (LMM) designed to efficiently process videos by dynamically compressing them into non-uniform cubes based on temporal density. This method, using Gumbel Softmax for partitioning, achieves a 45x compression rate while maintaining state-of-the-art performance on video understanding benchmarks.

Key concepts

Cubing Network
This network partitions a video into NQ cubes by analyzing the momentum of semantic feature differences between consecutive frames. It calculates an i-th momentum representation using a 2-layer MLP and LayerNorm to quantify frame significance, forming cubes based on keyframes.
Gumbel Softmax
This technique is used to dynamically partition the video frames into varying cubes based on their temporal density. It allows the model to adapt its perception strategy online, compressing dense parts of the video more aggressively while preserving information in sparser sections.
Visual Lag
A phenomenon observed during cube-based perception where terminal frames from preceding events are included in cubes containing subsequent events. This mechanism helps the model retain partial memory of past scenes to better understand current scenes.
Nonuniform Perception Paradigm
Instead of treating all video frames equally, this approach partitions the video into cubes whose size and content vary based on the temporal density of semantic features. This allows for adaptive compression tailored to specific parts of the video.

Terminology used across episodes

This episode discusses

The paper

Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax · Read on arXiv

Ji Qi, Yuan Yao, Yushi Bai, Bin Xu, Juanzi Li, Zhiyuan Liu

Tsinghua University · National University of Singapore

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax".

Paper summary: Tom: So we're looking at this paper, "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax," and the big idea is tackling the computational inefficiency of Large Multimodal Models when they try to process video.

Jane: Exactly. The paper argues that LMMs uniformly perceive every single frame in a video, which is just computationally wasteful when some moments are much more important or detailed than others.

Lu: That's right. Quicksviewer proposes a new way of perceiving video by partitioning it into cubes based on how dense the temporal information is within those segments.

Meng: So instead of treating every frame equally, you dynamically compress the video online by using this nonuniform density to decide what gets processed deeply and what gets compressed lightly.

Tom: Right, that's the core claim. They use a method called Gumbel Softmax to do this partitioning, which they describe as "cubing network" in their architecture.

Jane: And they call it simple and intuitive because it partitions the video based on the momentum of semantic feature differences between frames. That sounds like they're using what happens between frames to find key moments.

Lalam: From my perspective, this cubing mechanism is how the model identifies those important keyframes, which they call Fli, along with consecutive non-keyframe cubes making up a cube Qi plus one.

Tom: Okay, so the architecture has four main pieces: a visual encoder that makes tokens from frames, this cubing network to partition the video into cubes, a resampler to compress each slice or cube into a fixed number of tokens, and finally an LLM backbone that takes those visual tokens and some user prompt.

Jane: And what's interesting is how they train it. They start with three progressive stages, moving from training on image-text pairs to large-scale pretraining data, and finally using extensive video-text paired corpus for the robust video understanding part.

Lu: What really stands out is integrating the cubing network into the end-to-end training using Gumbel Softmax with an improved noise annealing mechanism. They say this substantially improves both training stability and effectiveness without needing boundary labels for that cubing layer itself.

Meng: I’m wondering about the practical side here. They mention achieving a forty-five times compression rate, which is huge, but how does that translate to actual deployment costs when you're dealing with ultra-long videos?

Tom: They say this nonuniform perception paradigm leads to a forty-five times compression rate while significantly reducing spatiotemporal redundancy. And they show the results are very solid compared to direct baseline methods employing a fixed partitioning strategy.

Jane: The performance gains are pretty significant too; they report outperforming that direct baseline by up to eight point seventy-two in accuracy on VideoMME, and they achieve state-of-the-art performance on VideoMME using just up to five percent of the tokens per frame required by baselines.

Lu: They also reveal this "Visual Lag" phenomenon during cube-based video perception, where terminal frames from preceding scenes get incorporated into cubes that contain subsequent event scenes, which they say helps the model retain partial memory.

Tom: That sounds like a clever way to handle long-term context without needing to keep every frame's full detail in the active processing window.

Jane: And they validate this understanding qualitatively across lengthy documentary and sports videos, as well as tasks like traffic sign recognition and even physics problem solving. That shows it’s not just about summarizing; it’s about deep comprehension.

Lalam: I also see that their inference capabilities are quite robust; they can query specific events, like the time period of an attack or if a penguin was injured, and handle complex reasoning tasks too.

Meng: For deployment, the ablation study is important. They found that the optimal configuration included three dee positional encoding and Gumbel noise annealing with a zero point zero zero one penalty weight along with full trainable parameters <ref:2504.15270#pg1>.

Tom: And they confirm that adding that three dee positional encoding actually gives a consistent accuracy improvement of plus three point fifteen percent, which is solid evidence for its benefit <ref:2504.15270#pg1>.

Jane: So, looking at the whole thing—the title "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax"—it suggests a focus on making these massive models more efficient right from the beginning of their design.

Lu: It really is about finding a way to manage that inherent temporal redundancy in video streams without losing the detailed information that an LMM needs to understand complex scenes.

Meng: For someone who actually has to run this in production, the fact that they need extensive training data, even though they only use zero point eight million total video-text samples for training, is something I'd want to know more about <ref:2504.15270#pg1>.

Tom: They say they also developed an infrastructure that allows for dynamic sequence lengths to support training on ultra-long videos and this is a practical addition for scaling up.

Jane: Ultimately, the implication here is that we might see video models become much more efficient when they start learning how to selectively compress their input based on the actual content of the video stream.

Conclusion: Tom: So, Quicksviewer is an LMM that cuts down on video processing time by compressing the video into cubes based on how much action is actually happening in those parts, using this Gumbel Softmax thing to do it.

Jane: It sounds like they took a huge problem—video isn't uniform—and gave it a dynamic solution, partitioning the video into manageable pieces so you only spend big processing power where it matters.

Tom: The authors are focusing on how to make these massive models efficient right from the start, with this early compression method baked into the design itself.

Jane: That means instead of feeding every single pixel or frame into the model all at once, you're using this smarter partitioning to save a ton of computation for just the right moments.

Tom: It suggests that future video models don't have to be giant compute hogs just because they handle long videos, if they can learn to ignore the boring parts intelligently.

Jane: But it raises a big question about what kind of understanding you lose when you compress it down like that; does the model miss subtle things in the compressed cubes?

Tom: That’s exactly what we need to figure out—if that compression is too aggressive, does the video comprehension break down?

Jane: Well, we'll see how their results hold up when they show us these SOTA scores on benchmarks like VideoMME.

More episodes

← Home