Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax".
Paper summary: Tom: So we're looking at this paper, "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax," and the big idea is tackling the computational inefficiency of Large Multimodal Models when they try to process video.
Jane: Exactly. The paper argues that LMMs uniformly perceive every single frame in a video, which is just computationally wasteful when some moments are much more important or detailed than others.
Lu: That's right. Quicksviewer proposes a new way of perceiving video by partitioning it into cubes based on how dense the temporal information is within those segments.
Meng: So instead of treating every frame equally, you dynamically compress the video online by using this nonuniform density to decide what gets processed deeply and what gets compressed lightly.
Tom: Right, that's the core claim. They use a method called Gumbel Softmax to do this partitioning, which they describe as "cubing network" in their architecture.
Jane: And they call it simple and intuitive because it partitions the video based on the momentum of semantic feature differences between frames. That sounds like they're using what happens between frames to find key moments.
Lalam: From my perspective, this cubing mechanism is how the model identifies those important keyframes, which they call Fli, along with consecutive non-keyframe cubes making up a cube Qi plus one.
Tom: Okay, so the architecture has four main pieces: a visual encoder that makes tokens from frames, this cubing network to partition the video into cubes, a resampler to compress each slice or cube into a fixed number of tokens, and finally an LLM backbone that takes those visual tokens and some user prompt.
Jane: And what's interesting is how they train it. They start with three progressive stages, moving from training on image-text pairs to large-scale pretraining data, and finally using extensive video-text paired corpus for the robust video understanding part.
Lu: What really stands out is integrating the cubing network into the end-to-end training using Gumbel Softmax with an improved noise annealing mechanism. They say this substantially improves both training stability and effectiveness without needing boundary labels for that cubing layer itself.
Meng: I’m wondering about the practical side here. They mention achieving a forty-five times compression rate, which is huge, but how does that translate to actual deployment costs when you're dealing with ultra-long videos?
Tom: They say this nonuniform perception paradigm leads to a forty-five times compression rate while significantly reducing spatiotemporal redundancy. And they show the results are very solid compared to direct baseline methods employing a fixed partitioning strategy.
Jane: The performance gains are pretty significant too; they report outperforming that direct baseline by up to eight point seventy-two in accuracy on VideoMME, and they achieve state-of-the-art performance on VideoMME using just up to five percent of the tokens per frame required by baselines.
Lu: They also reveal this "Visual Lag" phenomenon during cube-based video perception, where terminal frames from preceding scenes get incorporated into cubes that contain subsequent event scenes, which they say helps the model retain partial memory.
Tom: That sounds like a clever way to handle long-term context without needing to keep every frame's full detail in the active processing window.
Jane: And they validate this understanding qualitatively across lengthy documentary and sports videos, as well as tasks like traffic sign recognition and even physics problem solving. That shows it’s not just about summarizing; it’s about deep comprehension.
Lalam: I also see that their inference capabilities are quite robust; they can query specific events, like the time period of an attack or if a penguin was injured, and handle complex reasoning tasks too.
Meng: For deployment, the ablation study is important. They found that the optimal configuration included three dee positional encoding and Gumbel noise annealing with a zero point zero zero one penalty weight along with full trainable parameters <ref:2504.15270#pg1>.
Tom: And they confirm that adding that three dee positional encoding actually gives a consistent accuracy improvement of plus three point fifteen percent, which is solid evidence for its benefit <ref:2504.15270#pg1>.
Jane: So, looking at the whole thing—the title "Quicksviewer: An LMM with Early Video Compression using Annealed Gumbel-Softmax"—it suggests a focus on making these massive models more efficient right from the beginning of their design.
Lu: It really is about finding a way to manage that inherent temporal redundancy in video streams without losing the detailed information that an LMM needs to understand complex scenes.
Meng: For someone who actually has to run this in production, the fact that they need extensive training data, even though they only use zero point eight million total video-text samples for training, is something I'd want to know more about <ref:2504.15270#pg1>.
Tom: They say they also developed an infrastructure that allows for dynamic sequence lengths to support training on ultra-long videos and this is a practical addition for scaling up.
Jane: Ultimately, the implication here is that we might see video models become much more efficient when they start learning how to selectively compress their input based on the actual content of the video stream.
Conclusion: Tom: So, Quicksviewer is an LMM that cuts down on video processing time by compressing the video into cubes based on how much action is actually happening in those parts, using this Gumbel Softmax thing to do it.
Jane: It sounds like they took a huge problem—video isn't uniform—and gave it a dynamic solution, partitioning the video into manageable pieces so you only spend big processing power where it matters.
Tom: The authors are focusing on how to make these massive models efficient right from the start, with this early compression method baked into the design itself.
Jane: That means instead of feeding every single pixel or frame into the model all at once, you're using this smarter partitioning to save a ton of computation for just the right moments.
Tom: It suggests that future video models don't have to be giant compute hogs just because they handle long videos, if they can learn to ignore the boring parts intelligently.
Jane: But it raises a big question about what kind of understanding you lose when you compress it down like that; does the model miss subtle things in the compressed cubes?
Tom: That’s exactly what we need to figure out—if that compression is too aggressive, does the video comprehension break down?
Jane: Well, we'll see how their results hold up when they show us these SOTA scores on benchmarks like VideoMME.
Ji Qi, Yuan Yao, Yushi Bai, Bin Xu, Juanzi Li, Zhiyuan Liu
Tsinghua University · National University of Singapore
cs.CV, cs.CL
Submitted: 2025-04-21
Updated: 2026-11-18
Project page: https://quicksviewer.github.io
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
The gist: Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal information density The paper presents Quicksviewer…
Key concepts
- Cubing Network
- This network partitions a video into NQ cubes by analyzing the momentum of semantic feature differences between consecutive frames. It calculates an i-th momentum representation using a 2-layer MLP and LayerNorm to quantify frame significance, forming cubes based on keyframes.
- Gumbel Softmax
- This technique is used to dynamically partition the video frames into varying cubes based on their temporal density. It allows the model to adapt its perception strategy online, compressing dense parts of the video more aggressively while preserving information in sparser sections.
- Visual Lag
- A phenomenon observed during cube-based perception where terminal frames from preceding events are included in cubes containing subsequent events. This mechanism helps the model retain partial memory of past scenes to better understand current scenes.
- Nonuniform Perception Paradigm
- Instead of treating all video frames equally, this approach partitions the video into cubes whose size and content vary based on the temporal density of semantic features. This allows for adaptive compression tailored to specific parts of the video.
Terminology
Summary
Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal information density The paper presents Quicksviewer, an LMM with new perceiving paradigm that partitions a video of nonuniform density into varying cubes using Gumbel Softmax This simple and intuitive approach dynamically compress video online based on its temporal density, significantly reducing spatiotemporal redundancy (overall 45× compression rate). p. 0
Model Architecture Components
The overall architecture of the presented model is shown in Figure 2 The model consists of four basic components: (1) a visual encoder fv(·) that encode image slices or video frames into visual tokens, (2) a cubing network fq(·) that partitions the video frames into NQ cubes, (3) a resampler fr(·) which compress each slice or cube into a fixed number of tokens, and (4) an LLM fl(·) which accept concatenation of visual tokens and user prompt for response generation. The cubing network partitions the video frames into nonuniform cubes based on the momentum of semantic feature differences between frames
Cubing Mechanism Details
Cubing It is natural to leverage the semantic difference of features to find keyframes The cubing network calculate the i th momentum representation as: ∆i = α(Fi − Fi−1) + (1 − α)∆i−1 A 2-layers MLP with LayerNorm (Ba et al., 2016) is then applied to the mean of visual tokens momentums to quantify the significance of the frame The keyframe Fli, along with its subsequent consecutive non-keyframes, forms the cube Qi+1 = [Fli, …, Fli+1). p. 3
Training Strategy and Optimization
We train our models with three progressive stages starting from LLM backbones Stage-3 primarily leverage extensive video-text paired corpus to train our models in this stage, enabling robust video understanding capabilities. p. 4 We integrate the learning of the cubing network into the end-to-end training of the LMM using the Gumbel Softmax (Jang et al. p. 1, 2016) method with an improved noise annealing mechanism. p. 1 The Gumbel noise annealing substantially improves both training stability and effectiveness. p. 7
Performance Gains and Analysis
Our model outperforms the direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in accuracy Quicksviewer achieves SOTA performance on VideoMME using just up to 5% of the tokens per frame required by baselines. p. 0 We reveal a Visual Lag
phenomenon during cube-based video perception of the model: terminal frames from preceding event scenes are incorporated into cubes containing subsequent event scenes. The median cube length approximates 5 frames based on the predefined FPQ. p. 6
Ablation Study Findings
The optimal configuration demonstrated superior performance, which included 3D positional encoding, Gumbel noise annealing with 0.001 penalty weight, and full trainable parameters. p. 7 The ablation study shows that the 3D positional encoding results in a consistent accuracy improvement of +3. p. 8 p. 815%. Comprehensive parameter training yields superior benchmark performance compared to fine-tuning only the ViT and resampler parameters. p. 8 p. 8
Qualitative Understanding Validation
Representative qualitative analysis shows that Quicksviewer effectively understands lengthy documentary and sports videos, as well as informative single and multiple images. The model demonstrates comprehensive video understanding by identifying the nature of the unexpected attack, precisely locating its temporal occurrence, and summarizing the eventual outcome. p. 8 It also validates image understanding capabilities across extensive scenarios including traffic sign recognition in driving environments and physics problem solving. p. 8
Related Work Context
MiniCPMv2.6 is a highly adaptable multimodal model, demonstrating proficiency in single-frame, multi-frame, and full-video comprehension. p. 8 p. 8 LongVA (Zhang et al. p. 5 p. 5, 2024a) and LongVila (Xue et al. p. 1 p. 1, 2024) introduce the Needle-in-a-Haystack
evaluation framework for long video comprehension. NewsVideoQA primarily deals with news footage, where text plays a crucial role in conveying information, necessitating models that integrate both visual and textual cues for accurate question answering. The paper introduces Quicksviewer as an LMM designed for efficient video understanding through a nonuniform perception paradigm. p. 0
Conclusion
In this paper, we introduced Quicksviewer, an LMM designed for efficient video understanding through a nonuniform perception paradigm. By dynamically partitioning videos into nonuniform cubes and applying adaptive resampling, our approach achieves a 45× compression rate while maintaining a consistent representation for both images and videos. p. 1 We demonstrated that integrating the cubing network into end-to-end training via Gumbel Softmax with an improved noise annealing mechanism, enables efficient learning without boundary labels. p. 1 Our model, trained on just 0. p. 68M videos, attains SOTA performance on VideoMME with significantly fewer tokens per frame than baseline methods. p. 5 We also developed an infra that allows dynamic sequence lengths to support training on ultra-long videos. p. 1
References
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization Bai, S. p. 0, Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report Buckler et al. p. 6 Exploiting temporal redundancy in live computer vision Chen, G. p. 0 H., Chen, S., Zhang, R. p. 1, Chen, J., Wu, X., Zhang, Z. p. 1, Chen, Z., Li, J., Wan, X., and Wang B. p. 1 Allava: Harnessing gpt4v-synthesized data for lite vision-language models Chen et al. p. 1 Sharegpt4video: Improving video understanding and generation with better captions Chen, L. p. 4, Wei, X., Li, J., Dong, X., Zhang, P. p. 1, Zang, Y., Chen, Z., Duan, H., Tang, Z., Yuan, L., et al. Longvila: Scaling long-context visual language models for long videos Cheng et al. p. 1 Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms Deepmind, G. p. 1 Gemini 2.0 Farré et al. p. 4 Finevideo Fu et al. p. 4 Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis Jang, E. p. 8, Gu, S., and Poole, B. Categorical reparameterization with gumbel-softmax Herrmann et al. p. 1 Channel selection using gumbel softmax Jin et al. p. 0 Chat-univi: Unified visual representation empowers large language models with image and video understanding Jang, Y. p. 1, Song, Y., Yu, Y., Kim, Y., and Kim G. Tgif-qa: Toward spatio-temporal reasoning in visual question answering Krishna et al. p. 8 Dense-captioning events in videos Laurençon et al. p. 4 Obelics: An open web-scale filtered dataset of interleaved image-text documents Lei et al. p. 4 Tvqa: Localized, compositional video question answering Li et al. p. 6 Llava-onevision: Easy visual task transfer Li et al. p. 4 Mvbench: A comprehensive multi-modal video understanding benchmark Tosh, G. p. 8, Mathew, M., Garcia, S., Karatzas, D., and Jawahar C. V. Reading between the lanes: Text videoqa on the road Touvron et al. p. 7 Llama: Open and efficient foundation language models Wang et al. p. 5 Internvideo2. 5: Empowering video mllms with long and rich context modeling Wenger, S. p. 1 Video redundancy coding in h. p. 0 263+ Xu et al. Msr-vtt: A large video description dataset for bridging video and language Xu et al. p. 0 Video question answering via gradually refined attention over appearance and motion Xue et al. p. 0 Longvila: Scaling long-context visual language models for long videos Yang et al. p. 1 Qwen2. 5 technical report Yao et al. p. 6 Minicpm-v: A gpt-4v level mllm on your phone Ye et al. p. 2 mplug-owl3: Towards long image-sequence understanding in multi-modal large language models Yu et al. p. 2 Activitynet-qa: A dataset for understanding complex web videos via question answering Zeng et al. p. 0 Leveraging video descriptions to learn video question answering Zhai et al. p. 1 Sigmoid loss for language image pre-training Zhang et al. p. 1 Videollama 3: Frontier multimodal foundation models for image and video understanding Zhang et al. p. 4 Long context transfer from language to vision Zhang et al. p. 1 Direct preference optimization of video large multimodal models from language model reward Zhang Y. p. 1, Li B., Liu H., Lee Y., Gui L., Fu D., Feng J., Liu Z., and Li C. Llava-next: A strong zero-shot video understanding model Zhao et al. p. 8 Towards video text visual question answering: Benchmark and baseline Zhou et al. p. 8 Mlvu: A comprehensive benchmark for multi-task long video understanding Zohar et al. p. 8 Apollo: An exploration of video understanding in large multimodal models
--- Page 1 ---
Quicksviewer: An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes Large Multimodal Models (LMMs) uniformly perceive video frames, creating computational inefficiency for videos with inherently varying temporal information density This paper presents Quicksviewer, an LMM with new perceiving paradigm that partitions a video of nonuniform density into varying cubes using Gumbel Softmax
How the training signal is built
We train our models starting from the LLM backbones through three progressive stages, each stage incorporating lengthy videos on average of 420s/1fps by benefiting from the efficient perception mechanism. Stage-1 utilizes both interleaved and captioning image-text corpuses, and video-text captioning corpus to train our models, establishing fundamental alignment between visual encoder and LLM backbones with in-context learning capabilities. p. 4 Stage-2 employs large-scale pretraining data, primarily consisting of image-text multi-task data, to pre-train models establishing general multimodal capabilities across broad visual scenarios. p. 4 Stage-3 primarily leverage extensive video-text paired corpus to train our models in this stage, enabling robust video understanding capabilities.
Where the gains come from
The nonuniform perception paradigm, which is solely driven by the properties of input video, together with the subsequent resampling enables an efficient video encoding with 45× compression reate. With only 0.8M total video-text samples for training, our model achieves SOTA performance on VideoMME (Fu et al. p. 1, 2024) using just up to 5% of the tokens per frame required by baselines. p. 1 Our approach achieves SOTA on MLVU while demonstrating competitive performance on MVbench, despite utilizing substantially less training data (only 28% of Video-MME samples). p. 5
Ablation Study Results
The optimal configuration: 3D positional encoding, Gumbel noise annealing with 0.001 penalty weight, and full trainable parameters, demonstrating superior performance. p. 7 The ablation study shows that the 3D positional encoding results in a consistent accuracy improvement of +3.15%. Comprehensive parameter training yields superior benchmark performance compared to fine-tuning only the ViT and resampler parameters.
Qualitative Analysis Insights
We reveal a Visual Lag
phenomenon during cube-based video perception of the model: terminal frames from preceding event scenes are incorporated into cubes containing subsequent event scenes. This mechanism enables the model to retain partial memory of preceding scenes to facilitate current scene understanding. p. 6 The model successfully processes lengthy documentary and sports videos, as well as informative single and multiple images. p. 9 We further validate our model’s image understanding capabilities across extensive scenarios including traffic sign recognition in driving environments and physics problem solving. p. 8 The model demonstrates dual capability of recognizing individual athletes in sports footage and logically summarizing dynamic game situations. p. 8
Inference Capabilities
Quicksviewer achieves SOTA performance on Video-MME under modest sequence lengths using just up to 5% of tokens per frame required by baselines. The model supports querying about specific events, such as the time period of an attack and whether penguins were injured. p. 9 It also performs complex reasoning tasks, such as identifying biological structures and specifying their computational counterparts in CS disciplines. p. 8
Related Works Context
The rapid progress of Multimodal Large Language Models (MLLMs) has greatly enhanced video comprehension capabilities MiniCPMv2.6 is a highly adaptable multimodal model, demonstrating proficiency in single-frame, multi-frame, and full-video comprehension. LongVA (Zhang et al., 2024a) and LongVila (Xue et al., 2024) introduce the Needle-in-a-Haystack
evaluation framework for long video comprehension.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Dynamic Video Compression for Efficient Understanding: The system can achieve
High Compression Rate (64×) during all training stages
by utilizing a cubing network that partitions video into nonuniform cubes based on semantic feature momentum, which reduces spatiotemporal redundancy byoverall 45× compression rate.
-
Large Receptive Field with High Efficiency: The model can handle long-context processing because the cubing network enables
Large Receptive Field (420 frames) with High Compression Rate (64×) during all training stages,
allowing for efficient learning on videos averaging420s/1fps.
-
Online Video Stream Processing: The system can perform online cube partitioning, which is feasible for practical scenarios where
videos typically originate from lengthy offline recordings or online video streams
by using a cubing network that can beconducted online in streaming scenarios.
-
Enhanced Training Stability via Noise Annealing: The training process can be stabilized and optimized by adding an annealing strategy to Gumbel noise, which
substantially improves both training stability and effectiveness
during later stages of training. -
Improved Performance with Progressive Stages: The system can attain State of the Art (SOTA) performance on VideoMME using only
0.8M total video-text samples for training,
demonstrating that the modeloutperforms the direct baseline employing a fixed partitioning strategy by a maximum of 8.72 in accuracy.
-
Robust Multimodal Alignment: The system establishes a consistent representation for both images and videos by integrating learning of the cubing network into end-to-end training using Gumbel Softmax, which
enables efficient learning on videos without boundary labels.
Sources
- Layer Normalization
- Qwen2.5-VL Technical Report
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis
- Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
- Categorical Reparameterization with Gumbel-Softmax
- TVQA: Localized, Compositional Video Question Answering
- LLaVA-OneVision: Easy Visual Task Transfer
- VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
- Decoupled Weight Decay Regularization
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
- Reading Between the Lanes: Text VideoQA on the Road
- LLaMA: Open and Efficient Foundation Language Models
- LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture
- InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
- Qwen2.5 Technical Report
Related papers
- Efficient All-in-One Weather Restoration using Spectral Harmonization
- Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment
- Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
- WithEveryone: Unified Planning and Identity Grounding for Group Image Generation
- SciFigure2Code: An AI-Reconstructed Benchmark for Scientific Figure-to-Code
- RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space