Visual Token Coding for Video Multimodal Large Language Models
summary
The gist
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC)…
In short
Visual Token Coding (VTC) proposes a video coding-inspired method to compress visual tokens for MLLMs by structuring them into semantic I and P frames. VTCDy enhances this with dynamic resolution, token allocation, and spatial coverage to achieve superior compression. Results show VTCDy maintains high performance even with tight token budgets, proving structured compression is effective for long videos.
Key concepts
- Visual Token Coding (VTC)
- This paradigm organizes visual tokens like video frames into semantic I-frames (for scene references) and P-frames (for changes). By comparing the current frame to a predicted reference, VTC identifies redundant information, allowing for targeted token pruning based on temporal residuals.
- Dynamic Resolution Input (DyRSO)
- Instead of using all high-resolution frames, DyRSO organizes the video into a multi-resolution Group of Pictures (GOP). This replaces some full-resolution input with lower-resolution source frames, effectively expanding the video's temporal coverage while keeping the visual encoding cost manageable.
- Dynamic Token Allocation (DyTA)
- Inspired by HEVC rate control, DyTA distributes a limited token budget across different frames. It allocates more tokens to frames that exhibit higher coding complexity or significant temporal changes, ensuring critical information is prioritized over less important ones.
- Spatial Coverage Top-K (SC-TopK)
- This technique divides the visual token grid into distinct regions. It selects tokens by prioritizing those in regions showing large changes while ensuring regional coverage remains intact. This balances capturing significant motion with maintaining a complete view of the scene.
Terminology used across episodes
This episode discusses
- Visual Token Coding for Video Multimodal Large Language Models · Paper Radio
- LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence
- Qwen2.5-VL Technical Report
- Qwen3-VL Technical Report
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
- AdaCodec: A Predictive Visual Code for Video MLLMs
- Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
- LLaVA-OneVision: Easy Visual Task Transfer
- VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
- Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing
- CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
- LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
- PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction
- ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
- Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling
- Long Context Transfer from Language to Vision
The paper
Visual Token Coding for Video Multimodal Large Language Models · Read on arXiv
Xiamen University
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this base paradigm, we further extend VTC with a set of dynamic budgeting designs towards more cost-effective token coding, termed VTC dy. To validate VTC and VTC dy, we apply them to three MLLMs and conduct experiments on a set of video understanding benchmarks. The experimental results show that as a base paradigm, VTC can achieve an average performance retention of 98.4% on Qwen3-VL under a 50% token compression ratio, and the enhanced VTC dy can further improve the retention to 100.3%, outperforming a set of compared methods. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is given in the supplementary materials.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Visual Token Coding for Video Multimodal Large Language Models".
Title and authors: Tom: So, to recap, they’re using video compression ideas—like those from HEVC—to structure how we process video tokens for these massive multimodal models called MLLMs.
Jane: Basically, instead of treating every single frame as an equal piece of data, they organize the visual information into something like intra frames and predictive frames.
Tom: Exactly. The core mechanism is predicting what comes next and then looking at the difference between what you predicted and what actually happened to see if that difference is just noise or actual new information.
Lu: They call this residual coding, where small residuals mean the content is already covered by the reference frame, so you can compress it more aggressively.
Jane: And that’s why they call it Visual Token Coding, or VTC. It’s about treating the video like a compressed file so the model only has to pay attention to what actually changes over time.
Tom: But they took that baseline and built VTCDy, which adds dynamic resolution input, token allocation based on complexity, and spatial coverage top-k for better results.
Lu: The point of those extra parts is that they can handle very long videos—like a whole movie—with only fifty percent of the tokens needed.
Jane: I mean fifty percent! And they show that even when you cut the budget way down to twenty five percent, you still keep almost ninety eight point eight percent of the performance.
Tom: That’s a big retention rate for token pruning, especially on Qwen3-VL.
Meng: And from a deployment standpoint, they managed to reduce latency significantly, cutting it by two and a half times on an RTX three thousand nine hundred and ten graphics card when only using twenty five percent of the budget.
Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.
Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just guessing which tokens are important.
Tom: It moves the conversation from just picking important frames to using temporal residuals to expose redundancy at a whole video level.
Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.
Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.
Lu: And they also found that performance remained stable even when they changed the threshold for semantic I frames between one point two and two point zero.
Meng: That’s a good sign; it means the method isn't overly sensitive to tiny adjustments in how we define what counts as a reference frame.
Tom: It moves the focus from just picking important frames to using temporal residuals to expose redundancy at a whole video level.
Meng: And that structural organization is what makes VTCDy work better than just randomly removing tokens by importance scores.
Jane: That opens up a lot of possibilities for how we build these systems if we can apply that video coding structure across other complex data types.
The paper's summary: Tom: So we’re looking at how they actually made VTCDy better than the baseline VTC, and it comes down to these three specific dynamic mechanisms they introduced.
Jane: It basically says that just having a basic I and P frame structure isn't enough for really demanding video tasks.
Lu: Right. They added Dynamic Resolution Input, or DyRSO, which means they’re not sticking to one fixed resolution across the whole video timeline anymore.
Tom: So instead of using one resolution everywhere, the system uses a multi-resolution organization along the timeline, building a group of pictures that might use lower resolution frames where things are stable.
Jane: That helps with temporal coverage without just increasing the visual encoding cost proportionally.
Meng: So if you’re watching something mostly static, you don't need high-resolution data for every single second, which is smart when thinking about running models on limited hardware.
Tom: And then there’s Dynamic Token Allocation, or DyTA.
Lu: This part is inspired by how HEVC handles bit budgets; it distributes a set token budget across all the frames based on how complex or dependent those frames are on each other.
Jane: So if a frame is super busy with action, it gets more of that precious visual token budget, while a simpler frame gets less.
Tom: That seems like a very practical way to manage resource usage in real-time video processing without needing the model itself to be retrained for every scenario.
Meng: If you’re deploying this, that adaptive allocation sounds like it will make the inference much more efficient at scale.
Lu: And finally, Spatial Coverage Top-K, or SC-TopK. This is about organizing the entire grid of visual tokens into different regions and picking tokens from those regions to keep coverage in every area while prioritizing the ones that show big changes across those areas.
Jane: That’s a way to ensure you don't miss important local details while still capturing the overall motion happening at a broader scale.
Tom: And I have to say, when I look at the ablation study results for those three parts, DyRSO really helps with fine-grained detail and testing on things like LVBench.
Lu: But SC-TopK actually produced the strongest combined result overall when they looked at their MLVU average score.
Meng: So even though it’s more complex, that combined approach seems to give the best performance metric for this video token pruning technique.
Jane: It really shows that these three components aren't just one thing doing all the work; they complement each other nicely to give you better results.
Tom: And I think what this means is that structured prediction and residual coding can effectively expose redundancy at the video level, not just frame by frame.
The paper's improvements: Tom: So to wrap up, this paper on Visual Token Coding for Video Multimodal Large Language Models basically shows that you can use video compression principles to structure how these models process visual information into tokens.
Jane: It means they’ve found a way to organize video data like an I and P frame system so the AI only focuses on the actual changes between frames, which saves a ton of computational effort.
Lu: Exactly. The idea is that by modeling the video as a sequence of semantic reference frames and residual changes, you can prune tokens based on whether they are redundant or actually new information at that specific time point.
Tom: It’s about structured prediction overriding just simple frame-by-frame importance scores with this temporal context.
Jane: I mean for someone listening, it’s like teaching the AI to stop wasting its attention on things that haven't changed in a long time, which is crucial when dealing with those long videos we see every day.
Meng: From a deployment view, if this works as well as they say without needing massive model retraining, that drastically lowers the cost of running these multimodal systems.
Tom: I gotta ask, though—what does it actually cost to run this VTCDy system in production?
Lu: Well, they showed a latency reduction from four thousand one hundred nine milliseconds down to one thousand six hundred forty-four milliseconds on an RTX three thousand nine hundred and ten when only using twenty five percent of the budget.
Jane: That’s a two point five times speedup, which is a big deal for real-time or near real-time applications.
Lalam: Yes, I'm sure about this application of residual coding to video tokens. It feels like the most direct way to handle the high dimensionality of visual data efficiently without sacrificing too much fidelity on complex scenes.
Lu: They also noted that the ablation study showed that DyRSO and SC-TopK work together best for getting the highest score on their MLVU average, which is a solid piece of evidence for their combined approach.
Jane: So, to summarize, this paper suggests that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.
Tom: It's about using dynamic resolution and temporal budgeting to achieve superior compression without needing additional model tuning on the core architecture.
Lu: And it’s a big step because it moves us toward better representation learning that understands video structure, not just pixel importance.
Meng: If they can keep this level of performance retention when cutting the budget in half, that opens up much larger applications for these models.
Jane: The limitation they mentioned is mostly related to the complexity of implementing those dynamic allocation and coverage mechanisms perfectly at scale, but the paper provides a clear path forward for how structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level.
Tom: So, for anyone working on long-video MLLMs right now, this VTCDy approach seems like it gives you a solid, implementable framework that doesn't require additional model tuning.
Meng: It’s a structured prediction and residual coding idea applied to the visual token space.
Jane: I think what this paper is doing is showing that video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets.
Conclusion: Tom: So we're looking at this paper Visual Token Coding for Video Multimodal Large Language Models now. It comes from Chenxin Fang and the team at Xiamen University, and they’re proposing a new method called VTC.
Jane: What does that actually mean for us in practical terms?
Tom: Basically, it means they're trying to figure out how to handle the massive amount of video data efficiently using a token budget.
Lu: The core idea is organizing the video into I and P frames, which is a structure from video compression.
Jane: So an I frame holds the main scene information, and the P frames only hold what changed since that reference frame.
Tom: And they use those differences between what they predicted versus what actually happened to estimate token redundancy in terms of tokens.
Meng: But they don't stop at just that baseline VTC; they build VTCDy, which is Visual Token Coding with Dynamic enhancements.
Jane: What are these new dynamic parts that make it better?
Tom: They introduce Dynamic Resolution Input, or DyRSO, which uses a multi-resolution Group of Pictures or GOP along the video timeline.
Lu: That means they can handle different resolutions depending on where you are in the video, expanding the temporal coverage without increasing visual encoding cost.
Jane: And then there's Dynamic Token Allocation, or DyTA.
Tom: Then they have Spatial Coverage Top-K, or SC-TopK. This part divides the entire token grid into regions to make sure they keep coverage in each area while prioritizing tokens that show large changes across those areas.
Lu: It's about balancing local detail with global motion information in the selection process.
Jane: So, if I’m driving and I need to understand a long sequence of video, this VTCDy approach lets the model focus its attention where it matters most temporally and spatially.
Tom: The results they show are pretty strong; VTCDy achieves an average performance retention of one point zero zero one percent when you use fifty percent of the token budget for Qwen3-VL.
Meng: And they still keep ninety seven point eight percent average performance even when you cut that token budget down to twenty five percent.
Jane: That’s quite a jump, keeping nearly ninety eight point eight percent accuracy even with a quarter of the tokens.
Lu: It suggests that this structured compression method is genuinely applicable to existing multimodal models without needing massive retraining or tuning on the model itself.
Jane: So, the qualitative analysis they did shows how this allocation works on a video timeline; for example, assigning about seven point seven percent to what’s almost a duplicate of previous content and nearly thirty percent to substantial content change.
Jane: It’s interesting how those three components—resolution, budget allocation, and spatial coverage—play complementary roles; they aren't just one thing doing all the heavy lifting.
Tom: And that adaptive allocation is especially important when the budget gets more restrictive, where you need to be smart about how you spend every single token.
Lu: What this really changes is that you don't need massive fine-tuning just to make these models handle longer video sequences efficiently under tight token constraints.
Jane: It means the structure of video compression itself provides a practical way to save tokens for long-form multimodal understanding, instead of just
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation