DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception

summary

Video file (mp4)

The gist

The gist DVD proposes a dynamic vector decoding method that unifies 2D and 3D perception tasks for efficient integration with MLLMs.

In short

The DVD method proposes dynamic vector decoding to unify 2D and 3D perception for efficient integration with Multimodal Large Language Models (MLLMs). It transforms diverse shapes like boxes and masks into a sequence of vectors, maps them to discrete tokens, and then decodes these tokens back into original representations. This achieves high localization accuracy while significantly reducing token overhead and inference latency.

Key concepts

Perceptual Representation
This involves converting different spatial data—such as 2D bounding boxes, 2D masks, or 3D boxes—into a simple sequence of numbers (a vector). For example, a 3D box is represented by its six dimensions (x, y, z, w, h, l), and for a mask, points are sampled along the edges. This step standardizes varied input formats into a format suitable for vector processing.
Dynamic Vector Representation Learning
This is an adaptive encoder-decoder architecture designed to learn how to map the sequences of perceptual vectors into a compact set of discrete tokens. The system learns this mapping end-to-end using a combined loss function that ensures accurate reconstruction, proper quantization, and faithful geometric structure preservation across different tasks.
Dynamic Vector Decoding
This is the inference step where the learned discrete tokens are converted back into the original 2D or 3D perceptual data. A lightweight de-tokenizer handles this process, allowing MLLMs to seamlessly integrate these compact tokens with their language understanding capabilities without losing the original spatial information.
Geometric Training Loss
This is a task-agnostic loss function based on the bidirectional chamfer distance used during training. It provides unified supervision for both 2D and 3D perception tasks by ensuring that the reconstructed geometric structures (like bounding boxes) are spatially accurate, regardless of whether they are in a 2D or 3D context.

Terminology used across episodes

This episode discusses

The paper

DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception · Read on arXiv

Jinghua Hou, *Zhe Liu*, ×Hengshuang Zhao–Corresponding author

The University of Hong Kong

Multimodal large language models have made remarkable progress in bridging vision and language, facilitating various perception tasks essential for human-machine interaction, robotics, and autonomous driving. However, existing MLLM-based perception methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. To address these challenges, we propose a dynamic vector decoding method named DVD, which unifies the representation of 2D and 3D perception tasks. Specifically, we first transform diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into 1D vector sequences, which are then mapped to compact discrete tokens in the high-dimensional space. Then, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. Extensive experiments on 2D and 3D perception benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes demonstrate that DVD achieves superior performance in 2D and 3D tasks and reduces significantly the token overhead and inference latency. DVD provides an efficient and general framework for integrating perception capabilities into MLLMs, overcoming the inherent limitations of existing methods.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception".

Jane: The gist DVD proposes a dynamic vector decoding method that unifies 2D and 3D perception tasks for efficient integration with MLLMs.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up this discussion on "DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception," this paper is focused on creating a generalizable way to handle both 2D and three dee perception tasks simultaneously within an MLLM <ref:2610.12266#pg1,DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception>.

Jane: The authors, Jinghua Hou, Zhe Liu, and Hengshuang Zhao from the University of Hong Kong put forward this dynamic vector decoding method as a solution to the issues of token overhead and fixed-range constraints in current perception representations.

Lu: The main implication is that we can move towards more efficient multimodal models that integrate vision and language by using compact discrete tokens instead of just text coordinates or fixed quantization.

Meng: For practical application, this suggests a pathway for building more capable models on the edge because of the speedups they showed in inference latency.

Tom: The core idea is transforming diverse perceptual data into these high-dimensional tokens and then decoding them back to the original 2D and three dee representations using a lightweight de-tokenizer <ref:2610.12266#pg1>.

Jane: It’s about unifying perception capabilities, which means MLLMs can handle more complex reasoning tasks that require understanding both spatial dimensions at once.

Lu: The title highlights the dynamic nature of the decoding, emphasizing that it can adapt to different perception needs without being locked into fixed ranges.

Meng: So in simple terms, it’s a way to make these large vision models smarter about how they perceive the world by using a more efficient token format.

Tom: That's the essence of "DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception."

Conclusion: Tom: So we've been digging into DVD, Dynamic Vector Decoding for Efficient MLLM-based Perception. Basically, this paper is about taking all those messy 2D and three dee perception data—boxes and masks—and squeezing them into these small, neat little tokens that an MLLM can actually handle without getting bogged down <ref:2610.12266#pg1>.

Jane: It’s about unifying the way we represent things in vision tasks. We're talking about turning different kinds of visual info into a single 1D vector sequence that then gets mapped to these discrete tokens <ref:2610.12266#pg1>.

Lu: The core innovation is that it uses a dynamic approach, not just some fixed codebook you train once. It learns the best representation on the fly for whatever perception task you're doing.

Meng: From an engineering standpoint, what's really cool is how they manage that geometric loss part. They make sure those tokens still reconstruct the actual shape of the object correctly, not just some fuzzy guess in latent space.

Lalam: Lalam sees this as huge for culture because if we can get these models to perceive 2D and three dee things this compactly, it means our future AI can be much more flexible and integrated into complex real-world robotics <ref:2610.12266#pg1>.

Tom: Right, so the main point is efficiency—reducing that token overhead you talked about earlier, making these massive vision models faster and leaner for everyday use.

Jane: And it’s not just about speed; it’s about making the representation cleaner, less reliant on fixed ranges that were causing problems in three dee work.

Lu: The authors show that with the right setup, they can get high precision localization in three dee without needing an infinite number of tokens.

Meng: I'm interested in how this plays out when we actually deploy it on something like a car sensor or a robot; does the decoding step add too much delay?

Tom: That’s what we look at next, Meng, because the paper shows some really solid numbers on inference time that make this feel practical right now.

Jane: Before we get into the specifics of those numbers, let's just wrap up on what they actually proposed for the title and who wrote it.

More episodes

← Home