DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception

The gist DVD proposes a dynamic vector decoding method that unifies 2D and 3D perception tasks for efficient integration with MLLMs.

Introduction and Motivation

Existing methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. These limitations raise the question: Can we achieve 2D and 3D perception representations with unlimited range and high precision using a small number of tokens? The authors draw inspiration from Vector Quantized Variational Autoencoders (VQ-VAE) to propose dynamic vector decoding (DVD) for MLLMs that can seamlessly unify the representation of 2D and 3D perception tasks. The core innovation lies in transforming diverse perception representations (2D boxes, 2D masks, and 3D boxes) into a 1D vector sequence and projecting the sequence into discrete tokens in high-dimensional space.

Methodology

The DVD framework involves several key steps for representation learning and decoding.

1. Perceptual Representation: Diverse perceptual representations including 2D bounding boxes, 2D masks, and 3D bounding boxes are converted into 1D vector sequences. For 2D boxes, the left-top and right-down corner points are flattened (x1, y1, x2, y2), and for masks, fixed points are sampled uniformly along the edge of the mask. For 3D boxes, standard structures (x, y, z, w, h, l) are used directly to ensure precise localization in 3D space.

2. Dynamic Vector Representation Learning: A 1D encoder-decoder architecture is designed to adaptively map these sequences into latent space and generate corresponding discrete tokens by learning a unified limited codebook with reconstruction of multiple perception tasks. The autoencoder is trained end-to-end with a combined loss function, which includes the reconstruction loss (LL1), VQ loss (LVQ), and geometric loss (Lgeo).

3. Geometric Training Loss: To ensure faithful recovery of geometric structures, a task-agnostic geometric loss based on the bidirectional chamfer distance is introduced to provide unified geometric supervision for both 2D and 3D perception tasks. The total loss is defined as Ltotal = λ1 · LL1 + λ2 · LVQ + λ3 · Lgeo.

4. Dynamic Vector Decoding: In inference, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. This process involves mapping discrete tokens to code vectors and using the pre-trained 1D decoder to reconstruct the original sequence.

Experimental Validation

Extensive experiments were conducted across various benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes for 3D grounding tasks and RefCOCO, RefCOCO+, and RefCOCOg for 2D grounding and referring expression segmentation (RES) tasks.

1. Performance Comparison: DVD-2B surpasses SOTA methods on 3D grounding with 40.0%, 15.7%, and 19.8% AP3D on SUN-RGBD [38], Hypersim [37], and nuScenes [7], and achieves 89.9% Precision and 70.4% cIoU on RefCOCOg val [22]. DVD-8B further reaches 40.3%, 18.4%, and 24.4% AP3D on the three 3D datasets, and achieves 88.7% Precision and 73.4% cIoU on RefCOCOg val [22].

2. Efficiency Gains: DVD significantly reduces token overhead and inference latency compared to text-based coordinate representation methods, demonstrating its superiority in both performance and efficiency within a unified framework. The latency comparison shows that DVD achieves 499 ms for Box3D, 143 ms for Box2D, and 8 tokens for the special token representation.

3. Reconstruction Quality: The quality of 1D vector reconstruction directly affects the performance of subsequent perception tasks, and the autoencoder achieves negligible reconstruction error (90.8% AP3D, 100.0% Precision, and 84.8% cIoU for 3D grounding, 2D grounding, and 2D RES task).

Ablation Study Findings

Ablation studies illustrated the effectiveness of various components of the DVD framework.

1. Codebook Size: The codebook size of 4096 achieves the best performance across all three tasks, and a small codebook (2048) limits representational capacity, while an overly large codebook (8192) leads to degraded reconstruction.

2. Geometric Loss: Incorporating geometric guidance consistently improves the reconstruction performance across all tasks, with particularly notable gains on 3D grounding (+10.1% AP3D) and 2D RES (+10.0% cIoU). This confirms that the geometric loss is an essential component for learning a unified vector representation.

3. Downsampling Scale: Setting K=2 reduces the token count by half while maintaining competitive performance, verifying that K=2 achieves the best trade-off between reconstruction precision and token cost.

4. Latent Hidden Size: Setting z=16 provides sufficient capacity with stable quantization, achieving the best overall performance, as a narrow latent bottleneck (z=8) caused a notable drop on the 3D grounding task.

5. Multi-task Training: Joint multi-task training improves performance on both 3D grounding task and 2D RES task, while the performance of 2D grounding remains stable without obvious degradation.

Conclusion and Broader Impacts

DVD provides a new generalizable paradigm for efficient MLLM-based perception by unifying heterogeneous perception representations into compact discrete tokens. This framework benefits autonomous driving and robotics by reducing overhead and unifying 2D and 3D perception tasks, while also reducing the difficulty of deploying large models on the edge. However, due to its powerful perception capabilities, the research could be misused in closed-circuit television surveillance systems and affect personal privacy. Future work will focus on optimizing the dynamic vector representation learning process and exploring end-to-end training of the encoder-decoder and MLLM.

--- Page 1 ---

DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception Jinghua Hou, Zhe Liu∗, Hengshuang Zhao† The University of Hong Kong https://almoonysl.github.io/projects/DVD

--- Page 2 ---

Abstract Multimodal large language models have made remarkable progress in bridging vision and language, facilitating various perception tasks essential for humanmachine interaction, robotics, and autonomous driving. However, existing MLLMbased perception methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. To address these challenges, we propose a dynamic vector decoding method named DVD, which unifies the representation of 2D and 3D perception tasks. Specifically, we first transform diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into 1D vector sequences, which are then mapped to compact discrete tokens in the high-dimensional space. Then, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. Extensive experiments on 2D and 3D perception benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes demonstrate that DVD achieves superior performance in 2D and 3D tasks and reduces significantly the token overhead and inference latency. DVD provides an efficient and general framework for integrating perception capabilities into MLLMs, overcoming the inherent limitations of existing methods.

--- Page 3 ---

1 Introduction Multimodal large language models (MLLMs) [2, 4, 3, 46, 52, 9, 10, 16] have achieved breakthroughs in bridging vision and language enabling a wide range of perception tasks that are critical for human-machine interaction robotics and autonomous driving. Many previous works [4, 3, 24, 55, 35, 13, 40, 21] integrate 2D or 3D perception capabilities into MLLMs to achieve more general perceptual understanding benefited from the powerful general understanding and reasoning capabilities of MLLMs yet existing approaches still face fundamental limitations in representation and quantization that hinder their scalability and performance especially in 3D domains.

1. textbfDynamic Vector Representation Unification for Perception Tasks: The improved system can unify 2D and 3D perception representations by converting diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into a 1D vector sequence. This allows the model to process heterogeneous data under a single learning framework, overcoming the limitations of existing methods that use separate decoders for each task type (task-specific decoders lack generality).

2. textbf Adaptive Compression and Quantization via Unified Codebook: The system utilizes a unified codebook in high-dimensional space learned through a 1D encoder-decoder architecture to map continuous 1D vectors into compact discrete tokens. This achieves compact and high precision representations with fewer tokens, delivering consistent latency reduction across 2D and 3D tasks.

3. textbf Geometric Fidelity via Task-Agnostic Loss: The improved system incorporates a task-agnostic geometric loss based on the bidirectional chamfer distance to ensure geometric accuracy. This loss addresses the weakness of standard losses by providing unified geometric supervision for both 2D and 3D tasks, which is crucial because the L1 reconstruction loss only measures per-dimension numerical differences of the flattened sequence, which is sensitive to point ordering and parameterization rather than the underlying geometry.

4. textbf Efficient MLLM Integration via Lightweight De-tokenizer: The system integrates seamlessly with existing MLLMs by using a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. This process is designed to be minimal computational overhead compared to text-based methods, drastically improving inference speed.

5. textbf Superior Localization in Challenging Scenarios: The system achieves high localization accuracy even under difficult conditions, as evidenced by qualitative results showing that DVD can achieve satisfactory results even in dark and complex 3D scenes and consistently achieves precise 3D localization in challenging environments like rainy weather, cluttered 3D scenes with dense objects.

Multimodal large language models have made remarkable progress in bridging vision and language, facilitating various perception tasks essential for human-machine interaction, robotics, and autonomous driving. However, existing MLLM-based perception methods predominantly rely on text-based coordinate representation, which suffers from excessive token overhead, or fixed-range quantization, which suffers from range and precision constraints, especially for 3D domains with unbounded spatial range and high localization accuracy requirements. To address these challenges, we propose a dynamic vector decoding method named DVD, which unifies the representation of 2D and 3D perception tasks. Specifically, we first transform diverse perceptual representation (i.e., 2D bounding boxes, 2D masks, and 3D bounding boxes) into 1D vector sequences, which are then mapped to compact discrete tokens in the high-dimensional space. Then, a lightweight de-tokenizer enables seamless integration with MLLMs by decoding output tokens back to original 2D and 3D perceptual representations. Extensive experiments on 2D and 3D perception benchmarks including RefCOCO series, SUN-RGBD, KITTI, Hypersim, nuScenes demonstrate that DVD achieves superior performance in 2D and 3D tasks and reduces significantly the token overhead and inference latency. DVD provides an efficient and general framework for integrating perception capabilities into MLLMs, overcoming the inherent limitations of existing methods.