From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving
The study investigates how Vision-Language Models (VLMs) and vision-only encoders differ internally and whether these differences persist after end-to-end planning. Analysis shows that while policy learning creates shared features, non-transferable 'residual factors' remain. This leads to behavioral complementarity: VLMs excel in complex scenarios, allowing for the design of hybrid systems that combine the strengths of both models for better performance and efficiency.