Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol
The Bench2Drive evaluation protocol has flaws that misrepresent driving models regarding comfort and behavior. The original comfort metric rewards standing still, suggesting comfort trades against speed rather than driving quality. The authors propose a distance-based metric to correct this bias and identify two navigation signal pitfalls: steering overruling and target point mismatch, which are crucial for fair cross-model comparison.