nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

arXiv:2605.31572 · cs.CV · Submitted 2026-05-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving".

Jane: Here's the gist of the paper. Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions The dataset contains 20K clips, each 20 seconds long, collected across multiple cities, with synchronized multi-camera images, LiDAR data, HD maps, object annotations, and human-verified reasoning annotations spanning Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning Models trained with reasoning annotations in our dataset achieve significantly better reasoning and planning performance Dataset Introduction nuReasoning is introduced as a large-scale real-world dataset and benchmark for reasoning and end-to-end learning in AD, following the lineage of nuScenes

42: and nuPlan

47, 62: The dataset comprises 20K real-world driving clips, each 20 seconds long at 10 Hz, mined from large-scale driving logs to maximize scenario diversity and difficulty Each clip includes synchronized multi-modal inputs, including multi-camera images, LiDAR point clouds, HD maps, and object annotations Furthermore, it provides three complementary reasoning annotations: Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning The dataset contains 20K clips which are divided into training, validation, and private test splits Annotation Methodology The reasoning annotations are organized into three categories: Spatial, Decision, and Counterfactual Spatial reasoning describes the scene and the surrounding agents using both image observations from each camera view and 3D geometric information in the ego-centric coordinate system Decision reasoning explains the intended behavior of the ego vehicle and the basis for that behavior under the observed scene context Counterfactual reasoning captures the consequences of alternative decisions other than ground-truth actions Spatial reasoning is annotated from 3 seconds into the clip at 1 Hz to track dynamic scene geometry, while Decision and counterfactual reasoning are annotated from 5 seconds before the keyframe at 0.2 Hz Benchmark Design The nuReasoning Benchmark evaluates both reasoning accuracy and planning performance, providing a unified testbed for VLM and VLA systems The benchmark groups question types by the driving capabilities they assess into Geometry, Motion, Driving, and Counterfactual categories Evaluation metrics include choice accuracy for multiple-choice questions, tolerance-based accuracy for scalar numerical questions, coordinate hit rate and L2 error for coordinate questions, and trajectory hit rate and mean L2 error for trajectory prediction The planning score (NPS) is defined as a weighted combination of safety-gated metrics such as non-at-fault collision (sNC), driveable-area compliance (sDA), ego route progress (sEP), comfort (sCF), and human-likeness (sHL) Experimental Results Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific visual question answering, especially for spatial grounding, decision-making, and counterfactual assessment Incorporating reasoning supervision into VLA training further improves planning performance even when textual reasoning outputs are disabled at inference time The full nuVLA model obtains the highest NPS of 73.09 and the lowest ADE of 1.

Paper summary: Tom: So looking at nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving Zhiyu Huang et al., the main thing is they put together a massive dataset of twenty thousand real-world clips with rich reasoning annotations > <ref:2605.31572#pg1>

Jane: And their core contribution is proving that training models with these specific reasoning skills leads to much better performance in both how they understand the world and how they plan actions >

Lu: The authors are essentially showing that spatial reasoning, decision reasoning, and counterfactual reasoning are not just academic concepts; they are concrete things that help AI systems navigate complex driving situations >

Meng: For those of us building models, it means we need to start thinking about incorporating these types of supervision into our training pipelines instead of just focusing on raw perception metrics >

Jane: And ultimately, this paper suggests that for AI to drive safely in the long tail of driving, it needs this kind of deep reasoning embedded in its training data >

Conclusion: Tom: So, to wrap up this look at nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving, we're talking about how they set out a new way to test autonomous driving systems.

Jane: They didn't just build another dataset of pictures and sensor data; they added these specific reasoning layers—spatial, decision, and counterfactual—to every single clip.

Lu: What this means is that we’re moving past models that are good at seeing a car or predicting where it will go on the next second. We're demanding they understand *why* things move the way they do and *what happens* if they choose a different path.

Meng: For us in production, the real question is how much of this extra reasoning supervision we need to actually deploy something that doesn't break when it hits an unexpected road situation.

Tom: The results show that fine-tuning vision models on this data gives a serious boost to their ability to ground those high-level decisions and plan for the future.

Jane: It’s about building systems that can handle the messy, unpredictable stuff that happens when you're actually driving in the real world, not just in a simulation.

Lu: The authors are arguing that this structured supervision is what helps AI learn common sense navigation in those long-tail scenarios where standard datasets fall short.

Tom: So, it’s a call for us to shift our focus toward teaching AI not just perception, but the actual reasoning process behind those actions.

Jane: And that leads us to thinking about how we can structure these training pipelines moving forward.

University of California, Los Angeles

cs.CV

Submitted: 2026-05-29

Updated: 2026-11-04

Comments: NeurIPS 2026 (Evaluations and Datasets Track). Project page: https://nureasoning.github.io/

Project page: https://nureasoning.github.io/Abstract

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions…

Key concepts

nuReasoning Dataset
This is a large collection of 20K real-world driving clips, each 20 seconds long, collected across multiple cities. It includes synchronized multi-modal data such as camera images, LiDAR point clouds, HD maps, and object annotations to provide diverse and challenging scenarios for training autonomous driving systems.
Spatial Reasoning Annotation
This annotation describes the scene and surrounding agents using both visual observations from cameras and 3D geometric information in the ego vehicle's coordinate system. It is tracked dynamically over 3 seconds into each clip at a rate of one point per second to capture changing scene geometry.
Decision Reasoning Annotation
This annotation explains the intended behavior of the ego vehicle and the underlying basis for that behavior within a specific scene context. It is annotated from 5 seconds before a keyframe at 0.2 Hz, helping models understand why a vehicle chooses a particular action.
Counterfactual Reasoning Annotation
This captures the potential consequences of alternative decisions that are different from the ground-truth actions taken in the clip. It helps train models to assess how different choices would have impacted the outcome of a driving situation.

Terminology

Summary

Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions The dataset contains 20K clips, each 20 seconds long, collected across multiple cities, with synchronized multi-camera images, LiDAR data, HD maps, object annotations, and human-verified reasoning annotations spanning Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning Models trained with reasoning annotations in our dataset achieve significantly better reasoning and planning performance

Dataset Introduction

nuReasoning is introduced as a large-scale real-world dataset and benchmark for reasoning and end-to-end learning in AD, following the lineage of nuScenes [42] and nuPlan [47, 62] The dataset comprises 20K real-world driving clips, each 20 seconds long at 10 Hz, mined from large-scale driving logs to maximize scenario diversity and difficulty Each clip includes synchronized multi-modal inputs, including multi-camera images, LiDAR point clouds, HD maps, and object annotations Furthermore, it provides three complementary reasoning annotations: Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning The dataset contains 20K clips which are divided into training, validation, and private test splits

Annotation Methodology

The reasoning annotations are organized into three categories: Spatial, Decision, and Counterfactual Spatial reasoning describes the scene and the surrounding agents using both image observations from each camera view and 3D geometric information in the ego-centric coordinate system Decision reasoning explains the intended behavior of the ego vehicle and the basis for that behavior under the observed scene context Counterfactual reasoning captures the consequences of alternative decisions other than ground-truth actions Spatial reasoning is annotated from 3 seconds into the clip at 1 Hz to track dynamic scene geometry, while Decision and counterfactual reasoning are annotated from 5 seconds before the keyframe at 0.2 Hz

Benchmark Design

The nuReasoning Benchmark evaluates both reasoning accuracy and planning performance, providing a unified testbed for VLM and VLA systems The benchmark groups question types by the driving capabilities they assess into Geometry, Motion, Driving, and Counterfactual categories Evaluation metrics include choice accuracy for multiple-choice questions, tolerance-based accuracy for scalar numerical questions, coordinate hit rate and L2 error for coordinate questions, and trajectory hit rate and mean L2 error for trajectory prediction The planning score (NPS) is defined as a weighted combination of safety-gated metrics such as non-at-fault collision (sNC), driveable-area compliance (sDA), ego route progress (sEP), comfort (sCF), and human-likeness (sHL)

Experimental Results

Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific visual question answering, especially for spatial grounding, decision-making, and counterfactual assessment Incorporating reasoning supervision into VLA training further improves planning performance even when textual reasoning outputs are disabled at inference time The full nuVLA model obtains the highest NPS of 73.09 and the lowest ADE of 1. p. 6

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reasoning supervision in Vision-Language-Action (VLA) training improves planning performance even when textual reasoning outputs are disabled at inference time, establishing reasoning annotations as useful intermediate supervision for learning better planning representations.

  2. Fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, especially for spatial grounding, decision-making, and counterfactual assessment.

  3. Incorporating reasoning supervision into VLA training further improves planning performance by leveraging structured reasoning to learn better planning representations even when explicit reasoning is not generated at inference time.

  4. The nuReasoning benchmark provides a unified testbed for VLMs and VLA systems by evaluating both reasoning accuracy and planning performance, enabling a direct study of how reasoning supervision affects driving performance.

  5. Models trained with reasoning annotations in the nuReasoning dataset achieve significantly better reasoning and planning performance compared to models trained without such supervision.

Sources

Related papers