Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner

arXiv:2412.18086 · cs.RO, cs.AI, cs.CL, cs.GR, cs.LG · Submitted 2024-12-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner".

Dev: Here's the gist of the paper. Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework leverages Large Language Models to translate user-specified text descriptions into scripts, which are then used by a simulator like CARLA to produce traffic scenarios As the method can generate abundant safety-critical traffic scenarios, they are used as synthetic training data for motion planners Experiments show that motion planners trained with this data significantly outperform those trained solely on real-world data.

Rosa: Framework Overview The AutoSceneGen framework is a universal, general, and cost-effective method designed to automatically enhance the heterogeneity of traffic scenarios through scenario descriptions It leverages in-context learning (ICL) of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator The system architecture involves a filtering process that ensures simulator compatibility by replacing incompatible terms with those aligned to the simulator’s documented APIs Furthermore, the validator checks each API call against a documented list, replacing unsupported terms with approved synonyms or removing them to prevent errors The final step involves launching the simulator to run the generated configuration files How the training signal is built The method builds a training signal by generating abundant safety-critical traffic scenarios that are then used as synthetic training data for motion planners The generated scenarios are automatically categorized based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios The framework automates AV evaluation by efficiently generating traffic scenarios from user-defined descriptions, ensuring cost-effective simulation across diverse open-world environments The generated datasets are logically realistic and heterogeneous, which is demonstrated by running 41 AI-generated scenarios with varying numbers of vehicles and pedestrians within a single scenario logic Where the gains come from The primary gains stem from overcoming the limitations of manual scenario creation, which traditionally requires significant human effort The framework addresses challenges like low-resource scenarios, such as corner cases and accidents, by leveraging LLMs’ ICL capabilities to generate tailored configurations for rare scenarios The use of ICL eliminates the need for continuous fine-tuning or training of separate models for generation tasks The generated data, when combined with existing datasets like ApolloScapes, reduces displacement error by introducing valuable new scenarios This combination demonstrates that the original NGSIM dataset is augmented by the dataset collected via AutoSceneGen, indicating that the original NGSIM dataset is augmented by our dataset collected via AutoSceneGen Limitations The framework has several limitations, including performance, simulator integration, and data availability The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction If the chosen LLM lacks ICL capabilities or the simulator cannot handle complex scenarios, the framework’s utility is significantly reduced, making it difficult to generate corner cases The explainability of the generated scenario logic is limited to the explainability of the scenario descriptions and ICL examples If example data for ICL is insufficient, some simulators may not respond effectively, necessitating post-revision of the responses The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction AutoSceneGen is designed for generating traffic scenarios using foundation models The resulting scenarios can be simulated within virtual environments in simulators, within the real world, or even both concurrently The framework achieves the improvement of work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world It is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment AutoSceneGen harnesses the power of foundation models to simulate extensive arrays of traffic scenarios, thereby providing a robust testing environment for AVs.

Dev: The framework is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment References Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; et al.

Paper summary: Rosa: We're diving into how this AutoSceneGen framework actually works and what it claims about its results. They lay out the framework overview, which is designed to be universal and cost-effective by using scenario descriptions to automatically increase the heterogeneity of traffic situations.

Dev: It uses in-context learning of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batches inside a physics-based simulator. They even have a filtering process where they replace incompatible terms with things that match the simulator's documented APIs, and then a validator checks each call against a documented list to prevent errors.

Taro: That sounds like the system is trying to make sure whatever the AI generates actually works inside the simulator, which is important when you are building safety cases for something moving in the real world. I wonder how robust that API compatibility check actually is when dealing with complex accident reconstructions, though.

Rosa: It’s designed to ensure that because it’s generating scenarios automatically, they eliminate the need for manual creation and annotation of those safety-critical scenarios. They build a training signal by automatically categorizing these generated scenarios based on their descriptions, which means you don't have to annotate them downstream.

Dev: That automation is significant because it lets them generate logically realistic and heterogeneous datasets; they even ran forty-one AI-generated scenarios showing varying numbers of vehicles and pedestrians within a single scenario logic. That’s not just a few examples; that shows the diversity they are achieving.

Taro: So if the generation is this automated, does it really cover those low-resource scenarios they mentioned? Like corner cases or accidents, which are usually the hardest things to get in real data?

Rosa: That's where they claim their use of ICL helps overcome those limitations by generating tailored configurations for rare scenarios. The authors say that using ICL eliminates the need for continuous fine-tuning or training separate models just for the generation tasks.

Dev: And when you combine this generated data with existing datasets, like ApolloScapes, it actually reduces displacement error in the motion planners. They showed that their dataset augments things like the NGSIM dataset, indicating that they add valuable new scenarios to what was already there.

Conclusion: Rosa: So wrapping up this look at "Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner," the authors are essentially pointing toward a framework called AutoSceneGen that uses foundation models to create traffic scenarios. These scenarios can then be simulated in virtual environments, or even both virtually and in the real world simultaneously.

Dev: The main point they're making is that this framework significantly improves work efficiency when it comes to generating enough safety-critical scenarios needed for testing autonomous vehicles in an open world. It’s particularly useful for enhancing reliability testing of autonomous vehicles planned for real-world deployment.

Taro: From my side, what stands out is how they harness the power of these foundation models to simulate a huge array of traffic situations, which provides a very robust testing environment for AVs. I'm still thinking about the limitations they mentioned, though—their effectiveness really depends on the capabilities of both the Large Language Model and the simulator itself for tasks like scenario generation and accident reconstruction.

Rosa: Right, that’s right. The framework is designed to harness foundation models to simulate extensive arrays of traffic scenarios. This gives AV developers a way to create a very thorough testing environment before they put these systems out into public streets or complex environments.

Aizierjiang Aiersilan

University of Macau

cs.RO, cs.AI, cs.CL, cs.GR, cs.LG

Submitted: 2024-12-24

Updated: 2025-05-01

Comments: We are excited to announce that this paper has been accepted for oral presentation at the AAAI 2025 Main Conference. We are grateful for the insightful feedback from the reviewers and look forward to contributing to the discussions at AAAI

Journal ref: Proceedings of the AAAI Conference on Artificial Intelligence, 39(14), 14539-14547, 2025

DOI: 10.1609/aaai.v39i14.33593

Project page: https://ezharjan.github.io/AutoSceneGen

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework…

Key concepts

AutoSceneGen
A universal framework designed to automatically create varied traffic scenarios from simple text descriptions. It uses LLMs' in-context learning (ICL) to generate optimized configurations, which are then simulated in a physics-based environment to produce training data.
In-Context Learning (ICL)
A technique where a Large Language Model learns to perform new tasks based on examples provided directly within the prompt, without needing formal fine-tuning. This allows the LLM to generate specific traffic scenario configurations tailored to user text requests.
Scenario Generation for Training
The process of automatically creating a large volume of safety-critical traffic scenarios. These generated scenarios serve as synthetic training data, allowing motion planners to learn from rare events like accidents and corner cases efficiently.

Terminology

Summary

Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework leverages Large Language Models to translate user-specified text descriptions into scripts, which are then used by a simulator like CARLA to produce traffic scenarios As the method can generate abundant safety-critical traffic scenarios, they are used as synthetic training data for motion planners Experiments show that motion planners trained with this data significantly outperform those trained solely on real-world data. p. 0

Where the paper says it

To demonstrate the value of generated scenarios, we train existing motion planners on our synthetic data, real-world datasets, and a combination of both. Our experiments show that motion planners trained with our data significantly outperform those trained solely on realworld data, showing the usefulness of our synthetic data and the effectiveness of our data generation method.

Framework Overview

The AutoSceneGen framework is a universal, general, and cost-effective method designed to automatically enhance the heterogeneity of traffic scenarios through scenario descriptions It leverages in-context learning (ICL) of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator The system architecture involves a filtering process that ensures simulator compatibility by replacing incompatible terms with those aligned to the simulator’s documented APIs Furthermore, the validator checks each API call against a documented list, replacing unsupported terms with approved synonyms or removing them to prevent errors The final step involves launching the simulator to run the generated configuration files

How the training signal is built

The method builds a training signal by generating abundant safety-critical traffic scenarios that are then used as synthetic training data for motion planners The generated scenarios are automatically categorized based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios The framework automates AV evaluation by efficiently generating traffic scenarios from user-defined descriptions, ensuring cost-effective simulation across diverse open-world environments The generated datasets are logically realistic and heterogeneous, which is demonstrated by running 41 AI-generated scenarios with varying numbers of vehicles and pedestrians within a single scenario logic

Where the gains come from

The primary gains stem from overcoming the limitations of manual scenario creation, which traditionally requires significant human effort The framework addresses challenges like low-resource scenarios, such as corner cases and accidents, by leveraging LLMs’ ICL capabilities to generate tailored configurations for rare scenarios The use of ICL eliminates the need for continuous fine-tuning or training of separate models for generation tasks The generated data, when combined with existing datasets like ApolloScapes, reduces displacement error by introducing valuable new scenarios This combination demonstrates that the original NGSIM dataset is augmented by the dataset collected via AutoSceneGen, indicating that the original NGSIM dataset is augmented by our dataset collected via AutoSceneGen

Limitations

The framework has several limitations, including performance, simulator integration, and data availability The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction If the chosen LLM lacks ICL capabilities or the simulator cannot handle complex scenarios, the framework’s utility is significantly reduced, making it difficult to generate corner cases The explainability of the generated scenario logic is limited to the explainability of the scenario descriptions and ICL examples If example data for ICL is insufficient, some simulators may not respond effectively, necessitating post-revision of the responses The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction

Conclusions

AutoSceneGen is designed for generating traffic scenarios using foundation models The resulting scenarios can be simulated within virtual environments in simulators, within the real world, or even both concurrently The framework achieves the improvement of work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world It is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment AutoSceneGen harnesses the power of foundation models to simulate extensive arrays of traffic scenarios, thereby providing a robust testing environment for AVs. The framework is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment

References

Achiam, J.; Adler, S. p. 7

Where the paper says it

References Achiam, J. ; Adler, S.

; Agarwal, S. p. 7
Where the paper says it

; Adler, S. ; Agarwal, S.

; Ahmad, L. p. 7
Where the paper says it

; Agarwal, S. ; Ahmad, L.

; Akkaya, I. p. 7
Where the paper says it

; Ahmad, L. ; Akkaya, I.

; et al. 2023. Gpt-4 technical report. p. 7
Where the paper says it

; Akkaya, I. Gpt-4 technical report.

arXiv preprint arXiv:2303. p. 7
Where the paper says it

Gpt-4 technical report. arXiv preprint arXiv:2303.

08774

Alam, M. S. 2020. Automatic generation of critical driving scenarios

Alinezhad Noghre, G.; Katariya, V. p. 7

Where the paper says it

Alinezhad Noghre, G. ; Katariya, V.

; Danesh Pazho, A. p. 7
Where the paper says it

; Katariya, V. ; Danesh Pazho, A.

; Neff, C.; and Tabkhi, H. p. 7
Where the paper says it

; Danesh Pazho, A. ; and Tabkhi, H.

2023. Pishgu: Universal path prediction network architecture for real-time cyber-physical edge systems

Amersbach, C.; and Winner, H. p. 0

Where the paper says it

Scenario-based evaluation methods have largely focused on coverage, unsafe scenarios, and indicator-estimation tests (Jesenski et al. 2019; Amersbach and Winner 2019).

2019. Defining required and feasible test coverage for scenario-based validation of highly automated vehicles

Awad, G.; Butt, A. A.; Curtis, K. p. 7

Where the paper says it

In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 425–430. ; Curtis, K.

; Lee, Y.; Fiscus, J. p. 7
Where the paper says it

; Curtis, K. ; Fiscus, J.

; Godil, A. p. 7
Where the paper says it

; Fiscus, J. ; Godil, A.

; Delgado, A. p. 7
Where the paper says it

; Godil, A. ; Delgado, A.

, et al. 2020. Trecvid 2019: An evaluation campaign to benchmark video activity detection

Bock, J.; Krajewski, R. p. 0

Where the paper says it

2023), Waymo (Ettinger et al. 2021), Round (Krajewski et al.

; Moers, T. p. 7
Where the paper says it

; Krajewski, R. ; Moers, T.

p. 7
Where the paper says it

; Krajewski, R. ; Moers, T.

; Runde, S. p. 7
Where the paper says it

; Moers, T. ; Runde, S.

; Vater, L. p. 7
Where the paper says it

; Runde, S. ; Vater, L.

p. 7
Where the paper says it

; Runde, S. ; Vater, L.

; and Eckstein, L. p. 7
Where the paper says it

; Vater, L. ; and Eckstein, L.

p. 7
Where the paper says it

; Vater, L. ; and Eckstein, L.

p. 7
Where the paper says it

; Vater, L. ; and Eckstein, L.

2020. The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections

Bogdoll, D.; Breitenstein, J. p. 7

Where the paper says it

Bogdoll, D. ; Breitenstein, J.

; Heidecker, F. p. 7
Where the paper says it

; Breitenstein, J. ; Heidecker, F.

; Bieshaar, M. p. 7
Where the paper says it

; Heidecker, F. ; Bieshaar, M.

; Sick, B.; Fingscheidt, T. p. 7
Where the paper says it

; Bieshaar, M. ; Fingscheidt, T.

; and Zollner, M. p. 7
Where the paper says it

; Fingscheidt, T. ; and Zollner, M.

2021. Description of corner cases in automated driving: Goals and challenges

Caesar, H.; Bankiti, V. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1023–1028. ; Bankiti, V.

; Lang, A. H.; Vora, S.; Liong, V. p. 7
Where the paper says it

; Bankiti, V. ; Liong, V.

E.; Xu, Q.; Krishnan, A. p. 7
Where the paper says it

; Liong, V. ; Krishnan, A.

, et al. 2020. nuscenes: A multimodal dataset for autonomous driving

Chandra, R.; Bhattacharya, U. p. 7

Where the paper says it

Chandra, R. ; Bhattacharya, U.

; Bera, A.; and Manocha, D. p. 7
Where the paper says it

; Bhattacharya, U. ; and Manocha, D.

p. 7
Where the paper says it

; Bhattacharya, U. ; and Manocha, D.

2019. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions

Chang, M.-F.; Lambert, J. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8483–8492. ; Lambert, J.

p. 7
Where the paper says it

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8483–8492. ; Lambert, J.

; Sangkloy, P. p. 7
Where the paper says it

; Lambert, J. ; Sangkloy, P.

; Singh, J. p. 7
Where the paper says it

; Sangkloy, P. ; Singh, J.

p. 7
Where the paper says it

; Sangkloy, P. ; Singh, J.

; Bak, S.; Hartnett, A. p. 7
Where the paper says it

; Singh, J. ; Hartnett, A.

; Wang, D., et al. 2019. Argoverse: 3d tracking and forecasting with rich maps

Dai, D.; Sun, Y.; Dong, L.; Hao, Y.; Ma, S.; Sui, Z.; and Wei, F. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8748–8757. ; and Wei, F.

2022. Why can gpt learn in-context? p. 7
Where the paper says it

; and Wei, F. Why can gpt learn in-context?

language models implicitly perform gradient descent as meta-optimizers

Deng, Y.; Yao, J.; Tu, Z.; Zheng, X. p. 7

Where the paper says it

arXiv preprint arXiv:2212. ; Zheng, X.

; Zhang, M. p. 0
Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

; and Zhang, T. p. 0
Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

2023. Target: Traffic rule-based test generation for autonomous driving systems

Ding, W.; Xu, C.; Arief, M. p. 7

Where the paper says it

arXiv preprint arXiv:2305. ; Arief, M.

; Lin, H.; Li, B.; and Zhao, D. p. 7
Where the paper says it

; Arief, M. ; and Zhao, D.

2023. A survey on safety-critical driving scenario generation—A methodological perspective

Dosovitskiy, A.; Ros, G.; Codevilla, F. p. 7

Where the paper says it

Dosovitskiy, A. ; Codevilla, F.

; Lopez, A. p. 7
Where the paper says it

; Codevilla, F. ; Lopez, A.

; and Koltun, V. p. 7
Where the paper says it

; Lopez, A. ; and Koltun, V.

2017. CARLA: An open urban driving simulator

Ettinger, S.; Cheng, S. p. 7

Where the paper says it

Ettinger, S. ; Cheng, S.

; Caine, B. p. 7
Where the paper says it

; Cheng, S. ; Caine, B.

; Liu, C.; Zhao, H.; Pradhan, S. p. 7
Where the paper says it

; Caine, B. ; Pradhan, S.

; Chai, Y., et al. 2021. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset

Feng, L.; Li, Q.; Peng, Z.; Tan, S.; and Zhou, B. p. 7

Where the paper says it

In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9710–9719. ; and Zhou, B.

p. 7
Where the paper says it

In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9710–9719. ; and Zhou, B.

2023. Trafficgen: Learning to generate diverse and realistic traffic scenarios

Fremont, D. J.; Dreossi, T. p. 7

Where the paper says it

Fremont, D. ; Dreossi, T.

; Ghosh, S. p. 7
Where the paper says it

; Dreossi, T. ; Ghosh, S.

; Yue, X.; SangiovanniVincentelli, A. p. 7
Where the paper says it

; Ghosh, S. ; SangiovanniVincentelli, A.

L.; and Seshia, S. p. 7
Where the paper says it

; SangiovanniVincentelli, A. ; and Seshia, S.

A. 2019. Scenic: a language for scenario specification and scene generation

Geiger, A.; Lenz, P.; Stiller, C. p. 7

Where the paper says it

In Proceedings of the 40th ACM SIGPLAN conference on programming language design and implementation, 63–78. ; Stiller, C.

; and Urtasun, R. p. 7
Where the paper says it

; Stiller, C. ; and Urtasun, R.

2013. Vision meets robotics: The kitti dataset

Go, K.; and Carroll, J. p. 1

Where the paper says it

The concept of a “scenario” revolves around the “logic” governing objects within a scene. Go and Carroll (2004) defined a scenario as a detailed account involving actors, background details, objectives, and sequences of actions.

M. 2004. The blind men and the elephant

Goyal, S.; Griggio, A. p. 7

Where the paper says it

Interactions, 11(6): 44–53. ; Griggio, A.

p. 7
Where the paper says it

Interactions, 11(6): 44–53. ; Griggio, A.

; Kimblad, J. p. 7
Where the paper says it

; Griggio, A. ; Kimblad, J.

; and Tonetta, S. p. 7
Where the paper says it

; Kimblad, J. ; and Tonetta, S.

2023. Automatic Generation of Scenarios for System-level Simulation-based Verification of Autonomous Driving Systems

Huang, W.; Wang, K.; Lv, Y.; and Zhu, F. p. 7

Where the paper says it

arXiv preprint arXiv:2311. ; and Zhu, F.

2016. Autonomous vehicles testing methods review

Huang, X.; Cheng, X. p. 7

Where the paper says it

Ettinger, S. ; Cheng, S.

; Geng, Q.; Cao, B.; Zhou, D.; Wang, P., et al. 2018. The apolloscape dataset for autonomous driving

Jesenski, S.; Stellet, J. p. 7

Where the paper says it

Jesenski, S. ; Stellet, J.

E.; Schiegg, F. p. 7
Where the paper says it

; Stellet, J. ; Schiegg, F.

; and Zollner, J. p. 7
Where the paper says it

; Fingscheidt, T. ; and Zollner, M.

M., 2019. Generation of scenes in intersections for the validation of highly automated driving functions

Kalra, N.; and Paddock, S. p. 6

Where the paper says it

The framework achieves the improvement of the work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world, as well as the convenience of traffic accident reconstruction. As highlighted by Kalra and Paddock (2016), AVs face the challenge of needing to cover vast distances — potentially hundreds of millions to even billions of miles — to establish their reliability in terms of minimizing fatalities and injuries.

M. 2016. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability

Konig, L.; Heinzemann, C. p. 7

Where the paper says it

Transportation Research Part A: Policy and Practice, 94: 182–193. ; Heinzemann, C.

; Griggio, A.; Klauck, M. p. 7
Where the paper says it

; Griggio, A. ; Klauck, M.

; Cimatti, A. p. 7
Where the paper says it

; Klauck, M. ; ¨ Cimatti, A.

; Henze, F. p. 7
Where the paper says it

; ¨ Cimatti, A. ; Henze, F.

, et al. 2024. Towards Safe Autonomous Driving: Model Checking a Behavior Planner during Development

Kovvali, V. G.; Alexiadis, V. p. 0

Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

; and Zhang PE, L. p. 0
Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

2007. Videobased vehicle trajectory data collection

Krajewski, R.; Bock, J.; Kloeker, L. p. 8

Where the paper says it

Krajewski, R. ; Kloeker, L.

; and Eckstein, L. 2018. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems

Krajewski, R.; Moers, T.; Bock, J.; Vater, L.; and Eckstein, L. 2020. The round dataset: A drone dataset of road user trajectories at roundabouts in germany

Lerner, A.; Chrysanthou, Y. p. 0

Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

; and Lischinski, D. p. 0
Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

2007. Crowds by example

Li, D.; Liu, B.; Huang, Z. p. 0

Where the paper says it

2019), and MSRL (Wu et al. 2023), rely on real-world datasets like ApolloScapes (Huang et al.

; Hao, Q.; Zhao, D.; and Tian, B. p. 8
Where the paper says it

; Huang, Z. ; and Tian, B.

2023. Safe motion planning for autonomous vehicles by quantifying uncertainties of deep learning-enabled environment perception

Li, Q.; Peng, Z. M.; Feng, L.; Liu, Z.; Duan, C.; Mo, W.; and Zhou, B. 2024a. Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling

Li, X.; Liu, E.; Shen, T.; Huang, J. p. 0

Where the paper says it

2019), and MSRL (Wu et al. 2023), rely on real-world datasets like ApolloScapes (Huang et al.

; and Wang, F. p. 8
Where the paper says it

; Huang, J. ; and Wang, F.

-Y. 2024b. ChatGPT-based scenario engineer: A new framework on scenario generation for trajectory prediction

Ma, Y.; Zhu, X.; Zhang, S. p. 0

Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

; Yang, R.; Wang, W.; and Manocha, D. 2019. Trafficpredict: Trajectory prediction for heterogeneous traffic-agents

Majzik, I.; Semerath, O. p. 8

Where the paper says it

In Proceedings of the AAAI conference on artificial intelligence, volume 33, 6120–6127. ; Semerath, O.

; Hajdu, C. p. 8
Where the paper says it

; Semerath, O. ; Hajdu, C.

; Marussy, K. p. 8
Where the paper says it

; Hajdu, C. ; Marussy, K.

; Szatmari, Z. p. 8
Where the paper says it

; Marussy, K. ; Szatmari, Z.

; Micskei, Z. p. 8
Where the paper says it

; Szatmari, Z. ; Micskei, Z.

, et al. 2019. Towards system-level testing with coverage guarantees for autonomous vehicles

Mann, B.; Ryder, N. p. 8

Where the paper says it

In 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS). ; Ryder, N.

; Subbiah, M. p. 8
Where the paper says it

; Ryder, N. ; Subbiah, M.

; Kaplan, J. p. 8
Where the paper says it

; Subbiah, M. ; Kaplan, J.

; Dhariwal, P. p. 8
Where the paper says it

; Kaplan, J. ; Dhariwal, P.

; Neelakantan, A. p. 8
Where the paper says it

; Dhariwal, P. ; Neelakantan, A.

, et al. 2020. Language models are few-shot learners

Min, S.; Lyu, X.; Holtzman, A. p. 8

Where the paper says it

arXiv preprint arXiv:2005. ; Holtzman, A.

; Artetxe, M. p. 8
Where the paper says it

; Holtzman, A. ; Artetxe, M.

; Lewis, M. p. 8
Where the paper says it

; Artetxe, M. ; Lewis, M.

; Hajishirzi, H. p. 8
Where the paper says it

; Lewis, M. ; Hajishirzi, H.

; and Zettlemoyer, L. p. 8
Where the paper says it

; Hajishirzi, H. ; and Zettlemoyer, L.

2022. Rethinking the role of demonstrations: What makes in-context learning work

Oh, S.; Hoogs, A. p. 8

Where the paper says it

arXiv preprint arXiv:2202. ; Hoogs, A.

; Perera, A. p. 8
Where the paper says it

; Hoogs, A. ; Perera, A.

; Cuntoor, N. p. 8
Where the paper says it

; Perera, A. ; Cuntoor, N.

; Chen, C.-C.; Lee, J. T., et al. 2011. A large-scale benchmark dataset for event recognition in surveillance video

Pellegrini, S.; Ess, A.; Schindler, K. p. 8

Where the paper says it

Pellegrini, S. ; Schindler, K.

; and Van Gool, L. p. 8
Where the paper says it

; Schindler, K. ; and Van Gool, L.

2009. You’ll never walk alone: Modeling social behavior for multitarget tracking

Shah, S.; Dey, D.; Lovett, C. p. 8

Where the paper says it

In 2009 IEEE 12th international conference on computer vision, 261–268. ; Lovett, C.

; and Kapoor, A. p. 8
Where the paper says it

; Lovett, C. ; and Kapoor, A.

2018. AirSim: High-fidelity visual and physical simulation for autonomous vehicles

Shet, R.; Iyer, N. C.; Mirje, M. p. 8

Where the paper says it

Cham: Springer International Publishing. ; Mirje, M.

; Bikkannavar, K. p. 8
Where the paper says it

; Mirje, M. ; Bikkannavar, K.

V.; and Rokhade, S. p. 8
Where the paper says it

; Bikkannavar, K. ; and Rokhade, S.

2023. Path Planning of Autonomous Vehicle for Real World Scenario Using CARLA

Sun, H.; Feng, S.; Yan, X.; and Liu, H. p. 8

Where the paper says it

In International Conference on Information and Communication Technology for Competitive Strategies, 81–92. ; and Liu, H.

X., 2021. Corner case generation and analysis for safety assessment of autonomous vehicles

Sun, S.; Gu, Z.; Sun, T.; Sun, J.; Yuan, C.; Han, Y.; Li, D., and Ang, M. p. 8

Where the paper says it

Transportation research record, 2675(11): 587– 600. ; and Ang, M.

H., 2024. Drivescenegen: Generating diverse and realistic driving scenarios from scratch

Tan, S.; Ivanovic, B. p. 8

Where the paper says it

IEEE Robotics and Automation Letters. ; Ivanovic, B.

; Weng, X.; Pavone, M. p. 8
Where the paper says it

; Ivanovic, B. ; Pavone, M.

; and Kraehenbuehl, P. p. 8
Where the paper says it

; Pavone, M. ; and Kraehenbuehl, P.

2023. Language Conditioned Traffic Generation

Webson, A., and Pavlick, E. p. 1

Where the paper says it

However, doubts persist about whether LLMs truly understand prompts. For instance, Webson and Pavlick (2021) found that models performed similarly with irrelevant prompts, suggesting that improved performance might not equate to genuine comprehension.

2021. Do prompt-based models really understand the meaning of their prompts? p. 8
Where the paper says it

; and Pavlick, E. Do prompt-based models really understand the meaning of their prompts?

arXiv preprint arXiv:2109. p. 8
Where the paper says it

Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.

01247

Wilson, B.; Qi, W.; Agarwal, T. p. 7

Where the paper says it

; Adler, S. ; Agarwal, S.

; Lambert, J.; Singh, J.; Khandelwal, S. p. 8
Where the paper says it

; Singh, J. ; Khandelwal, S.

, et al. 2023. Argoverse 2: Next generation datasets for self-driving perception and forecasting

Wu, Y.; Wang, L.; Zhou, S.; Duan, J.; Hua, G.; and Tang, W. p. 8

Where the paper says it

arXiv preprint arXiv:2301. ; and Tang, W.

2023. Multi-stream representation learning for pedestrian trajectory prediction

Xinxin, Z.; Fei, L., and Xiangbin, W. p. 1

Where the paper says it

Related Work Scenario Generation and Gaps Existing works, such as LCTGen (Tan et al. 2023), CSG (Xinxin, Fei, and Xiangbin 2020), and TrafficGen (Feng et al.

2020. Csg: Critical scenario generation from real traffic accidents

Yu, Y.; Liu, S.; Jin, P. J.; Luo, X.; and Wang, M. p. 8

Where the paper says it

; Huang, J. ; and Wang, F.

2020. Multiplayer dynamic game-based automatic lane-changing decision model under mixed autonomous vehicle and humandriven vehicle environment

Zhang, X.; Tao, J.; Tan, K.; Torngren, M. p. 8

Where the paper says it

Transportation research record, 2674(11): 165–183. ; Torngren, M.

; Šanchez, J. M. G., et al. 2023. Finding critical scenarios for automated driving systems: A systematic mapping study

Zhang, Z.; Zhang, A. p. 0

Where the paper says it

2023), rely on real-world datasets like ApolloScapes (Huang et al. 2018), NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY (Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.

; Li, M.; and Smola, A. p. 8
Where the paper says it

; Zhang, A. ; and Smola, A.

2022.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Universal Scenario Generation Framework: The AutoSceneGen framework can automatically enhance traffic scenario heterogeneity through scenario descriptions, thereby accelerating simulation and testing processes by using in-context learning of Large Language Models (LLMs).

  2. Automated Data Categorization: The system can automatically categorize the generated scenarios based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios.

  3. Cost-Effective Scenario Creation: It can generate abundant safety-critical traffic scenarios using prompt-based, configurable, and AI-driven methods, which is more cost-effective than collecting real-world datasets such as Argoverse2 or nuScenes.

  4. Enhanced Motion Planner Training: Motion planners trained with the synthetic data generated by AutoSceneGen significantly outperform those trained solely on real-world data, showing the usefulness of the synthetic data and the effectiveness of our data generation method.

  5. Scenario Logic Definition: The framework leverages LLMs’ in-context learning capability to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator, reflecting the logic governing objects within a scene.

Abstract

Motion planning is a crucial component in autonomous driving. State-of-the-art motion planners are trained on meticulously curated datasets, which are not only expensive to annotate but also insufficient in capturing rarely seen critical scenarios. Failing to account for such scenarios poses a significant risk to motion planners and may lead to incidents during testing. An intuitive solution is to manually compose such scenarios by programming and executing a simulator (e.g., CARLA). However, this approach incurs substantial human costs. Motivated by this, we propose an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners. First, we represent traffic scenarios as scripts, which are then used by the simulator to generate traffic scenarios. Next, we develop a method that accepts user-specified text descriptions, which a Large Language Model translates into scripts using in-context learning. The output scripts are sent to the simulator that produces the corresponding traffic scenarios. As our method can generate abundant safety-critical traffic scenarios, we use them as synthetic training data for motion planners. To demonstrate the value of generated scenarios, we train existing motion planners on our synthetic data, real-world datasets, and a combination of both. Our experiments show that motion planners trained with our data significantly outperform those trained solely on real-world data, showing the usefulness of our synthetic data and the effectiveness of our data generation method. Our source code is available at https://ezharjan.github.io/AutoSceneGen.

Sources

Related papers