Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner".
Dev: Here's the gist of the paper. Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework leverages Large Language Models to translate user-specified text descriptions into scripts, which are then used by a simulator like CARLA to produce traffic scenarios As the method can generate abundant safety-critical traffic scenarios, they are used as synthetic training data for motion planners Experiments show that motion planners trained with this data significantly outperform those trained solely on real-world data.
Rosa: Framework Overview The AutoSceneGen framework is a universal, general, and cost-effective method designed to automatically enhance the heterogeneity of traffic scenarios through scenario descriptions It leverages in-context learning (ICL) of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator The system architecture involves a filtering process that ensures simulator compatibility by replacing incompatible terms with those aligned to the simulator’s documented APIs Furthermore, the validator checks each API call against a documented list, replacing unsupported terms with approved synonyms or removing them to prevent errors The final step involves launching the simulator to run the generated configuration files How the training signal is built The method builds a training signal by generating abundant safety-critical traffic scenarios that are then used as synthetic training data for motion planners The generated scenarios are automatically categorized based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios The framework automates AV evaluation by efficiently generating traffic scenarios from user-defined descriptions, ensuring cost-effective simulation across diverse open-world environments The generated datasets are logically realistic and heterogeneous, which is demonstrated by running 41 AI-generated scenarios with varying numbers of vehicles and pedestrians within a single scenario logic Where the gains come from The primary gains stem from overcoming the limitations of manual scenario creation, which traditionally requires significant human effort The framework addresses challenges like low-resource scenarios, such as corner cases and accidents, by leveraging LLMs’ ICL capabilities to generate tailored configurations for rare scenarios The use of ICL eliminates the need for continuous fine-tuning or training of separate models for generation tasks The generated data, when combined with existing datasets like ApolloScapes, reduces displacement error by introducing valuable new scenarios This combination demonstrates that the original NGSIM dataset is augmented by the dataset collected via AutoSceneGen, indicating that the original NGSIM dataset is augmented by our dataset collected via AutoSceneGen Limitations The framework has several limitations, including performance, simulator integration, and data availability The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction If the chosen LLM lacks ICL capabilities or the simulator cannot handle complex scenarios, the framework’s utility is significantly reduced, making it difficult to generate corner cases The explainability of the generated scenario logic is limited to the explainability of the scenario descriptions and ICL examples If example data for ICL is insufficient, some simulators may not respond effectively, necessitating post-revision of the responses The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction AutoSceneGen is designed for generating traffic scenarios using foundation models The resulting scenarios can be simulated within virtual environments in simulators, within the real world, or even both concurrently The framework achieves the improvement of work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world It is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment AutoSceneGen harnesses the power of foundation models to simulate extensive arrays of traffic scenarios, thereby providing a robust testing environment for AVs.
Dev: The framework is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment References Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; et al.
Paper summary: Rosa: We're diving into how this AutoSceneGen framework actually works and what it claims about its results. They lay out the framework overview, which is designed to be universal and cost-effective by using scenario descriptions to automatically increase the heterogeneity of traffic situations.
Dev: It uses in-context learning of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batches inside a physics-based simulator. They even have a filtering process where they replace incompatible terms with things that match the simulator's documented APIs, and then a validator checks each call against a documented list to prevent errors.
Taro: That sounds like the system is trying to make sure whatever the AI generates actually works inside the simulator, which is important when you are building safety cases for something moving in the real world. I wonder how robust that API compatibility check actually is when dealing with complex accident reconstructions, though.
Rosa: It’s designed to ensure that because it’s generating scenarios automatically, they eliminate the need for manual creation and annotation of those safety-critical scenarios. They build a training signal by automatically categorizing these generated scenarios based on their descriptions, which means you don't have to annotate them downstream.
Dev: That automation is significant because it lets them generate logically realistic and heterogeneous datasets; they even ran forty-one AI-generated scenarios showing varying numbers of vehicles and pedestrians within a single scenario logic. That’s not just a few examples; that shows the diversity they are achieving.
Taro: So if the generation is this automated, does it really cover those low-resource scenarios they mentioned? Like corner cases or accidents, which are usually the hardest things to get in real data?
Rosa: That's where they claim their use of ICL helps overcome those limitations by generating tailored configurations for rare scenarios. The authors say that using ICL eliminates the need for continuous fine-tuning or training separate models just for the generation tasks.
Dev: And when you combine this generated data with existing datasets, like ApolloScapes, it actually reduces displacement error in the motion planners. They showed that their dataset augments things like the NGSIM dataset, indicating that they add valuable new scenarios to what was already there.
Conclusion: Rosa: So wrapping up this look at "Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner," the authors are essentially pointing toward a framework called AutoSceneGen that uses foundation models to create traffic scenarios. These scenarios can then be simulated in virtual environments, or even both virtually and in the real world simultaneously.
Dev: The main point they're making is that this framework significantly improves work efficiency when it comes to generating enough safety-critical scenarios needed for testing autonomous vehicles in an open world. It’s particularly useful for enhancing reliability testing of autonomous vehicles planned for real-world deployment.
Taro: From my side, what stands out is how they harness the power of these foundation models to simulate a huge array of traffic situations, which provides a very robust testing environment for AVs. I'm still thinking about the limitations they mentioned, though—their effectiveness really depends on the capabilities of both the Large Language Model and the simulator itself for tasks like scenario generation and accident reconstruction.
Rosa: Right, that’s right. The framework is designed to harness foundation models to simulate extensive arrays of traffic scenarios. This gives AV developers a way to create a very thorough testing environment before they put these systems out into public streets or complex environments.
Aizierjiang Aiersilan
University of Macau
cs.RO, cs.AI, cs.CL, cs.GR, cs.LG
Submitted: 2024-12-24
Updated: 2025-05-01
Comments: We are excited to announce that this paper has been accepted for oral presentation at the AAAI 2025 Main Conference. We are grateful for the insightful feedback from the reviewers and look forward to contributing to the discussions at AAAI
Journal ref: Proceedings of the AAAI Conference on Artificial Intelligence, 39(14), 14539-14547, 2025
DOI: 10.1609/aaai.v39i14.33593
Project page: https://ezharjan.github.io/AutoSceneGen
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 75/100
The gist: Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework…
Key concepts
- AutoSceneGen
- A universal framework designed to automatically create varied traffic scenarios from simple text descriptions. It uses LLMs' in-context learning (ICL) to generate optimized configurations, which are then simulated in a physics-based environment to produce training data.
- In-Context Learning (ICL)
- A technique where a Large Language Model learns to perform new tasks based on examples provided directly within the prompt, without needing formal fine-tuning. This allows the LLM to generate specific traffic scenario configurations tailored to user text requests.
- Scenario Generation for Training
- The process of automatically creating a large volume of safety-critical traffic scenarios. These generated scenarios serve as synthetic training data, allowing motion planners to learn from rare events like accidents and corner cases efficiently.
Terminology
Summary
Motion planning is crucial in autonomous driving and this paper proposes an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners The framework leverages Large Language Models to translate user-specified text descriptions into scripts, which are then used by a simulator like CARLA to produce traffic scenarios As the method can generate abundant safety-critical traffic scenarios, they are used as synthetic training data for motion planners Experiments show that motion planners trained with this data significantly outperform those trained solely on real-world data. p. 0 To demonstrate the value
of generated scenarios, we train existing motion planners on
our synthetic data, real-world datasets, and a combination of
both. Our experiments show that motion planners trained with
our data significantly outperform those trained solely on realworld data, showing the usefulness of our synthetic data and
the effectiveness of our data generation method.Where the paper says it
Framework Overview
The AutoSceneGen framework is a universal, general, and cost-effective method designed to automatically enhance the heterogeneity of traffic scenarios through scenario descriptions It leverages in-context learning (ICL) of Large Language Models to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator The system architecture involves a filtering process that ensures simulator compatibility by replacing incompatible terms with those aligned to the simulator’s documented APIs Furthermore, the validator checks each API call against a documented list, replacing unsupported terms with approved synonyms or removing them to prevent errors The final step involves launching the simulator to run the generated configuration files
How the training signal is built
The method builds a training signal by generating abundant safety-critical traffic scenarios that are then used as synthetic training data for motion planners The generated scenarios are automatically categorized based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios The framework automates AV evaluation by efficiently generating traffic scenarios from user-defined descriptions, ensuring cost-effective simulation across diverse open-world environments The generated datasets are logically realistic and heterogeneous, which is demonstrated by running 41 AI-generated scenarios with varying numbers of vehicles and pedestrians within a single scenario logic
Where the gains come from
The primary gains stem from overcoming the limitations of manual scenario creation, which traditionally requires significant human effort The framework addresses challenges like low-resource scenarios, such as corner cases and accidents, by leveraging LLMs’ ICL capabilities to generate tailored configurations for rare scenarios The use of ICL eliminates the need for continuous fine-tuning or training of separate models for generation tasks The generated data, when combined with existing datasets like ApolloScapes, reduces displacement error by introducing valuable new scenarios This combination demonstrates that the original NGSIM dataset is augmented by the dataset collected via AutoSceneGen, indicating that the original NGSIM dataset is augmented by our dataset collected via AutoSceneGen
Limitations
The framework has several limitations, including performance, simulator integration, and data availability The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction If the chosen LLM lacks ICL capabilities or the simulator cannot handle complex scenarios, the framework’s utility is significantly reduced, making it difficult to generate corner cases The explainability of the generated scenario logic is limited to the explainability of the scenario descriptions and ICL examples If example data for ICL is insufficient, some simulators may not respond effectively, necessitating post-revision of the responses The framework’s effectiveness depends on the capabilities of both LLMs and the simulator, particularly for tasks such as scenario generation and accident reconstruction
Conclusions
AutoSceneGen is designed for generating traffic scenarios using foundation models The resulting scenarios can be simulated within virtual environments in simulators, within the real world, or even both concurrently The framework achieves the improvement of work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world It is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment AutoSceneGen harnesses the power of foundation models to simulate extensive arrays of traffic scenarios, thereby providing a robust testing environment for AVs. The framework is particularly advantageous for enhancing the reliability testing of AVs slated for real-world deployment
References
Achiam, J.; Adler, S. p. 7 References
Achiam, J. ; Adler, S. ; Adler, S. ; Agarwal, S. ; Agarwal, S. ; Ahmad, L. ; Ahmad, L. ; Akkaya,
I. ; Akkaya,
I. Gpt-4 technical report. Gpt-4 technical report. arXiv preprint
arXiv:2303.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Alam, M. S. 2020. Automatic generation of critical driving scenarios
Alinezhad Noghre, G.; Katariya, V. p. 7 Alinezhad Noghre, G. ; Katariya, V. ; Katariya, V. ; Danesh Pazho, A. ; Danesh Pazho, A. ; and Tabkhi, H.Where the paper says it
Where the paper says it
Where the paper says it
Amersbach, C.; and Winner, H. p. 0 Scenario-based evaluation methods have largely focused on
coverage, unsafe scenarios, and indicator-estimation tests
(Jesenski et al. 2019; Amersbach and Winner 2019).Where the paper says it
Awad, G.; Butt, A. A.; Curtis, K. p. 7 In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 425–430. ; Curtis, K. ; Curtis, K. ; Fiscus, J. ; Fiscus, J. ; Godil,
A. ; Godil,
A. ; Delgado, A.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Bock, J.; Krajewski, R. p. 0 2023), Waymo (Ettinger et al. 2021),
Round (Krajewski et al. ; Krajewski, R. ; Moers, T. ; Krajewski, R. ; Moers, T. ; Moers, T. ; Runde, S. ; Runde, S. ; Vater, L. ; Runde, S. ; Vater, L. ; Vater, L. ; and
Eckstein, L. ; Vater, L. ; and
Eckstein, L. ; Vater, L. ; and
Eckstein, L.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Bogdoll, D.; Breitenstein, J. p. 7 Bogdoll, D. ; Breitenstein, J. ; Breitenstein, J. ; Heidecker, F. ; Heidecker, F. ; Bieshaar, M. ; Bieshaar, M. ; Fingscheidt, T. ; Fingscheidt, T. ; and Zollner, M.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Caesar, H.; Bankiti, V. p. 7 In Proceedings of the IEEE/CVF International Conference
on Computer Vision, 1023–1028. ; Bankiti, V. ; Bankiti, V. ; Liong, V. ; Liong, V. ; Krishnan, A.Where the paper says it
Where the paper says it
Where the paper says it
Chandra, R.; Bhattacharya, U. p. 7 Chandra, R. ; Bhattacharya, U. ; Bhattacharya, U. ; and Manocha, D. ; Bhattacharya, U. ; and Manocha, D.Where the paper says it
Where the paper says it
Where the paper says it
Chang, M.-F.; Lambert, J. p. 7 In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 8483–8492. ; Lambert, J. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 8483–8492. ; Lambert, J. ; Lambert, J. ; Sangkloy, P. ; Sangkloy, P. ; Singh, J. ; Sangkloy, P. ; Singh, J. ; Singh, J. ;
Hartnett, A.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Dai, D.; Sun, Y.; Dong, L.; Hao, Y.; Ma, S.; Sui, Z.; and Wei, F. p. 7 In Proceedings of
the IEEE/CVF conference on computer vision and pattern
recognition, 8748–8757. ; and Wei,
F. ; and Wei,
F. Why can gpt learn in-context?Where the paper says it
Where the paper says it
Deng, Y.; Yao, J.; Tu, Z.; Zheng, X. p. 7 arXiv
preprint arXiv:2212. ; Zheng, X. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.Where the paper says it
Where the paper says it
Where the paper says it
Ding, W.; Xu, C.; Arief, M. p. 7 arXiv preprint
arXiv:2305. ; Arief, M. ; Arief, M. ; and Zhao, D.Where the paper says it
Where the paper says it
Dosovitskiy, A.; Ros, G.; Codevilla, F. p. 7 Dosovitskiy, A. ; Codevilla, F. ; Codevilla, F. ; Lopez, A. ; Lopez, A. ; and
Koltun, V.Where the paper says it
Where the paper says it
Where the paper says it
Ettinger, S.; Cheng, S. p. 7 Ettinger, S. ; Cheng, S. ; Cheng, S. ; Caine, B. ; Caine, B. ; Pradhan, S.Where the paper says it
Where the paper says it
Where the paper says it
Feng, L.; Li, Q.; Peng, Z.; Tan, S.; and Zhou, B. p. 7 In Proceedings of the IEEE/CVF International
Conference on Computer Vision, 9710–9719. ; and Zhou, B. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, 9710–9719. ; and Zhou, B.Where the paper says it
Where the paper says it
Fremont, D. J.; Dreossi, T. p. 7 Fremont, D. ; Dreossi, T. ; Dreossi, T. ; Ghosh, S. ; Ghosh, S. ; SangiovanniVincentelli, A. ; SangiovanniVincentelli, A. ; and Seshia, S.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Geiger, A.; Lenz, P.; Stiller, C. p. 7 In
Proceedings of the 40th ACM SIGPLAN conference on programming language design and implementation, 63–78. ; Stiller, C. ; Stiller, C. ; and Urtasun, R.Where the paper says it
Where the paper says it
Go, K.; and Carroll, J. p. 1 The concept of a “scenario” revolves around the “logic”
governing objects within a scene. Go and Carroll (2004)
defined a scenario as a detailed account involving actors,
background details, objectives, and sequences of actions.Where the paper says it
Goyal, S.; Griggio, A. p. 7 Interactions, 11(6): 44–53. ; Griggio, A. Interactions, 11(6): 44–53. ; Griggio, A. ; Griggio, A. ; Kimblad, J. ; Kimblad, J. ; and Tonetta, S.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Huang, W.; Wang, K.; Lv, Y.; and Zhu, F. p. 7 arXiv preprint arXiv:2311. ; and Zhu, F.Where the paper says it
Huang, X.; Cheng, X. p. 7 Ettinger, S. ; Cheng, S.Where the paper says it
Jesenski, S.; Stellet, J. p. 7 Jesenski, S. ; Stellet, J. ; Stellet, J. ; Schiegg, F. ; Fingscheidt, T. ; and Zollner, M.Where the paper says it
Where the paper says it
Where the paper says it
Kalra, N.; and Paddock, S. p. 6 The framework achieves the improvement of the work efficiency on generating sufficient safety-critical scenarios conveniently for testing AVs in an open world, as well as
the convenience of traffic accident reconstruction. As highlighted by Kalra and Paddock (2016), AVs face the challenge
of needing to cover vast distances — potentially hundreds
of millions to even billions of miles — to establish their reliability in terms of minimizing fatalities and injuries.Where the paper says it
Konig, L.; Heinzemann, C. p. 7 Transportation Research Part
A: Policy and Practice, 94: 182–193. ; Heinzemann, C. ; Griggio, A. ; Klauck, M. ; Klauck, M. ; ¨
Cimatti, A. ; ¨
Cimatti, A. ; Henze, F.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Kovvali, V. G.; Alexiadis, V. p. 0 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.Where the paper says it
Where the paper says it
Krajewski, R.; Bock, J.; Kloeker, L. p. 8 Krajewski, R. ; Kloeker, L.Where the paper says it
Krajewski, R.; Moers, T.; Bock, J.; Vater, L.; and Eckstein, L. 2020. The round dataset: A drone dataset of road user trajectories at roundabouts in germany
Lerner, A.; Chrysanthou, Y. p. 0 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.Where the paper says it
Where the paper says it
Li, D.; Liu, B.; Huang, Z. p. 0 2019), and MSRL (Wu et al. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. ; Huang, Z. ; and Tian, B.Where the paper says it
Where the paper says it
Li, Q.; Peng, Z. M.; Feng, L.; Liu, Z.; Duan, C.; Mo, W.; and Zhou, B. 2024a. Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling
Li, X.; Liu, E.; Shen, T.; Huang, J. p. 0 2019), and MSRL (Wu et al. 2023), rely on
real-world datasets like ApolloScapes (Huang et al. ; Huang, J. ; and Wang, F.Where the paper says it
Where the paper says it
Ma, Y.; Zhu, X.; Zhang, S. p. 0 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al.Where the paper says it
Majzik, I.; Semerath, O. p. 8 In Proceedings of the AAAI
conference on artificial intelligence, volume 33, 6120–6127. ; Semerath, O. ; Semerath, O. ; Hajdu, C. ; Hajdu, C. ; Marussy, K. ; Marussy, K. ; Szatmari,
Z. ; Szatmari,
Z. ; Micskei, Z.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Mann, B.; Ryder, N. p. 8 In 2019
ACM/IEEE 22nd International Conference on Model Driven
Engineering Languages and Systems (MODELS). ; Ryder, N. ; Ryder, N. ; Subbiah, M. ; Subbiah, M. ; Kaplan, J. ; Kaplan, J. ; Dhariwal, P. ; Dhariwal, P. ;
Neelakantan, A.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Min, S.; Lyu, X.; Holtzman, A. p. 8 arXiv
preprint arXiv:2005. ; Holtzman, A. ; Holtzman, A. ; Artetxe, M. ; Artetxe, M. ; Lewis, M. ; Lewis, M. ; Hajishirzi, H. ; Hajishirzi, H. ; and Zettlemoyer, L.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Oh, S.; Hoogs, A. p. 8 arXiv preprint arXiv:2202. ; Hoogs, A. ; Hoogs, A. ; Perera, A. ; Perera, A. ; Cuntoor, N.Where the paper says it
Where the paper says it
Where the paper says it
Pellegrini, S.; Ess, A.; Schindler, K. p. 8 Pellegrini, S. ; Schindler, K. ; Schindler, K. ; and Van Gool, L.Where the paper says it
Where the paper says it
Shah, S.; Dey, D.; Lovett, C. p. 8 In 2009 IEEE 12th international conference
on computer vision, 261–268. ; Lovett, C. ; Lovett, C. ; and Kapoor, A.Where the paper says it
Where the paper says it
Shet, R.; Iyer, N. C.; Mirje, M. p. 8 Cham:
Springer International Publishing. ; Mirje, M. ; Mirje, M. ; Bikkannavar, K. ; Bikkannavar, K. ; and
Rokhade, S.Where the paper says it
Where the paper says it
Where the paper says it
Sun, H.; Feng, S.; Yan, X.; and Liu, H. p. 8 In International
Conference on Information and Communication Technology
for Competitive Strategies, 81–92. ; and Liu, H.Where the paper says it
Sun, S.; Gu, Z.; Sun, T.; Sun, J.; Yuan, C.; Han, Y.; Li, D., and Ang, M. p. 8 Transportation research record, 2675(11): 587–
600. ;
and Ang, M.Where the paper says it
Tan, S.; Ivanovic, B. p. 8 IEEE Robotics
and Automation Letters. ; Ivanovic, B. ; Ivanovic, B. ; Pavone, M. ; Pavone, M. ; and Kraehenbuehl, P.Where the paper says it
Where the paper says it
Where the paper says it
Webson, A., and Pavlick, E. p. 1 However, doubts persist about whether LLMs truly understand prompts. For instance, Webson and Pavlick (2021) found that models performed similarly with irrelevant prompts, suggesting that
improved performance might not equate to genuine comprehension. ; and Pavlick, E. Do prompt-based models really understand the meaning of their prompts? Do prompt-based models really understand the meaning of their prompts? arXiv
preprint arXiv:2109.Where the paper says it
Where the paper says it
Where the paper says it
Wilson, B.; Qi, W.; Agarwal, T. p. 7 ; Adler, S. ; Agarwal, S. ; Singh, J. ;
Khandelwal, S.Where the paper says it
Where the paper says it
Wu, Y.; Wang, L.; Zhou, S.; Duan, J.; Hua, G.; and Tang, W. p. 8 arXiv preprint arXiv:2301. ; and Tang, W.Where the paper says it
Xinxin, Z.; Fei, L., and Xiangbin, W. p. 1 Related Work
Scenario Generation and Gaps
Existing works, such as LCTGen (Tan et al. 2023), CSG
(Xinxin, Fei, and Xiangbin 2020), and TrafficGen (Feng
et al.Where the paper says it
Yu, Y.; Liu, S.; Jin, P. J.; Luo, X.; and Wang, M. p. 8 ; Huang, J. ; and Wang, F.Where the paper says it
Zhang, X.; Tao, J.; Tan, K.; Torngren, M. p. 8 Transportation research record,
2674(11): 165–183. ; Torngren, M.Where the paper says it
Zhang, Z.; Zhang, A. p. 0 2023), rely on
real-world datasets like ApolloScapes (Huang et al. 2018),
NGSIM (Kovvali, Alexiadis, and Zhang PE 2007), UCY
(Lerner, Chrysanthou, and Lischinski 2007), and ETH (Pellegrini et al. ; Zhang, A. ; and Smola, A.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Universal Scenario Generation Framework: The AutoSceneGen framework can automatically enhance traffic scenario heterogeneity through scenario descriptions, thereby accelerating simulation and testing processes by using in-context learning of Large Language Models (LLMs).
-
Automated Data Categorization: The system can automatically categorize the generated scenarios based on their descriptions, eliminating the need for downstream annotation and facilitating the training of motion planners in open-world scenarios.
-
Cost-Effective Scenario Creation: It can generate abundant safety-critical traffic scenarios using prompt-based, configurable, and AI-driven methods, which is more cost-effective than collecting real-world datasets such as Argoverse2 or nuScenes.
-
Enhanced Motion Planner Training: Motion planners trained with the synthetic data generated by AutoSceneGen significantly outperform those trained solely on real-world data, showing the usefulness of the synthetic data and the effectiveness of our data generation method.
-
Scenario Logic Definition: The framework leverages LLMs’ in-context learning capability to generate optimized scenario configurations from given templates, simulating these scenarios in batch within a physics-based simulator, reflecting
the logic governing objects within a scene.
Abstract
Motion planning is a crucial component in autonomous driving. State-of-the-art motion planners are trained on meticulously curated datasets, which are not only expensive to annotate but also insufficient in capturing rarely seen critical scenarios. Failing to account for such scenarios poses a significant risk to motion planners and may lead to incidents during testing. An intuitive solution is to manually compose such scenarios by programming and executing a simulator (e.g., CARLA). However, this approach incurs substantial human costs. Motivated by this, we propose an inexpensive method for generating diverse critical traffic scenarios to train more robust motion planners. First, we represent traffic scenarios as scripts, which are then used by the simulator to generate traffic scenarios. Next, we develop a method that accepts user-specified text descriptions, which a Large Language Model translates into scripts using in-context learning. The output scripts are sent to the simulator that produces the corresponding traffic scenarios. As our method can generate abundant safety-critical traffic scenarios, we use them as synthetic training data for motion planners. To demonstrate the value of generated scenarios, we train existing motion planners on our synthetic data, real-world datasets, and a combination of both. Our experiments show that motion planners trained with our data significantly outperform those trained solely on real-world data, showing the usefulness of our synthetic data and the effectiveness of our data generation method. Our source code is available at https://ezharjan.github.io/AutoSceneGen.
Sources
- GPT-4 Technical Report
- TRECVID 2019: An Evaluation Campaign to Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval
- Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
- TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles via Validated LLM-Guided Knowledge Extraction
- Automatic Generation of Scenarios for System-level Simulation-based Verification of Autonomous Driving Systems
- Language Models are Few-Shot Learners
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- Do Prompt-Based Models Really Understand the Meaning of their Prompts?
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting
- Automatic Chain of Thought Prompting in Large Language Models
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation