Foundation Model Driven Robotics: A Comprehensive Review
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Foundation Model Driven Robotics: A Comprehensive Review".
Dev: Here's the gist of the paper. Foundation Model Driven Robotics: A Comprehensive Review provides a structured synthesis of recent developments in foundation models like LLMs and VLMs as they transform robotics across simulation, open-world execution, sim-to-real transfer, and adaptable systems This critical review highlights integrated strategies and evaluates the practical feasibility of these models in real-world environments Foundation Models Overview The integration of language models into robotic systems is based on a clear understanding of how these models have evolved to support increasingly general, interpretable, and context-aware reasoning As LLMs become central to enabling high-level cognition in robots, from interpreting user intent to generating executable plans, their architectural and functional progression provides critical context for evaluating their suitability in embodied settings VLMs extend LLMs with visual perception capabilities, enabling multimodal reasoning for tasks such as image captioning and referring expression comprehension Early efforts introduced joint architectures to align visual and textual characteristics, laying the foundation for perception-driven interaction Integration in Robotics Robotic systems are composed of several interdependent subsystems that govern how an agent perceives, plans, acts, and interacts These typically include Perception which processes raw sensor data to build an environmental model; Planning which translates high-level goals into sequences of actions; Control which governs low-level actuation and motion execution; and Human-Robot Interaction HRI which manages communication and collaboration with users Simulation-Driven Robotics Foundation models are reshaping robot simulation frameworks by automating environment synthesis, procedural content creation, simulation scripting, and multi-robot coordination Key applications include: 1.
Rosa: Procedural Environment and Task Generation where foundation models are applied to the generation of procedural environments by directly translating natural language prompts into simulation assets.
Rosa: Code-Driven Simulation Design where foundation models act as autonomous code generators that translate high-level textual task descriptions into executable simulation scripts.
Title and authors: Rosa: So, we're looking at this paper called "Foundation Model Driven Robotics: A Comprehensive Review," and the title itself tells you a lot about what this work is trying to do. It’s not just another paper listing off different models; it’s trying to tie everything together.
Dev: Yeah, it seems like the main goal is to give us a structured look at how foundation models, like LLMs and VLMs, are actually changing robotics right now across different areas like simulation and open-world execution. It's trying to move past just looking at isolated capabilities and see how these systems fit into real robot tasks.
Taro: I wonder if that structure helps us figure out where the real bottlenecks are, you know, what’s actually hard about putting this stuff onto a physical machine versus just running it in a simulation?
Rosa: Exactly. It highlights things like procedural scene generation and policy generalization as key trends, but it also points out some big problems that need solving, like the issue of limited embodiment and the lack of multimodal data.
Dev: Right. And what I’m seeing in this review is that it’s trying to connect those high-level model capabilities—like instruction following or code generation—to the actual physical execution and real-world demands of embodied intelligence.
The paper's summary: Rosa: So, when we look at the summary of "Foundation Model Driven Robotics: A Comprehensive Review," it’s basically saying that integrating language models into robots is a huge shift because these models give robots better semantic understanding and high-level reasoning.
Dev: They are moving beyond just simple commands to actually interpreting user intent and then generating executable plans for complex tasks, which is where the real power seems to be coming from.
Taro: I think the paper emphasizes that VLMs add a whole new layer by giving robots visual perception, so they can reason about what they see alongside what a human might be saying.
Rosa: That’s right. It talks about joint architectures that align the visual and textual characteristics early on, which sets the stage for more integrated perception-driven interaction in robotics.
Dev: And it covers this whole spectrum, from how foundation models are used in simulation-driven design to how they can be applied to executing tasks in unstructured, open-world environments.
Taro: I’m interested in the part where they discuss multi-robot coordination and decomposing complex instructions into structured subtasks for things like that.
The paper's improvements: Rosa: The review suggests that the real improvement here is moving from just looking at individual model successes to focusing on integrated, system-level strategies. It’s about how these different components—perception, planning, control—can work together effectively.
Dev: I think the key suggestion is to look at specific techniques like semantic alignment and multimodal representations as ways to bridge the gap between simulation and reality by using high-level features like object identities or spatial relations as common anchors.
Taro: That makes sense for sim-to-real transfer, but what about making those models actually robust when things go wrong in the real world? I’m thinking about how they handle unexpected failures.
Rosa: The paper touches on this by discussing policy learning and transfer with large models, where foundation models shape the policy directly by automating simulator design and reward definition.
Dev: And another point is their focus on enabling adaptable systems, where LLMs act as high-level brains to adapt low-level control or regenerate task plans when the initial strategy fails.
Taro: I see that modular architecture idea too, where you have separate models handling different things, like a language reasoner and a physics model for understanding the environment. That seems like a way to build in some kind of transparency for debugging later on.
Conclusion: Rosa: So wrapping up "Foundation Model Driven Robotics: A Comprehensive Review," it’s clear that the direction this research is heading is toward combining massive prior knowledge with tailored integration into simulations, real-world deployments, and transfer pipelines.
Dev: The main implication for me is that we’re moving away from brittle learning methods toward more holistic solutions where perception, reasoning, and action are unified by these general models.
Taro: I just want to stress the practical challenges they flag: the computational constraints of these models right now and the real-world data scarcity needed for them to be reliable.
Rosa: Right. And they also highlight some safety concerns where LLMs can produce unsafe actions if you don't have rigorous validation, which is a big thing for deployment.
Dev: It’s a lot of work connecting all these pieces—the planning, the control, the interaction—in a way that works reliably outside of a controlled lab setting.
Taro: Yeah, so the future work needs to focus on making those models more grounded in physical affordances and giving them closed-loop reasoning so they can actually self-correct when things get messy.
Rosa: That sounds like the next big step then. We'll keep an eye on how these integrated strategies translate into reliable systems for actual embodied intelligence.
Muhammad Tayyab Khan, Ammar Waheed
Nanyang Technological University · Texas A&M University
cs.RO
Submitted: 2025-07-14
Updated: 2026-11-13
Code: https://github.com/AlexKaravaev/world-creator
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: Foundation Model Driven Robotics: A Comprehensive Review provides a structured synthesis of recent developments in foundation models like LLMs and VLMs as they transform robotics across simulation…
Key concepts
- Foundation Models Overview
- These are large language models (LLMs) and vision-language models (VLMs) that have evolved to support general reasoning. They are now central to robots for high-level cognition, allowing them to interpret user intent and generate executable plans, moving beyond narrow programming.
- Integration in Robotics
- Robotic systems involve interdependent parts: Perception (understanding the world), Planning (creating action sequences), Control (executing movement), and Human-Robot Interaction. Foundation models are integrated into these subsystems to improve how robots perceive, plan, act, and interact with their surroundings.
- Robots in Unstructured Environments
- This area focuses on moving robots beyond simple tasks into open-world settings. Key methods involve using multi-modal reasoning to break down complex goals into sequential actions and employing generalist policy models that learn end-to-end mappings from sensory input to continuous motor outputs.
- Bridging the Sim-to-Real Gap
- This addresses the difficulty of transferring skills learned in simulation to real robots. Foundation models help by creating consistent semantic representations between simulation and reality, generating diverse training data for better adaptation, and shaping policy learning directly.
Terminology
Summary
Foundation Model Driven Robotics: A Comprehensive Review provides a structured synthesis of recent developments in foundation models like LLMs and VLMs as they transform robotics across simulation, open-world execution, sim-to-real transfer, and adaptable systems This critical review highlights integrated strategies and evaluates the practical feasibility of these models in real-world environments
Foundation Models Overview
The integration of language models into robotic systems is based on a clear understanding of how these models have evolved to support increasingly general, interpretable, and context-aware reasoning As LLMs become central to enabling high-level cognition in robots, from interpreting user intent to generating executable plans, their architectural and functional progression provides critical context for evaluating their suitability in embodied settings VLMs extend LLMs with visual perception capabilities, enabling multimodal reasoning for tasks such as image captioning and referring expression comprehension Early efforts introduced joint architectures to align visual and textual characteristics, laying the foundation for perception-driven interaction
Integration in Robotics
Robotic systems are composed of several interdependent subsystems that govern how an agent perceives, plans, acts, and interacts These typically include Perception which processes raw sensor data to build an environmental model; Planning which translates high-level goals into sequences of actions; Control which governs low-level actuation and motion execution; and Human-Robot Interaction HRI which manages communication and collaboration with users
Simulation-Driven Robotics
Foundation models are reshaping robot simulation frameworks by automating environment synthesis, procedural content creation, simulation scripting, and multi-robot coordination Key applications include:
-
Procedural Environment and Task Generation where foundation models are applied to the generation of procedural environments by directly translating natural language prompts into simulation assets. p. 5
Where the paper says it
Recent studies utilize foundation models to construct rich simulation scenarios in various types of robots and tasks with minimal human intervention, minimizing data scarcity and improving generalization [125], [126]. Procedural Environment and Task Generation Foundation models are increasingly applied to the generation of procedural environments by directly translating natural language prompts into simulation assets, scene layouts, and
-
Code-Driven Simulation Design where foundation models act as autonomous code generators that translate high-level textual task descriptions into executable simulation scripts. p. 6
Where the paper says it
Such structured variation improves dataset diversity and supports better generalization to unseen scenarios, though errors in relational grounding may introduce logically inconsistent task setups when not properly constrained. Code-Driven Simulation Design Foundation models increasingly act as autonomous code generators that translate high-level textual task descriptions into executable simulation scripts [131].
-
Multi-Robot and Multi-Modal Simulation Tasks which support multi-robot coordination by decomposing complex instructions into structured subtasks and cooperative scheduling strategies. p. 6
Where the paper says it
Although this approach offers lower computational overhead and faster response times, it presumes accurate and complete translation from language to program, making system performance vulnerable to ambiguity, underspecification, and context drift. Multi-Robot and Multi-Modal Simulation Tasks Foundation models also support multi-robot coordination by decomposing complex instructions into structured subtasks, sub-agent assignments, and cooperative scheduling strategies [137].
Robots in Unstructured, Open-World Environments
Recent advances show potential to expand robotic autonomy beyond narrowly scripted tasks into open-world, unstructured environments Principal methods include:
-
Multi-Modal Reasoning and Hierarchical Task Decomposition which involves decomposing high-level commands into chain-of-action sequences that map abstract goals to executable locomotion and manipulation primitives. p. 7
Where the paper says it
Multi-Modal Reasoning and Hierarchical Task Decomposition To address the complexity of open-world environments, several methods integrate multimodal perception with languageconditioned reasoning to generate adaptive task plans. A prominent strategy involves decomposing high-level commands into chain-of-action sequences that map abstract goals to executable locomotion and manipulation primitives [142].
-
Generalist Policy Models and Cross-Embodiment Control which addresses adaptability through large-scale vision-language action models that learn end-to-end mappings from sensory input and task descriptions to continuous motor outputs. p. 8
Where the paper says it
Generalist Policy Models and Cross-Embodiment Control Beyond high-level planning, open-world operation requires adaptable control policies that can be generalized across tasks, environments, and embodiments. Generalist policy architectures address this through large-scale vision-language action models that learn end-to-end mappings from sensory input and task descriptions to continuous motor outputs.
-
Bridging the Sim-to-Real Gap which facilitates reliable transfer through improved domaininvariant representations and improved policy generalization. p. 8
Where the paper says it
BRIDGING THE SIM-TO-REAL GAP Bridging the simulation-to-reality (Sim2Real) gap remains challenging due to unmodeled dynamics and sensor discrepancies [146]. Foundation models offer promising solutions by facilitating reliable transfer through improved domaininvariant representations and improved policy generalization [147].
Limitations and Future Directions
Key limitations include:
-
Real-Time & Computational Constraints where current foundation models are computationally heavy and often too slow for on-board robotic deployment. p. 5
Where the paper says it
First, today’s foundation models still struggle with real-time on-board operation due to slow inference and vast computational requirements, which conflict with the speed and resource constraints of robotic systems [121]. Second, there is a lack of robot-specific training data and experience; models mostly pre-trained on internet-scale text or image data often lack the physical grounding and long-tail robotic corner cases needed for reliable deployment [122].
-
Data Scarcity & Physical Grounding where models primarily trained on internet-scale data lack the physical grounding and long-tail robotic corner cases needed for reliable deployment. p. 5
Where the paper says it
First, today’s foundation models still struggle with real-time on-board operation due to slow inference and vast computational requirements, which conflict with the speed and resource constraints of robotic systems [121]. Second, there is a lack of robot-specific training data and experience; models mostly pre-trained on internet-scale text or image data often lack the physical grounding and long-tail robotic corner cases needed for reliable deployment [122].
-
Limited Multimodal Integration where current foundation models struggle to incorporate other modalities like tactile or audio that robots rely on. p. 5
Where the paper says it
Third, current approaches are largely limited in modality and scope, where many LLM-based policies rely solely on language or vision and cannot yet ingest the full spectrum of real-world inputs (e. , audio, tactile, or proprioceptive data) [123].
-
Safety & Reliability Concerns where LLMs can produce actions or instructions that are biased or unsafe, especially if prompt attacks exploit their lack of rigorous validation. p. 5
Where the paper says it
Finally, safety and ethical challenges pose non-technical barriers. LLM-driven robots can produce actions or instructions that are biased or unsafe, especially if prompt attacks or distribution shifts exploit their lack of rigorous validation [124].
-
Lack of Interpretability & Transparency where the decisionmaking process of foundation models is largely a black box, posing challenges for debugging and trust. p. 0
Where the paper says it
Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust.
Bridging the Sim-to-Real Gap
Foundation models facilitate reliable transfer through several mechanisms:
-
Semantic Alignment and Multimodal Representations where foundation models facilitate consistent semantic representations across simulation and reality by leveraging high-level features such as object identities, spatial relations, and task context as common semantic anchors. p. 8
Where the paper says it
Recent surveys emphasize the potential of foundation model-based methods in addressing longstanding Sim2Real challenges [148]. Semantic Alignment and Multimodal Representations Foundation models increasingly facilitate consistent semantic representations across simulation and reality by leveraging high-level features such as object identities, spatial relations, and task context as common semantic anchors [149].
-
Generative Simulation and Domain Adaptation where generative foundation models enhance Sim2Real transfer by generating realistic and diverse training data, thus improving domain randomization and adaptation. p. 9
Where the paper says it
Generative Simulation and Domain Adaptation Generative foundation models (e. , generative adversarial networks (GANs), diffusion models, pretrained simulators) enhance Sim2Real transfer by generating realistic and diverse training data, thus improving domain randomization and adaptation [153].
-
Policy Learning and Transfer with Large Models where foundation models shape policy learning directly by automating simulator design, reward definition, and training curricula. p. 9
Where the paper says it
Collectively, generative and representation-focused foundation models enrich simulation by producing high-quality data and robust feature spaces and reducing reliance on extensive real-world training. Policy Learning and Transfer with Large Models Foundation models increasingly shape policy learning directly, enhancing Sim2Real transfer by automating simulator design, reward definition, and training curricula.
Enabling Adaptable Robotic Systems
Emerging directions involve leveraging foundation models to enhance robotic adaptability:
-
LLM-Guided Controller Adaptation and Self-Repair where LLMs can serve as ’high-level brains’ to adapt low-level control to a robot on the fly. p. 4
Where the paper says it
, “go to the kitchen”) Path generation incorporating terrain constraints and verbal mission goals Control Adaptive motion generation from high-level prompts (e. , “walk over and wave”) Low-level control code synthesis for stacking, sorting, or manipulating parts Dynamic obstacle avoidance based on text-guided policy modulation Gait and balance adaptation via LLM-suggested behavior strategies Interaction Multimodal dialogue including gesture, gaze, and context-rich conversation Instruction refinement through natural language correction (e.
-
Adaptive Planning and Strategy Regeneration where foundation models have shown effectiveness in adapting task plans on the fly when a robot’s initial strategy fails or new constraints arise. p. 7
Where the paper says it
Multi-Modal Reasoning and Hierarchical Task Decomposition To address the complexity of open-world environments, several methods integrate multimodal perception with languageconditioned reasoning to generate adaptive task plans. A prominent strategy involves decomposing high-level commands into chain-of-action sequences that map abstract goals to executable locomotion and manipulation primitives [142].
-
Modular Architectures and Self-Assessment for Resilience where researchers decompose the problem into separate pretrained models to mirror the sense-plan-act pipeline and improve transparency.
Conclusion on Future Trajectory
The common theme across these four directions is an ambition to transcend the fragmented, brittle nature of current robot learning by combining massive prior knowledge with tailored integration into simulations, real-world deployments, transfer learning pipelines, and adaptive control loops Robotics research is moving toward more holistic solutions where perception, reasoning, and action are unified by general models.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Grounding language in physical affordances allows robots to interpret abstract symbols, commands, or language to link them to
the perceptual understanding and physical actions of the robot in the real world
(page 1). This enables agents like PaLMSayCan to propose high-level actions while simultaneously using anaffordance model evaluates the likelihood of successful execution
(page 4). -
Implementing closed-loop reasoning with feedback allows foundation models to iteratively re-plan or self-correct by giving the LLM access to environment observations and allowing it to
iteratively re-plan or self-correct
(page 5). This is demonstrated in systems that incorporateconversational feedback during navigation (e.g., “turn after the red sign”)
(page 5). -
Developing code generation policies enables robots to respond to scenarios never explicitly seen by synthesizing scripts that
invoke motion primitives and perception routines
(page 6). This allows for dynamic policy synthesis, enabling the robot tostitch together known functions in new ways
without task-specific training. -
Utilizing multi-modal architectures like PaLM-E enables embodied reasoning by combining a large language model with a vision transformer so that it can
plan actions directly from raw images and textual goals
(page 5). This allows the robot to handle multimodal reasoning tasks, includingmultimodal manipulation, scene understanding, and reward detection.
-
Employing generative foundation models for simulation design enables the creation of
procedural environments by directly translating natural language prompts into simulation assets
(page 7). This allows for rapid scaling of training data in diverse manipulation and navigation scenarios. -
Enabling adaptive planning allows robots to handle unexpected changes by using frameworks like RePLan, which
combines vision and language models to enable real-time replanning in long-horizon tasks
(page 11). This system can incorporate feedback from past interactions via Retrieval Augmented Generation (RAG) to refine its current reasoning. -
Creating modular architectures allows for a decomposition where separate models handle distinct functions, such as a GPT-based language reasoner for high-level task planning and a video-based physics model to understand the environment (page 12). This hierarchical approach improves interpretability by allowing observers to
inspect intermediate plans or visual predictions to understand what the robot ’thinks’ will happen.
-
Training models using generative approaches, such as diffusion models, enhances sim-to-real transfer by
generating realistic and diverse training data, thus improving domain randomization and adaptation
(page 10). This allows for the creation of novel simulation scenarios that continuously introducenovel simulation scenarios,
which yields better real-world navigation performance. -
Integrating pretrained visual models with language encoders facilitates semantic alignment across modalities, enabling
open-vocabulary perception
where robots can performzero-shot recognition of arbitrary objects by name
(page 4). This capability is further extended by mapping touch data to a multimodal semantic space, allowing robots to perform touch-based tasks in a zero-shot fashion. -
Developing generalist policy models, such as those inspired by DeepMind’s Gato or RT-2, allows robots to execute commands based on high-level concepts rather than task-specific training (page 6). This enables
cross-domain policy flexibility
and the ability toexecute commands based on high-level concepts, such as identifying and manipulating an object described in a language even if it was not seen during training.
Abstract
The rapid emergence of foundation models, particularly large language models and vision-language models, has introduced a transformative paradigm in robotics. The models offer powerful capabilities in semantic understanding, high-level reasoning, and cross-modal generalization, enabling significant advances in perception, planning, control, and human-robot interaction. This critical review provides a structured synthesis of recent developments and organizes applications into four directions: simulation-driven robotics, open-world operation, sim-to-real transfer, and adaptable robotic systems. Unlike existing surveys that emphasize isolated capabilities, this work highlights integrated, system-level strategies and evaluates their practical feasibility in real-world environments. Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust. The review concludes with a roadmap of proposed future research directions for bridging semantic reasoning and physical intelligence through more robust, interpretable, and embodied models.
Sources
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models
- Large Language Models for Robotics: A Survey
- Large Language Models for Multi-Robot Systems: A Survey
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- Emergent Abilities of Large Language Models
- Leveraging Vision-Language Models for Manufacturing Feature Recognition in CAD Designs
- NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions
- Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction
- Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
- CogVLM2: Visual Language Models for Image and Video Understanding
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Evaluating Large Language Models Trained on Code
- RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis
- LLMs for Coding and Robotics Education
- 3P-LLM: Probabilistic Path Planning using Large Language Model for Autonomous Robot Navigation
- Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
- LLaMA: Open and Efficient Foundation Language Models
- CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
- Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation