Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models".
Title and authors: Rosa: So we're looking at this paper titled Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models. It seems like they're tackling that big problem of full automation in cities where things change constantly, needing real-time adaptation instead of just following a map.
Dev: Right, it's about using Large Language Models to generate Answer Set Programming rules by translating those informal navigation instructions into structured logic-based reasoning. That's pretty cool because it moves away from relying solely on predefined maps.
Taro: So, the core idea is that this approach uses non-monotonic reasoning provided by Answer Set Programming ASP so the vehicle can adapt to changing scenarios without needing a static map. But that sounds complicated when you think about real-time control loops and failure modes.
Rosa: Exactly, Taro. They're showing how these LLMs can generate rules that encode real-world urban driving logic into this formal knowledge representation. It's basically translating what a human might say into something the car can actually reason about logically.
Dev: From an engineering standpoint, I wonder how fast this translation happens and what the latency would be when you're dealing with dynamic environments, you know?
Taro: That’s a real concern. The paper mentions that they generate these ASP constraints dynamically based on specific navigation instructions and real-time perception. It suggests this could automate the creation of dynamic rules needed for specific navigation goals based on what the vehicle sees right then.
Rosa: And that's where it gets interesting for me because it lets the system perform cognitively inspired navigation. Instead of just following precise metric distances, it interprets vague natural language instructions like “turn left at the bus stop” and adapts when things change, like if an intersection gets replaced by a roundabout.
Dev: That sounds much more flexible than sticking to coordinates on a map, but I'm still thinking about the safety case. How do we formally verify these rules generated by an LLM to make sure they respect all the traffic regulations and safety constraints?
Taro: The authors address that by ensuring these navigation rules are not just human-like but also logically verifiable because they are encoded using Answer Set Programming ASP, which gives us a formal knowledge representation. That’s supposed to enhance the system’s ability to interpret high-level instructions while making sure it operates safely in complex urban scenarios.
Rosa: So, instead of just having a hardcoded path, we have this system that builds its own reasoning rules based on what we tell it and what the sensors tell it is happening right now. This decouples navigation from existing cartography, which seems like a big win for reliability when maps are unreliable or missing.
Dev: Decoupling from maps does sound good for dynamic situations, but we still have to worry about the system processing environmental facts as inputs into this non-monotonic reasoning framework as the vehicle is moving. That suggests a lot of computational load in real-time loops.
Title and authors: Taro: The paper points out that this means the system can dynamically adjust its plan by processing those environmental facts as inputs, which lets it respond intelligently to incomplete or changing information as it drives along. It’s about handling uncertainty gracefully during motion.
Rosa: And on top of all that, the paper provides an explainable framework for dynamic navigation planning through the traceable reasoning contained in those generated answer sets. That inherent explainability is crucial for debugging and verifying safety constraints when we're aiming for SAE Level five automation <ref:2505.16498#pg2>.
Dev: Explainability is key, but I gotta ask how robust this is when the LLM makes a translation error in a complex scenario. If the initial instruction to the LLM was slightly ambiguous, does that translate into an unsafe rule?
Taro: The paper focuses on using this approach to improve adaptability by enabling semantic-based decisionmaking that aligns closely with how humans communicate navigational intent. It’s about making sure the logic derived from those instructions respects safety constraints even when the input is informal.
Rosa: So, what we're seeing is that this LLM-assisted knowledge engineering can automatically generate part of the extrinsic knowledge base by interpreting informal human instructions and translating them into formal DLV constraints compatible with intrinsic traffic rules. That automates creating those dynamic rules needed for specific navigation goals based on real-time perception.
Dev: That sounds like a lot of work for the LLM to do in real time, generating these DLV constraints that integrate with existing traffic knowledge. I wonder about the complexity of that constraint generation process itself.
Taro: It's about automating the creation of those dynamic rules based on what's happening right now. The authors showed how this helps bridge the gap between high-level human intent and low-level formal logic constraints, which is a big step in making navigation truly semantic.
Rosa: And that leads into the next part, where they show how this method allows autonomous vehicles to adapt to evolving scenarios without needing those fixed cartographic maps. That adaptation based on semantic meaning is what makes it so different from traditional metric-based localization.
Dev: So we've covered the core idea of using LLMs for ASP rule generation and how it offers semantic navigation, but where does this leave us when we think about the practical deployment outside of a controlled lab setting?
Taro: The paper suggests that this system can perform cognitively inspired navigation, interpreting vague instructions rather than relying only on precise metric distances. It’s about making the vehicle behave more like a person navigating the city.
Rosa: And what I really like is how they show it can handle things like adapting when an intersection is replaced by a roundabout because it adjusts its understanding dynamically based on semantic meaning, not just pre-programmed geometry.
Title and authors: Dev: But that dynamic adjustment means we need to be very confident in the safety margins when the system is constantly re-reasoning its constraints based on new perception inputs. What about failure modes when the LLM's interpretation of the instruction is fundamentally flawed?
Taro: The paper does address this by focusing on generating rules that respect traffic regulations and safety constraints, which implies a layer of filtering or validation over the LLM's raw output before it becomes an executable rule. It’s about grounding that semantic understanding in known physical limitations.
Rosa: So we're seeing a framework where the system can dynamically adjust its plan by processing environmental facts as inputs into this non-monotonic reasoning framework, allowing it to respond intelligently to incomplete or changing information as it drives along. That’s a major step toward handling the messiness of real-world driving.
Dev: It sounds like we're moving closer to a system that can handle those spontaneous detours better than just looking up coordinates on a map, but I still need more concrete data on the performance metrics when compared directly against current metric-based localization systems.
Taro: The paper does show an experimental evaluation where they generate ASP constraints that encode real-world urban driving logic into this formal knowledge representation, which is how they measure the effectiveness of this translation process. It gives us a way to quantify how much better the semantic approach performs compared to just using fixed maps.
Rosa: So, the main point here is that we can achieve full automation in dynamic urban environments by automating the translation of informal instructions into structured logic rules, which improves adaptability through semantic-based decisionmaking that mirrors human intent.
Dev: It seems like this work tackles the knowledge engineering bottleneck head-on by using LLMs to generate rules, and it offers a way to improve explainability through traceable reasoning in answer sets.
Taro: I think the biggest implication is decoupling navigation from existing cartography, which adds a layer of reliability when maps are unreliable or missing, moving us closer to that level five autonomy goal.
Rosa: So we're wrapping up on this paper, Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models. It shows how semantic navigation can handle evolving scenarios by translating natural language into formal logic rules that the car can reason with dynamically.
Dev: I think the challenge remains in making sure that LLM-generated rules are both robust enough for real-time loops and that we have a solid safety case for when things go wrong in those dynamic scenarios.
Taro: We're really excited about the idea of this system adapting to unexpected detours based on semantic meaning, moving beyond just following a pre-set path defined by coordinates.
Rosa: That’s what it does, and it shows how we can start building systems that think more like people when they are navigating complex city streets.
The paper's summary: Rosa: So, to wrap up what we just talked about, this paper is proposing a way for autonomous cars to navigate cities semantically instead of just following lines on a map by using Large Language Models to create rules that understand human instructions and formal logic.
Dev: Exactly, it's taking those vague sentences you might say like "turn left at the bus stop" and turning them into structured rules that the car can reason about logically, without needing a fixed map for every single street.
Rosa: The summary says this method lets the vehicle adapt to things changing on the fly, like if an intersection suddenly becomes a roundabout, because it understands the meaning of what you're saying instead of just relying on precise distances.
Dev: It’s about using Answer Set Programming, or ASP, which is this formal logic system. So they are generating these rules that combine general traffic laws with the specific instructions from the driver and whatever sensors are seeing right now to create a dynamic set of navigation constraints.
Rosa: That’s where I get excited; it moves us away from metric-based localization, which means you don't need perfect GPS coordinates for every single turn, you just need to understand the intent behind what you're driving toward.
Dev: The paper makes a big point about explainability too. Because these rules are generated through this logical process, we get a traceable reasoning path—we can actually see *why* the car decided to take a certain turn based on the rules it created.
Rosa: That’s crucial for safety verification, because if something goes wrong, we can look at the answer set and figure out exactly which rule caused the unexpected behavior, which is way better than a black box system.
Dev: But I still have to ask about the speed of this process. Generating these rules on demand based on real-time perception and then running that logic in a tight control loop needs to be incredibly fast if we want this to work for actual driving without massive latency issues.
Rosa: The authors say they're focused on how the LLM can automate generating that extrinsic knowledge base, which means it helps build the specific rules needed for a navigation goal based on what the vehicle perceives in real time.
Dev: So, it automates the creation of dynamic rules that fit existing traffic constraints—that’s a lot of heavy lifting for an AI model to do instantly. It's basically turning natural language into executable logic constraints, right?
Rosa: Right, and this is what I think changes things for the world: we could get vehicles that navigate city streets with the same kind of intuitive understanding as a person does.
Dev: That’s a huge leap if it holds up under real-world stress tests, but we need to see how they handle those failure modes where the LLM might misinterpret an instruction in a very messy situation.
Rosa: We're going to look at those specific failure modes in the next segment, seeing what happens when the world really misbehaves and this semantic navigation framework has to recover gracefully.
The paper's improvements: Taro: So, we're looking at how they suggest making this system even better than just translating instructions into rules. They focus on decoupling navigation from traditional map systems by relying on these generated rules instead.
Rosa: That’s right, so the main improvement is that the car doesn't need a perfect pre-built map for every street; it can use these logic rules to figure out where to go based on what it sees and what you tell it.
Dev: They also emphasize that this semantic navigation allows for cognitively inspired driving, meaning the car can handle vague instructions like "turn left at the bus stop" instead of needing perfect metric distances.
Rosa: That’s a big win because if an intersection gets changed—say it turns into a roundabout—the system should adjust its understanding dynamically based on what it actually perceives in that moment.
Dev: From my side, I'm interested in the fact that they ensure these rules are not just made up by the LLM but are also logically verifiable because they use Answer Set Programming. That gives us a formal way to check if the generated rule set respects all the safety constraints we need.
Taro: It’s about making sure that even though it’s learning from human intent, the final output is something we can prove is safe, which is pretty important for getting that high level of autonomy.
Rosa: And they mention how this LLM-assisted knowledge engineering lets the system automatically generate parts of its own rules by interpreting those informal human instructions and turning them into formal constraints compatible with existing traffic rules.
Dev: So it automates creating the dynamic rules needed for a specific navigation goal based on what the vehicle sees right then, which cuts down on how much manual coding we have to do for every new scenario.
Taro: This capability to dynamically adjust its plan by processing environmental facts as inputs into this non-monotonic reasoning framework means it can respond intelligently even when the information it has is incomplete or changing along the way.
Rosa: So what this means for us on the ground, outside of a lab setting, is that we could have vehicles that are way more adaptable to unexpected detours than systems strictly following pre-set routes.
Dev: It shifts the focus from just path following to actual understanding of intent and adapting to the messiness of real urban life, even when those instructions aren't perfectly worded.
Taro: And the explainability part is still key; they keep stressing that this framework provides a traceable reasoning path through those answer sets, which helps us debug and verify safety constraints in a way that’s hard with standard deep learning models.
Rosa: So we're moving toward a system where the navigation isn't just about geometry anymore; it’s about reasoning semantically about the world around it, guided by human-like instructions.
Dev: And while they show this works in controlled settings, I still wonder how robust this entire translation process is when the LLM makes a small error and generates a rule that could be dangerous.
Taro: The paper does acknowledge that filtering is needed, but the goal here is to use this semantic understanding to create rules that inherently respect traffic regulations and safety constraints from the start.
Rosa: It sounds like we’re building something where the vehicle thinks more like a person when navigating complex city streets, which is a big step toward true full automation.
Conclusion: Rosa: So we’ve covered how this paper uses Large Language Models to translate informal navigation instructions into formal logic rules using Answer Set Programming for semantic navigation in autonomous driving systems.
Dev: Basically, it shows how we can bridge that gap between what a human says and what the car needs to reason about logically in real time.
Taro: It’s promising because it moves us away from just following fixed coordinates on a map toward a system that truly understands the meaning of the driving situation.
Rosa: The main implication is that we could get vehicles that adapt to unexpected city changes without needing constant remapping, which is huge for reliability in dynamic environments.
Dev: I still have to stress the need for high performance; if this rule generation takes too long in a real-time loop, it’s not useful for anything beyond a controlled lab setting.
Taro: I agree about the speed but also about the recovery; we need to know how gracefully this system handles those moments where the LLM might generate an incorrect rule due to an ambiguous instruction.
Rosa: The paper does flag that the system relies on grounding these rules in intrinsic traffic rules and commonsense driving behaviors, which is a necessary safety layer.
Dev: So it’s not just pure LLM output; it’s the LLM generating a draft that gets checked against a set of known physical constraints before becoming an executable plan.
Taro: That verification process is what makes me interested; it shows they are thinking about making sure the semantic understanding translates into safe, verifiable actions rather than just clever suggestions.
Rosa: In short, "Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Models" gives us a powerful framework to handle the messiness of urban driving through formal logic.
Dev: It’s a solid contribution to knowledge engineering because it automates the creation of dynamic rules based on what the car perceives right then.
Taro: I think this work is really about pushing autonomy toward that level five goal by giving vehicles a way to reason semantically about their surroundings instead of just being reactive.
Rosa: That’s what it boils down to; moving from simple navigation to genuine intent-based reasoning in complex urban settings.
Dev: We’ll keep watching how they handle the latency and the safety case when this system is actually deployed on a vehicle operating at high speeds.
Taro: Next time, we'll look at some papers that focus more on those actual real-world failure modes and contingency planning for safety-critical autonomous vehicles.
Augusto Luis Ballardini, É Miguel Angel Sotelo
University of Alcala
cs.RO, cs.AI
Submitted: 2025-05-22
Updated: 2026-11-17
Comments: 7 pages, 5 figures, internal report
Code: https://github.com/invett/HUMANAV-ConferencePaper
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 75/100
The gist: Achieving full automation in self-driving vehicles remains a challenge, especially in dynamic urban environments where navigation requires real-time adaptability.
Key concepts
- Knowledge Engineering Bottleneck
- This refers to the difficulty of manually creating and maintaining the vast knowledge bases required for autonomous systems. Traditional methods rely on fixed maps or rigid rules, which struggle when situations change unexpectedly. This research aims to automate this process by using LLMs to bridge the gap between human language and formal logic rules.
- Answer Set Programming (ASP)
- ASP is a formal logic programming paradigm used to represent knowledge as a set of rules and constraints. It allows the system to perform complex, non-monotonic reasoning, meaning it can draw conclusions based on what is known while also considering possibilities that might change. This provides a mathematically rigorous way to encode driving regulations and safety constraints.
- Semantic Navigation
- Instead of relying solely on precise GPS coordinates (metric navigation), semantic navigation interprets the *meaning* of instructions, such as 'turn left at the bus stop.' This allows the vehicle to adapt dynamically when physical landmarks change or routes evolve, making its decision-making cognitively inspired and more flexible than map-based systems.
Terminology
Summary
Achieving full automation in self-driving vehicles remains a challenge, especially in dynamic urban environments where navigation requires real-time adaptability. p. 0 16498v1 [cs. RO] 22 May 2025
Human-like Semantic Navigation for
Autonomous Driving using Knowledge
Representation and Large Language Models
Augusto Luis Ballardini Member, IEEE and Miguel Angel Sotelo ´ Fellow, IEEE
Abstract—Achieving full automation in self-driving vehicles remains a challenge, especially in dynamic urban
environments where navigation requires real-time adaptability. Existing systems struggle to handle navigation plans
when faced with unpredictable changes in road layouts,
spontaneous detours, or missing map data, due to their
heavy reliance on predefined cartographic information. In
this work, we explore the use of Large Language Models
to generate Answer Set Programming rules by translating
informal navigation instructions into structured, logicbased reasoning. By automating the
translation of informal navigation instructions into logical
rules, our method improves adaptability and explainability
in autonomous navigation. Results show that LLM-driven
ASP rule generation supports semantic-based decisionmaking, offering an explainable framework for dynamic
navigation planning that aligns closely with how humans
communicate navigational intent.Where the paper says it
Where the paper says it
Where the paper says it
The core problem addressed is the knowledge engineering bottleneck
The research focuses on automating the translation of informal navigation instructions into logical rules by leveraging Large Language Models (LLMs) to generate Answer Set Programming (ASP) rules. This approach aims to move beyond traditional metric-based localization by enabling semantic navigation that adapts to evolving scenarios without relying on predefined maps. p. 0 Index Terms—Autonomous Driving, Knowledge Representation, Scene Understanding, Urban Navigation, Semantics, Answer Set Programming, Explainability, Large
Language Models. INTRODUCTION
This work explores the development of a semantic
navigation system for autonomous vehicles, proposing
and implementing an approach that moves beyond traditional metric-based localization. We present
an experimental evaluation in which LLMs generate ASP
constraints that encode real-world urban driving logic into
a formal knowledge representation. By automating the
translation of informal navigation instructions into logical
rules, our method improves adaptability and explainability
in autonomous navigation.Where the paper says it
Where the paper says it
Semantic Navigation using ASP
The proposed method translates informal navigation instructions into a structured representation using ASP, defined as a set of rules and constraints combined with a complementary program that encodes road-driving rules. This transformation allows the system to perform cognitively inspired navigation, interpreting vague natural language instructions in a way that respects traffic regulations and safety constraints. p. 2 Semantic Navigation using ASP
We focus on translating informal navigation instructions into a structured representation using ASP, defined
as a set of rules and constraints combined with a complementary program that encodes road-driving rules. This
transformation allows the system to perform cognitively
inspired navigation, interpreting vague natural language
instructions in a way that respects traffic regulations
and safety constraints.Where the paper says it
The process involves:
-
Translating informal driving instructions into structured representation using ASP. p. 2
Where the paper says it
This approach also enables the system to adapt to situations when an intersection is replaced by a roundabout. Semantic Navigation using ASP We focus on translating informal navigation instructions into a structured representation using ASP, defined as a set of rules and constraints combined with a complementary program that encodes road-driving rules.
-
Using an intrinsic Knowledge Base composed of general traffic rules and commonsense driving behaviors. p. 3
Where the paper says it
Within our system, the KB is divided into two parts. First, an intrinsic KB is manually designed, containing general traffic rules and commonsense driving behaviors.
-
Generating an extrinsic Knowledge Base dynamically depending on the specific navigation instruction and real-time perception. p. 3
Where the paper says it
Importantly, this intrinsic knowledge is fixed and independent of any specific navigation instruction, ensuring that the vehicle always operates in accordance with standard road regulations. Conversely, an extrinsic KB must be generated dynamically, depending both on the specific navigation instruction received and the real-time perception of the environment.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated translation of informal navigation instructions into formal logic rules allows autonomous vehicles to adapt to evolving scenarios without relying on predefined maps. This improves adaptability by enabling semantic-based decisionmaking that aligns closely with how humans communicate navigational intent.
-
The system can perform cognitively inspired navigation, interpreting vague natural language instructions like “turn left at the bus stop” rather than relying on precise metric distances. This enables adaptation when an intersection is replaced by a roundabout, as the system adjusts its understanding dynamically based on semantic meaning.
-
The proposed approach ensures that navigation rules are not only derived from human-like reasoning but are also logically verifiable because they are encoded using Answer Set Programming (ASP). This provides a formal knowledge representation that enhances the system’s ability to interpret high-level instructions while ensuring safe operation in complex urban scenarios.
-
The LLM-assisted knowledge engineering can automatically generate part of the extrinsic Knowledge Base by interpreting informal human instructions and translating them into formal DLV constraints compatible with intrinsic traffic rules. This automates the creation of dynamic rules needed for specific navigation goals based on real-time perception.
-
The improved AI system can decouple navigation from existing cartography, adding a layer of reliability in dynamic environments where maps are unreliable or missing. It allows the vehicle to dynamically adjust its plan by processing environmental facts as inputs to a non-monotonic reasoning framework, enabling it to respond intelligently to incomplete or changing information as the vehicle moves.
-
The system can provide an explainable framework for dynamic navigation planning through the traceable reasoning contained in generated answer sets. This inherent explainability helps in debugging and verifying safety constraints, which is crucial for achieving SAE Level 5 automation under rigorous terms.
Abstract
Achieving full automation in self-driving vehicles remains a challenge, especially in dynamic urban environments where navigation requires real-time adaptability. Existing systems struggle to handle navigation plans when faced with unpredictable changes in road layouts, spontaneous detours, or missing map data, due to their heavy reliance on predefined cartographic information. In this work, we explore the use of Large Language Models to generate Answer Set Programming rules by translating informal navigation instructions into structured, logic-based reasoning. ASP provides non-monotonic reasoning, allowing autonomous vehicles to adapt to evolving scenarios without relying on predefined maps. We present an experimental evaluation in which LLMs generate ASP constraints that encode real-world urban driving logic into a formal knowledge representation. By automating the translation of informal navigation instructions into logical rules, our method improves adaptability and explainability in autonomous navigation. Results show that LLM-driven ASP rule generation supports semantic-based decision-making, offering an explainable framework for dynamic navigation planning that aligns closely with how humans communicate navigational intent.
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation