LLM-Flax: LLM-Generated Rules and Object Guidance for Neuro-Symbolic Task Planning

summary

The gist

Deploying a neuro-symbolic task planner on a new domain today requires significant manual effort: a domain expert must author relaxation and complementary rules…

In short

LLM-Flax is a three-stage framework that automates neuro-symbolic task planning for new domains using only a PDDL file. It eliminates manual effort by automatically generating necessary rules, implementing LLM-guided failure recovery, and replacing traditional object scoring with zero-shot LLM guidance. This allows deployment on new domains without extensive expert engineering or training data.

Key concepts

Stage 1: Rule Generation
This stage uses a local open-source LLM to automatically create necessary relaxation and complementary rules for the planner. It employs structured prompting with strict format validation and self-correction to ensure the generated rules are syntactically correct, saving domain experts from writing these rules manually.
Stage 2: Failure Recovery
This mechanism intervenes when a planning step times out by having an LLM identify which objects are likely causing the blockage. It uses a budget policy that reserves API latency cost before each call to ensure the system doesn't run out of time before attempting a fallback relaxation.
Stage 3: Zero-Shot Object Scoring
This replaces a trained Graph Neural Network with an LLM that scores object importance without any prior training data. The LLM receives the goal and state facts to determine which objects are most relevant, drastically reducing the need for domain-specific training.
LLM-Flax Framework
The overall system is a three-stage pipeline designed to deploy complex neuro-symbolic planners on entirely new domains using only a PDDL domain file. It integrates automated rule generation, intelligent failure recovery, and zero-shot scoring to minimize manual setup time from hours to just one domain file.

Terminology used across episodes

This episode discusses

The paper

LLM-Flax: LLM-Generated Rules and Object Guidance for Neuro-Symbolic Task Planning · Read on arXiv

Seongmin Kim, Daegyu Lee

Jeonbuk National University

Adapting a neuro-symbolic task planner to a new domain can require hand-authored symbolic rules and solved training instances for a learned object scorer. This paper presents LLM-Flax, which applies large language models (LLMs) to offline rule generation, failure recovery, and object scoring within the Flax planning pipeline. Generated rules undergo validation and corrective feedback before reuse, while runtime guidance operates within a shared planning budget. Experiments on MazeNamo and two additional domains show that the evaluated rule configuration improves planning success over manually authored rules while retaining the trained object scorer. Across eight MazeNamo settings, the mean success rate increases by approximately three percentage points. Controlled comparisons distinguish the effects of rule content from the choice of generator. Runtime recovery and object scoring do not establish an overall performance advantage under the evaluated planning budgets. These findings demonstrate the value of reusable LLM-generated rules and identify information quality and inference efficiency as central challenges for runtime guidance.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "LLM-Flax: LLM-Generated Rules and Object Guidance for Neuro-Symbolic Task Planning".

Paper summary: Rosa: So we're looking at this paper called LLM-Flax, and it claims to automate the whole process of setting up a neuro-symbolic task planner on a brand new domain. It says that instead of needing a human expert to write all those relaxation and complementary rules, and then solving hundreds of training problems for a graph neural network object scorer, you just give the LLM a PDDL domain file.

Dev: Right, so the core idea is this three-stage framework that aims to eliminate all those manual engineering steps by using a locally hosted LLM. Stage one automatically generates those relaxation and complementary rules through structured prompting with some validation and self-correction built in.

Taro: And then stage three is pretty ambitious—it replaces that whole domain-trained graph neural network object scorer entirely with zero-shot LLM object importance scoring, meaning it requires no training data at all.

Rosa: That sounds like a big shift because the manual effort you mentioned could take hours from an expert to just a single PDDL file. The paper is looking at how this system performs across different grid sizes, specifically ten by ten twelve by twelve and fifteen by fifteen grids in the MazeNamo benchmark <ref:2604.26569#pg1>.

Dev: And they report that the full system achieves an average success rate of zero point nine four five versus a manual baseline of zero point eight two eight, which is an improvement of one point seventeen seven. That suggests it’s doing better than just using those hand-crafted rules we usually have to write ourselves.

Taro: But they also hit a wall at the larger scale, specifically on the fifteen by fifteen Hard grid where the manual planner fails completely with zero success rate <ref:2604.26569#pg1>. They attribute that collapse to a problem in context, saying that problems have over two hundred objects, but the LLM prompt is capped at eighty state facts.

Rosa: So they found something interesting: while it works well on smaller and medium problems, like those twelve by twelve grids where it solves twenty-two out of thirty problems in an average of five point two four seconds, it struggles when the problem gets too big <ref:2604.26569#pg1>.

Dev: And they’re calling that a context-window bottleneck at scale, which points to the main challenge for future work. The LLM prompt size is limited to eighty state facts, and that's not enough information when you have hundreds of objects in a complex scenario like on that fifteen by fifteen grid <ref:2604.26569#pg1>.

Taro: It makes sense from a failure mode standpoint; if the context window isn't big enough, the zero-shot scoring just doesn't have enough data to make an accurate importance decision for all those objects. How does that affect how gracefully the system recovers when it hits that context limit?

Paper summary: Rosa: That brings us to stage two, which is where they introduce LLM-guided failure recovery with a budget policy. This policy explicitly reserves the API latency cost before each LLM call to make sure that if the system needs to fall back to those domain-trained relaxation rules, it doesn't get starved of resources.

Dev: They implemented a feasibility check there too, meaning recovery is skipped if the remaining time before the Step two deadline is less than tLLM plus treplan minimum plus tstep two minimum, which they conservatively set to eleven seconds. That’s a safety net built into the loop so you don't get stuck waiting for an API call when time is running out.

Taro: From my side, I wonder if that budget policy is robust enough when the world misbehaves in a real-time scenario; does reserving that latency cost actually prevent a dangerous stall? If the LLM recovery call takes longer than expected, you're still capped at that eleven second limit before you have to decide whether to proceed or not.

Rosa: It seems like they are trying to manage the trade-off between letting the system try and failing gracefully versus cutting off the plan generation because of time constraints. The whole point is that this framework lets you deploy on a new domain with no manual engineering required for those three stages.

Dev: And looking at it from an engineering side, it means a PDDL domain file becomes the main input instead of needing hours of expert configuration to build the necessary rules and training data for a GNN scorer. That drastically cuts down the deployment cost on new domains.

Taro: I think what's really interesting is that they’ve managed to replace the need for hundreds of training problems with zero-shot LLM importance scoring in stage three, which is a huge contribution for cross-domain generalization. It proves you don't always need massive datasets when you have a capable LLM doing the scoring.

Rosa: So, to wrap up this overview of LLM-Flax: it’s a system that automates rule generation, includes budget-aware recovery, and uses zero-shot LLMs for scoring. The paper shows strong performance on smaller problems but highlights the context window limitation as a major hurdle for scaling to larger domains.

Dev: And the conclusion they draw is that open-source LLMs have enough comprehension of PDDL to replace those domain-specific engineering components for smaller and medium problems, opening a practical path toward neuro-symbolic planning that doesn't require constant manual effort.

Taro: The implication for me is that we need to keep pushing on how the system handles those large state spaces; if you can solve the context window issue, this framework could become much more applicable to complex, real-world robotics where things get messy.

Rosa: It sounds like the next step for this research is focusing on expanding that context understanding so that it doesn't just stop working when the problem gets too big. We’ll see how they tackle that in future work.

Conclusion: Rosa: So we're wrapping up on LLM-Flax, which is titled "LLM-Generated Rules and Object Guidance for Neuro-Symbolic Task Planning." It’s about taking a PDDL domain file and automatically building the entire planning setup without needing an expert to do all that manual rule writing.

Dev: Right, so it’s this three-stage framework where one stage creates those rules, another handles recovery if things go wrong, and the third replaces the old object scoring with zero-shot LLM importance guidance. It's all built using a locally hosted LLM.

Taro: The authors are showing how you can use an open-source LLM to do most of the heavy lifting for setting up a neuro-symbolic planner on new domains, which used to take hours of manual engineering work.

Rosa: Yeah, what it means is that deploying a planning system in a brand new area doesn't require hiring an expert just to configure the rules; you just need the domain file.

Dev: And the most significant part they highlight is how stage three replaces that whole trained graph neural network object scorer with zero-shot LLM scoring, which means no more training data needed for that part.

Taro: It’s a big step toward cross-domain generalization because you can use this method on problems you haven't seen before without having to train a model on specific examples of those problems.

Rosa: But the paper does show its limits, especially when the problem gets really big—the context window size becomes a real bottleneck for accurate object importance scoring.

Dev: That context window issue is definitely where we need to focus next; if the system can't handle bigger state spaces, it won't scale well in complex real-world scenarios.

Taro: Exactly, so while it’s great for smaller or medium problems right now, the challenge is making that zero-shot guidance robust enough when there are hundreds of objects involved.

Rosa: It shows that open-source LLMs have enough understanding of PDDL to replace those hard domain-specific engineering parts for smaller tasks.

Dev: So the implication here is a much lower barrier to entry for building these kinds of planning systems, moving away from needing massive amounts of task-specific training data.

Taro: The next thing we need to watch is how they plan to fix that context limitation so this framework can actually tackle those huge, messy problems in the real world.

More episodes

← Home