Unifying Deep Predicate Invention with Pre-trained Foundation Models
cs.RO
Submitted: 2025-12-19
Updated: 2026-09-25
Comments: Published in IEEE TRANSACTIONS ON ROBOTICS (TR-O), Special Issue on Foundation Models for Robotics
Code: https://github.com/thu-ml/RDT2
Project page: https://unipred.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-horizon robotic tasks are hard due to continuous state-action spaces and sparse feedback.
Terminology
Abstract
Long-horizon robotic tasks are hard due to continuous state-action spaces and sparse feedback. Symbolic world models help by decomposing tasks into discrete predicates that capture object properties and relations. Existing methods learn predicates either top-down, by prompting foundation models without data grounding, or bottom-up, from demonstrations without high-level priors. We introduce UniPred, a bilevel learning framework that unifies both. UniPred uses large language models (LLMs) to propose predicate effect distributions that supervise neural predicate learning from low-level data, while learned feedback iteratively refines the LLM hypotheses. Leveraging strong visual foundation model features, UniPred learns robust predicate classifiers in cluttered scenes. We further propose a predicate evaluation method that supports symbolic models beyond STRIPS assumptions. Across five simulated and two real-robot domains, UniPred achieves 2-4 times higher success rates than top-down methods and 3-4 times faster learning than bottom-up approaches, advancing scalable and flexible symbolic world modeling for robotics. For more information and project materials (video, code...) visit https://unipred.github.io/
Sources
- DINOv3
- GPT-4 Technical Report
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
- From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models
- InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning
- Gemini: A Family of Highly Capable Multimodal Models
- OpenVLA: An Open-Source Vision-Language-Action Model
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- Learning Transferable Visual Models From Natural Language Supervision
- PaLM-E: An Embodied Multimodal Language Model
- Guiding Long-Horizon Task and Motion Planning with Vision Language Models
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
- SAM 2: Segment Anything in Images and Videos
- Relational inductive biases, deep learning, and graph networks
- Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
- Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
- A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
- SLAP: Shortcut Learning for Abstract Planning
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies