Hierarchical Planning with Latent World Models

arXiv:2604.03208 · cs.LG · Submitted 2026-04-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Hierarchical Planning with Latent World Models".

Tom: The gist The Hierarchical Planning with Latent World Models (HWM) framework enables zero-shot, non-greedy manipulation on a real robot from a single goal image, where single-level world-model planners fail.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: We've looked at the title and authors of "Hierarchical Planning with Latent World Models." It’s a mouthful, but it sums up the core idea: using hierarchy to plan with latent world models.

Jane: The authors are Zhang, Terver, Zholus, Chitnis, Sutaria, Assran, Balestriero—they bring together expertise from Meta and NYU and other places.

Lu: What this means in simple terms is that they are proposing a new way to structure the planning process itself.

Meng: Instead of one giant planner trying to figure out everything at once across a long sequence of steps, they break it down into two or more specialized planners working together.

Lalam: It’s like having a manager who sets big, abstract goals while a worker handles the fine details of getting there.

Tom: That's exactly right, Jane. The core idea is decomposition. They are taking the complexity of long-horizon tasks and splitting it up into shorter subproblems that are easier to solve individually.

Jane: And they do this all within a shared latent space, which is important because it keeps the information consistent across both levels of planning.

Lu: The paper introduces this specific coupling mechanism where predictions from the coarser model feed as subgoals for the finer model to optimize primitive actions in latent space.

Meng: So the high-level planner figures out a macro-action, and that prediction acts as a target point for the low-level planner to hit with actual movement commands.

Lalam: It’s about making sure the high-level plan is actually achievable by ensuring that there's a detailed, low-level plan matching its latent prediction.

Tom: It’s about creating this feedback loop where both levels replan as new observations come in, which is crucial for real robot control.

Jane: So, when you look at the overall structure of "Hierarchical Planning with Latent World Models," it’s a system designed to manage complexity by layering planning decisions.

The paper's summary: Tom: We're now going over the actual summary of "Hierarchical Planning with Latent World Models." This paper outlines the architecture and the specific planning paradigm they are using.

Jane: They describe how they train these world models solely via next-latent prediction, which is a key part of their training strategy.

Lu: The training involves learning world models at multiple temporal resolutions within that shared latent space, so they have different levels of detail available.

Meng: This means the low-level model F(one) predicts the very next state based on current state and action, while the high-level model F(two)phi predicts dynamics over longer stretches <ref:2604.03208#pg1>.

Lalam: And they use a learned action encoder to compress those long sequences of primitive actions into these latent macro-actions for the high-level planner.

Tom: The planning procedure itself is hierarchical: a high-level planner optimizes abstract macro-actions using the long-horizon world model to reach the final goal embedding.

Jane: The output of that high level plan, which is the first predicted latent state from the rollout, serves as the subgoal for a low-level planner.

Lu: That low-level planner then optimizes primitive actions to reach that specific subgoal using its short-horizon world model.

Meng: The energy functions they define are E2 for the high level and E1 for the low level, which guide each planner toward its respective objective.

Lalam: This structure is what allows them to achieve things like seventy percent success on Franka pick-and-place from a single goal image, compared to zero percent for VJEPA2-AC one <ref:2604.03208#pg1>.

Tom: The paper also emphasizes that this entire framework works in a closed-loop MPC fashion, meaning it replans every k interactions using the hierarchical planner.

Jane: So, the core summary is this: training multiple temporal models at different scales, coupling them via latent subgoal matching to allow for two levels of planning that work together.

The paper's improvements: Tom: Let’s talk about what the authors claim are the actual improvements they’ve achieved with this framework. They aren't just presenting a new idea, they are showing results compared to existing methods.

Jane: The main improvement is that HWM solves pick-and-place from a single goal image at seventy percent success where prior single-level planning failed completely, achieving zero percent on VJEPA2-AC.

Lu: They also show consistent gains across three latent world model backbones—VJEPA2-AC for manipulation, DINO-WM for push manipulation, and PLDM for maze navigation.

Meng: And these gains are significant; they show that hierarchical planning improves both performance by up to plus forty percent and efficiency by up to plus thirty-nine percent absolute improvement on long-horizon tasks.

Tom: The efficiency gain is really notable here because it shows that this structure helps reduce search complexity, requiring up to three times less planning compute under a fixed planning budget.

Jane: That computational saving means we can actually run these more complex plans on hardware that might otherwise struggle with the computation needed for single-level planners.

Lu: The analysis of the components is interesting because it shows that manual subgoals isolate planning from representation, and HWM automatically recovers most of that oracle performance, getting seventy percent for pick-and-place from a single goal image.

Meng: They also found that the learned macro-actions they produce are consistently better than both concatenating primitive actions and hand-crafted summaries like net end-effector displacement.

Lalam: The paper suggests that using a macro-action dimension of four seems to strike the right balance for tasks like Franka tasks, which is useful information for how we structure these latent summaries.

Conclusion: Tom: We’re wrapping up with the conclusion of "Hierarchical Planning with Latent World Models." This paper sets a lot of expectations for what's possible in embodied AI.

Jane: Essentially, HWM is presented as the first world-model planner to demonstrate zero-shot, non-greedy real-robot manipulation from pixels using only a single goal image.

Lu: The authors suggest that hierarchy is a practical mechanism for extending latent world-model MPC beyond short, greedy behaviors by coupling multiple latent world models through shared latent subgoal matching.

Meng: It’s about enabling zero-shot planning with learned world models trained from offline trajectories, which is a big step for making these systems more general.

Lalam: This framework opens up possibilities where the AI can plan much more robustly than before because of this hierarchical structure, especially when dealing with long sequences of observations.

Tom: The results show that even though success rates degrade with task horizon for every method they evaluate, HWM still keeps performing better on those longer tasks compared to single-level planners.

Jane: So, the big implication is that for anyone working on embodied control, hierarchy seems like a practical way to move beyond simple greedy behaviors by managing the planning complexity more effectively.

Lu: The paper shows consistent gains across VJEPA2-AC, DINO-WM, and PLDM backbones across different task suites, proving that the hierarchical approach is robust across different types of tasks.

Meng: We’re just waiting to see how much this translates into real deployment on a robot that isn't in a perfect simulation environment.

Lalam: It’s an interesting piece of research because it shows how structure in the AI system can directly translate into better performance on the actual physical world.

FAIR at Meta · New York University

cs.LG

Submitted: 2026-04-03

Updated: 2026-10-12

Comments: Accepted at CoRL 2026. 27 pages, including references and appendices. Project page: https://kevinghst.github.io/HWM/

Code: https://github.com/kevinghst/HWM_PLDM

Project page: https://kevinghst.github.io/HWM

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The Hierarchical Planning with Latent World Models (HWM) framework enables zero-shot, non-greedy manipulation on a real robot from a single goal image, where single-level world-model

Key concepts

Latent World Models: These are AI models trained on visual data that learn how the world evolves over time in a compressed, abstract latent space. They predict future states or dynamics based on current observations and actions. HWM uses multiple versions of these models at different resolutions to capture both short-term movements and long-term task progression.
Hierarchical Planning: This is a planning strategy that breaks down a large, complex problem into smaller, manageable subproblems. In HWM, a high-level planner determines abstract 'macro-actions' toward the final goal using a long-horizon model. These macro-actions then serve as targets for a lower-level planner which focuses on executing the fine details of primitive actions.
Action Encoder: This is a learned component that takes sequences of low-level, small actions and compresses them into a single, meaningful latent 'macro-action'. This allows the high-level planner to work efficiently over long time horizons by focusing on these summarized, abstract steps rather than tracking every tiny movement.
Energy Functions (E1 and E2): These are mathematical functions used by the planners to guide their search for the best sequence of actions. The high-level energy function guides the selection of macro-actions based on reaching a goal state, while the low-level energy function guides primitive actions to achieve those specific latent subgoals set by the high-level planner.

Terminology

World Model — 6× in this paper · explained in 7 episodes
Hierarchical Planning with Latent World Models — 2× in this paper
Latent World Models — 2× in this paper · explained in 3 episodes
Latent Space — 1× in this paper · explained in 7 episodes
Model Predictive Control (MPC) — 1× in this paper · explained in 7 episodes

Summary

The gist The Hierarchical Planning with Latent World Models (HWM) framework enables zero-shot, non-greedy manipulation on a real robot from a single goal image, where single-level world-model planners fail.

How it works

HWM is an architecture and planning paradigm for hierarchical model predictive control (MPC) directly on visual world models trained solely via next-latent prediction. It trains world models at multiple temporal resolutions within a shared latent space, and couples them at planning time by letting predictions from the coarser model serve as subgoals that the finer-scale MPC matches in latent space. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions.

Hierarchical Planning Procedure

The top-down hierarchical planning strategy operates entirely in this latent space. A high-level planner optimizes abstract macro-actions toward the goal using a long-horizon world model, and the resulting latent predictions serve as subgoals for a low-level planner optimizing over primitive actions. The high-level energy function is defined as E2(ˆl1:H; z1, zg) ≜ zg − F(2)(ˆl1:H; z1). The low-level energy function is defined as E1(ˆa1:h; z1, z˜1) ≜ z˜1 − F(1)(ˆa1:h; z1). The agent re-plans every k interactions using the hierarchical planner.

World Model Architecture

HWM requires two world models operating at different time scales in a shared latent space. The low-level model F(1) predicts zt+1 = f(zt, at) and the high-level model F(2)ϕ predicts dynamics across temporally extended segments conditioned on latent macro-actions. Macro-actions are produced by a learned action encoder Aψ that summarizes temporally extended sequences of low-level actions.

Key Findings and Results

HWM solves pick-&-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3× less planning compute. The framework demonstrates consistent gains across three latent world-model backbones — VJEPA2-AC (manipulation), DINO-WM (push manipulation), and PLDM (maze navigation). In the Diverse Maze evaluation, HWM nearly doubles the success rate from 44% to 83% on the hardest setting.

Analysis of Planning Components

The analysis shows that manual subgoals isolate planning from representation. HWM recovers most of this oracle performance automatically (70%/60% from a single goal image). Learned macro-actions consistently outperform both concatenating primitive actions and hand-crafted summaries such as net end-effector displacement. A macro-action dimension of 4 strikes this balance for the Franka tasks.

Conclusion

HWM is the first world-model planner to demonstrate zero-shot, non-greedy real-robot manipulation from pixels with a single goal image. The results suggest that hierarchy is a practical mechanism for extending latent world-model MPC beyond short, greedy behaviors. The framework enables zero-shot planning with learned world models trained from offline trajectories.

Limitations

Success rates degrade with task horizon for every method we evaluate. HWM uses a single latent space across hierarchy levels. Subgoal quality is shaped by choices at training and inference. It differs from prior hierarchical model-based methods by coupling multiple latent world models through shared-latent subgoal matching <ref:2604.

Improvements for AI systems

Bold header: Hierarchical Planning with Latent World Models (HWM) for Zero-Shot Manipulation

The improved system can solve non-greedy real-robot manipulation from pixels with a single goal image, achieving 70% success on tasks like pick-and-place where the optimal trajectory must temporarily move away from the goal.

Bold header: Task Agnostic Performance Improvement

HWM consistently improves performance across three different world model backbones—VJEPA2-AC, DINO-WM, and PLDM—demonstrating that hierarchical planning improves both performance (up to +44% and +39% absolute) and efficiency (up to a 3× reduction in planning compute).

Bold header: Enhanced Long-Horizon Reasoning

The system can handle long-horizon tasks like Push-T with greater robustness, as it shows that the hierarchical planner remains substantially more robust at longer horizons compared to single-level planners, even when task difficulty increases.

Bold header: Improved Compute Efficiency in Planning

HWM achieves superior efficiency by reducing search complexity; the paper states it requires up to 3× less planning compute than single-level planners while matching or exceeding performance under a fixed planning budget.

Bold header: Robustness Against Model Capacity

The system's gains are shown to stem from structure rather than size, as flat models with comparable or larger parameter counts fail to match hierarchical performance—in fact, it often degrades it.

Abstract

World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long horizon model serve as subgoals for the short horizon model via latent matching, without task-specific rewards or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.