WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

arXiv:2610.12459 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "WorldGuide: Goal-Directed Video World Model for Procedural Task Execution".

Tom: The gist The WorldGuide model introduces a video world model that turns procedural video generation into closed-loop task execution by coupling a ContextPlanner and an Executor to select and carry…

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about this paper called WorldGuide: Goal-Directed Video World Model for Procedural Task Execution. It's basically about taking those video generators that make cool videos, and turning them into something that actually plans and executes tasks step by step in the real world.

Jane: Right. The big idea here is to move away from just making a long video based on one prompt where the model just keeps going until it runs out of time or hits a fixed end point. Instead, this system figures out what to do next based on what it’s actually seen and what its goal is.

Lu: It frames procedural video generation as closed-loop task execution in visual world space, which means every step requires the model to decide what action to take and then realize that action through a generated clip before deciding the next move or stopping.

Meng: So instead of just generating a whole sequence upfront, you have this loop where the planner looks at the current visual state and history, picks an atomic action, generates a video for that action, and then uses that new video to decide what to do next.

Lalam: The core setup involves two main parts: a ContextPlanner and an Executor. The Planner predicts the next atomic action or task completion from the goal, current visual state, and history.

Tom: And the Executor is responsible for generating the actual video clip based on that predicted action and context. They're trained together in a specific way to make sure they work well together.

Jane: The paper says they are trained sequentially on the same demonstrations but with different conditions, which is important because it lets them learn complementary skills.

Lu: The training setup is specific: the ContextPlanner learns to predict actions by supervised fine-tuning Qwen2 point 5-VL-7B on demonstration histories and their next action or completion targets.

Meng: And the Executor gets trained on those same demonstrations, but it’s conditioned on the frozen ContextPlanner’s action embeddings so it learns to render exactly what the planner proposes. That coupling is key for making sure the prediction actually translates into video.

Lalam: The design choices they made were pretty interesting, especially around memory management for long tasks. They used K equals three recent clip-action pairs as history for the ContextPlanner, which is separate from what the Executor uses <ref:2610.12459#pg1>.

Tom: And that Executor has a hierarchical, compressed visual memory to keep track of the state across long execution horizons without the memory cost growing indefinitely.

Jane: That leads into how they frame these improvements. They show that this closed-loop approach really helps task success rates when compared to just open-loop generation, which is where things get interesting.

Lu: The results on WorldGuide Bench show that closed-loop execution improves Task Success by twenty-one point six two percent over the open-loop variant, and visual feedback alone improves it by eighteen point six one percent <ref:2610.12459#pg1>.

Title and authors: Meng: That's a solid lift, but they also looked at other aspects like plan success and reducing repeat or skip actions during the execution phase.

Jane: They found that closed-loop execution improves Plan Success by five point three four percent and reduces Repeat/Skip by four point four six percent, which shows it helps the model actually follow the plan better <ref:2610.12459#pg1>.

Tom: The memory component also gave them gains of five point seven seven percent on WorldGuide Bench and seventeen point nine one percent on Video-CraftBench, showing that having that persistent visual context is valuable for long tasks too <ref:2610.12459#pg1>.

Lu: They also found their proposed approach achieves the highest Aesthetic Quality and Overall Consistency on WorldGuide Bench, which is a good sign about the visual output quality of this method.

Jane: So what's the big picture here? It’s shifting from just making a video to building a system that actually executes a plan using what it sees, deciding when to stop based on what has been accomplished.

Meng: It means we can build agents that don't just hallucinate the next frame; they commit to an action, see the result, and adjust their plan if things didn't go as expected.

Lu: This validates the idea that procedural video generation needs to be grounded in generated visual state rather than just relying on action history alone for planning.

Tom: And it really highlights a gap they identified earlier: the problem where an executor might not realize an arbitrary action proposed by the planner, and how closed-loop execution addresses that trust issue.

Jane: The paper also points out a real hurdle: existing datasets don't provide demonstrations that pair atomic actions with their visual consequences and explicit completion signals, so they had to create step-level supervision for this kind of training.

Meng: From an engineering standpoint, the need for that explicit step-level supervision is clear. You can't just train a video model and expect it to learn to stop perfectly without teaching it that specific sequence of actions leads to completion.

Lu: The paper’s formulation of closed-loop task execution in visual world space is what really motivates this work, moving the concept into a more actionable framework for video models.

Tom: So, overall, WorldGuide demonstrates that by coupling planning with learned execution in a recurrent loop, you get much better task completion than just open-loop synthesis.

Jane: It supports the idea that instead of thinking about generating a finished product, we should think about executing a sequence of decisions grounded in the visual reality as we go.

Lu: This is taking the next step toward how video world models can use their generated consequences to decide what to do next and when to stop, guiding step-by-step execution toward the final goal.

Meng: For practical AI deployment, this means these models are more robust because they have a mechanism for self-correction in long sequences.

Tom: So, that’s it for WorldGuide: Goal-Directed Video World Model for Procedural Task Execution. It shows that step-by-step execution grounded in visual feedback significantly boosts task success compared to open-loop generation.

The paper's summary: Tom: So, WorldGuide is essentially taking video generation and turning it into a guided task execution system where the model actually plans what to do next based on what it sees happening in real time.

Jane: Right, so instead of just making a long video based on a prompt where the model drifts off course, this system runs this loop: plan, execute action, see result, then plan again. It's about making the AI decide what to do next based on what has actually been accomplished.

Lu: The core idea is that every step needs to be grounded in the visual state and history of what’s already happened, not just a fixed sequence of actions from the start.

Meng: I'm thinking about how practical this is for building agents. So, they train two parts together—a planner and an executor—so the planner learns to propose moves that the executor actually knows how to render correctly.

Lalam: Exactly. The ContextPlanner predicts the next action or when a task is done, and then the Executor renders that specific video clip based on what the planner just told it to do, using this joint training setup.

Tom: And they used this hierarchical memory for the executor's visual memory, which keeps track of things over a long period without growing infinitely with every new frame.

Jane: That’s crucial because long tasks need that state continuity, and it’s what gives them those gains in performance compared to just open-loop generation.

Lu: The results show that this closed-loop approach boosts task success by over twenty percent compared to the older methods they tested.

Meng: Twenty percent is a big jump for procedural video tasks, especially when you factor in how much better it is at reducing those repeat or skip actions during execution.

Lalam: And on top of that, the system actually achieves the best visual quality and consistency on their main benchmark because the planning and executing are tightly coupled.

Tom: So what this really means for us is a shift away from just synthesizing a finished video to building systems that can actually follow complex instructions step by step, using visual feedback to guide their decisions.

Jane: It’s about moving toward agents that aren't just generating text or images, but are capable of executing multi-step procedures grounded in what they observe as they go.

Lu: This validates the idea that a video world model needs to use its generated consequences—the actual visual results—to make decisions about what to do next and when to stop.

Meng: And it highlights a real challenge they pointed out earlier, which is that the executor has to be trained properly on those specific atomic actions before the planner can trust it with complex tasks.

Lalam: It's also interesting how they had to create their own step-level supervision because existing video datasets don't provide demonstrations pairing atomic actions with their visual consequences and explicit completion signals.

Tom: So, this work pushes the boundary on how we train these models—we need that kind of specific, action-consequence supervision for true closed-loop reasoning in video generation.

Jane: And it opens up a whole new way to think about goal-directed video modeling where the execution and planning are inseparable components of the same loop.

Lu: This sets the stage nicely for us to look at how these kinds of decision loops could be applied across different domains, not just procedural videos.

The paper's improvements: Tom: So we’re looking at what WorldGuide suggests as improvements to this closed-loop execution setup. It basically says that just having a loop isn't enough; you need specific ways to make that planning and executing process better.

Jane: Right, it talks about how we can actually improve the planner's ability to predict the next move when things get tricky, and how we can better ensure the executor actually performs what it’s supposed to do.

Lu: They suggest focusing on making sure the planner is really grounded in that visual state as much as possible, so it doesn't just rely on remembering a list of past actions.

Meng: From an engineering standpoint, they are looking at ways to make that planning step more robust against visual noise or when the environment changes unexpectedly during execution.

Tom: They’re also discussing how we can use the visual feedback in a more direct way to help the system replan if it realizes it’s on a wrong track.

Jane: It means we are looking at training methods that give us better visual consequences back to the decision-making part of the model, not just relying on some abstract reward signal.

Lu: They also touched on memory management again, suggesting even smarter ways to compress that visual history so it stays efficient for really long sequences.

Meng: I’m interested in how practical this is for deployment; if we can make the state tracking more efficient, it makes sense for running these long procedures on standard hardware.

Tom: So the main idea is that we need to move beyond just a sequence of steps and build mechanisms where the visual progress actively informs every single decision point along that path.

Jane: It’s about making the feedback loop tighter so that what you see actually changes what you do in a more intelligent, responsive way.

Lu: This points toward future work involving more sophisticated ways to encode and retrieve that visual context during the planning phase, making it less about just recent history and more about meaningful state representation.

Meng: And for the engineering side, they’re hinting at developing better methods for generating those step-level supervision examples so we can train these models effectively on real-world tasks.

Tom: So, the direction is clear: move from simple closed-loop execution to a system that has actively learned how to use its visual output to correct its own path in real time.

Conclusion: Tom: So we’re wrapping up WorldGuide: Goal-Directed Video World Model for Procedural Task Execution by summarizing what this work really means for AI video generation and task completion systems.

Jane: Basically, this paper shows that moving to closed-loop execution—where the model decides what to do next based on the visual results of the action it just took—is way more effective than just making a long video based on an initial prompt.

Lu: It proves that coupling a ContextPlanner with an Executor in this recurrent loop significantly boosts task success rates over open-loop generation, hitting numbers like twenty-one point six two percent improvement.

Meng: For us in the engineering world, the practical implication is that we’re not just training models to generate frames; we’re training them to actually be reliable agents that can execute a sequence of steps.

Lalam: I think it really changes how we view procedural generation, moving it from an open-ended synthesis problem to a concrete planning and execution problem in visual space.

Tom: It confirms that the ability for the AI to use its generated consequences to decide what to do next is a much more powerful mechanism than just relying on action history alone.

Jane: And that leads into how we can build systems that are truly goal-directed, not just blindly following a fixed path.

Lu: The methodology, focusing on joint training between the planner and executor conditioned on the frozen embeddings, seems to be a solid way to ensure that the proposed atomic actions actually get rendered correctly.

Meng: I still have some questions about scaling this up; getting that hierarchical memory to work reliably over very long horizons without just blowing up the token cost is something we’ll need to watch closely as we build larger agents.

Tom: Yeah, scalability is definitely a factor, but the performance gains they showed on benchmarks like WorldGuide Bench are pretty compelling for what it achieves right now.

Jane: It sets a new standard for how video world models should operate—not just as generators, but as sequential task-executing entities.

Lu: This work is really paving the way for future research where we might see these closed-loop concepts applied to more complex environments than just procedural videos, perhaps in robotics or simulation.

Tom: So, WorldGuide gives us a much clearer blueprint for achieving reliable, step-by-step execution in video generation by tying planning directly to realized visual state.

Jane: It’s a strong piece of research that shows the value of this integrated approach over older methods.

Lu: We're excited to see what researchers do next with these concepts, especially how they can adapt this closed-loop framework for even more complex, real-world scenarios.

Ankan Deria, Komal Kumar, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan

Mohamed bin Zayed University of Artificial Intelligence

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 34 pages, 14 figures, 15 Tables

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The gist The WorldGuide model introduces a video world model that turns procedural video generation into closed-loop task execution by coupling a ContextPlanner and an Executor to select and carry

Key concepts

Closed-loop task execution in visual world space
This frames procedural video generation as a continuous loop where decisions are grounded in the generated visual state. The system doesn't just generate a sequence; it plans an action, executes it visually, observes the result, and then decides what to do next or when to stop based on that actual outcome.
ContextPlanner
This component is responsible for predicting the next atomic action or task completion. It takes the initial image, the goal, and a history of past actions and visual states. It learns to decide what step should be taken next by looking at the current progress towards the overall objective.
Executor
The Executor is trained to directly realize the predicted atomic actions from the planner. When given an action suggestion, it generates a corresponding video clip that visually executes that specific step. It learns to render exactly what the planner proposes, ensuring that decisions are grounded in actual visual consequences.

Terminology

Summary

The gist The WorldGuide model introduces a video world model that turns procedural video generation into closed-loop task execution by coupling a ContextPlanner and an Executor to select and carry out atomic actions grounded in generated visual state

How it works

WorldGuide formulates procedural video generation as closed-loop task execution in visual world space, where the ContextPlanner predicts the next atomic action or task completion from the goal, current visual state, and history. The Executor then generates the corresponding video clip conditioned on this predicted action and visual context. This loop continues until a completion token is predicted or a step budget is reached.

The system utilizes two main components trained sequentially on the same demonstrations 1. The ContextPlanner learns to predict the next atomic action or task completion from the generated visual state and action history. It is trained by supervised fine-tuning of Qwen2.5-VL-7B on demonstration histories and their next-action or completion targets. 2. The Executor is trained on the same demonstrations, conditioned on the frozen ContextPlanner’s action embeddings, so it learns to render the same atomic actions the planner is trained to propose.

Key Design Choices

WorldGuide Bench comprises approximately 59K procedural videos across 245 tasks and 27 procedural categories, segmented into temporally grounded atomic action clips with explicit completion signals. The ContextPlanner uses K=3 recent clip-action pairs as history, which is distinct from the Executor’s compressed latent memory. The Executor maintains a hierarchical, compressed visual memory to maintain state across long-horizon execution with bounded history token cost.

Performance and Validation

Experiments validate these design choices, showing that closed-loop execution improves Task Success by 21.62% over the open-loop variant. WorldGuide achieves a 33.33% Task Success on WorldGuide-Bench, compared with 29.90% for the strong recent video model MiniMax-H3. The approach also outperforms MiniMax-H3 on Video-CraftBench under goal-only conditioning (47.69% versus 32.73%).

Ablation and Analysis

The study investigates several empirical questions, including whether closed-loop execution improves procedural task completion and if generated visual feedback improves replanning. Ablation results show that removing step-wise replanning or visual feedback lowers Task Success, demonstrating that the planner must select appropriate atomic actions and the executor must realize their consequences. The hierarchical memory provides gains of 5.77% on WorldGuide Bench and 17.91% on Video-CraftBench.

Conclusion

WorldGuide demonstrates that closed-loop procedural video generation substantially improves task completion, supporting a shift from open-loop future synthesis to closed-loop task execution. The model successfully uses its generated consequences to decide what to do next and when to stop, guiding step-by-step execution toward the final goal. This work takes a step towards how video world models can use their generated consequences to decide what to do next and when to stop, guiding step-by-step execution toward the final goal. The results support a shift from open-loop future synthesis to closed-loop task execution.

--- Page 1 ---

WorldGuide Bench comprises approximately 59K procedural videos across 245 tasks and 27 procedural categories, segmented into temporally grounded atomic action clips with explicit completion signals

Given an initial image x0 and task goal g, WorldGuide performs goal-directed video world modeling through a recurrent planner–executor loop (Fig. 2)

The ContextPlanner and Executor are initialized from Qwen2.5-VL-7B and HunyuanVideo-1.5 respectively

WorldGuide trains both the ContextPlanner and Executor on the same demonstrations, conditioned on the frozen ContextPlanner’s action embeddings, so it learns to render the same atomic actions the planner is trained to propose

The central idea is that a procedure should unfold through decisions grounded in the generated visual state: what to do next and when to stop depend on what has actually been accomplished.

--- Page 2 ---

Video-based world models predict how visual states evolve under actions, goals, or other control signals

Open-loop generation follows a fixed prompt with no intermediate decisions, so it can drift into a wrong step and keep generating after the task is done

Closed-loop planning with a pretrained video executor A correct plan can still fail when the executor cannot realize the requested atomic action (Fig. 1b)

WorldGuide trains the ContextPlanner and Executor on the same demonstrations, conditioned on the frozen ContextPlanner’s action embeddings, so it learns to render the same atomic actions the planner is trained to propose

None of these approaches select actions from visual progress, execute them through directly supervised video generation, and decide completion within a single closed loop

--- Page 3 ---

Given an initial image x0 and task goal g, WorldGuide performs goal-directed video world modeling through a recurrent planner–executor loop (Fig. 2)

If at = DONE, the rollout terminates; otherwise, the Executor fϕ generates the next visual state xt+1, a short video clip, conditioned on at, xt, and visual memory Mt

The history ht holds the latest K=3 actions, each paired with the clip it generated: ht + at appends at together with its clip xt+1 and drops pairs older than K steps

--- Page 4 ---

This gap raises five challenges that a closed-loop procedural video model must address First, if the executor is frozen or only indirectly supervised through a reward model, it cannot be trusted to correctly realize an arbitrary planner-proposed action; the executor itself needs to be trained on the atomic actions it is asked to perform Second, a planner conditioned only on its previously requested actions cannot determine whether those actions were successfully executed, partially completed, or resulted in visual drift. Therefore, planning must be grounded in the generated visual state, not action history alone Third, task completion must be predicted from visual progress rather than determined by a fixed generation length or external retry budget Fourth, because every generated clip affects subsequent decisions, long-horizon execution requires persistent visual context without unbounded memory growth Fifth, existing instructional-video datasets do not provide demonstrations that pair atomic actions with their visual consequences and explicit completion signals. Therefore, closed-loop procedural training requires dedicated step-level action–video supervision

--- Page 5 ---

Goal-directed generation therefore requires a planner grounded in the evolving visual state and an executor trained to realize the actions it predicts

Closed-loop execution improves Task Success by 21.62% over the open-loop variant visual feedback improves Task Success by 18.61% and Plan Success by 5.34% while reducing Repeat/Skip by 4.46% while memory provides gains of 5.77% on WorldGuide Bench and 17.91% on Video-CraftBench Our proposed approach achieves the highest Aesthetic Quality and Overall Consistency on WorldGuide Bench

--- Page 6 ---

None of these approaches select actions from visual progress, execute them through directly supervised video generation, and decide completion within a single closed loop WorldGuide instead learns next-action selection, atomic-action execution, and completion prediction from the same demonstrations, coupling planning with learned execution in a recurrent closed loop

--- Page 7 ---

This motivates our formulation of closed-loop task execution in visual world space First, if the executor is frozen or only indirectly supervised through a reward model, it cannot be trusted to correctly realize an arbitrary planner-proposed action; the executor itself needs to be trained on the atomic actions it is asked to perform Second, a planner conditioned only on its previously requested actions cannot determine whether those actions were successfully executed, partially completed, or resulted in visual drift.

Improvements for AI systems

  1. Bold header: Closed-loop procedural execution capability

WorldGuide predict[s] each action-consequences and termination from generated progress, which allows it to decide[to] stop based on visual results, overcoming the limitation of Open-loop generation cannot adapt to execution outcomes.

  1. Bold header: Joint training of planner and executor

The system utilizes a sequential training approach where the ContextPlanner is first fine-tuned, and then frozen while the Executor is fine-tuned on action captions embedded by it, ensuring that the Executor learns to render the same atomic actions the planner is trained to propose.

  1. Bold header: Bounded visual memory for long-horizon state continuity

The Executor employs a hierarchical, compressed visual memory where older frames are progressively compressed, which keeps the number of memory tokens bounded as the generated video becomes longer, preventing unbounded memory growth while maintaining state continuity across successive steps.

  1. Bold header: Enhanced planning grounded in visual state

The ContextPlanner is trained to predict the next atomic action or a completion token from the current visual state xt and the clip–action history ht, meaning planning must be grounded in the generated visual state, not action history alone.

  1. Bold header: Improved procedural task success rate

By implementing closed-loop execution, WorldGuide achieves a significant improvement, increasing Task Success by 21.62% over the open-loop variant and reaching 33.33% Task Success on WorldGuide-Bench.

Abstract

Video generators and video-based world models can synthesize plausible visual trajectories, but long-horizon procedural tasks require generation to adapt to what has actually been produced. A model must determine the next action from its generated state, execute that action, and recognize when the task is complete. Open-loop generation cannot adapt to execution outcomes, while existing closed-loop systems often rely on pretrained executors or indirect verification. This leaves a gap between deciding an action and successfully realizing it. We formulate procedural video generation as closed-loop task execution in visual world space and introduce WorldGuide. Given only an initial image and a task goal, WorldGuide predicts an atomic action, generates its corresponding video clip, and uses the generated result to select the next action or terminate. The Planner and Executor are trained on the same step-level procedural demonstrations: the Planner learns to predict the next atomic action or task completion from visual progress, while the Executor is directly trained to realize the predicted actions. Hierarchical visual memory maintains state across long-horizon execution with bounded history token cost. Due to the lack of step-level action-video supervision for joint planner-executor training, we introduce WorldGuide Bench: approximately 59K step-annotated videos across 245 tasks and 27 procedural categories. WorldGuide achieves a 33.33% Task Success on WorldGuide-Bench, compared with 29.90% for the strong recent video model MiniMax-H3, even though MiniMax-H3 receives reference action plans, and achieves 47.69% on VideoCraft-Bench compared with 32.73% for MiniMax-H3 under goal-only conditioning. These results demonstrate the importance of coupling planning with learned execution for goal-directed procedural video generation.

Sources

Related papers