Feedback World Model Enables Precise Guidance of Diffusion Policy

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Feedback World Model Enables Precise Guidance of Diffusion Policy".

Dev: Here's the gist of the paper. The gist The proposed feedback world model enables precise guidance of diffusion policy by using real-time observations to correct latent predictions online, which substantially improves prediction accuracy and policy performance under distribution shift.

Rosa: World models aim to improve robotic decision making by predicting the consequences of actions<ref:2605.15705#pg2> However, in practice, their predictions often become unreliable once the robot encounters states outside the training distribution, limiting their effectiveness at deployment<ref:2605.15705#pg3> We observe that execution itself provides a natural but underutilized signal: after each action, the robot directly observes the true next state, revealing the mismatch between predicted and actual outcomes<ref:2605.15705#pg4> Building on this insight, we propose feedback world model, a new paradigm that closes the loop between prediction and observation at inference time<ref:2605.15705#pg5> Instead of treating the world model as a static open-loop predictor, our method maintains a lightweight feedback state that is updated online to iteratively correct future predictions, compensating for model errors using real-time observations without additional training data or parameter updates<ref:2605.15705#pg5> We show that this process can be interpreted as a latent-space observer and admits convergence guarantees under mild conditions<ref:2605.15705#pg5> We further introduce action-aware guidance to better translate corrected predictions into control by emphasizing action-controllable components while suppressing irrelevant variations<ref:2605.15705#pg5> Experiments on LIBERO-Plus, Robomimic, and real-world manipulation tasks demonstrate that our method substantially improves both prediction accuracy and policy performance under distribution shift<ref:2605.15705#pg5> In particular, it reduces world model prediction error by up to 76.4% and improves out-of-distribution (OOD) success rate by 30%<ref:2605.15705#pg5> These results show that incorporating real-time feedback at inference time provides a simple yet powerful VGGT (Wang et al., 2025) scheme combines one global sparse pass with dense local passes, enabling long-horizon manipulation videos to be processed without fitting the entire sequence in memory.

Dev: Page 17 µ0: A Scalable 3D Interaction-Trace World Model A.4 Progressive 3D Tracking Across Chunks Running a 3D point tracker independently per chunk discards continuity: the same physical point would be re-discovered with a new identity in every chunk, and any object missed by semantic clustering in one chunk would simply vanish.

Rosa: We instead track progressively.

Paper summary: Rosa: So, we’re looking at this new thing called Feedback World Model that aims to fix how robots make decisions by getting real-time feedback during the actual planning phase. Basically, they’re tackling the problem where world models get unreliable when the robot sees something it wasn't trained on.

Dev: Right. The main idea here is that execution itself gives us a signal we usually ignore: after the robot takes an action, it directly sees the actual next state and compares it to what its prediction said happened. It’s like checking your GPS against reality while you’re driving; you see where you are wrong right away >

Taro: So, instead of just throwing away that mismatch when the robot hits something unexpected, this model keeps a small feedback state that updates online to correct the future predictions it's about to make. It uses those real-time observations to compensate for errors without needing any new training data or updating the main model parameters >

Rosa: Exactly. The paper proposes this feedback world model as a way to close that loop between prediction and observation right at inference time, instead of just treating the world model like a static predictor >

Dev: And they show that this process can be seen as some kind of latent-space observer. They claim it has convergence guarantees under certain conditions, specifically when the dynamics residuals are bounded and asymptotically convergent with a stable feedback gain >

Taro: It sounds like they’re not just patching errors; they’re building a mechanism that actively learns from the immediate experience during deployment to keep predictions accurate under distribution shift >

Rosa: And they've tested this on tasks like LIBERO-Plus and Robomimic, showing substantial improvements in both prediction accuracy and policy performance under those challenging conditions >

Dev: They report reducing world model prediction error by up to seventy-six point four percent in some cases, which is a pretty big number when you’re dealing with out-of-distribution states >

Taro: And they even show it can improve the success rate for out-of-distribution scenarios by about thirty percent >

Conclusion: Rosa: So, wrapping up this discussion on Feedback World Model Enables Precise Guidance of Diffusion Policy, it sounds like the core contribution is this feedback mechanism that lets the model correct its own future predictions using real-time data during policy inference >

Dev: It moves world models from being just static predictors to being active tools that adapt their forecasts based on what’s happening right now, which means they are much more robust when the robot runs into situations it hasn't seen in training >

Taro: The implication for autonomous systems is that you don't need massive amounts of new labeled data every time you encounter a tricky situation; you just need the ability to observe and correct errors online >

Rosa: That’s right. It’s a simpler way to handle distribution shift than constantly retraining the whole model or collecting more data, which makes deployment much more practical for real-world robotics >

Dev: The authors show this approach can even be applied to existing foundation models like Fast-WAM-Joint without having to update their parameters, which points toward a really plug-and-play capability for other systems >

Taro: It’s interesting how they also added action-aware guidance on top of the corrected predictions, which helps translate those refined predictions into better actual control actions by focusing on what parts of the model are actually controllable >

Rosa: So, in simple terms, this paper shows that incorporating real-time feedback at inference time provides a way to get significantly better prediction accuracy and policy performance when you’re working with world models in the real world >