VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents

arXiv:2606.19729 · cs.RO, cs.AI · Submitted 2026-06-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents".

Dev: The gist: VOiLA, a framework for approximate POMDP solving without a priori models,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Now, looking at what the paper actually says about VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents, it summarizes that this framework learns task-agnostic POMDP models for online planning under uncertainty.

Dev: It explains that a POMDP problem is defined by a seven-tuple ⟨S, A, O, T, Z, R, γ⟩ and the agent maintains a belief b which is a probability distribution over states <ref:2606.19729#pg3,a 7-tuple ⟨S, A, O, T, Z, R, γ>.

Rosa: The paper focuses on learning transition and observation samplers using conditional diffusion models and learning observation-likelihood models for particle-based belief updates.

Taro: It also mentions that the goal is to enable efficient online planning by distilling those diffusion samplers into compact feedforward generators for the transition sampler fT.

Dev: The summary highlights how they learn these from offline generated simulation data, which means no a priori models are needed for learning the dynamics.

Rosa: So it’s about building reusable models from simulation and then using those learned models online to plan in real-time belief space.

Taro: What I find important here is that this lets you generate good real-world strategies using only simulation data, which is what the paper says.

Dev: That means we don't need to spend huge amounts of time collecting a massive, task-specific dataset to get started with planning.

The paper's summary: Rosa: The paper points out several improvements they made to this approach, focusing on making it practical for online use.

Dev: They emphasize that the transition sampler is modeled as s prime equals fT of s, a, omega T where omega T is a latent noise variable capturing transition stochasticity.

Rosa: And they distill the diffusion samplers into compact feedforward generators so that they are efficient enough to serve as generative models during online POMDP planning.

Taro: That addresses the issue of computational cost directly because it makes the learned generative models practical for real-time use.

Dev: They also introduce a conditional observation autoencoder to model high-dimensional observations by projecting them into a lower-dimensional latent space Z with dim(Z) less than dim(O).

Rosa: So, they're compressing the observation space first, then using diffusion models in that compressed latent space before distilling it into the compact fZ generator.

Taro: That compression step is key for handling things like images efficiently during online belief-space planning.

The paper's improvements: Dev: So to wrap up, the paper on VOiLA shows how you can learn efficient POMDP models from relatively small amounts of training data and generalize to unseen environment configurations.

Rosa: It also shows that they can transfer these models successfully from simulation to real-world planning problems without needing additional training on the physical robot itself.

Taro: The ability to generalize to unseen environments is pretty important because recurrent policies often drop off when they hit things they haven't seen before.

Dev: I think this combination of learning task-agnostic models with scalable online planning is a promising direction for robotic sequential decision making under uncertainty, according to the VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents paper.

Rosa: It seems like they've built something that handles complex dynamics and high-dimensional observations while keeping the planning efficient enough for deployment on physical robots.

Taro: I think having models that are learned from simulation data rather than being handcrafted is a big step toward making robots more robust in the real world.

Dev: It really shows how learning these components can be integrated with online planning in a way that is scalable across different tasks and environments.

Conclusion: Rosa: So VOiLA is essentially showing us how you can learn task-agnostic POMDP models just from simulation data, without needing an upfront model of what the robot or environment does, and then use those learned models for online planning in belief space.

Dev: Right, it’s about building these reusable transition and observation samplers from offline generated data—that means you don't need a massive dataset specific to every single task you want the robot to do.

Taro: The real-world test, though, is whether those models actually generalize when the world throws something unexpected at them in a live environment.

Rosa: They show that VOiLA can transfer these learned models successfully from simulation all the way over to physical robots for tasks they haven't explicitly trained on.

Dev: That’s significant because usually, if you train a policy in simulation, it struggles when the real sensor noise or dynamics are just slightly different.

Taro: I think that ability to handle unseen environments is what makes this approach interesting when you look at failure modes; it suggests the learned dynamics capture more general physics than a task-specific model would.

Rosa: And they get massive computational efficiency by distilling those expressive conditional diffusion models down into compact feedforward generators for the transition sampler, which makes online planning actually feasible.

Dev: That distillation step is key for loop rate concerns, because if you have a complex generative model running every time you need to plan a move, it won't work in real-time.

Taro: It’s good that they handled the high-dimensional observations too; they use an autoencoder to map those images into a smaller latent space before sampling.

Rosa: So, the paper on "VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents" proves that learning these generative components and integrating them with VOPP works really well for sequential decision making under uncertainty.

Dev: It’s a solid framework for handling continuous belief spaces because they extended VOPP with something called Progressive Widening to handle those observation branches smoothly.

Taro: The limitation they mention is that the success rate depends on how much data you put into the simulation, but even then, it shows robustness compared to other methods like recurrent SAC when tested on unseen environments.

Rosa: Overall, VOiLA suggests that combining these learned POMDP models with scalable online planning is a really promising direction for making robots handle uncertainty in real-world scenarios.

Dev: It’s a step toward systems that can learn their own dynamics without needing perfect initial mathematical descriptions of the world.

Taro: Next up, we'll look at some other work on how diffusion models are actually capturing the geometry of constraint manifolds when dealing with inverse kinematics.

Australian National University · The University of Sydney

cs.RO, cs.AI

Submitted: 2026-06-18

Updated: 2026-10-15

Comments: Accepted at the 2026 International Symposium of Robotics Research (ISRR)

Code: https://github.com/twni2016/pomdp-baselines

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: VOiLA, a framework for approximate POMDP solving without a priori models, combines diffusion-based generative modeling with model distillation to learn task-agnostic transition and

Key concepts

POMDP
Partially Observable Markov Decision Process is a mathematical framework used to model decision-making in situations where the agent cannot see the full state of the world. It involves states, actions, and observations, allowing robots to make decisions under uncertainty about what they actually perceive.
Diffusion Models
These are generative models that learn to create data by gradually adding noise to random input until it becomes structured data. In VOiLA, these models are used to learn how the robot's next state and observations are generated from the current state and action, capturing complex dynamics.
Model Distillation
This process takes a large, complex generative model (like a diffusion sampler) and compresses it into a much smaller, faster version (a feedforward generator). This distillation makes the learned POMDP models efficient enough to be used for quick online planning during robot operation.
Task-Agnostic Models
These are world models that learn general rules of physics and observation structure from diverse simulation data, rather than learning specific behaviors for one task. Because they capture general dynamics, these learned models can be applied effectively to completely new environments without needing extensive retraining.

Terminology

Summary

The gist: VOiLA, a framework for approximate POMDP solving without a priori models, combines diffusion-based generative modeling with model distillation to learn task-agnostic transition and observation samplers and observation-likelihood models that support both particle-based belief updates and online belief-space planning.

Overview of VOiLA

VOiLA is a sequential decision-making framework that combines the recently proposed massively parallel online POMDP planning VOPP with learned POMDP Models Online Belief Space Planning with VOPP In particular, our method learns three POMDP model components: (a) a transition sampler fT: S×A 7→ S to approximately sample next states according to the transition function T(s, a, s′), (b) an observation sampler fZ: S ×A 7→ O that approximately samples observations according to Z(s′, a, o), where highdimensional observations are modeled in a learned latent observation space and (c) a likelihood model l: S ×A×O 7→ R which approximates the (unnormalized) POMDP observation density Z. A core requirement of the learned transition and observation samplers is that they must be sufficiently expressive to capture complex stochastic robot dynamics (e.g., legged locomotion) and high-dimensional observation distributions (e.g., images), while remaining efficient enough to serve as generative models during online POMDP planning.

Learning Task-Agnostic POMDP Models

VOiLA learns task-agnostic transition and observation models purely from offline generated simulation data. Training data for the learned POMDP models is generated entirely offline using high-fidelity robot simulators such as Isaac Sim. Formally, we construct an offline dataset D = ξ (n) N n=1, where each trajectory ξ (n) = (s (n) i, a (n) i, o (n) i) K i=1 consists of a sequence of states, actions, and observations generated through interaction with the simulator. Because the data collection policy is independent of the downstream task and reward structure, the resulting dataset captures general dynamics and observation structure rather than task-specific behaviors.

Learning Model Components

VOiLA learns three efficient model components for online planning: a transition sampler fT that approximately samples successor states according to the transition distribution T(s, a, s′), an observation sampler fZ that approximately samples observations according to the observation distribution Z(s′, a, o), and a likelihood model l(s′, a, o) that approximates the (unnormalized) observation density Z. The transition sampler is modeled as s′ = fT (s, a, ωT), where ωT ∼ N (0, I) is a latent noise variable that captures transition stochasticity. Real-world transition dynamics are often highly nonlinear and multimodal due to contact dynamics, actuation errors, and robot–environment interactions. Capturing such dynamics therefore requires expressive generative models. To enable efficient online planning, we distill an ODE-based diffusion sampler [28] into the compact model fT, which directly maps (s, a, ωT) to a next-state sample s′, such that s′ is approximately distributed according to T(s, a, s′).

Distillation and Likelihood Modeling

To enable efficient online planning, the diffusion samplers are distilled into compact feedforward generators. The observation sampler fZ is learned analogously to the transition sampler described in the previous paragraph. For high-dimensional observations such as images, we introduce a lower-dimensional latent observation space Z with dim(Z) ≪ dim(O) using a conditional observation autoencoder consisting of an encoder E and decoder D, such that z = E(s′, a, o) and o = D(s′, a, z). We then train a conditional diffusion model in the latent observation space using tuples (s′, a, z), where z = E(s′, a, o). Following the same diffusion and distillation procedure used for the transition sampler, we finally distill the diffusion sampler into the compact feedforward observation sampler z = fZ(s′, a, ωZ), where ωZ ∼ N (0, I) is latent noise. To support continuous observation spaces, we extend VOPP with a fully vectorized implementation of progressive widening [29].

Online Planning with VOPP

The learned transition, observation, and likelihood models are integrated into VOPP for both online belief-space planning and particle-based belief updates. During planning, vectorized forward simulations are generated directly using the offline learned transition and observation samplers instead of relying on handcrafted models. To handle continuous observation spaces, we extend VOPP with a vectorized implementation of Progressive Widening (PW) [29]. PW incrementally limits the number of observation branches associated with each action node in T. The likelihood model l(s′, a, o) is then used to compute relative weights for successor states with respect to the observation stored at the selected branch, after which successor states are resampled according to these weights.

Experimental Results and Generalization

Experimental results on three benchmark problems demonstrate that VOiLA can learn efficient POMDP models from relatively small amounts of training data, generalize to unseen environment configurations, and transfer successfully from simulation to real-world planning problems without additional training. In FloorPositioning, VOiLA requires only 50K transitions, compared to 2.6M transitions for recurrent SAC and 32M transitions for VTS, while still achieving a 100% success rate. Similarly, in LeggedNavigation, VOiLA uses 50K transitions, compared to 750K transitions for recurrent SAC, and achieves competitive or higher success rates across all tested environments. A key advantage of learning task-agnostic POMDP models is generalization to unseen environments. This is demonstrated in the LeggedNavigation problem, where training data was collected only in environments 1 and 2. Although recurrent SAC achieves strong performance on the training environments, its success rate drops to 12% and 10% in unseen environments 3 and 5, respectively. In contrast, VOiLA maintains success rates between 72% and 80% across all unseen environments and consistently achieves the highest average discounted return. Results on TargetFinding To see whether POMDP models learned in simulation transfer to a real robot, we deployed VOiLA on the TargetFinding problem. Across 10 real-world test runs, including five runs for each possible target location, VOiLA successfully completed the task in every trial.

Conclusion

VOiLA enables efficient belief-space planning while retaining the ability to represent complex transition and high-dimensional observation functions. Experimental results on three benchmark problems demonstrate that VOiLA can learn efficient POMDP models from relatively small amounts of training data, generalize to unseen environment configurations, and transfer successfully from simulation to real-world planning problems without additional training. These results suggest that combining reusable learned POMDP models with scalable online planning is a promising direction for robotic sequential decision making under uncertainty.

Acknowledgements

This work was supported by the ARC Research Hub in Intelligent Robotic Systems for Real-Time Asset Management (IH210100030), in collaboration with the University of Sydney and Nexxis Technology.

References

[1] Arulampalam, M., Maskell, S., Gordon, N., Clapp, T.

Improvements for AI systems

  1. textbf Learned Task-Agnostic POMDP Models for Generalization in Online Planning: The VOiLA Framework enables generalization to unseen environment configurations by learning transition and observation models purely from simulation data, allowing policies to be used across tasks sharing the same environment dynamics.

  2. textbf Massive Computational Efficiency via Model Distillation: The system reduces sampling cost by up to nearly three orders of magnitude by distilling expressive conditional diffusion models into compact feedforward generators, making learned generative POMDP models practical for online planning.

  3. textbf Robust Real-World Deployment Without Extensive Training: VOiLA enables the deployment of models trained solely from simulation data onto physical robots, achieving successful task accomplishment in physical robot evaluation runs and demonstrating robustness to unseen environments where a recurrent policy fails.

Sources

Related papers