PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning

arXiv:2606.21139 · cs.RO, cs.AI, cs.LG · Submitted 2026-06-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning".

Dev: The gist Latent action pretraining learns representations of visual change from pairs of observations,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called "PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning". It sounds like they're trying to fix a problem where latent actions, which are supposed to summarize visual changes, get tangled up.

Dev: Yeah, it seems like the existing methods usually encode every transition as one single unstructured representation that mixes the size of the change with what kind of change it is. It means a short transition and a long transition might look very similar in that latent space, but they're not explicitly encouraged to be related.

Taro: That entanglement is bad because if you're learning a policy, you want to know whether the robot needs to perform a small adjustment or a big maneuver, and that distinction gets lost in one vector.

Rosa: Exactly. PoLAR introduces Polar Latent Actions with Radial structure, which they are encouraging radius to encode the transition extent and direction to retain the transition mode. It's like separating how far you've moved versus what kind of movement it is.

Dev: And they do this by using temporal offset between observation pairs as a weak proxy for that physical transition extent, meaning larger time gaps lead to latent actions occupying larger radii. That gives the radius a job, and leaves the direction to handle the mode.

Taro: It makes sense that if you're looking at two frames far apart in time, you're likely seeing a much bigger movement than if they were only one frame apart. So tying time to radius is an intuitive way to structure that relationship.

Rosa: They instantiate this radial structure in hyperbolic space, which is interesting because the volume expands with the radius, so it gives more room for diverse transition modes at larger extents. This geometry helps keep the directions distinct when you're looking at those larger transitions.

Dev: The math they use involves adding two losses to structure these prequantized IDM outputs, one of which is Lord, which enforces that the farther transition should lie farther from the local anchor than the intermediate one.

Title and authors: Taro: That Lord loss sounds like it's explicitly structuring the distance between transitions along a radial line in some sense. It’s not just letting them float randomly in space.

Rosa: And they also have a loss called Lrad, which is added to structure the prequantized IDM outputs z-hat, where Lord enforces that the farther transition should lie farther from the local anchor than the intermediate transition. Both these losses matter, and removing either one weakens how well the downstream policy performs.

Dev: That’s a significant finding because it shows that both enforcing this distance structure and this radial layout are important for getting good results on real tasks. They use a full objective function that's LLAM plus lambda ord plus lambda radLrad, which is pretty comprehensive.

Taro: So, if you look at the discrete latent actions, they replace the flat codebook with a radial-direction VQ interface where each action gets one radial token and four direction tokens. It’s a way to handle things when you have discrete choices but still want that structure.

Rosa: They use this factorization because it allows them to select one shared radial index at a time, t-ell, from the aggregate norm of the latent action and then pick a direction index for each of those slots. It’s how they handle the discrete part while keeping the radial structure intact.

Dev: The experimental results are pretty strong. They tested this across continuous and discrete latent action parameterizations in both in-task and large-scale pretraining regimes, showing improvement on RoboMimic and MimicGen simulations with continuous actions.

Taro: And when they moved to real-world robot manipulation tasks, PoLAR achieved the highest final success rate on all three of those tasks, outperforming π0 point 5, UniVLA, SmolVLA, and Villa-X. That's a big win for real application.

Rosa: On the real world side too, they found that temporal offset has the strongest Spearman correlation with both object and robot state distances across all datasets. That means time separation is a very reliable way to get a measure of how far apart things are physically in the scene.

Dev: And when they compared it against other methods like DINOv2 or ResNet18 for predicting those distances, the temporal offset was better than both, which confirms that this proxy is robust across different visual understanding techniques.

Title and authors: Taro: So what does this mean for a system deployed on a robot? It means you can use time differences to guide how the latent action space organizes itself around physical distance and movement type.

Rosa: It gives you control over the latent representation, allowing you to explicitly separate scale from style in your pretraining, which is crucial for robust policy learning where similar transitions at different horizons should be related.

Dev: And because of this explicit separation, the resulting latent actions are more action-informative overall because PoLAR achieves the highest mutual information estimates and probe R2 compared to baselines like those using flat codebooks.

Taro: That higher information content is what leads to better control, and they also showed it improves robustness to prediction errors because hyperbolic PoLAR yields lower wrong-token latent error than Flat and UniVLA.

Rosa: So, the implication here is that the geometry of your latent action space itself is a design choice that can really help your robot policy learning improve.

Dev: And if you're training for multi-horizon tasks, PoLAR can benefit from concatenating continuous latent actions for different future offsets, leading to gains of thirty-six point zero and sixty point zero points on tasks like Coffee and Stack Three.

Taro: That means if you're doing complex manipulation that spans a long sequence of steps, having this structured representation helps the AI manage that complexity better than just one flat vector would allow.

Rosa: So, to wrap up on "PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning", it introduces a latent action learning framework that uses temporal ordering to impose hyperbolic radial structure on latent actions.

Dev: Across various policies, simulation benchmarks, and real-world robot experiments, PoLAR consistently improves downstream policy performance as a pretraining method.

Taro: And I'd add that the fact that they found this works across both continuous and discrete latent action parameterizations is pretty impressive for versatility in deployment.

Rosa: It suggests that latent action geometry is a useful design choice for robot policy learning, giving you a much better foundation than just any unstructured representation.

The paper's summary: Rosa: So PoLAR is basically taking those latent actions that summarize what’s changing visually, and it’s trying to fix how they are structured inside that space so we can actually use them better for controlling a robot.

Dev: Right, because usually, one latent action vector tries to do too much—it mixes up how big the movement is with what kind of movement it is. PoLAR separates those two things out.

Rosa: Exactly. They introduce this radial structure on the latent action space where the radius handles how far we've moved physically and the direction handles what kind of transition it is, like a push or a pull, or a slide.

Dev: And they use temporal offset between observation pairs as this weak signal for that physical extent because larger time gaps in the video usually mean bigger movements. So, larger radii correspond to longer time gaps.

Rosa: They do all this by using hyperbolic space because that geometry lets the radius get bigger and bigger while still giving you enough room for different modes at those larger scales. It’s a natural fit for complex actions when they get big.

Dev: And they add these specific losses, like Lord and Lrad, to make sure that the farther transitions are actually farther apart in this radial sense than the intermediate ones. Both of these losses are important because if you drop one of them, your policy performance drops.

Rosa: What this means for me out on the field is that when I see a robot doing a big maneuver versus a small adjustment, PoLAR should represent those two things in distinctly different parts of its latent space.

Dev: And from a control standpoint, it’s about making the action information richer overall because they find that hyperbolic PoLAR gives us higher mutual information estimates and probe R2 compared to the flat methods.

Rosa: That higher information content is what helps us build better policies, especially when we look at real-world manipulation tasks where accuracy matters.

Dev: Plus, they showed it’s more robust to those prediction errors that happen during training; the wrong-token latent error is lower when you use PoLAR compared to flat methods.

Rosa: It sounds like the main thing is that this radial structure gives us a much better foundation for learning robot policies, moving beyond just a single messy vector.

Dev: And if you’re doing multi-horizon training, they show you can concatenate those continuous latent actions for different future offsets and get significant gains on tasks like stacking things or getting coffee.

Rosa: So PoLAR is suggesting that the geometry of the latent action space itself is a choice we should make to really improve how robot policies learn complex, long-horizon behaviors.

Dev: It’s a design choice that seems to actually help in separating scale from style, which I think is crucial for building reliable control systems.

Rosa: We’ll keep digging into those specific losses later, but for now PoLAR seems like a solid way to structure the latent action learning process.

The paper's improvements: Taro: So, PoLAR isn't just adding one new trick; it’s fundamentally restructuring how we think about what those latent actions actually represent by separating scale from mode.

Rosa: Right, because the paper suggests that this explicit separation—radius for extent and direction for mode—is key to getting better policy performance across the board.

Dev: It changes how we train these models because instead of a flat representation where everything is mixed up, you get a structured action space that makes sense physically.

Taro: And the paper points out that this structure is actually better supported by hyperbolic geometry because the angular capacity at larger radii helps keep those modes distinct when the transitions are bigger.

Rosa: So, what does this mean for my trials? It means if I’m looking at a robot making a huge movement, PoLAR should capture that scale differently than it captures a small adjustment.

Dev: From an engineer's view, it means we get more reliable action information because the mutual information and probe R2 are higher when you use this radial structure compared to just using flat codebooks.

Taro: That improved information content is what lets us handle those real-world failure modes better because the latent representation is more accurate about what’s actually happening physically.

Rosa: And it also helps with robustness; the paper found that hyperbolic PoLAR produces lower wrong-token latent error compared to flat methods like UniVLA.

Dev: That error reduction, especially after decoding it back into an actual action, suggests that separating those components makes the learned representations more trustworthy for real control loops.

Taro: It seems like they’re showing us that for complex robotic tasks, designing a structured latent space isn't just academic; it directly translates to better performance in simulation and on the robot itself.

Rosa: Exactly, and this approach is flexible enough to work whether you’re using continuous or discrete latent actions, which is pretty impressive versatility.

Dev: And for those working on multi-horizon tasks, the method lets you stack these structured actions together for different future offsets to see big gains in performance.

Taro: So the implication here is that we should start thinking more about the geometry of our latent spaces as a deliberate design choice rather than just something that happens by accident during pretraining.

Rosa: It moves us toward building better foundations for policy learning, and I think this is going to help us tackle those really long-horizon manipulation tasks we've been struggling with.

Conclusion: Rosa: So PoLAR is basically taking those latent actions that summarize what’s changing visually, and it’s trying to fix how they are structured inside that space so we can actually use them better for controlling a robot.

Dev: Right, because usually, one latent action vector tries to do too much—it mixes up how big the movement is with what kind of movement it is. PoLAR separates those two things out.

Taro: It changes how we think about what those latent actions actually represent by separating scale from mode.

Taro: That makes sense; if you’re looking at two frames far apart in time, you're likely seeing a much bigger movement than if they were only one frame apart.

Taro: I mean, those losses matter a lot for ensuring graceful recovery when the world misbehaves; it’s about enforcing that structure consistently.

Dev: From an engineer's view, it’s about making the action information richer overall because they find that hyperbolic PoLAR gives us higher mutual information estimates and probe R2 compared to just using flat codebooks.

Taro: That higher information content is what lets us handle those real-world failure modes better because the latent representation is more accurate about what’s actually happening physically.

Rosa: Plus, it helps with robustness; the paper found that hyperbolic PoLAR produces lower wrong-token latent error compared to flat methods like UniVLA.

Rosa: Exactly, and this approach is flexible enough to work whether you’re using continuous or discrete latent actions, which is pretty impressive versatility.

Dev: And for those working on multi-horizon tasks, the method lets you stack these structured actions together for different future offsets to see big gains in performance.

Rosa: It moves us toward building better foundations for policy learning, and I think this is going to help us tackle those really long-horizon manipulation tasks we've been struggling with.

Taro: I just want to emphasize how much they focused on the hyperbolic part because that’s what really unlocks the ability for the direction tokens to retain mode information when we look at those bigger transitions.

Rosa: I agree, Taro, it seems like that geometric choice is what makes this framework distinct from just applying standard VQ codebooks.

Dev: From a loop rate perspective, if these losses are calculated efficiently, this structure should fit well into real-time training setups without adding too much latency to the pretraining phase.

Taro: Yeah, and I wonder how robust this radial structure is when we move to truly chaotic environments where temporal offsets might become unreliable signals.

Rosa: That’s a valid point; the paper flags that temporal ordering as weak supervision, so if you have cyclic behavior or long pauses, those time-based radii might become noisy.

Dev: So the safety case here is that while it's not perfect for every scenario, it provides a strong baseline structure for policy learning across many different datasets.

Taro: It’s a solid starting point for autonomy research, but we still need to test how well it handles those edge cases where time signals break down completely.

Rosa: Anyway, that wraps up our look at PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning.

Dev: It’s a solid piece of work showing that latent action geometry is a design choice that can genuinely improve robot policy learning across various tasks.

Taro: Definitely something worth looking into if you're building next-generation embodied AI systems.

Youngjoon Jeong, Jihwan Yu, Minsoo Jo, Junha Chun, Taesup Kim

Seoul National University

cs.RO, cs.AI, cs.LG

Submitted: 2026-06-19

Updated: 2026-10-14

Comments: Project Page: https://joon-stack.github.io/PoLAR, Accepted to CoRL 2026

Project page: https://joon-stack.github.io/PoLAR

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: The gist Latent action pretraining learns representations of visual change from pairs of observations, but existing methods typically encode each transition as a single unstructured representation

Key concepts

Polar Latent Actions with Radial structure (PoLAR): PoLAR imposes a radial structure on the latent action space. The radius of the latent action is specifically designed to encode how much change occurs (transition extent), while the direction encodes what kind of change it is (transition mode). This separation helps downstream policies map these structured actions more effectively.
Temporal Offset as Extent Proxy: The paper uses the time difference between two observation pairs as a weak signal for transition extent. Larger temporal gaps between observations are used to encourage latent actions with larger radii, meaning longer transitions are associated with larger radial distances in the latent space.
Hyperbolic Space Geometry: PoLAR instantiates its structure within hyperbolic space because its expanding volume naturally accommodates diverse transition modes at larger extents. This geometry is beneficial because it allows the angular capacity (direction) to retain information about the transition mode even when the radius is large and encoding a long extent.

Terminology

Action Space — 1× in this paper · explained in 1 episode

Summary

The gist Latent action pretraining learns representations of visual change from pairs of observations, but existing methods typically encode each transition as a single unstructured representation that entangles transition extent and transition mode; PoLAR introduces Polar Latent Actions with Radial structure to impose a radial-direction structure on latent actions, encouraging radius to encode transition extent and direction to retain transition mode; this geometry is instantiated in hyperbolic space where angular capacity grows with radius, leading to improved downstream policy performance in simulation and real-world robot experiments.

How it works

PoLAR imposes a radial structure on the latent action space where the radius is encouraged to encode observed transition extent, reducing the burden on direction to encode extent and leaving direction to capture transition mode How it works; PoLAR imposes a radial structure on the latent action space where the radius is encouraged to encode observed transition extent, reducing the burden on direction to encode extent and leaving direction to capture transition mode.
Temporal ordering between observation pairs is used as a weak proxy for transition extent, encouraging larger gaps to occupy larger radii. The paper states that PoLAR uses temporal offset between two observations as a weak proxy for transition extent, encouraging latent action from observation pairs separated by larger temporal gaps to occupy larger radii.
This radial structure is instantiated in hyperbolic space, where the expanding volume with radius offers a natural fit for more diverse transition modes at larger extents. The paper states that we instantiate this structure in hyperbolic space, whose expanding volume with radius offers a natural fit for more diverse transition modes at larger extents.
Radial losses are added to structure the pre-quantized IDM outputs zˆ, where Lord enforces that the farther transition should lie farther from the local anchor than the intermediate transition. The paper states that PoLAR adds two losses to structure the pre-quantized IDM outputs zˆ. First, the farther transition should lie farther from the local anchor than the intermediate transition: Lord = softplus (d(zt,0, zt,j) − d(zt,0, zt,k)).
The full objective is defined as L = LLAM + λord + λradLrad where LLAM denotes the base latent action objective. The paper states that the full objective is L = LLAM + λord + λradLrad, where LLAM denotes the base latent action objective.
For discrete latent actions, a factorized radial-direction codebook is used where each discrete latent action is represented by one radial token and four direction tokens. The paper states that for discrete latent actions, we replace the flat VQ codebook with a radial-direction VQ interface. Given the prequantized continuous latent action zt,l with C latent slots, the quantizer selects one shared radial index at t,l from the aggregate norm of zt,l and one direction index bt,l,m for each slot m from the normalized direction of slot zt,l.

Experimental Setup and Results

PoLAR is evaluated across continuous and discrete latent action parameterizations in in-task and large-scale pretraining regimes. The paper states we evaluate PoLAR across continuous and discrete latent action parameterizations, in-task and large-scale pretraining regimes.
In simulation results on RoboMimic & MimicGen, PoLAR improves continuous latent action conditioned diffusion policies on RoboMimic and MimicGen. The paper states PoLAR improves continuous latent action conditioned diffusion policies on RoboMimic and MimicGen.
Across real-world robot manipulation tasks, PoLAR achieves the highest final success rate on all three tasks and the best overall average, outperforming π0.5 as well as UniVLA, SmolVLA, and Villa-X. The paper states PoLAR achieves the highest final success rate on all three tasks and the best overall average, outperforming π0.5 as well as UniVLA, SmolVLA, and Villa-X.
Analysis shows that temporal offset has the strongest Spearman correlation with both object and robot state distances across all datasets. The paper states Across all datasets, temporal offset has the strongest Spearman correlation with both object and robot state distances, outperforming pretrained DINOv2 [45], ResNet18 [49], and pixel distances in all reported settings.

Analysis of Structure

Temporal offset serves as an effective proxy for physical transition extent by comparing it against low-dimensional state distances. The paper states we audit temporal offset as a weak proxy for physical transition extent by comparing it against low-dimensional state distances.
The Euclidean variant has higher cosine similarity between directions at adjacent temporal offsets than hyperbolic PoLAR, but lower direction-only probe R2 after removing radius. The paper states the Euclidean variant has higher cosine similarity between directions at adjacent temporal offsets than hyperbolic PoLAR (0.974 vs 0.908), but lower direction-only probe R2 after removing radius (0.088 vs 0.208).
Hyperbolic geometry better supports PoLAR’s intended structure because as radius tracks transition extent, the expanding angular capacity at larger radii helps direction retain information for distinguishing transition modes. The paper states hyperbolic geometry better supports PoLAR’s intended structure: as radius tracks transition extent, the expanding angular capacity at larger radii helps direction retain information for distinguishing transition modes.
Both radial losses matter, and removing either Lord or Lrad weakens downstream policy performance. The paper states Both radial losses matter, and removing either Lord or Lrad weakens downstream policy performance.

Advantages of PoLAR

Action informativeness is quantified using mutual information estimates and probe R2 from latent actions to ground-truth action chunks. The paper states we quantify how much information learned latent actions contain about ground-truth robot action chunks on BridgeData V2.
Hyperbolic PoLAR achieves the highest mutual information estimates and probe R2, suggesting that codebook factorization alone is insufficient; radial structure in hyperbolic space helps retain action-related information in the learned latent actions. The paper states Together with the downstream gains over Fact. in Fig. 3 (b), this suggests that codebook factorization alone is insufficient; radial structure in hyperbolic space helps retain action-related information in the learned latent actions.
Robustness to prediction errors is improved because hyperbolic PoLAR yields lower wrong-token latent error than Flat and UniVLA. The paper states Hyperbolic PoLAR yields lower wrong-token latent error (normalized) than Flat and UniVLA (0.311 vs 0.447 and 0.689), and the same trend holds after decoding to actions (0.143 vs 0.238 and 0.191).

Limitations

The paper notes that PoLAR infers latent action structure from visual observation pairs, and its experiments use a fixed third-person, top-view camera. The paper states PoLAR infers latent action structure from visual observation pairs, and our experiments use a fixed third-person, top-view camera. Furthermore, the assumption that temporal ordering is an effective signal for observed transition extent may break down under cyclic behavior or pauses. The paper states PoLAR also uses temporal ordering as weak supervision for transition extent. This assumption is well suited to goal-directed demonstrations, where larger temporal offsets often correspond to larger task-relevant changes, but it may break down under cyclic behavior, pauses, recovery motions, or repeated back-and-forth moves.

Conclusion

PoLAR introduces a latent action learning framework that uses temporal ordering to impose hyperbolic radial structure on latent actions. The paper states We introduced PoLAR, a latent action learning framework that uses temporal ordering to impose hyperbolic radial structure on latent actions. Across various policies, simulation benchmarks, and real-world robot experiments, PoLAR consistently improves downstream policy performance as a pretraining method. Together, these results suggest that latent action geometry is a useful design choice for robot policy learning. The paper states Across various policies, simulation benchmarks, and real-world robot experiments, PoLAR consistently improves downstream policy performance as a pretraining method.

Improvements for AI systems

Bold header: Radial structure for latent actions

PoLAR imposes a radial-direction structure on latent actions, encouraging radius to encode transition extent and direction to retain transition mode. This allows the system to explicitly separate scale (extent) from style (mode), which is crucial for robust policy learning where similar transitions at different horizons should be related.

Bold header: Temporal offset as weak supervision

The system can use temporal ordering as weak supervision for transition extent, where larger gaps often correspond to larger robot, object, or task-state changes. This enables the policy to learn that longer temporal gaps imply larger physical transitions by encouraging latent actions from observation pairs separated by larger temporal gaps to occupy larger radii.

Bold header: Enhanced action informativeness

The resulting latent actions are more action-informative overall because PoLAR achieves the highest mutual information estimates and probe R2 compared to baselines. This means the learned representations provide richer information about ground-truth robot actions, leading to better performance in downstream control tasks like Pick & Place Banana or Cup Stack.

Bold header: Robustness to prediction errors

The system demonstrates lower wrong-token latent error (normalized) when using PoLAR compared to Flat and UniVLA. This suggests that because extent and mode are represented separately, some token errors can remain close to the intended transition instead of mapping to an unrelated flat code, improving reliability in real-world manipulation.

Bold header: Multi-horizon latent policy training

The system can benefit from multi-horizon latent policy training which improves performance on tasks like Coffee and Stack Three by concatenating continuous latent actions for different future offsets, leading to gains of 36.0 and 60.0 points.