OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulation".
Dev: Here's the gist of the paper. OASIS proposes a visuomotor policy that aligns intermediate representations with the action space via SE(3) end-effector trajectory prediction, which is significant because it provides a geometric inductive bias for action generation Core Insight and Design Principle The key insight is that predicting an SE(3) end-effector trajectory aligns the intermediate representation with the action space, providing a geometric inductive bias rather than enriching The paper formalizes this by stating that an intermediate representation is geometrically aligned when its pose-supervised component provides a readout of the target SE(3) state for action generation This design principle is instantiated by predicting a camera-frame SE(3) end-effector trajectory which satisfies the geometric-alignment property with g = T−1c→w, the inverse of the camera-to-world extrinsic The intermediate representation is geometrically aligned when it provides an explicit pose readout r(mt+h) = g · Tt+h in SE(3) at each horizon step, where g ∈ SE(3) is a fixed rigid transform that may be unknown OASIS Architecture and Components OASIS couples three components trained end-to-end: a 3D-aware feature encoder, an SE(3) trajectory predictor, and an action decoder The 3D-aware feature encoder fuses vision-language features with metric-depth features into a shared representation that supplies calibrated 3D spatial information On this representation, an SE(3) trajectory predictor forecasts a sequence of camera-frame end-effector states The action decoder, conditioned on these pose-supervised hidden states and the current robot state, learns to approximate the relative-action recovery of Eq.
Rosa: (2) while absorbing frame conversion, prediction noise, contact dynamics, and gripper timing as residuals Training Objectives and Loss Functions The total objective combines trajectory supervision with executable-action supervision: Ltotal = λLtraj + Lact The trajectory loss Ltraj supervises the local pose parameterization with ground-truth pose vectors obtained by transforming world-frame demonstrations into the camera frame using its extrinsic matrix The action loss Lact supervises the output action chunk against expert demonstrations, where at+h−1 =
∆pt+h−1, ∆θt+h−1: is the 6-DoF relative action and gt+h−1 is the gripper command The total objective combines trajectory supervision with executable-action supervision, Ltotal = λLtraj + Lact where λ balances the two losses and is set to 0.1 Experimental Results and Ablation Analysis Extensive experiments show that OASIS outperforms VLA and WAM baselines in success rate and out-of-distribution generalization On LIBERO, OASIS achieves a 97.6% average success rate across the four suites, leading the next-best baseline Unified-VLA
51: by 2.1% Ablations show that adding the SE(3) trajectory predictor to the 3D-aware feature encoder and decoder raises success from 89.5 to 95.2% on LIBERO-Long, which is the largest additional gain among ablated components Across real-world platforms, OASIS reaches an 89.2% average success rate across multi-task, spatial-relationship, and long-horizon suites Generalization and Robustness Under out-of-distribution (OOD) perturbations of the Goal task, covering unseen backgrounds, an altered camera viewpoint, and human interference, OASIS attains an average success rate of 90.8% The metric-depth feature keys off scene geometry under unseen backgrounds so that distractor objects and tabletops barely shift the decoder input Furthermore, the residual robustness of π0.5 and RDT likely reflects their large-scale robotic pretraining OASIS matches the observed 25-demonstration success rate of π0.5 using only 10 demonstrations on this Long task Practical Implications OASIS proposes a new design principle for visuomotor policies, aligning the intermediate representation with the action space via SE(3) end-effector trajectory prediction A key strength of OASIS lies in its training efficiency and accessibility, as it requires neither large-scale robotic pretraining nor annotated spatial labels The explicit separation of geometric reasoning from action execution further makes the intermediate representation inspectable, supporting transparent debugging and safer deployment than monolithic image-to-action models This design can benefit assistive household robots, dexterous industrial manipulators, and tabletop laboratory automation where precise SE(3) control under modest data budgets is essential Limitations and Future Work Our current method focuses on tabletop manipulation with a single robotic arm, so the SE(3) intermediate aligns with the action space of a single end-effector Extending this design principle to richer action spaces is a natural next step, including coupled SE(2) × SE(3) trajectories for mobile manipulation and contact-conditioned trajectories for dexterous hands The framework requires neither large-scale robotic pretraining nor annotated spatial labels, with supervision derived solely from standard expert demonstrations With only 0.18B trainable parameters trained on four GPUs, the framework lowers the cost barrier for research groups and small organizations developing manipulation policies on commodity hardware The explicit separation of geometric reasoning from action execution further makes the intermediate representation inspectable, supporting transparent debugging and safer deployment than monolithic image-to-action models With only 0.
Paper summary: Rosa: So we're looking at this paper called OASIS: Observation-Action Space Alignment via SE(three) Trajectory Prediction for Robotic Manipulation. The main idea is that we need to make sure what the AI understands about the scene actually matches how the robot moves things in three dimensions.
Dev: Exactly, and they propose using something called SE(three) trajectory prediction to align that observation-action space with the actual geometry of robot movement, which gives it a geometric bias for generating actions.
Taro: What I find interesting is that this isn't just adding more data; they are using this geometric alignment as a fundamental design principle for action generation, meaning the intermediate representation has an inherent structure to follow.
Rosa: Right, and what that means for us on the ground is that instead of just learning a messy mapping from pixels to movement commands, we're forcing the model to predict a specific path in three dimensions before it even tries to decide what action to take.
Dev: That's a big deal because if the intermediate representation is geometrically aligned with the action space, then when you generate an action, there's this built-in inductive bias that makes it much more likely to be physically plausible.
Taro: I mean, when we test this on real-world platforms outside of a perfect lab setup, like those multi-task suites they mention, it holds up pretty well because that trajectory prediction part gives the system something solid to anchor itself to when things get messy.
Rosa: It does seem robust under these out of distribution conditions where the background changes or the viewpoint shifts, which shows it's not just memorizing specific scenarios but learning a better way to reason about space.
Dev: I'm still focused on the engineering side, though because they mention that this whole framework doesn't require massive datasets or huge robotic pretraining to get started, which makes deploying something like this much more accessible for smaller teams.
Taro: That accessibility is definitely a point; you can see it applied to things like assistive robots or even simpler industrial manipulators where you don't have the luxury of millions of dollars in training infrastructure.
Rosa: It really shifts the focus from just getting a high success rate number to building systems that have this underlying geometric understanding, which I think will be crucial for making robots truly useful in everyday environments.
Dev: So, we've been talking about OASIS, which is this paper proposing a way to make sure what an AI understands geometrically actually matches how a robot moves things in three dimensions.
Rosa: Exactly, and the title itself is pretty telling because it focuses on aligning that observation-action space using something called SE(three) trajectory prediction.
Taro: What I find interesting about the authors is that they aren't just throwing more data at it; they’re using this geometric alignment as a fundamental design principle for action generation, which means the intermediate representation has an inherent structure to follow.
Taro: It really shifts the focus from just getting a high success rate number to building systems that have this underlying geometric understanding, which I think will be crucial for making robots truly useful in everyday environments.
Conclusion: Rosa: So we've talked about OASIS, which is this paper proposing a way to make sure what an AI understands geometrically actually matches how a robot moves things in three dimensions.
Dev: Exactly, and the title itself is pretty telling because it focuses on aligning that observation-action space using something called SE(three) trajectory prediction.
Taro: What I find interesting about the authors is that they aren't just throwing more data at it; they’re using this geometric alignment as a fundamental design principle for action generation, which means the intermediate representation has an inherent structure to follow.
Rosa: Right, and what this means for us on the ground is that instead of just learning a messy mapping from pixels to movement commands, we're forcing the model to predict a specific path in three dimensions before it even tries to decide what action to take.
Dev: That's a big deal because if the intermediate representation is geometrically aligned with the action space, then when you generate an action, there's this built-in inductive bias that makes it much more likely to be physically plausible.
Taro: I mean, when we test this on real-world platforms outside of a perfect lab setup, like those multi-task suites they mention, it holds up pretty well because that trajectory prediction part gives the system something solid to anchor itself to when things get messy.
Rosa: It does seem robust under these out of distribution conditions where the background changes or the viewpoint shifts, which shows it's not just memorizing specific scenarios but learning a better way to reason about space.
Dev: I'm still focused on the engineering side, though, because they mention that this whole framework doesn't require massive datasets or huge robotic pretraining to get started, which makes deploying something like this much more accessible for smaller teams.
Taro: That accessibility is definitely a point; you can see it applied to things like assistive robots or even simpler industrial manipulators where you don't have the luxury of millions of dollars in training infrastructure.
Rosa: It really shifts the focus from just getting a high success rate number to building systems that have this underlying geometric understanding, which is something I think will be crucial for making robots truly useful in everyday environments.
Xinzhe Chen, Sihua Ren, Liqi Huang, Haowen Sun, Mingyang Li
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University
cs.RO, cs.AI
Submitted: 2026-05-25
Updated: 2026-11-07
Comments: Accepted to NeurIPS 2026 (Spotlight). Project page: https://oasis-neurips.github.io
Project page: https://npuhandsome.github.io/OASIS_web
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 90/100
The gist: OASIS proposes a visuomotor policy that aligns intermediate representations with the action space via SE(3) end-effector trajectory prediction…
Key concepts
- Geometric Inductive Bias
- This is the core design principle where the intermediate representation is forced to align with the action space by predicting a target SE(3) state. Instead of just enriching representations, this explicitly provides a geometric structure that guides how actions should be generated, making the learning process more efficient and structured.
- SE(3) Trajectory Prediction
- This involves forecasting where the robot's end-effector will be in 3D space (position and orientation) over several future time steps. This prediction serves as a crucial geometric readout, directly connecting the visual input to a specific, structured movement plan that the action decoder can use.
- Intermediate Representation Alignment
- The goal is to ensure that the internal features learned by the system are geometrically consistent with what is needed for physical movement. By making this alignment explicit through pose supervision, OASIS ensures that the features used for decision-making directly correspond to physically executable poses.
- Action Loss (Lact)
- This loss function trains the action decoder to produce specific robot commands, such as joint velocity or gripper movements. It learns to correct errors by absorbing complexities like frame conversions and contact dynamics, ensuring the final output actions are executable based on expert demonstrations.
Terminology
Summary
OASIS proposes a visuomotor policy that aligns intermediate representations with the action space via SE(3) end-effector trajectory prediction, which is significant because it provides a geometric inductive bias for action generation
Core Insight and Design Principle
The key insight is that predicting an SE(3) end-effector trajectory aligns the intermediate representation with the action space, providing a geometric inductive bias rather than enriching The paper formalizes this by stating that an intermediate representation is geometrically aligned when its pose-supervised component provides a readout of the target SE(3) state for action generation This design principle is instantiated by predicting a camera-frame SE(3) end-effector trajectory which satisfies the geometric-alignment property with g = T−1c→w, the inverse of the camera-to-world extrinsic The intermediate representation is geometrically aligned when it provides an explicit pose readout r(mt+h) = g · Tt+h in SE(3) at each horizon step, where g ∈ SE(3) is a fixed rigid transform that may be unknown
OASIS Architecture and Components
OASIS couples three components trained end-to-end: a 3D-aware feature encoder, an SE(3) trajectory predictor, and an action decoder The 3D-aware feature encoder fuses vision-language features with metric-depth features into a shared representation that supplies calibrated 3D spatial information On this representation, an SE(3) trajectory predictor forecasts a sequence of camera-frame end-effector states The action decoder, conditioned on these pose-supervised hidden states and the current robot state, learns to approximate the relative-action recovery of Eq. (2) while absorbing frame conversion, prediction noise, contact dynamics, and gripper timing as residuals
Training Objectives and Loss Functions
The total objective combines trajectory supervision with executable-action supervision: Ltotal = λLtraj + Lact The trajectory loss Ltraj supervises the local pose parameterization with ground-truth pose vectors obtained by transforming world-frame demonstrations into the camera frame using its extrinsic matrix The action loss Lact supervises the output action chunk against expert demonstrations, where at+h−1 = [∆pt+h−1, ∆θt+h−1] is the 6-DoF relative action and gt+h−1 is the gripper command The total objective combines trajectory supervision with executable-action supervision, Ltotal = λLtraj + Lact where λ balances the two losses and is set to 0.1
Experimental Results and Ablation Analysis
Extensive experiments show that OASIS outperforms VLA and WAM baselines in success rate and out-of-distribution generalization On LIBERO, OASIS achieves a 97.6% average success rate across the four suites, leading the next-best baseline Unified-VLA [51] by 2. p. 61% Ablations show that adding the SE(3) trajectory predictor to the 3D-aware feature encoder and decoder raises success from 89. p. 15 to 95.2% on LIBERO-Long, which is the largest additional gain among ablated components Across real-world platforms, OASIS reaches an 89. p. 12% average success rate across multi-task, spatial-relationship, and long-horizon suites
Generalization and Robustness
Under out-of-distribution (OOD) perturbations of the Goal task, covering unseen backgrounds, an altered camera viewpoint, and human interference, OASIS attains an average success rate of 90.8% The metric-depth feature keys off scene geometry under unseen backgrounds so that distractor objects and tabletops barely shift the decoder input Furthermore, the residual robustness of π0. p. 85 and RDT likely reflects their large-scale robotic pretraining OASIS matches the observed 25-demonstration success rate of π0. p. 85 using only 10 demonstrations on this Long task
Practical Implications
OASIS proposes a new design principle for visuomotor policies, aligning the intermediate representation with the action space via SE(3) end-effector trajectory prediction A key strength of OASIS lies in its training efficiency and accessibility, as it requires neither large-scale robotic pretraining nor annotated spatial labels The explicit separation of geometric reasoning from action execution further makes the intermediate representation inspectable, supporting transparent debugging and safer deployment than monolithic image-to-action models This design can benefit assistive household robots, dexterous industrial manipulators, and tabletop laboratory automation where precise SE(3) control under modest data budgets is essential
Limitations and Future Work
Our current method focuses on tabletop manipulation with a single robotic arm, so the SE(3) intermediate aligns with the action space of a single end-effector Extending this design principle to richer action spaces is a natural next step, including coupled SE(2) × SE(3) trajectories for mobile manipulation and contact-conditioned trajectories for dexterous hands The framework requires neither large-scale robotic pretraining nor annotated spatial labels, with supervision derived solely from standard expert demonstrations With only 0.18B trainable parameters trained on four GPUs, the framework lowers the cost barrier for research groups and small organizations developing manipulation policies on commodity hardware The explicit separation of geometric reasoning from action execution further makes the intermediate representation inspectable, supporting transparent debugging and safer deployment than monolithic image-to-action models With only 0.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
SE(3) Trajectory Prediction Alignment: The system improves by coupling a 3D-aware feature encoder with an SE(3) trajectory predictor that produces a camera-frame end-effector trajectory, which
aligns the intermediate representation with the action space, providing a geometric inductive bias for action generation.
-
Enhanced Geometric Inductive Bias: By using pose-supervised hidden states from the trajectory predictor to condition the action decoder, OASIS ensures that
the decoder generates action chunks consistent with rigid-body motion,
effectively solving the problem where existing models decode actions from representations thatdo not share the rigid-body geometry of the action space.
-
Robustness to Out-of-Distribution Perturbations: The system achieves better generalization by maintaining high success rates under OOD scenarios, such as
unseen backgrounds, an altered camera viewpoint, and human interference,
reaching an average success rate of 90.8% on these tasks. -
Training Efficiency and Accessibility: The improved system is highly accessible because it
requires neither annotated spatial labels nor pretraining on large-scale robotic demonstrations,
leveraging only standard expert demonstrations and requiring only 0.18B trainable parameters, whichlowers the cost barrier for research groups.
-
Superior Performance on Complex Tasks: OASIS demonstrates superior performance across benchmarks, such as achieving a 97.6% average success rate on LIBERO and outperforming baselines in real-world settings like CALVIN ABC->D.
Sources
- Qwen2.5-VL Technical Report
- PaliGemma: A versatile 3B VLM for transfer
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
- RT-1: Robotics Transformer for Real-World Control at Scale
- Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation
- WorldVLA: Towards Autoregressive Action World Model
- GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
- Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
- RVT-2: Learning Precise Manipulation from Few Demonstrations
- Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
- OpenVLA: An Open-Source Vision-Language-Action Model
- MolmoAct: Action Reasoning Models that can Reason in Space
- PointVLA: Injecting the 3D World into Vision-Language-Action Models
- What Matters in Building Vision-Language-Action Models for Generalist Robots
- QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models
- Depth Anything 3: Recovering the Visual Space from Any Views
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
- Pixel Motion as Universal Representation for Robot Control
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation