RigidFormer: Learning Rigid Dynamics using Transformers
cs.CV, cs.AI, cs.GR, cs.LG, cs.RO
Submitted: 2026-05-09
Updated: 2026-11-06
Comments: NeurIPS 2026. Project page: https://frank-zy-dou.github.io/projects/RigidFormer/
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons.
Terminology
Abstract
Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons. Most existing methods remain tied to mesh connectivity and vertex-level message passing, which limits their applicability to mesh-free inputs such as point clouds and leads to high computational cost. Efficiently modeling high-fidelity rigid-body dynamics from mesh-free representations therefore remains challenging. We introduce RigidFormer, an object-centric Transformer-based model that learns mesh-free rigid-body dynamics with controllable integration step sizes. RigidFormer reasons at the object level and advances each object through compact anchors; Anchor-Vertex Pooling enriches these anchors with local vertex features, retaining contact-relevant geometry without dense vertex-level interaction. We propose Anchor-based RoPE to make attention aware of the objects' spatial layout while respecting the unordered nature of objects and anchors: object-token processing is permutation-equivariant, and the mean-pooled anchor descriptor is invariant to anchor reindexing. RigidFormer further enforces rigidity by projecting updates onto the rigid-body manifold using differentiable Kabsch alignment. On standard benchmarks, RigidFormer outperforms or matches mesh-based baselines using point inputs, runs faster, generalizes to unseen point resolutions and across datasets, and scales to 200+ objects; we also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. Project page: https://frank-zy-dou.github.io/projects/RigidFormer/
Sources
- Learning rigid dynamics with face interaction graph networks
- A Compositional Object-Based Approach to Learning Physical Dynamics
- Learning Neural Event Functions for Ordinary Differential Equations
- Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
- DiffTaichi: Differentiable Programming for Physical Simulation
- PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation
- Attention Residuals
- Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
- MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- Learning 3D Particle-based Simulators from RGB-D Videos
- Modeling the Real World with High-Density Visual Particle Dynamics
- Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
- TesserAct: Learning 4D Embodied World Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs