RigidFormer: Learning Rigid Dynamics using Transformers

arXiv:2605.09196 · cs.CV, cs.AI, cs.GR, cs.LG, cs.RO · Submitted 2026-05-09 · Read on arXiv

cs.CV, cs.AI, cs.GR, cs.LG, cs.RO

Submitted: 2026-05-09

Updated: 2026-11-06

Comments: NeurIPS 2026. Project page: https://frank-zy-dou.github.io/projects/RigidFormer/

License: http://creativecommons.org/licenses/by-sa/4.0/

The gist: Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons.

Terminology

Abstract

Learning-based simulation of multi-object rigid-body dynamics remains difficult because contact is discontinuous and errors compound over long horizons. Most existing methods remain tied to mesh connectivity and vertex-level message passing, which limits their applicability to mesh-free inputs such as point clouds and leads to high computational cost. Efficiently modeling high-fidelity rigid-body dynamics from mesh-free representations therefore remains challenging. We introduce RigidFormer, an object-centric Transformer-based model that learns mesh-free rigid-body dynamics with controllable integration step sizes. RigidFormer reasons at the object level and advances each object through compact anchors; Anchor-Vertex Pooling enriches these anchors with local vertex features, retaining contact-relevant geometry without dense vertex-level interaction. We propose Anchor-based RoPE to make attention aware of the objects' spatial layout while respecting the unordered nature of objects and anchors: object-token processing is permutation-equivariant, and the mean-pooled anchor descriptor is invariant to anchor reindexing. RigidFormer further enforces rigidity by projecting updates onto the rigid-body manifold using differentiable Kabsch alignment. On standard benchmarks, RigidFormer outperforms or matches mesh-based baselines using point inputs, runs faster, generalizes to unseen point resolutions and across datasets, and scales to 200+ objects; we also show a preliminary extension to command-conditioned articulated bodies by treating body parts as interacting object-level components. Project page: https://frank-zy-dou.github.io/projects/RigidFormer/

Sources

Related papers