RiCo: Neural Simulation of Rigid-Body Interactions via Local Contact Reasoning

The gist: RiCo introduces Rigid-body Contact Reasoning, a neural simulator that models rigid-body interactions through sparse local contact reasoning.

Problem Formulation

To model the rigid-body dynamics, the problem involves considering a scene containing M rigid objects, where object i is represented by Ni sampled surface points x i t = [x i t,1,..., xi t,Ni]⊤ ∈ R Ni×3 with corresponding surface normals ν i t and center of mass c i t and physical properties ϕ i = [(mi)−1, µi, ei] (Page 3). Given two consecutive states st-1 and st, together with the static environment ξ, the goal is to build a dynamics model fθ (i.e., state transition model) to predict sˆt+1 = fθ(st-1, st, ϕ, ξ) (Page 3).

Sparse Contact Neighborhood Representation

RiCo models the state update in three stages: (1) constructing a sparse representation of local contacts, (2) reasoning over contacts within each object, and (3) predicting rigid-body motion from anchors (Page 3). The contact neighborhood construction involves an object-level broad-phase query over active object pairs using their surface-point AABBs to retain only pairs whose minimum AABB separation is no greater than the contact radius ρ (Page 13). For each retained unordered pair (i, j), the implementation computes a pairwise Euclidean-distance matrix using the surface samples of the two objects, retaining at most K nearest external samples for every source point (Page 14). The candidate positions are denoted by y i t,n,k K k=1 (Page 13). To characterize potential contact response, a 14-dimensional interaction descriptor η i t,n,k is formed using relative geometry and motion between the source point x i t,n and each external candidate y i t,n,k (Page 13).

Point State Representation

To model how the source object responds to local interactions, a point-wise state representation s i t,n is defined as s i t,n = x i t,n − c i t, ∆x i t,n ν i t,n ϕ i, xi t,n − x i 0,n ∈ R 15 (Page 13). Within this representation, x i t,n − c i t locates the point relative to the object’s center of mass and ∆x i t,n captures its motion over the preceding time step (Page 13). The object-level physical parameters ϕ i = [(mi)−1, µi, ei] specify the inverse mass, friction coefficient, and restitution coefficient that govern contact response (Page 13).

Intra-Object Contact Reasoning

Contact feature aggregation involves embedding the point state s i t,n and each local interaction descriptor η i t,n,k using MLPs ϕs and ϕc respectively (Page 14). The contact embeddings are aggregated by a masked, distance-dependent softmax where the weights are defined as h i 0 t n = RMSNorm(ϕs(s i t,n) + X k∈Ci t,n α i t,n,kϕc(η i t,n,k)) (Page 14). These contact-conditioned point tokens are then processed independently using a shared B-block point Transformer Tθ (Page 14). Importantly, since information from external surfaces is incorporated through the local contact fusion in Eq. 2, intra-object self-attention allows each point to access contact information from across its own object without dense cross-object attention (Page 14).

Anchor-Based Rigid Motion Prediction and Training

RiCo predicts each rigid object’s future motion from its contact-conditioned tokens H i t. To preserve rigidity, it follows prior anchor-based formulations and selects A anchor samples for each object using deterministic farthest point sampling (FPS) (Page 14). The corresponding sample indices are fixed throughout the rollout, such that the anchors track the same surface samples over time (Page 14). Motion updates are predicted only at these anchors and used to recover the rigid transformation of the entire object (Page 15). To advance the anchors from their current motion, a Verlet-style update is used: x i t+1,n a = 2x i t,n a − x i t-1,n a + ∆d2x i t,anc (Page 15). A proper rigid alignment is recovered for each active object through unweighted least-squares alignment between its current anchor positions and the unconstrained next positions using the Kabsch algorithm (Page 15).

Experimental Results

RiCo consistently achieves lower position and orientation errors than all baselines on MOVi-A and MOVi-B across all evaluated horizons (Page 6). On MOVi-A, compared with the strongest baseline, our model reduces position RMSE by approximately 35% and orientation RMSE by 38–43% over 50–100 prediction steps (Page 7). In terms of contact fidelity on MOVi-B, RiCo reduces relative contact violations to 11.0% of object-time steps, with an average relative penetration depth of 2.22 mm (Page 7). RiCo generalizes effectively across unseen distributions of 270 objects (Page 7). Furthermore, in real-world multi-ball collision experiments, RiCo achieves position RMSEs of 138.54, 214.69, and 272.42 mm at prediction horizons of 50, 75, and 100 frames (Page 7).

Ablation Studies

Ablation studies show that replacing the estimated signed distance with its absolute value substantially degrades contact fidelity, increasing ∆PTR from 11.0% to 23.6% and ∆MPD from 2.22 to 7.86 mm (Page 8). Increasing the number of retained contact candidates K to K=4 achieves the lowest ∆PTR and orientation RMSE (Page 9). Increasing point-cloud resolution from 512 to 1024 points yields only a marginal change in position RMSE, but substantially improves contact fidelity by decreasing ∆PTR from 23.4% to 11.0% (Page 9).

Scalability and Sim-to-Real Transfer

RiCo generalizes effectively across unseen distributions of 270 objects (Page 7). In zero-shot sim-to-real transfer, RiCo is evaluated on five real-world experimental sessions, providing preliminary evidence that the learned contact dynamics can generalize to real-world multi-object interactions without additional training (Page 8). The model achieves position RMSEs of 138.54, 214.69, and 272.42 mm at prediction horizons of 50, 75, and 100 frames in real-world multi-ball collision experiments (Page 7).

Conclusion

RiCo introduces a neural simulator that models rigid-body interactions through sparse local contact reasoning, achieving accurate and physically consistent predictions with high computational efficiency (Page 10). The approach preserves fine-grained contact geometry while avoiding dense scene-level interactions (Page 10). Experiments demonstrate its ability to exploit high-resolution geometry, generalize across diverse object distributions, scale to scenes containing 270 objects and transfer to real world (Page 10). Nevertheless, our current framework focuses on passive rigid-body dynamics and does not account for deformable objects or actuated systems (Page 10).

How it works

1. Construct a sparse representation of local contacts by performing an object-level broad-phase query over active object pairs using their surface-point AABBs to retain only pairs whose minimum AABB separation is no greater than the contact radius ρ (Page 13).

2. For each candidate, form a 14-dimensional interaction descriptor η i t,n,k incorporating relative geometry and motion between the source point x i t,n and each external candidate y i t,n,k (Page 13).

3. Fuse local contact descriptors with each point’s state to form contact-conditioned tokens h i 0 t n using a distance-dependent softmax aggregation (Page 14).

4. Process these tokens independently using a shared B-block point Transformer Tθ, where intra-object self-attention allows each point to access contact information from across its own object without dense cross-object attention (Page 14).

5. Anchor the motion prediction by selecting A anchor samples for each object using deterministic farthest point sampling (FPS) and predicting motion updates only at these anchors, which are then used to recover the rigid transformation of the entire object via Kabsch alignment (Page 15).

Evaluation Metrics

Prediction accuracy is measured by autoregressive position RMSE and orientation RMSE at 50, 75, and 100 frames (Page 6). Contact fidelity is assessed by ground-truth-relative penetration metrics, specifically the differences in penetration-time ratio (∆PTR) and mean penetration depth (∆MPD) between predicted and ground-truth trajectories (Page 8).

1. Improved Rigid-Body Interaction Modeling: The system can now capture how local contacts between surfaces shape object motion by using Local Contact Reasoning (RiCo), which represents interactions through sparse neighborhoods of contact surface points. This allows for high accuracy in predicting rigid-body dynamics by combining each point’s state with the relative geometry, motion, and physical properties of nearby surfaces.

2. Enhanced Contact Fidelity: The improved system achieves higher physical consistency by reducing relative contact violations to 11.0% of object-time steps and a mean relative penetration depth of 2.22 mm, which provides a higher accuracy and contact fidelity compared to baselines like HOPNet and RigidFormer.

3. Scalable Scene Prediction: The AI system can now handle significantly larger environments by generalizing zero-shot from small-scale training scenarios to scenes containing 270 objects. This demonstrates the ability to perform accurate long-horizon rollouts in complex, multi-object scenes without retraining for each new configuration.

4. Robust Sim-to-Real Transfer: The system can perform reliable real-world interaction prediction by demonstrating preliminary evidence of sim-to-real transfer through real-world multi-ball collision experiments, suggesting the model's learned contact dynamics can generalize to physical interactions without additional training.

5. Resolution and Efficiency Trade-off: The system can maintain high geometric detail while remaining computationally efficient, as it shows that sparse local contact reasoning enables high-resolution geometric modeling without sacrificing computational efficiency, achieving 47.89 FPS on a single GPU even with 1024 surface points per object.

6. Geometry Sensitivity Analysis: The AI system is now sensitive to the quality of geometric information, as ablations show that explicitly encoding the signed geometric relationship between nearby surfaces helps the model better capture contact configurations and reduce interpenetration, even if trajectory prediction accuracy only changes marginally.