EquiContact: A Hierarchical SE(3) Vision-to-Force Equivariant Framework for Spatially Generalizable Contact-rich Tasks

summary

The gist

EquiContact presents a hierarchical SE(3) vision-to-force equivariant policy for spatially generalizable contact-rich tasks The proposed EquiContact consists of a Diffusion-Equivariant Descriptor…

In short

EquiContact is a hierarchical system for robotic manipulation that achieves SE(3) equivariance, meaning it performs correctly regardless of how the robot is moved in space. It uses a high-level planner (Diff-EDF) to estimate reference frames and a low-level policy (G-CompACT) that relies only on local sensor data like force readings and visual errors. This allows the system to generalize robustly to new, unseen task positions.

Key concepts

SE(3) Equivariance
This is a mathematical property ensuring that if you transform the entire scene or robot in 3D space (rotation and translation), the resulting control actions also transform predictably. It means the policy learns to handle spatial changes automatically without needing to be explicitly programmed for every possible position.
Diff-EDF
This is a high-level vision planner that processes scene point clouds to generate reference frames. These reference frames act as anchors, helping the system understand where objects are located in the world for tasks like pick-and-place, providing global context to the robot's actions.
G-CompACT
This is a low-level policy that uses real-time feedback (like force and visual errors) to decide how to move. It is 'compliant,' meaning it adjusts its motion based on physical contact forces, allowing for smooth, safe interaction during tasks like screwing or wiping surfaces.
Geometric Admittance Control (GAC)
This control module translates the policy's desired movements into actual robot commands by calculating dynamic gains. It uses the calculated gains to ensure the robot moves compliantly, resulting in lower interaction forces and better performance during contact-rich manipulation tasks.

Terminology used across episodes

This episode discusses

The paper

EquiContact: A Hierarchical SE(3) Vision-to-Force Equivariant Framework for Spatially Generalizable Contact-rich Tasks · Read on arXiv

Joohwan Seo, Arvind Kruthiventy, Soomi Lee, Megan Teng, Seoyeon Choi, Xiang Zhang, Jongeun Choi, Roberto Horowitz

University of California, Berkeley · Yonsei University

This paper presents a framework for learning vision-based robotic policies for contact-rich manipulation tasks, which generalize spatially across task configurations. We propose EquiContact, a novel hierarchical framework composed of a high-level vision planner, such as a Diffusion Equivariant Descriptor Field (Diff-EDF) and a low-level compliant visuomotor policy, such as a Geometric Compliant Action Chunking Transformer (G-CompACT). G-CompACT, operating using localized observations from wrist-mounted RGB images, end-effector pose feedback in the form of a geometrically consistent error vector (GCEV) and force-torque readings, produces actions defined on the end-effector frame. We show that the entire EquiContact pipeline is-equivariant under equivariant reference-frame estimation and left-invariant visual features, from perception to force control. Once the robot learns to execute a contact-rich task, such as inserting a peg-in-hole (PiH), it can execute similar tasks at different locations and orientations without retraining of the low-level policy. We also describe three key components for attaining spatially generalizable contact-rich policies: geometrically consistent compliance, the use of localized policies, and induced equivariance. Real-world peg-in-hole (PiH) experiments demonstrate near-perfect success rates under spatially out-of-distribution configurations, validating the proposed framework and principles, while additional screwing and surface-wiping experiments evaluate the transfer of the localized policy design beyond insertion. The experimental videos and more details can be found on the project website: https://equicontact.github.io/EquiContact-website/

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "EquiContact: A Hierarchical SE(3) Vision-to-Force Equivariant Framework for Spatially Generalizable Contact-rich Tasks".

Dev: Here's the gist of the paper. EquiContact presents a hierarchical SE(3) vision-to-force equivariant policy for spatially generalizable contact-rich tasks The proposed EquiContact consists of a Diffusion-Equivariant Descriptor Field (Diff-EDF) and a Geometric Compliant Action Chunking Transformer (G-CompACT) This framework achieves SE(3)-equivariance from perception to force control, enabling robust spatial generalization across task configurations Framework Architecture The EquiContact pipeline integrates a high-level vision planner (Diff-EDF) with a low-level compliant visuomotor policy (G-CompACT) and geometric admittance control (GAC) at the lowest level The Diff-EDF processes scene point clouds to generate reference frames for pick-and-place tasks, which anchor the G-CompACT Based on these estimated reference frames, the G-CompACT outputs relative pose and admittance gains from real-time feedback These outputs are then fed to geometric admittance control (GAC) which provides compliant motion command to the robot Conceptually, the framework follows "anchoring a localized policy on a globally estimated reference frame" Key Design Principles The paper identifies three key principles for spatially generalizable contact-rich manipulation: (1) left-invariant compliant control action via GAC

19: , (2) localized policy, and (3) induced equivariance These principles enable SE(3)-equivariant behavior without requiring explicitly equivariant neural networks The G-CompACT relies only on localized observations like the geometrically consistent error vector (GCEV), forcetorque readings, and wrist-mounted RGB images Assumption 1 states that the visual encoder produces features that are approximately left-invariant to task transformations Equivariance Proof The paper mathematically proves the SE(3) equivariance property of EquiContact under specific conditions Proposition 2 demonstrates that if an SE(3) equivariant reference frame estimator, such as Diff-EDF, is used, the resulting EquiContact policy possesses the equivariance property Corollary 1 shows that the GCompACT policy satisfies a left-invariance property in the end-effector frame under ideal invariant visual features The final proof shows that when the policy is left-transformed by an arbitrary element gl ∈ SE(3), the resulting trajectories are transformed to glgd, showing equivariance Experimental Validation Real-world experiments on peg-in-hole (PiH), screwing, and surface wiping tasks demonstrate a near-perfect success rate and robust generalization to unseen spatial configurations The method achieves perfect success rates on the translationally unseen flat platform in OOD scenarios The EquiContact framework is validated for various contact-rich tasks, including screwing and surface wiping Limitations include symmetry braking and the need for larger datasets for certain tasks like surface wiping The work concludes that the three principles—compliance, localized policy, and induced equivariance—offer a simple yet powerful design guideline for developing spatially generalizable and interpretable robotic policies This approach provides complementary insights to recent trends in robot learning by enabling a structured divide-and-conquer approach The framework successfully generalizes to unseen platform positions and orientations while maintaining low contact force and near-perfect success rates The EquiContact method is a vision-to-force equivariant policy for spatially generalizable contact-rich tasks The paper successfully proves the SE(3) equivariance of the policy under assumptions on point cloud and image observations The framework is feasible towards screwing and surface wiping tasks Page 1 --- EquiContact: A Hierarchical SE(3) Vision-to-Force Equivariant Policy for Spatially Generalizable Contact-rich Tasks Joohwan Seo, Arvind Kruthiventy, Soomi Lee, Megan Teng, Seoyeon Choi, Xiang Zhang, Jongeun Choi and Roberto Horowitz University of California, Berkeley; Yonsei University Pruning redundant reasoning tokens The proposed EquiContact consists of a Diffusion-Equivariant Descriptor Field (Diff-EDF) and a Geometric Compliant Action Chunking Transformer (G-CompACT) The Diff-EDF, the high-level planner, first processes the scene point cloud to produce reference frames for pick-and-place tasks for the G-CompACT to anchor on With the provided reference frames, the G-CompACT outputs the relative pose and admittance gains from real-time wrist cameras and proprioceptive feedback The output relative pose and admittance gains are then fed to geometric admittance control (GAC) that provides compliant motion command to the robot How the training signal is built To train a G-CompACT, expert demonstrations are collected via teleoperation at a fixed platform pose The dataset includes data with arbitrary visual distractors so that the policy can learn to reject background perturbations For Diff-EDF training, scene and grasp point clouds are collected together with target reference frames representing the desired poses of the end-effector for pick-andplace operations Noise is added to the reference frame gref during dataset preprocessing to provide an inductive bias to rely on eG values for rough alignment Where the gains come from The geometric admittance control (GAC) module implements geometric impedance control (GIC) proposed in

20, 21: The desired end-effector dynamics are written as MV˙ b + KdV b + fG = Fe The elastic wrench fG is given by a formula involving stiffness matrices Kp and KR The gains Kp and KR are defined on the desired frame, i.e., on the body-frame Experimental Validation Real-world experiments on peg-in-hole (PiH), screwing, and surface wiping tasks demonstrate a near-perfect success rate and robust generalization to unseen spatial configurations The EquiContact method achieves perfect success rates on the translationally unseen flat platform, as can be seen in the 4th row of Table I The G-CompACT for the PiH task showed almost perfect success rates on all extreme task transformations, validating that a well-trained G-CompACT policy can handle arbitrary task transformations The framework is feasible towards screwing and surface wiping tasks Limitations and Future Work The most prominent failure case of EquiContact is symmetry braking, specifically, manipulator singularities where pose tracking accuracy degrades A lack of dataset is noted for surface wiping and screwing tasks requiring better quality demonstrations Generalization to other visuomotor policies like diffusion policy (DP) can be explored in future work Vision encoders can be improved by investigating newer versions of vision-language models for better left-invariance In this work, we introduced EquiContact, a vision-toforce equivariant policy for spatially generalizable contactrich tasks By integrating a global reference frame estimator (Diff-EDF) with a fully localized visuomotor servoing policy module (G-CompACT), we demonstrate how compliance, localized policy, and induced equivariance can be unified to enable the peg-in-hole (PiH) task The principles offer a simple yet powerful design guideline for developing spatially generalizable and interpretable robotic policies The framework successfully generalizes to unseen platform positions and orientations while maintaining low contact force and near-perfect success rates The EquiContact method is a vision-toforce equivariant policy for spatially generalizable contact-rich tasks The paper successfully proves the SE(3) equivariance of the policy under assumptions on point cloud and image observations The framework is feasible towards screwing and surface wiping tasks Page 2 --- transformers (ACT)

37: and diffusion policy (DP)

4: are evaluated only within the limited spatial variations Furthermore, both methods exhibit near-linear performance growth as the demonstration dataset size increases, suggesting that the trained policies do not inherently generalize well to new spatial configurations, but rather tend to interpolate between seen demonstrations Pruning redundant reasoning tokens ACT and DP have been extended to other approaches, including compliance and force-reactive behaviors as well as structural improvements Our work is most closely related to CompliantACT (CompACT)

8: which integrates compliant control for visuomotor policies We have significantly improved CompACT by incorporating a provable SE(3) equivariant structure Our approach induces full SE(3) equivariance from vision to control force without relying on explicitly equivariant neural networks, but using structured observations and actions via geometrical canonicalization How the training signal is built The G-CompACT architecture receives GCEV eG (2), F/T sensor value Fe as proprioceptive inputs, and wrist camera images Iw The visual features are modulated by text tokens ttext from the CLIP text encoder via a Feature-wise Linear Modulation (FiLM) layer The training data includes expert demonstrations via teleoperation at a fixed platform pose, with various gain modes like low-gain mode, high-gain mode, insertion mode, and contact mode Where the gains come from The GAC operates with the (gd, Kp, KR) signal calculated from G-CompACT The desired end-effector pose is calculated via gd = ggrel The stiffness gains Kp and KR are defined on the desired frame, i.e., on the body-frame Experimental Validation The CompACT with force-torque sensor inputs and output gains shows lower interaction force in all directions CompACT consistently produces lower interaction forces, especially in the z-direction compared to baseline ACT EquiContact achieves perfect success rates on the translationally unseen flat platform, as can be seen in the 4th row of Table I Page 3 --- Fig.

Rosa: 2: Overview of the workspace for the peg-in-hole assembly task is presented Peg and hole assembly with 1mm of clearance Pruning redundant reasoning tokens Manipulation in Object Frame relies on representing the visuomotor policy in the end-effector frame for improved fidelity and robustness This choice explicitly links the reference frame choice to the equivariance property, allowing for full SE(3) transformations of the reference frame How the training signal is built The G-CompACT architecture receives observations defined by (1) Geometrically Consistent Error Vector (GCEV) eG, (2) FT sensor in the end-effector frame Fe, and (3) RGB images Iw from wrist cameras The visual feature z is modulated using a FiLM layer from text tokens ttext via equation (5) Where the gains come from The GAC module calculates the desired end-effector dynamics for the GAC setup using (8) and (9) The desired pose command is calculated using (8), which is then passed to the robot as a pose command signal Experimental Validation The CompACT with force-torque sensor inputs and output gains shows lower interaction force in all directions CompACT consistently produces lower interaction forces, especially in the z-direction EquiContact achieves perfect success rates on the translationally unseen flat platform, as can be seen in the 4th row of Table I Page 4 --- Fig.

Dev: 4: G-CompACT architecture is presented Pruning redundant reasoning tokens The G-CompACT receives observations defined by (1) Geometrically Consistent Error Vector (GCEV) eG, (2) FT sensor in the end-effector frame Fe, and (3) RGB images Iw The observation vector o(k) is defined as (eG, Fe, Iw)(k) The policy is written as a(k) = πθ(o(k)) How the training signal is built The G-CompACT outputs actions a(k) = (grel, Kp, KR)(k), where grel is the relative pose and Kp, KR are admittance gains The GCEV eG (g, gref) ∈ R 6 is defined as an error vector between the current end-effector frame and the reference frame Where the gains come from The desired pose gd(k) = g(k)grel(k) The GAC module outputs the geometrically consistent compliant motion from desired poses and admittance gains to enable equivariant force interaction Experimental Validation The CompACT with force-torque sensor inputs and output gains shows lower interaction force in all directions CompACT consistently produces lower interaction forces, especially in the z-direction Page 5 --- Fig.

Rosa: 5: Effects of the left group action gl to the end-effector pose g and the reference frame gref, and to the wrists cameras Iw,1 and Iw,2 Pruning redundant reasoning tokens Assumption 1 states that visual encoder features are approximately left-invariant to task transformations Under this assumption, the G-CompACT policy is left-invariant in the end-effector frame Proposition 1 shows that the G-CompACT policy πθ is left-invariant in the end-effector frame Corollary 1 shows that the GCompACT πθ represented in the spatial frame satisfies a SE(3) equivariance property How the training signal is built The visual encoder µϕ produces features that are approximately left-invariant to task transformations The language tokens encode object identity rather than pose, providing a conditioning signal invariant to global SE(3) transformations of the scene Where the gains come from The desired pose signal in the spatial frame gd is obtained via (with a slight abuse of notation) gd = ggrel = g · πθ(o(k)) The GAC module calculates the desired end-effector dynamics for the GAC setup using (8) and (9) Page 6 --- Fig.

Title and authors: Rosa: So let’s look at the title of this paper again: EquiContact: A Hierarchical SE(three) Vision-to-Force Equivariant Framework for Spatially Generalizable Contact-rich Tasks <ref:2507.10961#pg1,EquiContact: A Hierarchical SE(3) Vision-to-Force Equivariant>. It tells you immediately what the system is aiming to do, which is tackle contact manipulation problems in a spatially general way.

Dev: It’s not just about moving things; it’s about making sure the robot responds correctly to changes in its environment's orientation or position. The SE(three) equivariance part suggests that the system should be robust regardless of how you orient the world around it <ref:2507.10961#pg1>.

Taro: I’m curious about that SE(three) part <ref:2507.10961#pg1>. Does that mean if we rotate my entire workspace, the robot’s planned actions also rotate perfectly with it? Or is it just good at handling translations and rotations individually?

Rosa: It means a full transformation in three dee space—rotations, translations, scaling—should result in a corresponding transformation of the output <ref:2507.10961#pg1>. The paper says they prove this equivariance property under specific conditions regarding the observations they use, like scene point clouds and images.

Dev: The authors are proposing this hierarchical structure: Diff-EDF for high-level planning and GCompACT for low-level control, anchored by geometric admittance control. It’s a way to manage the complexity of a contact task by breaking it down into manageable steps.

Taro: Breaking it down sounds good, but what happens when the global estimate from the Diff-EDF is slightly wrong? Does that error propagate and cause chaos in the local policy?

Rosa: That’s where they show how anchoring helps. The GCompACT outputs relative pose and admittance gains based on those estimated frames, which acts as a bridge. It takes those estimates and uses them to generate the compliant motion command through the GAC module.

Dev: And that bridge is crucial because it’s what allows the system to handle spatial shifts without needing perfect global knowledge right away. The whole point is localized policy on a globally estimated frame, which you said earlier.

Taro: So if we're looking at this paper, we have the theory of equivariance and a structure that links vision to force control through compliance, but the main question remains how robust it is when things go sideways in the real world.

Rosa: Exactly. The robustness comes from combining those principles—compliance, localized policy, and induced equivariance. It’s a design guideline for building these kinds of systems that are both interpretable and generalizable across different task configurations.

Dev: And that structure allows them to test it on real-world tasks like peg-in-hole or screwing, showing near perfect success rates even on translationally unseen flat platforms. That’s the kind of validation we need to see before we put this in a robot arm.

The paper's summary: Rosa: So, let's summarize what EquiContact actually proposes. At its core, it’s a hierarchical policy combining the Diff-EDF and GCompACT modules to solve contact manipulation problems with spatial generalization.

Dev: Essentially, the Diff-EDF processes point clouds to create reference frames for tasks like pick-and-place operations, which then anchor the GCompACT. The GCompACT uses real-time sensor feedback—wrist cameras and proprioceptive feedback—to generate relative poses and admittance gains.

Taro: So, the low level isn't just reacting blindly; it’s using those reference frames to make smarter decisions about how much force to apply, which is interesting. It’s not just pure reflex anymore.

Rosa: Right. And that action—the relative pose and admittance gains—is then fed into geometric admittance control or GAC, which translates those commands into compliant motion commands for the robot's movement.

Dev: And the paper emphasizes that this entire pipeline achieves SE(three) equivariance by using structured observations and actions via geometrical canonicalization rather than relying on explicit equivariant neural networks <ref:2507.10961#pg1>.

Taro: So they’re inducing it through observation structure, not by building a whole new type of network architecture from scratch. That makes sense for deployment speed-wise.

Rosa: It's about using the structure of the observation and action to enforce the desired transformation behavior, which is what they call induced equivariance. This lets them get spatial generalization without needing explicitly equivariant neural networks, which can be computationally expensive or hard to develop in a real-time setting.

The paper's improvements: Dev: The authors highlight three key design principles that make this work: first, left-invariant compliant control action via GAC. Second, localized policy. And third, induced equivariance. These are the core ingredients they believe are needed for spatially generalizable contact-rich manipulation.

Taro: I like the idea of "localized policy" because it keeps the control decision-making focused on what’s immediately relevant to the current contact point, rather than trying to solve the entire world problem at once.

Rosa: Right. And that localized policy relies only on local information like the geometrically consistent error vector, or GCEV, force-torque readings from sensors in the end-effector frame, and wrist camera images.

Dev: Those are just local inputs for the GCompACT architecture. The visual features are also modulated by text tokens from a CLIP text encoder via a Feature-wise Linear Modulation layer.

Taro: So they’re using language to condition the visual input in a way that seems to be invariant to global scene changes, which is interesting because it’s not just about geometric structure.

Rosa: They are using language tokens for object identity rather than pose, which keeps the conditioning signal invariant to global SE(three) transformations of the scene <ref:2507.10961#pg1>. This supports Assumption one where visual encoder features are approximately left-invariant.

Conclusion: Dev: So, to wrap up EquiContact, the framework successfully integrates a global reference frame estimator with a fully localized visuomotor servoing policy module. It demonstrates how compliance, localized policy and induced equivariance can be unified for tasks like peg-in-hole.

Rosa: The major implication is that this provides a simple yet powerful design guideline for developing policies that are both spatially generalizable and interpretable. It’s a structured way to approach complex problems in robotics without needing overly complex or specialized neural networks.

Taro: I just hope they keep focusing on the practical deployment aspects, because the real world is where we see if this actually survives those tricky singularities we discussed earlier.

Dev: That’s a valid concern regarding those limitations—symmetry braking and the need for larger datasets for certain tasks like surface wiping. But they’ve shown it works on translationally unseen flat platforms with perfect success rates in some cases, which is solid data points.

Rosa: So, EquiContact is a vision-to-force equivariant policy for spatially generalizable contact-rich tasks that successfully proves the SE(three) equivariance of the policy under specific assumptions on point cloud and image observations <ref:2507.10961#pg1,vision-to-force equivariant policy for spatially generalizable contact-rich tasks>. It’s a significant piece of work for anyone working on this area.

Taro: I think it gives us a clear set of tools to move forward with designing policies that are more robust against spatial changes, even if they have to work through those known failure modes for now.

Dev: Indeed, it's a vision-to-force equivariant policy for spatially generalizable contact-rich tasks. We’re looking forward to seeing how this framework matures in the future.

More episodes

← Home