ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

arXiv:2606.03177 · cs.RO · Submitted 2026-06-02 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control".

Dev: The gist The authors introduce ConTrack, a reinforcement learning framework that treats object tracking as a constraint and allocates remaining control authority to motion fidelity,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called "ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control." It sounds like they've tackled that big problem of taking human demonstrations and making them work on real robots, especially for complex tasks.

Dev: Yeah, it seems to be focused on the gap between how a human moves their hand and what a robot can actually do when you try to follow that motion directly. It addresses tracking long sequences with contacts while keeping the joint movement and contact timing correct.

Taro: I'm curious about what they're trying to achieve here in terms of scaling up those demonstrations, because usually when you go from simulation to real hardware, things break pretty fast.

Rosa: Exactly, that's where this paper steps in by treating object tracking as a constraint and putting the remaining control power into motion fidelity. It lets the system adapt what it prioritizes online using this dual-variable update mechanism.

Dev: That dual-variable update sounds like they're essentially managing a trade-off between getting the object in position and making sure the hand movement looks physically realistic, which is a tough balance to strike.

Taro: So, instead of picking one thing to optimize at a time, it tries to handle both simultaneously by adjusting that weighting on the fly.

Rosa: Right, and they also have this adaptive mid-trajectory reset library which reuses simulator states near where the policy starts failing, which is a big deal for long learning episodes.

The paper's summary: Dev: So, if we look at the summary of "ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control," it lays out how they formulate the problem as a constrained reinforcement learning problem. They want the policy to maximize style fidelity while meeting a minimum level of task success defined by Equation three <ref:2606.03177#pg1>.

Rosa: The reward function itself is broken down into three parts, so you got tracking success from object pose, style fidelity from hand kinematics and contacts, and penalizing high-frequency motion. That's a structured way to tell the AI what it needs to do.

Taro: And the way they update that policy is through a mixed advantage function called A mix which mixes task advantage with style advantage based on a weight wtask that goes from zero to one using a logistic sigmoid.

Dev: That weight wtask isn't fixed; it's updated online using this dual-variable update, and if the task return J g drops below a target alpha, that weight increases to shift optimization toward tracking.

Rosa: That dynamic adjustment of the trade-off during training is what they call a dual-variable update, which is key because it lets the system react to performance changes as it learns.

Taro: It makes sense because when you're just learning, you might need to prioritize getting the object tracked correctly first, and then worry about making the motion look good later on.

The paper's improvements: Rosa: One of the main improvements they point out is this online adaptive task–style mixing mechanism we just talked about. It’s designed to let the system dynamically adjust its objective function based on performance metrics during training.

Dev: That's the dual-variable update in action, and it lets them shift optimization toward task tracking when the task success drops below a threshold of alpha, which is between zero and one.

Taro: And I think another really important part is this adaptive mid-trajectory reset library they use to stabilize long-horizon learning by restarting from simulator states near where the policy tends to fail quickly relative to the remaining horizon.

Rosa: That reset library reuses states for every frame k from zero up to T minus one, sampling those reset frames based on a probability proportional to exp negative u k over tau. It concentrates training on segments where the policy tends to fail early.

Dev: And they also have these contact priors which act as an auxiliary style prior by including a contact event overlap term that rewards agreement with the reference contact events c ref t,o,l.

Taro: That helps anchor the style objective to real interaction patterns, and it also penalizes the distance between your tracked object position and the reference object position when both indicate contact.

Conclusion: Rosa: So to wrap up on "ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control," these improvements mean they can handle long-horizon, contact-rich dexterous hand tracking by treating object tracking as a constraint and allocating remaining authority to motion fidelity.

Dev: It means the system can execute complex, multi-step manipulation tasks on real robots while preserving the demonstrated joint motion and contact timing, which helps bridge that kinematic gap with human demonstrations.

Taro: The online adaptive task–style mixing mechanism lets the AI dynamically adjust its objective function based on performance metrics during training so it can adapt the trade-offs online.

Rosa: And that reset library stabilizes long-horizon learning by reusing simulator states near failure points, and they validated this framework on a real-world bimanual xArm7+xHand platform.

Dev: The contact priors give complementary supervision for contact-rich tracking, and together these components help ConTrack narrow the gap between making progress and achieving high fidelity.

Taro: Overall, it shows that adapting the task–style balance online and stabilizing long-horizon learning through mid-trajectory resets really helps in bimanual interaction benchmarks under a fixed training budget.

Rosa: That's what we were talking about with "ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control." It’s a framework that allows the AI to adapt the trade-offs during training and stabilize long-horizon learning.

University of California San Diego

cs.RO

Submitted: 2026-06-02

Updated: 2026-10-13

Project page: https://www.lyt0112.com/projects/ConTrack

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The authors introduce ConTrack, a reinforcement learning framework that treats object tracking as a constraint and allocates remaining control authority to motion fidelity, allowing it to

Key concepts

Task Constraint vs. Style Fidelity: The framework models object tracking as a mandatory constraint that must be met. The remaining control effort is then dedicated to maximizing style fidelity, which refers to how closely the robot's movement matches the desired hand kinematics and contact timings from the demonstration. This forces the robot to balance achieving its goal with executing smooth, realistic movements.
Online Dual Controller (Dual-Variable Update): This component manages the trade-off between task success and style fidelity in real-time during training. It uses a mechanism where a weight variable adjusts based on the current performance of the task. If tracking lags, this controller increases the weight favoring task success, allowing the policy to adapt its focus dynamically.
Adaptive Mid-Trajectory Reset Library: To solve instability in long sequences, this library stores simulator states near points where previous attempts failed. When training encounters a difficult segment, it samples a reset point from this library. This restarts the learning process from a more promising state, preventing the policy from getting stuck in poor local optima during long-horizon tasks.

Terminology

Fidelity — 2× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes

Summary

The gist The authors introduce ConTrack, a reinforcement learning framework that treats object tracking as a constraint and allocates remaining control authority to motion fidelity, allowing it to adapt task–style trade-offs online using a dual-variable update.

How it works

ConTrack models the object tracking problem as a constraint and allocates remaining optimization pressure to style fidelity. This is achieved through three practical components:

An online dual controller that adjusts the task–style allocation.
An adaptive mid-trajectory reset library that restarts rollouts from policy-reachable states near the current failure boundary.
Contact priors that anchor the style objective to reference interaction patterns.

The core optimization is formulated as a constrained reinforcement learning problem where the policy maximizes style while meeting a minimum level of task success, defined by Equation 3. The reward function is decomposed into three parts:

**/r(s t,a t)=r g(s t,a t)+r s(s t,a t)+r p(s t,a t), (1) where rg measures task success through object pose tracking, rs measures style fidelity through hand kinematics and contacts, and rp penalizes high-frequency motion. The policy controls the robot through residual joint targets around the reference. The objective is to maximize style while meeting a minimum level of task success: max π J s(π) s.t. J g(π)≥ α J g⋆, where α ∈ (0, 1] is a target task ratio and J⋆ g is a clip-specific normalizer defined as a running maximum of a moving estimate of Jg during training. The policy updates using a mixed advantage A mix = wtask Ag + (1-wtask)As + Ap, where wtask ∈ [0, 1] is shared across all parallel environments of the clip. This mixing weight is mapped to a convex mixing weight w task = σ (λ), where σ is the logistic sigmoid. The dual update adjusts λ from an online estimate of the task return Jg using Equation 7, which increases wtask and shifts optimization toward task tracking when Jˆg/J⋆ g drops below α. This mechanism allows ConTrack to adapt the trade-off during training via a dual-variable update. The adaptive mid-trajectory reset library addresses long-horizon learning inefficiency by storing simulator states for each frame k ∈ [0, T − 1] and sampling reset frames with p(k) ∝ exp (−uk/τ). This concentrates training on segments where the policy tends to fail quickly relative to the remaining horizon. The contact priors provide an auxiliary style prior by including a contact event overlap term that rewards agreement with the reference contact events c ref t,o,l. This term also penalizes the distance between yt,o,l and y ref t,o,l when both the simulator and reference indicate contact. The observation specification uses an asymmetric actor–critic setup where the critic receives additional signals that reduce value-estimation variance and make long-horizon training stable. The critic input concatenates these terms: o critic t=[z t; v t; c t; z ref t:t+K; c ref t; φ t]. (13) The controller uses a target ratio α = 0.9 in the main results, step size η = 0.05 in Eq. 7, and initializes λ = 0 so that wtask = σ(λ) = 0.5 at the start of training. The reward terms are specified with weights such as tracking obj pos exp −mse(e p t)/σ squared p, where σp = 0.05 m +500. The observation vector for the actor is o actor t=[z t-d; v t-d] in R(2D+13O), where d ≥ 0 is an environment-step observation delay sampled during training. The critic receives binary contact indicators c ∈ [0, 1] LO that mark whether each of L tracked hand links is in contact with each object at index t. The final evaluation involves comparing ConTrack against baselines like ManipTrans, DexMachina, and SPIDER under a fixed simulator-step budget. The results show that ConTrack improves success and object pose accuracy while preserving joint and contact fidelity. This is demonstrated across three benchmark tiers of increasing contact complexity and validated on a real-world bimanual manipulation setup. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The adaptive update attains higher naturalness as measured by contact fidelity under the same budget for some achieved progress level. The reset library improves progress under the same budget compared to always resetting from the first frame and to a uniform mid-clip reset. The contact priors provide complementary supervision for contact-rich tracking, as removing either term reduces progress or contact accuracy. ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. This is achieved across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets. Across bimanual interaction and in-hand manipulation benchmarks under a fixed training budget, ConTrack improves progress and contact fidelity while retaining accurate object tracking. The results show that ConTrack narrows this gap by adapting the task–style balance online and by stabilizing long-horizon learning through mid-trajectory resets.

Improvements for AI systems

The ConTrack framework enables long-horizon, contact-rich dexterous hand tracking by treating object tracking as a constraint and allocates remaining control authority to motion fidelity. This allows the AI system to execute complex, multi-step manipulation tasks on real robots while preserving the demonstrated joint motion and contact timing, overcoming the kinematic gap between human demonstrations and robot actuation.
The online adaptive task–style mixing mechanism, driven by a dual-variable update, enables the system to dynamically adjust its objective function based on performance: When it drops below α, Eq. 7 increases wtask and shifts optimization toward task tracking. This allows the AI to adapt task–style trade-offs online during training, ensuring that the robot prioritizes object pose accuracy when necessary while maintaining style fidelity.
The adaptive mid-trajectory reset library stabilizes long-horizon learning by restarting from policy-reachable simulator states near the current failure boundary. This mechanism addresses inefficiency in long-horizon training by sampling reset frames with a probability proportional to the survival ratio, which concentrates training on segments where the policy tends to fail quickly relative to the remaining horizon.
The system incorporates contact priors as an auxiliary style prior by including a term that rewards agreement with reference contacts: The style reward includes a contact event overlap term that rewards agreement with the reference contact events c ref t,o,l. This ensures high-fidelity contact execution by anchoring the style objective to reference interaction patterns even when object pose errors alone are insufficient.
The observation specification utilizes an asymmetric actor–critic setup where the critic receives binary contact indicators ct in LO that mark whether each of L tracked hand links is in contact with each object at index t. This allows the critic to receive rich, uncorrupted information, which reduces value-estimation variance and make long-horizon training stable.

Abstract

Human demonstrations provide strong priors for robot manipulation, yet it is non-trivial to transfer them to execute on real robots due to the kinematic gap. In dexterous manipulation, it remains challenging to track long-horizon, contact-rich sequences even in simulators: a reference-tracking policy must keep objects on their target trajectories while preserving demonstrated joint motion and contact timing. Existing approaches often rely on hand-crafted reward tuning that require per-sequence tuning and break under limited interaction budgets. We introduce ConTrack, a reinforcement learning (RL) framework that scales with tracking data. ConTrack treats object tracking as a constraint and allocates remaining control authority to motion fidelity, which allows it to adapt task--style trade-offs online using a dual-variable update. In addition, ConTrack also stabilizes long-horizon learning with an adaptive mid-trajectory reset library that reuses policy-reachable simulator states. Our qualitative and quantitative results in simulation tracking and real robot demonstrate that ConTrack improves success and object pose accuracy significantly over prior arts while preserving joint and contact fidelity. Website: https://www.lyt0112.com/projects/ConTrack.