Same Action, Different Outcome: Variability in Dynamic Cloth Manipulation

The gist The authors quantify and characterize variability in dynamic cloth manipulation in 269 collected executions across four dynamic tasks with a total of 27 different trial configurations >

Interaction Variability

Interaction variability is defined as the non-repeatability of the cloth state under the same trajectory and a nominally identical initial state The authors formalize this by defining a rollout feature z as z(i):= ϕ y(i) 0:T, z(i) ∼ preal(z π), i = 1..., Nc They then measure the variability of a feature z as the expected distance between different rollouts, and how different it is from the floor value ε The configuration is deemed non-repeatable when "D¯c > εc This empirical form of the interaction variability is measured by calculating Dc(i, j) as Dc(i, j):= 1/Mij X m∈Mij q(i,m) − q(j,m)"

Key Metrics and Factors

The study utilizes several metrics to quantify variability across different conditions

1. The dispersion of the task configuration is estimated by "D¯c:= Nc/2-1 X i<j Dc(i, j)".

2. The floor variability εc is obtained by measuring the distance between markers at grasped corners as "Nc/2-1 X i<j 1/Hij X m∈Hij q(i,m) − q(j,m) 2".

3. The speed contrast of a cloth is defined as ΓD:= D¯c(cloth, min ts) / D¯c(cloth, max ts).

Experimental Setup and Tasks

The experiments involved executing the same trajectory ten times across four dynamic tasks, each tested with three cloths of very different properties and up to three execution speeds The setup utilized two Franka Research 3 arms with default Franka Hands placed side by side across a table The cloth objects used had sizes of 0.7×0.5 m and exhibited extremely different physical cloth properties, such as mass, area density, friction coefficient, elasticity along the short, long and diagonal edges, drape stiffness (Table I) The four dynamic tasks included Swing to place (right) and gripper inward motion (left), swing-toplace flat also known as fling, dynamic folding on the table, using a single hand, and two novel tasks involving moving the grippers inward toward each other, with and without a swing

Results on Variability Drivers

The results demonstrate that variability is significant across all experimental configurations and depends highly on cloth properties Specifically, the cloth explains 61 %, the task 10 % and the execution speed only 6 % when grouping non-regrasp configurations by task or speed The brown cloth, being the heaviest and highest friction fabric, is generally the most repeatable in every task except for the fast swing join where it reaches variability similar to red cloth The two light cloths scatter three to ten times more than the thick brown cloth Regarding speed contrast, the only one below 1 corresponds to the dynamic fold with the green satin cloth, for which ΓD = 0.97 In contrast, for most other tasks, higher velocity increases variability

Simulator Performance

The evaluation included replaying all rollouts in four calibrated modern cloth simulators to assess their fidelity The findings showed that none of them can reproduce the variability magnitude we observed in real cloth, nor its ordering The simulated markers were written back in the motioncapture file format for comparison with the real analysis code It was observed that the order is not well preserved in any simulator, with Clothilde being slightly better thanks to the swing to place task for which works very well

Conclusion on Sim-to-Real Gap

The study concludes that any sim-to-real approach with dynamic motions will always be a rough approximation unless we can reproduce this behavior in the simulation The authors open-sourced their labeled dataset of marked cloth keypoints from motion capture, along with synchronized stereo stream and proprioception, to aid in training perception methods and evaluating cloth simulators In future works, the goal is to tackle this sim-to-real gap and use the measured variability to improve the generalizability of trained policies across cloth types

Measuring Without Markers

The researchers also investigated measuring dispersion without markers using data from the stereo camera This involved segmenting terminal state images with SAM 2, followed by 3-D extraction via FoundationStereo and DINOv3 patches to compute Dc(i, j) using the mean distance between matched patches While vision supports a qualitative comparison of cloths and speeds, it does not capture the same variability values as OptiTrack precision

Analysis of Specific Tasks

For Task 1 (swing place), the thick cloth showed resting poses less than a centimeter apart at every speed, whereas light cloths scattered significantly more with speed For Task 2 (dynamic fold), the free corner landed away from its target in every configuration by 7 to 41 cm, with the thick fabric being best behaved For Tasks 3 and 4 (linear join and swing join), buckling was a non-negligible phenomenon, with the swing introducing more variability in magnitude d⊥ for the stiff wool fabric The soft red chiffon cloth showed that the swing motion does not seem to influence the outcome at all, at any speed

Regrasping Effect

Regrasping before each rollout raised the floor variability to 12–27 mm instead of 2–4 mm However, this increased floor variability was still exceeded by the outcomes by a factor of 2.6–4.2× compared to non-regrasped tasks The corner sits differently in the fingers after a regrasp, and the cloth adds its own variability on top

Final Summary

The authors have quantified and characterized the variability in dynamic cloth manipulation in 269 collected executions across four dynamic tasks with a total of 27 different trial configurations > The results show that variability is significant across all experimental configurations, and that it depends highly on the cloth properties, and to a lesser extent on speed > This work provides a dataset of motion-captured rollouts with synchronized stereo video, robot logs, and corresponding simulator twins for four dynamic cloth manipulation tasks > The authors believe this dataset can be useful to train perception methods, evaluate cloth simulators and learn cloth dynamic models >

The gist The authors quantify and characterize variability in dynamic cloth manipulation in 269 collected executions across four dynamic tasks with a total of 27 different trial configurations >

How it works

Interaction variability is defined as the non-repeatability of the cloth state under the same trajectory and a nominally identical initial state The authors formalize this by defining a rollout feature z as z(i):= ϕ y(i) 0:T, z(i) ∼ preal(z π), i = 1..., Nc They then measure the variability of a feature z as the expected distance between different rollouts, and how different it is from the floor value ε The configuration is deemed non-repeatable when "D¯c > εc This empirical form of the interaction variability is measured by calculating Dc(i, j) as Dc(i, j):= 1/Mij X m∈Mij q(i,m) − q(j,m)"

Experimental Setup and Tasks

The experiments involved executing the same trajectory ten times across four dynamic tasks, each tested with three cloths of very different properties and up to three execution speeds The setup utilized two Franka Research 3 arms with default Franka Hands placed side by side across a table The cloth objects used had sizes of 0.7×0.

1. Improve policy reliability by incorporating a formalism to show that the same dynamic manipulation of cloths yields different outcomes depending on the nature of the manipulation and the cloth’s features. This allows policies to explicitly account for interaction variability rather than assuming deterministic outcomes, which is crucial for real-world deployment.

2. Enhance sim-to-real transfer by requiring simulators to reproduce not just the magnitude we observed in real cloth, nor its ordering, but also ensuring that the simulated dispersion matches the real one across all tasks and cloth types. This directly addresses the finding that none of them can reproduce the variability magnitude we observed in real cloth, nor its ordering.

3. Develop policies robust to environmental effects by explicitly modeling or compensating for aerodynamic and frictional changes, as these are identified as key drivers of variability: variability is driven mainly by the cloth physical properties but also grows with speed. This enables policies to adapt their execution strategy based on predicted environmental influences.

4. Implement speed-adaptive control strategies informed by the calculated speed contrast of a cloth, defined in Eq. (6). By understanding when higher velocity increases variability versus when it might be beneficial, the system can adjust its trajectory time scale (ts) dynamically to minimize outcome dispersion while still achieving the goal.

5. Create perception systems that utilize markerless vision techniques (using data from the stereo camera that recorded all our experiments) to estimate cloth state features, acknowledging that perception struggles most in the dynamic fold task where color contrast is lost and patch matching fails, leading to under-estimation of variability.