A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures

arXiv:2506.10239 · cs.RO · Submitted 2025-06-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures".

Dev: Here's the gist of the paper. Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task, based on learned or perceived uncertainty<ref:2506.10239#pg2> While keeping the human in the loop remains essential, for instance, to ensure high precision, partial automation of certain task phases is critical for productivity<ref:2506.10239#pg2> We present a unified framework for probabilistic VFs that seamlessly switches between manual fixtures, semi-automated fixtures (with the human handling precise tasks), and full autonomy<ref:2506.10239#pg2> Our main contribution is a framework for the arbitration of probabilistic fixtures, ensuring both an 1 School of Computation, Information and Technology, Sensor Based Robotic Systems and Intelligent Assistance Systems, Technical University of Munich, Friedrich-Ludwig-Bauer-Str.

Rosa: 3, Garching, Germany<ref:2506.10239#pg2> Framework Components The framework introduces a novel probabilistic Dynamical System-based VF for coarse guidance<ref:2506.10239#pg2> For tasks requiring precise guidance, we extend probabilistic positionbased trajectory fixtures with automation, allowing for seamless human interaction, geometry-awareness and optimal impedance gains<ref:2506.10239#pg2> We introduce a novel variable stiffness formulation that models couplings between positional and orientational degrees of freedom (DoFs)<ref:2506.10239#pg2> The framework outputs a Cartesian wrench wVF which is applied to the end effector<ref:2506.10239#pg2> Fixture Arbitration and Fusion The core of our framework is a novel variable impedance control scheme as well as an optimal arbitration of all probabilistic fixture wrenches<ref:2506.10239#pg2> We form the fusion of different VFs in a principled way using an arbitration scheme (Section IV), thus solving the problem of selecting and switching between VFs in a flexible and adaptive manner based on learned and/or perceived uncertainties<ref:2506.10239#pg2> This involves an extended arbitration scheme taking different types of VFs (Table I) and different underlying geometries into account (Section IV-A)<ref:2506.10239#pg2> We propose a Product of Experts (PoE)

55: to arbitrate between different fixture wrenches expressed in the same manifold<ref:2506.10239#pg2> The resulting wˆ with the optimally weighted wrenches of the individual fixtures is then applied to the robot through (3)<ref:2506.10239#pg2> Learning and Uncertainty Modeling Different probabilistic models encoding assistive behaviors can be learned from demonstration data using Gaussian distributions<ref:2506.10239#pg2> We argue that fixture activation should be inversely proportional to uncertainty – both epistemic and aleatoric – such that strong guidance corresponds to low uncertainty levels<ref:2506.10239#pg2> Kernel-based methods such as Gaussian Processes (GPs)

52: and KMPs

53: output epistemic uncertainty which increases with the distance between training data and new inputs<ref:2506.10239#pg2> The KMP encodes a small covariance at s and a bigger covariance at t for the right motion policy<ref:2506.10239#pg2> Automation and Control Laws For position-based VFs, we introduce an automation of this fixture in Section VI-D<ref:2506.10239#pg2> We propose a novel control law for DS-based Virtual Fixtures wVF,i = DVF,i (x˙ VF,i − x˙ ee) (28)<ref:2506.10239#pg2> The resulting wrench is then added to the wrench of the impedance controller (4)<ref:2506.10239#pg2> Evaluation and Results Our framework is implemented on three robotic systems in different automation levels as and interaction modes for evaluating task-specific metrics<ref:2506.10239#pg2> The evaluation with expert users suggests high usability of our method, paired with low workload for human operators<ref:2506.10239#pg2> We find significant differences for measured forces and the NASA TLX score and a non-significant trend for the SUS metric in Table V<ref:2506.10239#pg2> The overall small standard deviations in both experiments show that despite only aiming for coarse guidance, without external perturbations, the repeatability of the DS based fixture is remarkably high<ref:2506.10239#pg2> In this work we introduce a unified, probabilistic Virtual Fixture framework providing different types of assistance to operators – particularly coarse, precise, and very precise guidance—where each type of fixture can either be manually defined or learned from human demonstrations<ref:2506.10239#pg2> We further introduce geometry-awareness in shared control through object-specific coordinate systems, including Cartesian, cylindrical and spherical frames<ref:2506.10239#pg2> Combined with a novel variable impedance formulation – which robustly captures correlations between DoFs—our framework brings together the different fixture types using a product of experts approach, enabling a principled and uncertainty-aware fusion of assistance commands<ref:2506.10239#pg2> We have validated that the approach can be readily applied across diverse use cases, thanks to its ease of programming and flexibility with respect to fixture representations, input modalities, and uncertainty sources<ref:2506.10239#pg2> The evaluation with 6 expert users suggests high usability of our method, paired with low workload for human operators<ref:2506.10239#pg2> While we demonstrated its use in factory automation and space scenarios, and in an initial experiment with expert users, we believe that the approach is also well-suited for medical and personal assistance robotics, which we plan to investigate in future work<ref:2506.10239#pg2> We furthermore envision developing methods to interactively modify position-based fixtures, allowing an operator to modify them adaptively based on novel task needs<ref:2506.10239#pg2> To ease finding working hyperparameters for the geometric visual servoing fixture, methods for automatic tuning should be investigated<ref:2506.10239#pg2> Finally, to enable the use of the proposed framework in high-latency applications such as on-orbit servicing, we plan to integrate it with controllers that can provide stable force feedback despite such delays<ref:2506.

Paper summary: Rosa: So looking at the whole paper again, "A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures" is about unifying how robots get guidance in tasks that require different levels of precision.

Dev: The authors are showing how you can create this single framework to handle coarse, precise, and very precise guidance by making the fixture selection adaptive based on uncertainty.

Taro: I think the real takeaway is that this isn't just about adding more fixtures; it’s about creating a principled way to blend manual setup with learned assistance using those probabilistic models.

Rosa: Exactly. They introduce geometry awareness through object-specific coordinate systems, like Cartesian or cylindrical frames, which lets the robot understand its physical surroundings better for different fixture types.

Dev: Coupled with that variable impedance formulation they developed—which robustly captures correlations between degrees of freedom—they use that product of experts approach to fuse all those assistance commands based on uncertainty.

Taro: It seems applicable beyond just factory automation, because they tested it in space scenarios and also suggested it could be useful for medical or personal assistance robotics.

Rosa: They've shown that the method is flexible enough to be applied across different use cases because you can define fixtures manually or learn them from human demonstrations.

Dev: The main implication is that this gives operators a tool where the level of help they get changes automatically based on how uncertain the robot thinks it is about what it's doing.

Conclusion: Rosa: So we've been looking at this paper from School of Computation, Information and Technology at TUM about this unified framework for virtual fixtures.

Dev: Yeah, they put together something that handles coarse guidance, precise guidance, and even very precise guidance all in one system using uncertainty modeling.

Rosa: It seems the whole point is to let a robot switch between manual control and full automation based on how sure it is about what it's doing.

Dev: Exactly. The main thing they did was create a way to arbitrate between these different fixture commands using a product of experts approach, weighted by that uncertainty level.

Rosa: I mean, so instead of having separate systems for each level of guidance, you have one brain deciding which tool to use right now based on the situation and its confidence.

Dev: Right. And they used a novel variable impedance formulation to tie together the positional and orientation movements really well. It's not just picking one fixture; it's blending them optimally.

Rosa: It makes sense for real-world robots where you can’t always be perfectly precise, but you still need reliable assistance when things get tricky or uncertain.

Dev: The results they showed suggest this approach is usable, even with expert users, and the repeatability of their coarse guidance fixture is pretty high without any external pushes.

Rosa: It opens up possibilities for applying this in areas where robots have to interact with humans or delicate objects where the required assistance level changes constantly.

Dev: It's definitely got some promise for making autonomous systems more adaptable to messy, real-world tasks and not just lab settings.

School of Computation, Information and Technology, Technical University of Munich · German Aerospace Center (DLR), Robotics and Mechatronics Center (RMC) · Idiap Research Institute · Ecole Polytechnique Férale de Lausanne (EPFL)

cs.RO

Submitted: 2025-06-11

Updated: 2026-11-10

Comments: for the supplementary video, see https://youtu.be/tm0S4H3Tze8

DOI: 10.1109/TRO.2026.3751889

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task…

Key concepts

Probabilistic Virtual Fixtures (VFs)
These are adaptive guidelines for a robot's movement that change based on uncertainty. Instead of fixed rules, VFs learn or predict how uncertain a situation is and choose the best guidance strategy—manual, semi-automated, or fully autonomous—to help the human operator.
Fixture Arbitration and Fusion
This is the core mechanism for deciding which fixture to use. It uses an optimal arbitration scheme, including a Product of Experts (PoE), to combine different virtual fixture wrenches into one best command. This allows the system to flexibly switch between various guidance types based on perceived uncertainty.
Uncertainty Modeling
The framework learns assistive behaviors using Gaussian distributions from data. It links fixture activation inversely to uncertainty; meaning, when the system is highly uncertain about a movement, it provides stronger guidance. Kernel-based methods like Gaussian Processes quantify this epistemic uncertainty.
Variable Stiffness Formulation
This mathematical model describes how positional and orientational movements are coupled together. It allows the framework to create fixtures that account for relationships between different degrees of freedom, leading to more robust guidance when modeling complex interactions.

Terminology

Summary

Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task, based on learned or perceived uncertainty<ref:2506. p. 010239#pg2> While keeping the human in the loop remains essential, for instance, to ensure high precision, partial automation of certain task phases is critical for productivity<ref:2506. p. 010239#pg2> We present a unified framework for probabilistic VFs that seamlessly switches between manual fixtures, semi-automated fixtures (with the human handling precise tasks), and full autonomy<ref:2506. p. 010239#pg2> Our main contribution is a framework for the arbitration of probabilistic fixtures, ensuring both an 1 School of Computation, Information and Technology, Sensor Based Robotic Systems and Intelligent Assistance Systems, Technical University of Munich, Friedrich-Ludwig-Bauer-Str. p. 0 3, Garching, Germany<ref:2506. p. 010239#pg2>

Framework Components

The framework introduces a novel probabilistic Dynamical System-based VF for coarse guidance<ref:2506.10239#pg2> For tasks requiring precise guidance, we extend probabilistic positionbased trajectory fixtures with automation, allowing for seamless human interaction, geometry-awareness and optimal impedance gains<ref:2506. p. 010239#pg2> We introduce a novel variable stiffness formulation that models couplings between positional and orientational degrees of freedom (DoFs)<ref:2506. p. 110239#pg2> The framework outputs a Cartesian wrench wVF which is applied to the end effector<ref:2506. p. 310239#pg2>

Fixture Arbitration and Fusion

The core of our framework is a novel variable impedance control scheme as well as an optimal arbitration of all probabilistic fixture wrenches<ref:2506.10239#pg2> We form the fusion of different VFs in a principled way using an arbitration scheme (Section IV), thus solving the problem of selecting and switching between VFs in a flexible and adaptive manner based on learned and/or perceived uncertainties<ref:2506. p. 010239#pg2> This involves an extended arbitration scheme taking different types of VFs (Table I) and different underlying geometries into account (Section IV-A)<ref:2506. p. 110239#pg2> We propose a Product of Experts (PoE) [55] to arbitrate between different fixture wrenches expressed in the same manifold<ref:2506. p. 410239#pg2> The resulting wˆ with the optimally weighted wrenches of the individual fixtures is then applied to the robot through (3)<ref:2506. p. 410239#pg2>

Learning and Uncertainty Modeling

Different probabilistic models encoding assistive behaviors can be learned from demonstration data using Gaussian distributions<ref:2506.10239#pg2> We argue that fixture activation should be inversely proportional to uncertainty – both epistemic and aleatoric – such that strong guidance corresponds to low uncertainty levels<ref:2506. p. 310239#pg2> Kernel-based methods such as Gaussian Processes (GPs) [52] and KMPs [53] output epistemic uncertainty which increases with the distance between training data and new inputs<ref:2506. p. 410239#pg2> The KMP encodes a small covariance at s and a bigger covariance at t for the right motion policy<ref:2506. p. 910239#pg2>

Automation and Control Laws

For position-based VFs, we introduce an automation of this fixture in Section VI-D<ref:2506.10239#pg2> We propose a novel control law for DS-based Virtual Fixtures wVF,i = DVF,i (x˙ VF,i − x˙ ee) (28)<ref:2506. p. 610239#pg2> The resulting wrench is then added to the wrench of the impedance controller (4)<ref:2506. p. 710239#pg2>

Evaluation and Results

Our framework is implemented on three robotic systems in different automation levels as and interaction modes for evaluating task-specific metrics<ref:2506.10239#pg2> The evaluation with expert users suggests high usability of our method, paired with low workload for human operators<ref:2506. p. 010239#pg2> We find significant differences for measured forces and the NASA TLX score and a non-significant trend for the SUS metric in Table V<ref:2506. p. 910239#pg2> The overall small standard deviations in both experiments show that despite only aiming for coarse guidance, without external perturbations, the repeatability of the DS based fixture is remarkably high<ref:2506.10239#pg2>

Conclusion

In this work we introduce a unified, probabilistic Virtual Fixture framework providing different types of assistance to operators – particularly coarse, precise, and very precise guidance—where each type of fixture can either be manually defined or learned from human demonstrations<ref:2506.10239#pg2> We further introduce geometry-awareness in shared control through object-specific coordinate systems, including Cartesian, cylindrical and spherical frames<ref:2506. p. 410239#pg2> Combined with a novel variable impedance formulation – which robustly captures correlations between DoFs—our framework brings together the different fixture types using a product of experts approach, enabling a principled and uncertainty-aware fusion of assistance commands<ref:2506.10239#pg2> We have validated that the approach can be readily applied across diverse use cases, thanks to its ease of programming and flexibility with respect to fixture representations, input modalities, and uncertainty sources<ref:2506.10239#pg2> The evaluation with 6 expert users suggests high usability of our method, paired with low workload for human operators<ref:2506. p. 010239#pg2> While we demonstrated its use in factory automation and space scenarios, and in an initial experiment with expert users, we believe that the approach is also well-suited for medical and personal assistance robotics, which we plan to investigate in future work<ref:2506.10239#pg2> We furthermore envision developing methods to interactively modify position-based fixtures, allowing an operator to modify them adaptively based on novel task needs<ref:2506. p. 010239#pg2> To ease finding working hyperparameters for the geometric visual servoing fixture, methods for automatic tuning should be investigated<ref:2506. p. 010239#pg2> Finally, to enable the use of the proposed framework in high-latency applications such as on-orbit servicing, we plan to integrate it with controllers that can provide stable force feedback despite such delays<ref:2506. p. 0

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Uncertainty-aware Dynamical System based Virtual Fixtures allow for coarse guidance modeling the relation between assistive velocities x˙ DS and robot state xee probabilistically p(x˙ DSxee) = N (x˙ DSµDS, ΣDS) which enables flexible task automation while keeping the operator in control.

  2. Geometry-aware mixture of experts gating functions allow for the modeling of visual servoing fixtures as a multi-modal distribution p(xVSxee) = X M m=1 hˆm(xee, µm)pm(xVSxee) which incorporates uncertainties resulting from ambiguous or occluded detections.

  3. A novel variable stiffness formulation models couplings between positional and orientational degrees of freedom by creating a full stiffness matrix with nonzero coupling terms Ktr, allowing for a full stiffness matrix with nonzero coupling terms Ktr to be realized while respecting physical stability limits.

  4. Probabilistic policy arbitration via products of experts enables an optimal combination of different VFs [8] by transforming the wrench into a common representation in the cotangent space of R3 × S3, facilitating a principled and uncertainty-aware fusion of assistance commands.

  5. Automated position-based fixtures can be used for precise guidance by calculating a preferred direction along the trajectory, resulting in a wrench wVF,aut,i which is added to the impedance controller's wrench to achieve a more precise guidance than with DSbased fixtures in complex tasks.

Sources

Related papers