OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation

arXiv:2510.17150 · cs.RO · Submitted 2025-10-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation".

Dev: Here's the gist of the paper. OmniVIC presents a self-improving variable impedance controller enhanced by a vision language model (VLM) and vision-language in-context learning (ICL) to improve safety and adaptation in contact-rich robotic manipulation tasks Core Innovation OmniVIC interprets task context derived from images and natural language to generate adaptive impedance parameters for a VIC controller The core mechanism involves a self-improving Retrieval-Augmented Generation (RAG) and in-context learning (ICL) strategy RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks ICL leverages these retrieved examples and the prompt of the current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario This self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios Control Architecture The classical Cartesian VIC modulates the robot’s stiffness K and damping D to achieve compliant interactions with the environment The control law is typically defined in task space as Fext = Dx˜˙ + Kx˜, (1) The desired stiffness matrix K ∈ R6×6 is a diagonal matrix with variable terms as K=diag

Kx, Ky, Kz, ϵKx, ϵKy, ϵKz: The desired damping matrix D ∈ R6×6 is also diagonal matrix with variable terms as D=diag

Dx, Dy, Dz, ζDx, ζDy, ζDz: In OmniVIC, the VLM processes visual and language inputs to derive the task context which is then used to generate adaptive impedance parameters for the VIC controller Data and Experience Management RAG data are acquired by deploying any robot policy and applying two success criteria: a maximum allowable contact force Fmax and a time budget Tmax A trial is labeled failure if the measured force exceeds Fmax at any time or if the task time exceeds Tmax; otherwise, it is labeled success For each successful execution, the RAG database appends (i) the instruction T, (iii) the end–effector twist vt ∈ R6 expressed in the world frame, (iv) the gravity–compensated wrench wt =

Ft, τt: ∈ R6 expressed in the world frame The phase label is determined by querying a VLM with the instruction T, the current image It captured by an overview camera, and the world-frame twist vt and the world-frame wrench wt When a new successful record arrives, if the bank is full, it pools with records belonging to this instruction Retrieval and Adaptation Mechanism RAG retrieval identifies the most relevant prior experiences from the database to inform impedance parameter generation for the current request The retrieval process involves a four-step progressive retrieval process Step 3 computes four modality-specific similarity scores between the query and each candidate record from the pool of phase-consistent records These scores are force, torque, linear velocity, and angular velocity similarity using cosine similarity The top five highest-scoring candidates are kept as exemplars for in-context learning Experimental Validation The evaluation metrics include task success rates, force violations, and overall interaction safety In simulation experiments, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC) The ablation study comparing RAG-only versus RAG+ICL shows that RAG+ICL achieves a 1.59-fold higher task success rate than RAG-only Real-world results show that OmniVIC maintains interaction forces within a safe threshold of 30N during the task push the plate to the front of the stove The average success rate increases from 27% (baseline) to 61.4% (OmniVIC) OmniVIC significantly improves task success rates and reduces force violations compared to baselines with the average success rate increasing from 27% (baseline) to 61.

Title and authors: Rosa: So we're looking at this paper now called "OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation." Rosa, what's the main idea behind this whole setup in plain English?

Dev: Basically, they've taken a standard variable impedance controller, which is how robots feel like they have springiness and damping when they touch things, and they’ve added a vision language model to make that feeling smart enough to understand the situation.

Taro: So it means instead of having fixed stiffness and damping settings, the robot can change those numbers based on what it sees and what you tell it in words about the task.

Rosa: Exactly. The core innovation here is that OmniVIC interprets context from both images and language to generate those adaptive parameters for the controller, aiming to make contact-rich tasks safer and more adaptable.

Dev: And they achieve this by using a self-improving Retrieval Augmented Generation, or RAG, combined with in-context learning, ICL. This means it looks up past experiences from a memory bank and uses those examples along with the current task instructions to ask the vision language model for the right impedance settings.

Rosa: So it's like if you're teaching a robot to push something, it doesn't just use one fixed setting; it remembers how it succeeded on similar pushes before and asks the AI what to do now based on what it sees and what you say.

Taro: That self-improving RAG sounds interesting for generalization, but I wonder if that memory bank is always reliable when the real world gets messy.

Dev: They test this by labeling trials as success or failure based on two criteria: a maximum allowable contact force and a time budget, so they get data from successful runs to build that memory bank.

Rosa: And for each success, they save the instruction, the end-effector twist in world coordinates, and the gravity-compensated wrench in world coordinates so it has rich experience to learn from.

Title and authors: Taro: But what happens when the robot encounters something completely new—a situation it hasn't seen before, or a contact force that falls outside of those past successful experiences?

Dev: That's where the ICL part comes in. They condition the vision language model with those retrieved examples and the current task context to generate parameters dynamically without needing to retrain the model itself.

Rosa: Which is smart because it lets it adapt quickly to new scenarios on the fly, which is something traditional controllers struggle with when things get unstructured.

Taro: I'm curious about how robust this adaptation is when the visual input itself is ambiguous or if the language instruction doesn't perfectly describe what's happening physically.

Dev: The retrieval step involves a four-step progressive process, where it computes similarity scores between the current query and every record in that memory bank based on force, torque, linear velocity, and angular velocity similarity using cosine similarity.

Rosa: So it doesn't just grab the closest match; it scores them across different physical aspects of the experience to find the most relevant past behavior.

Taro: If you have five top candidates from that retrieval process, how does that translate into a decision for setting those stiffness and damping matrices?

Dev: Those top five highest-scoring candidates become exemplars for in-context learning, meaning the vision language model uses them as specific examples to generate the context-aware and adaptive impedance parameters for the current manipulation scenario.

Rosa: It sounds like they’re building a system that learns from its own history to make real-time decisions about how stiff or soft it should be when interacting with an object.

Taro: So, what are the actual results showing? Did this setup actually improve safety and adaptation in the experiments?

Dev: In simulation experiments, they saw the average success rate go from twenty-seven percent in the baseline to sixty-one point four percent with OmniVIC. Plus, real-world tests showed interaction forces staying within a safe threshold of thirty newtons during a task pushing a plate toward a stove.

Rosa: That jump from twenty-seven to sixty-one is quite significant for contact manipulation tasks, especially in complex scenarios.

Title and authors: Taro: And the ablation study that compared just RAG versus RAG plus ICL showed that the combination achieved a one point five nine fold higher task success rate than using just RAG alone.

Dev: That confirms that combining the retrieval mechanism with the in-context learning strategy is what really boosts its performance and adaptability across different scenarios.

Rosa: It means this paper suggests that having a memory bank of past experiences and a way to dynamically query an AI based on those experiences is a solid way to improve how robots handle physical interaction uncertainty.

Taro: What are the limitations they mentioned? Where does this system stop working, or what's the authors' warning about deploying it?

Dev: The paper notes that the RAG mechanism relies on deploying any robot policy and applying success criteria like a maximum allowable contact force Fmax and a time budget Tmax to label trials as success or failure.

Rosa: And another limitation is that the system is still tied to the VLM's ability to process the visual and language inputs correctly, so if those inputs are poorly represented, its context understanding might fail.

Taro: So it’s not a perfect solution for every single unpredictable physical interaction, but it gets much better at handling unseen situations because of that self-improving loop.

Dev: To wrap up on this OmniVIC paper, the system significantly improves task success rates and reduces force violations compared to baselines, with the average success rate increasing from twenty-seven percent in the baseline to sixty-one point four percent.

Rosa: It’s a big step forward for making physical interactions safer by giving the controller a way to learn and adapt from its own mistakes in real time.

Taro: The implication is that we can expect robots to handle a much wider variety of contact-rich tasks without needing explicit, hand-tuned parameters for every single one.

Dev: We're looking forward to seeing how this kind of self-improving RAG and ICL actually translates into deployment on more complex, unstructured physical environments.

The paper's summary: Rosa: So, to quickly recap, OmniVIC is this variable impedance controller that uses a vision language model to adapt its stiffness and damping settings based on what it sees and what you tell it in words for safer physical tasks.

Dev: Yeah, exactly. It’s moving away from fixed settings by letting the robot change how stiff or soft it is when it touches something, all guided by visual context and natural language instructions.

Taro: It uses a retrieval augmented generation strategy to look at past successful interactions and use those examples to inform what the current impedance parameters should be.

Rosa: That's the self-improving part. So, if it fails once, it learns from that failure by putting that experience into its memory bank for future reference.

Dev: Right. And when a new task comes up, it retrieves similar past experiences and uses those to query the vision language model to generate parameters for the current situation without needing a full retraining cycle.

Taro: It’s about local adaptation, using retrieved examples and in-context learning to make decisions right there during manipulation.

Rosa: The experiments show that this approach makes tasks much more successful. We saw the average success rate jump from twenty-seven percent in the baseline to sixty-one point four percent with OmniVIC.

Dev: And that jump is significant, especially when we look at real-world results, where interaction forces stayed within a safe threshold of thirty newtons during a task pushing a plate to the front of a stove.

Taro: That safety aspect is key for me. It shows the system can maintain control even when dealing with messy physical interactions that weren't explicitly programmed.

Rosa: So, the implication is that we’re getting robots that can handle contact-rich manipulation tasks with much higher success rates and less risk of damaging things or exceeding safe force limits.

Dev: It suggests a way to make robotic control more intuitive for complex physical scenarios by letting the AI learn and adapt from its own trial-and-error history in real time.

Taro: The question is how robust this self-improvement stays when the world throws something completely unexpected at it, or if the visual input itself is ambiguous, because that’s where we always test failure modes.

Rosa: That's what we need to look at next. How does this system handle those situations where the context it gets from images and language isn't perfect?

The paper's improvements: Tom: So we're looking at what they actually improved in this paper today: they’re talking about how OmniVIC goes beyond just using a single model to make things better.

Rosa: Yeah, it’s about making the entire system more versatile. They developed a universal variable impedance controller that can handle a much broader range of physical interactions than before.

Dev: That means we're not stuck tuning parameters for every single manipulation task; the AI should adapt its behavior to the context it perceives, whether that’s a new object or just a slightly different way of pushing something.

Taro: It specifically focuses on zero-shot adaptation, meaning it should work in scenarios it hasn't been explicitly trained for before.

Rosa: Exactly. They integrated a self-improving RAG mechanism to pull relevant past experiences from a memory bank, which informs the impedance adaptation based on similar tasks already done.

Dev: And then they layered in in-context learning, or ICL, which uses those retrieved examples along with the current task prompt to generate the right parameters dynamically during inference.

Taro: That’s smart because it avoids retraining the core model every time we change a scenario; it just queries the AI based on what’s relevant right now.

Rosa: It also explicitly incorporated real-time force and torque feedback into that VIC controller, ensuring those interaction forces stay within safe thresholds, which is a big safety addition.

Dev: That's crucial for me. For control engineers, having a clear safety case where the system guarantees forces stay under some limit during operation is what makes this practical for deployment.

Taro: So it’s not just about success rates; it’s about graceful recovery when the world misbehaves and keeping things physically safe while it adapts.

Rosa: Right. The big picture here is that we move toward a system where robots can be much more flexible and safer in unstructured, contact-rich environments without needing a massive, hand-engineered library of every possible interaction.

Dev: It shifts the focus from pre-programming every single physical scenario to giving the AI the tools to learn from its own mistakes during operation.

Taro: The question remains how reliable that memory bank is when the visual context or language input itself is noisy or completely irrelevant, because that’s where we always test failure modes.

Conclusion: Rosa: So to wrap up, OmniVIC is this self-improving variable impedance controller that uses vision language models to adapt stiffness and damping for safer physical tasks, and it showed a success rate increase from twenty-seven percent to sixty-one point four percent.

Dev: It’s about giving the robot a way to learn and adapt from its own trial history in real time while keeping forces within safe limits, which is what we need for reliable control loops.

Taro: I think it really shows how autonomy can handle messy physical interactions gracefully instead of just crashing or failing when things get unexpected.

Rosa: The main implication is that we’re getting robots that can do contact-rich manipulation tasks with much higher success rates and less risk of damaging things or exceeding safe force limits.

Dev: It suggests a way to make robotic control more intuitive for complex physical scenarios by letting the AI learn from its own trial-and-error history in operation.

Taro: The question is how reliable that memory bank is when the visual context or language input itself is noisy or completely irrelevant, because that’s where we always test failure modes.

Rosa: We're still looking at those limitations, especially for deployment outside of a perfectly controlled lab setting.

Dev: Right. Next time we look at this paper, I want to see how the authors address the latency when you have all that RAG and ICL happening in real-time loops.

Heng Zhang, Wei-Hsing Huang, Gokhan Solak, Arash Ajoudani

Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genoa, Italy. · Georgia Institute of Technology, Atlanta, USA.

cs.RO

Submitted: 2025-10-20

Updated: 2026-10-29

Comments: Accepted by IEEE Humanoids 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 98/100

The gist: OmniVIC presents a self-improving variable impedance controller enhanced by a vision language model (VLM) and vision-language in-context learning (ICL) to improve safety and adaptation in…

Terminology

Summary

OmniVIC presents a self-improving variable impedance controller enhanced by a vision language model (VLM) and vision-language in-context learning (ICL) to improve safety and adaptation in contact-rich robotic manipulation tasks

Core Innovation

OmniVIC interprets task context derived from images and natural language to generate adaptive impedance parameters for a VIC controller The core mechanism involves a self-improving Retrieval-Augmented Generation (RAG) and in-context learning (ICL) strategy RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks ICL leverages these retrieved examples and the prompt of the current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario This self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios

Control Architecture

The classical Cartesian VIC modulates the robot’s stiffness K and damping D to achieve compliant interactions with the environment The control law is typically defined in task space as Fext = Dx˜˙ + Kx˜, (1) The desired stiffness matrix K ∈ R6×6 is a diagonal matrix with variable terms as K=diag[Kx, Ky, Kz, ϵKx, ϵKy, ϵKz] The desired damping matrix D ∈ R6×6 is also diagonal matrix with variable terms as D=diag[Dx, Dy, Dz, ζDx, ζDy, ζDz] In OmniVIC, the VLM processes visual and language inputs to derive the task context which is then used to generate adaptive impedance parameters for the VIC controller

Data and Experience Management

RAG data are acquired by deploying any robot policy and applying two success criteria: a maximum allowable contact force Fmax and a time budget Tmax A trial is labeled failure if the measured force exceeds Fmax at any time or if the task time exceeds Tmax; otherwise, it is labeled success For each successful execution, the RAG database appends (i) the instruction T, (iii) the end–effector twist vt ∈ R6 expressed in the world frame, (iv) the gravity–compensated wrench wt = [Ft, τt] ∈ R6 expressed in the world frame The phase label is determined by querying a VLM with the instruction T, the current image It captured by an overview camera, and the world-frame twist vt and the world-frame wrench wt When a new successful record arrives, if the bank is full, it pools with records belonging to this instruction

Retrieval and Adaptation Mechanism

RAG retrieval identifies the most relevant prior experiences from the database to inform impedance parameter generation for the current request The retrieval process involves a four-step progressive retrieval process Step 3 computes four modality-specific similarity scores between the query and each candidate record from the pool of phase-consistent records These scores are force, torque, linear velocity, and angular velocity similarity using cosine similarity The top five highest-scoring candidates are kept as exemplars for in-context learning

Experimental Validation

The evaluation metrics include task success rates, force violations, and overall interaction safety In simulation experiments, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC) The ablation study comparing RAG-only versus RAG+ICL shows that RAG+ICL achieves a 1. p. 6

Where the paper says it

3) Ablation study: RAG vs. RAG+ICL (OmniVIC): To quantify the contribution of ICL on top of RAG, we compare two configurations under identical settings: RAG-only, which applies the retrieved parameters without ICL, and RAG+ICL (OmniVIC), which feeds the retrieved exemplars to the VLM for in-context learning to predict the impedance parameters.

59-fold higher task success rate than RAG-only Real-world results show that OmniVIC maintains interaction forces within a safe threshold of 30N during the task push the plate to the front of the stove The average success rate increases from 27% (baseline) to 61. p. 5
Where the paper says it

3 illustrates the critical advantage of our approach in maintaining safe interaction forces from one episode of the task push the plate to the front of the stove. Our OmniVIC dynamically maintain contact forces within a safe threshold of 30N.

4% (OmniVIC)

Conclusion

OmniVIC significantly improves task success rates and reduces force violations compared to baselines with the average success rate increasing from 27% (baseline) to 61.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Develop a universal variable impedance controller enhanced by a vision language model (VLM) that interprets task context derived from images and natural language to generate adaptive impedance parameters for safe physical interaction. This system can facilitate zero-shot adaptation to a broad spectrum of previously unseen scenarios, even when the inputs of the VLM are not well-represented.

  2. Implement a self-improving Retrieval Augmented Generation (RAG) mechanism that retrieves relevant prior experiences from a structured memory bank to inform impedance adaptation based on similar past tasks. This approach can locally enhance generalization and adaptability across diverse manipulation scenarios.

  3. Integrate an in-context learning (ICL) strategy that conditions the VLM on retrieved examples and the current task context to dynamically generate impedance parameters without retraining. This enables rapid adaptation to new tasks and environments while alleviating heavy computation during inference.

  4. Incorporate real-time force/torque feedback into the VIC controller to ensure interaction forces remain within safe thresholds, further enhancing safety during manipulation by ensuring that interaction forces remain within safe thresholds.

  5. The resulting AI system can perform contact-rich manipulation tasks with improved success rates, increasing the average success rate from 27% (baseline) to 61.4%, and demonstrate a significant reduction in force violations compared to baselines.

Abstract

We present OmniVIC, a universal variable impedance controller (VIC) enhanced by a vision language model (VLM), which improves safety and adaptation in any contact-rich robotic manipulation task to enhance safe physical interaction. Traditional VIC have shown advantages when the robot physically interacts with the environment, but lack generalization in unseen, complex, and unstructured safe interactions in universal task scenarios involving contact or uncertainty. To this end, the proposed OmniVIC interprets task context derived reasoning from images and natural language and generates adaptive impedance parameters for a VIC controller. Specifically, the core of OmniVIC is a self-improving Retrieval-Augmented Generation(RAG) and in-context learning (ICL), where RAG retrieves relevant prior experiences from a structured memory bank to inform the controller about similar past tasks, and ICL leverages these retrieved examples and the prompt of current task to query the VLM for generating context-aware and adaptive impedance parameters for the current manipulation scenario. Therefore, a self-improved RAG and ICL guarantee OmniVIC works in universal task scenarios. The impedance parameter regulation is further informed by real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms baselines on a suite of complex contact-rich tasks, both in simulation and on real-world robotic tasks, with improved success rates and reduced force violations. OmniVIC takes a step towards bridging high-level semantic reasoning and low-level compliant control, enabling safer and more generalizable manipulation. Overall, the average success rate increases from 27% (baseline) to 61.4% (OmniVIC).

Sources

Related papers