Tell Robot What Not to Do: A Negation Understanding Perspective

arXiv:2610.11952 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Tell Robot What Not to Do".

Rosa: The gist The proposed NegaAlign framework enables vision-language-action models (VLAs) to follow negated instructions by extending pretrained VLAs through image–language supervision alone,

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So we're talking about this paper now called "Tell Robot What Not to Do: A Negation Understanding Perspective," and it really gets at the core problem of making robots follow instructions when you don't just want them to do one thing, but also explicitly avoid something else.

Dev: Exactly. It tackles how vision-language-action models, or VLAs, can handle those negative constraints—the stuff like "do this task, except for that part"—without needing tons of new video data for every single exclusion scenario.

Taro: It’s about the difficulty VLMs have with negation itself. The paper points out that existing models struggle to distinguish what should be done from what should be excluded when you use negative language, which is a big deal for real-world robot interaction.

Rosa: That's right, and this paper proposes NegaAlign as a way to fix that by using only image-language supervision. They introduce these Negation Transformation Layers into the vision-language backbone to reshape how the model understands those instructions internally.

Dev: The mechanism they use involves projecting the instruction tokens into a space where they can be pooled into content and polarity embeddings, then transforming that content code into a satisfied affirmative destination while calculating a gate for the polarity branch.

Taro: So it's essentially trying to create an internal representation that maps the negated instruction to a satisfied positive action plan, which helps ground those excluded constraints in the visual content.

Rosa: And they pair that up with something called Teacher-Guided Visual Alignment, which is designed to align those revised language representations with the actual visual tokens by calculating a cosine distance between the negated student and a satisfying teacher representation.

Dev: This alignment process uses an adaptive threshold to select instruction-relevant visual tokens based on how much contrast there is between the student and teacher representations, then it uses that to create an equivalence loss to align them with what the teacher actually did.

Taro: It sounds like they are using existing affirmative data, like successful instructions, alongside hand-crafted negated instructions to build this supervision structure without needing any new negated action trajectories.

Rosa: That's the key part for me—the training recipe only updates the inserted layers while keeping all the pretrained parameters frozen, including the action generator. The objective function is set up to revise those intermediate instruction representations and push that negation semantics onto the visual tokens.

Dev: They optimize a total loss that combines alignment loss with some semantic shift terms, specifically summing L l equiv and L l sem over all layers containing these Negation Transformation Layers.

Taro: This makes sense because it means they are not retraining the whole massive model; they are just fine-tuning the parts of the VLA that handle this specific type of instruction revision, which is much more parameter-efficient.

Title and authors: Rosa: The results show consistent improvements across several models, like pi0 point 5, where they see a negated-instruction success rate jump from two point six zero percent up to eighty-eight point four five percent on the NegaBench benchmark and also on real-world tasks.

Dev: And what's important is that this success happens while keeping performance on affirmative instructions pretty stable, showing they aren't compromising the original capabilities of the VLA.

Taro: It really demonstrates that you can transfer this understanding of negation into physical robot behavior without needing to collect specific negated action data for every task.

Rosa: In real-world tests with five xArm6 manipulation tasks, the affirmative-only policy hits about ninety-one point six percent success, but it only manages twelve point four percent on negated trials.

Dev: With NegaAlign applied, that average negated success rate climbs up to eighty-eight point eight percent, which is a gain of about seventy-six percentage points over the base performance.

Taro: That gain spans different types of constraints—it's not just one type of exclusion; it works on object constraints, attribute constraints, and even outcome constraints across various tasks.

Rosa: It shows that this approach is broadly applicable to physical manipulation problems where you have to respect explicit negative conditions in the environment.

Dev: The limitation they mentioned is that they are relying on existing pretrained VLAs, which means the system might not cover every possible weird or complex negated scenario out there.

Taro: And they also pointed out that performance could degrade if the negated instructions are really ambiguous or highly complex, which suggests future work will need to focus on those edge cases.

Rosa: So to wrap this up, NegaAlign is a parameter-efficient framework using Negation Transformation Layers and Teacher-Guided Visual Alignment to give VLAs a way to follow negated instructions using just image-language supervision.

Dev: It’s essentially teaching the model how to revise its internal representations of an instruction when it sees something like "give me a tool, except the knife," by aligning that revision with visual information from satisfying teacher demonstrations.

Taro: The implication is that we can start deploying these VLAs in more complex physical settings where users need to specify what *not* to do, not just what they *should* do.

Rosa: And for the listener driving or cooking, this means robots could finally understand nuanced requests like "make sure you don't use that sharp knife" instead of just blindly grabbing whatever is closest.

Dev: It moves the focus from purely following commands to respecting constraints, which is a much more realistic interaction in physical space.

Taro: We’re still seeing things where the model might struggle if the negation is really tricky, so they need to keep looking at how that internal grounding works under pressure.

Rosa: So that's what we have on "Tell Robot What Not to Do: A Negation Understanding Perspective," and it shows a solid path toward making robot instructions more nuanced.

The paper's summary: Rosa: So, essentially, this paper says that current vision-language models just don't naturally understand what to *exclude* when you give them an instruction, like "get me a wrench but leave the hammer behind."

Dev: Right. It builds this framework called NegaAlign which uses existing language and image data to teach the model how to handle those negative constraints without needing new video footage for every exclusion scenario.

Rosa: They’re introducing these Negation Transformation Layers that basically reshape the instruction inside the model so it sees a satisfied positive version of what you want, while also calculating a way to track what it needs to avoid.

Dev: And they pair that with something called Teacher-Guided Visual Alignment, which is like using successful examples—the teacher—to figure out exactly which visual parts matter when the instruction has been modified for negation.

Rosa: The big point here is that they don't need any specific action data showing what to do and what not to do; they build this supervision just from existing affirmative instructions and the negated instructions themselves.

Dev: And the training method is smart because it only tweaks those new layers, keeping the rest of the massive vision-language model totally frozen, which saves a ton of time and computational power.

Rosa: When they test it on benchmarks like NegaBench, they see success rates for negated instructions jump from really low numbers up to eighty-eight percent without hurting the model’s ability to follow normal commands.

Dev: That's a huge gain in performance, especially because that improvement comes specifically from learning the concept of negation itself, not just learning more general manipulation skills.

Rosa: It shows that you can transfer this understanding of what *not* to do into real-world robot behavior without having to collect tons of specialized data for every single exclusion task.

Dev: But the authors themselves are honest about the limitations, saying this method relies on existing pretrained models, so it might not handle every weird or extremely complicated way a human could phrase a negative request yet.

Rosa: That’s the caveat. They think future work will need to focus on those really tricky instructions where even this framework might start to struggle with ambiguity.

Dev: So what this means for us in the robotics world is that we can start deploying these VLAs in physical settings where users need to specify what *not* to do, not just blindly follow positive commands.

Rosa: It moves the conversation from just executing a task to respecting specific boundaries in a physical space.

Dev: And if you're thinking about setting up your robot's control loop, this kind of semantic revision inside the representation could actually help manage latency and failure modes when those negative constraints come into play during real-time operation.

The paper's improvements: Taro: So, we’re looking at how they suggest making these NegaAlign systems even better by focusing on those specific layers they put in there.

Rosa: Right, they suggest that because the instruction representations change depending on which part of the vision-language backbone you’re looking at, you should insert those transformation layers into different middle layers for different kinds of instructions.

Dev: That makes sense from a loop perspective; if you have a complex task with many sub-steps, maybe one layer handles object constraints and another handles attribute constraints.

Rosa: Exactly. They found that putting the transformation layers in the right spot gives better performance for both following positive commands and handling those tricky negative ones simultaneously.

Dev: And what’s interesting is they showed that removing the Teacher-Guided Visual Alignment component makes a huge difference; without that alignment, the negated success rate drops way down to about twenty percent.

Rosa: That means you can't just rely on the internal language revision alone; you need that visual grounding from the teacher representations to make sure it actually translates into correct physical action.

Dev: It confirms my concern about latency; if the alignment loss isn't working well, the revised semantics might not propagate fast enough to keep up with real-time control demands.

Rosa: So, for someone just listening who’s driving or cooking, what does this mean practically? It means robots that can handle those complex "do this but don't touch that" requests are moving from lab experiments toward being useful in messy, real-world kitchens or workshops.

Dev: It implies we can design control systems where the VLA has a built-in mechanism to revise its plan on the fly based on explicit negative rules, rather than having a whole new model for every possible constraint.

Rosa: The authors did mention that this still isn't perfect; they noted that if an instruction is really ambiguous or super complicated, the system’s performance might actually start to degrade.

Dev: That’s the caveat we need to watch out for when deploying these systems; ambiguity in the input language could lead to unpredictable behavior in a physical setting.

Rosa: So, while NegaAlign gives us a massive boost in success rates, we still have that edge case of really hard-to-understand negative instructions that they haven't fully solved yet.

Dev: We’ll see how robust those inserted layers are when the visual evidence is conflicting or overwhelming during high-speed movements.

Conclusion: Rosa: So to wrap up, this paper on "Tell Robot What Not to Do: A Negation Understanding Perspective" shows that we can give vision-language models a way to follow negative instructions by just adding a few layers and using existing image data for supervision.

Dev: It’s basically a parameter-efficient trick, keeping the main model frozen while only fine-tuning these new parts so it understands negation semantics without needing tons of new video footage.

Rosa: The big takeaway is that we're getting much better at handling explicit exclusions in physical manipulation, like telling a robot to grab something but specifically avoiding a certain object or feature.

Dev: I still see the performance ceiling being tied to how well those intermediate representations align with the teacher data; if that alignment breaks down, the entire negated action plan could get messy and unpredictable in real-time control.

Taro: From an autonomy standpoint, this means we can start building systems that respect complex user constraints in dynamic environments without needing a separate training pipeline just for negatives.

Rosa: It’s a step toward making robot interactions more nuanced, moving past simple "do this" commands to detailed "do this, but not that."

Dev: And I'm still watching how the loop rate holds up when these semantic shifts are happening; we need to make sure the latency introduced by these transformation layers doesn't cause instability in the physical control system.

Taro: The limitation they flagged is that if a negative instruction is really weird or ambiguous, the current setup might just fail to generalize properly, which means we still have a lot of work ahead in making it truly robust for every possible human request.

Rosa: So "Tell Robot What Not to Do: A Negation Understanding Perspective" gives us a solid foundation for incorporating negation into VLA behavior using only image-language supervision.

Dev: It's definitely an interesting piece of work because it shows how much we can learn about complex instruction following without having to collect all that expensive, specific negative action data.

Taro: We’ll keep watching how these models handle those tricky edge cases where the negation is really subtle or contradictory, because that’s where the real autonomy test will be.

South China University of Technology · Harbin Institute of Technology

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The proposed NegaAlign framework enables vision-language-action models (VLAs) to follow negated instructions by extending pretrained VLAs through image–language supervision alone,

Key concepts

Negation Transformation Layers (NTLs)
These layers are inserted into the vision-language backbone to modify intermediate instruction representations. They project tokens into content and polarity embeddings, then use a Trans. Layer to transform the content code based on whether the instruction is satisfied or negated, allowing for a semantic shift in the representation.
Teacher-Guided Visual Alignment (TGVA)
TGVA aligns visual tokens with revised language representations by comparing them against teacher representations derived from satisfying instructions. It computes a cosine distance and selects instruction-relevant visual tokens based on this contrast to ensure the model focuses on the correct visual information for the negated task.
Negation Supervision Construction
Supervision is created using existing affirmative demonstrations and hand-crafted negated instructions, avoiding the need for new negative action data. The constraint is formalized as a relationship between a negated instruction, satisfying instructions, and excluded behaviors to guide the adaptation process.
Parameter-Efficient Framework
NegaAlign is designed to be parameter-efficient by only updating the newly inserted layers (NTLs) during training. All other pretrained parameters, including the action generator, remain frozen. This allows for effective learning of negation semantics without retraining the entire large model.

Terminology

Summary

The gist The proposed NegaAlign framework enables vision-language-action models (VLAs) to follow negated instructions by extending pretrained VLAs through image–language supervision alone, achieving significant improvements in negated instruction following success rates while preserving affirmative capabilities

How it works

The core of NegaAlign involves a parameter-efficient, plug-and-play framework that extends pretrained VLAs to follow negated instructions through image–language supervision alone. Specifically, the framework introduces two main components: Negation Transformation Layers and Teacher-Guided Visual Alignment.

  1. Negation Transformation Layers (NTLs) are introduced into selected layers of the vision–language backbone to reshape intermediate instruction representations. These layers first project instruction tokens into a compact space to pool them into content and polarity embeddings, and then a Trans. Layer transforms the content code into a satisfied affirmative destination r l, while the polarity branch produces a gate g l = Gate(p l). The resulting semantic shift is modulated by each token representation to obtain ∆h l t, which is then added to the original hidden state h l t to get h̃ l t.

  2. Teacher-Guided Visual Alignment (TGVA) is designed to align the instruction-relevant visual tokens with the revised language representations. TGVA extracts visual keys and values from attention blocks and computes a cosine distance d l b,i between the negated student and satisfying teacher representations. It then selects instruction-relevant visual tokens based on an adaptive threshold to form Ψ l b, where weights w l b,i are proportional to the teacher contrast. The equivalence loss L l equiv aligns the negated student visual Key and Value with Oa T on the selected tokens.

Negation Supervision Construction

The framework constructs supervision from existing affirmative demonstrations and hand-crafted negated instructions without additional negation-specific action data. This is achieved by formalizing the negation constraint as a relational supervision R = (l n, A, l e), where l n is a negated instruction, A = the set of affirmative instructions satisfying its constraint, and l e describes the excluded behavior. During adaptation, four language conditions are evaluated using existing affirmative data: I a S (affirmative student), I n S (negated student), I a T (satisfying teacher), and I e T (excluded teacher).

Training Recipe

The training phase updates only the inserted layers, keeping all pretrained parameters frozen, including the action generator. The training objective is defined as Ltotal = Σ L l equiv + αL l sem + αL l gate over all layers with NTLs. This objective is optimized to revise the intermediate instruction representations of negated instructions and propagate the negation semantics to the instruction-relevant visual tokens. The learning rate schedule is defined in Eq. 16, with a warmup period W and total steps T.

Evaluation and Results

The capability of reliable instruction following is evaluated using NegaBench, a simulation benchmark spanning 10 scenarios across five domains for systematically evaluating manipulation under negated constraints. Experiments across GR00T, π0, and π0.5 demonstrate consistent improvements in negated instruction following. For instance, NegaAlign increases the negated-instruction success rate of π0.5 from 2.60% to 88.45% on NegaBench and from 12.4% to 88.8% on real-world tasks, while retaining performance on affirmative instructions. The results show that NegaAlign reaches a negated success rate of 57.75%, 63.30%, and 88.45% on GR00T, π0, and π0.5 without using any negated action trajectories.

Ablation Studies

Ablation studies confirm the necessity of both components of NegaAlign. Removing TGVA reduces negated success from 88.45% to 21.35%, while affirmative success remains relatively high at 91.27%. Inserting NTLs into middle layers yields the best performance in both affirmative and negated instruction following, indicating that the instruction-dependent visual representations have different characteristics in different layers. The NTL prevents revision from compromising original capabilities of policy while TGVA propagates revised semantics through action-relevant visual tokens.

Real-World Performance

In real-world robot experiments on five xArm6 manipulation tasks, the affirmative-only policy achieves a high affirmative success rate of 91.6%, but succeeds in only 12.4% of negated trials. With NegaAlign, the average negated success rate rises to 88.8%, a gain of 76.4 percentage points, while affirmative performance remains essentially unchanged at 90.8%. This demonstrates that our approach effectively transfers negation understanding into real-world robot behavior without compromising the original manipulation capabilities of policy. The gain spans constraints on objects, attributes, and outcomes rather than being driven by one task family.

Conclusion

NegaAlign is a parameter-efficient framework that combines Negation Transformation Layers and Teacher-Guided Visual Alignment to enable negated instruction following in pretrained VLAs without additional negated action trajectories. We also introduce NegaBench to evaluate this capability in robotic manipulation. Simulation and real-world experiments demonstrate substantial improvements in negated instruction following while preserving affirmative performance. The limitations include the reliance on existing pretrained VLAs, which may not cover all possible negated scenarios, and the potential for performance degradation in highly complex or ambiguous negated instructions that leads our direction of future work.

AI Use Statement

In this work, we used generative AI tools to assist with language editing, figure presentation, checking mathematical notation, and checking the consistency between the manuscript and existing code. We did not use generative AI to generate research data, formulate or prove mathematical claims, design experiments, or analyze or interpret experimental results. The authors reviewed all AI-assisted text and suggestions and verified their accuracy against the original code, experimental records, and relevant literature. The authors take full responsibility for the final text, claims, and research artifacts in this work.

References

Bhuvan Aggarwal et al. Seeing what’s not there: negation understanding needs more than training. In International Conference on Learning Representations, volume 2026, pp. 69200–69217, 2026

Kumail Alhamoud et al. Vision-language models do not understand negation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29612–29622, IEEE, 2025

Shuai Bai et al. Finevla: Fine-grained instruction alignment for steerable vision-language-action policies, 2026

Johan Bjorck et al. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025

Nhat-Tan Bui et al. Nein: Telling what you don’t want. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3132–3140, June 2025

Tianxing Chen et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In Forty-third International Conference on Machine Learning, 2026a

Zixing Chen et al. Hazardarena: Evaluating semantic safety in vision-languageaction models. arXiv preprint arXiv:2604.12447, 2026b

Yuchen Cui et al. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, pp. 93–101, 2023

Radina Dobreva and Frank Keller. Investigating negation in pre-trained vision-and-language models. In Proceedings of the fourth blackboxnlp workshop on analyzing and interpreting neural networks for nlp, pp. 350–362, 2021

Xuan Dong et al. When does language matter? multilingual instructions reveal step-wise language sensitivity in vision-language-action models. In Maria Liakata et al., Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.

Improvements for AI systems

  1. Bold header: NegaAlign Framework Implementation

NegaAlign enables vision-language-action models (VLAs) to follow negated instructions by updating only a few parameters, as it updates only the inserted layers, keeping all pretrained parameters frozen, including the action generator. This allows for parameter-efficient adaptation without costly negation-specific embodied data.

  1. Bold header: Negation Transformation Layers (NTLs)

The NTLs revise intermediate instruction representations to distinguish negated instructions from affirmative ones by transforming them into a satisfied affirmative destination, thereby enabling the VLA to understand negation semantics.

  1. Bold header: Teacher-Guided Visual Alignment (TGVA)

TGVA aligns the revised language representations with visual tokens by computing a loss that minimizes the weighted distance between the negated student and the satisfying teacher to ensure that semantic revision is propagated to instruction-relevant visual tokens.

  1. Bold header: NegaBench Evaluation

NegaBench provides a systematic evaluation environment, spanning 10 scenarios across five domains (object exclusion, attribute exclusion, relative comparison, goallocation exclusion, and combinations of multiple constraints), allowing for the systematic evaluating manipulation under negated constraints.

  1. Bold header: Real-World Constraint-Aware Manipulation

The improved system can reliably execute actions under explicit exclusions in physical environments; for instance, it can successfully perform tasks like Hand me a piece of tableware, but not the knife or Put a bowl into the microwave, except the stainless steel bowl.

Sources

Related papers