Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations

arXiv:2604.24661 · cs.RO · Submitted 2026-04-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations".

Paper summary: Rosa: So, we're talking about a paper called "Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations". Basically, it looks at how visual systems handle things getting messy in the real world, like weather changes or sensor noise.

Dev: Right. The main idea they are pushing is that current image restoration methods are too specialized and not robust enough when the corruption switches around. They ask two big questions: how does this restoration work when the corruption keeps changing, and does fixing the pixels actually keep what the downstream control system needs?

Taro: I'm interested in that second question. If we just fix pixels perfectly, are we accidentally feeding the downstream model junk information?

Rosa: Exactly. The paper shows that trying to faithfully reconstruct corrupted observations actually forces the internal representation to carry extra information about *what kind* of corruption it saw. They call this corruption-specific nuisance information, and that's bad for control.

Dev: It turns out, they use a concept called the Information Bottleneck perspective to argue that anchoring the representation to just the clean foreground is how you stop that contamination from happening. They prove that if you set a specific variable, they call it Z t equal to Ft exactly, you eliminate this corruption entanglement and still meet their information bottleneck task constraint of I z t Y greater than or equal beta where beta equals H Y minus eta >

Taro: That means the system doesn't need to remember the noise; it just needs to focus on what's actually important for the robot or whatever is controlling things. But how does this work when you don't know which corruption is active in real time?

Rosa: They introduce something called ACO-MoE, which stands for Agent-Centric Observations with Mixture of Experts. It’s this frozen adapter module that pairs a routed bank of restoration experts with a foreground mask branch.

Dev: That means it's plug and play, and it doesn't need any labels or oracle corruption information to work. The router just dynamically selects the right latent expert based on the input, which repairs both the degraded RGB and segments out that task-relevant foreground onto a black background.

Taro: So you’re not waiting for a label to tell you what's corrupted; the system figures it out on its own during inference time. That sounds really useful for real deployment, I like it.

Rosa: It does substantially mitigate the degradation that downstream visual-control backbones suffer across diverse corruption types. They tested this using a benchmark called VDCS, which generates Markov-switching physical degradations.

Dev: And the performance numbers are pretty strong when we look at their results on things like JPEG, Rain Noise, Color Hard, and Motion Blur. In the VDCS test, they show that ACO-MoE consistently recovers ninety-five point three percent of clean input performance even under those challenging Markov-switching corruptions <ref:2604.24661#pg2>.

Taro: So when you look at the comparison with other methods they tested—the frozen generic restorers—they found that ACO-MoE actually outperformed them by an average return increase of two hundred twenty point three percent, or about forty point three percent. That’s a solid lift in control performance.

Rosa: It confirms that input-adaptive expert capacity and having dual stream outputs are really important factors here. They also noted that removing the foreground supervision—just doing RGB repair without the mask—caused a much bigger drop, at fifty seven point one percent, compared to only removing the RGB repair, which was about ten point six percent.

Dev: That strongly suggests that getting that foreground composition right is the dominant factor in making this system robust under perturbation. The way it’s designed keeps perception separate from the perturbation itself during downstream backbone learning, which is key for stability.

Taro: It still leaves room for questions about what happens if you throw something at it that's totally unseen—out-of-distribution visual perturbations—that isn't covered in their pretraining data. The study mentions that severe unseen perturbations that destroy spatial structure or hide the task-relevant foreground regions remain a challenge.

Rosa: Exactly, and they are upfront about that limitation. They aren't claiming this is ready for deployment if you throw it at something completely new, even though it still maintains strong zero shot performance under the perturbations they did train on.

Dev: So when we look at the full picture of "Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations," what it really means is that you don't have to build a custom restoration pipeline for every possible noise scenario. You just put this plug and play module in front of your existing visual control system, and it adapts on the fly using only the corrupted input.

Taro: It shifts the focus from building perfect pixel reconstruction to building an observation adapter that preserves task relevance, which is a different kind of problem entirely.

Rosa: Right. And for anyone working on real-time loops or physical trials, knowing that this module can handle non stationary corruption switching without needing new training labels is a big deal for safety and reliability.

Dev: It does move the evaluation from just pixel fidelity to actual control success under changing conditions, which is what matters most when we are talking about closed loop systems.

Conclusion: Rosa: So, this paper is about an observation adapter called ACO-MoE that helps visual control systems handle messy things that keep changing, like sudden sensor noise or weather shifts.

Dev: Right, Rosa? It’s essentially a plug and play module designed to make sure the system keeps its focus even when the input image is corrupted in different ways.

Taro: I'm looking at the authors, and they focused on how to anchor that observation representation to things that are definitely clean, like what we call foreground masks.

Rosa: They show that if you try to fix every single pixel perfectly, the system starts memorizing the corruption itself instead of learning what it needs for control. That’s bad for a robot trying to navigate.

Dev: Because they prove mathematically that this setup makes sure the representation isn't contaminated by the noise information, which is a big deal for real-time stability. They show that setting a specific term to be exact solves this entanglement problem and keeps the system focused on what matters for control.

Taro: It sounds like they are tackling the hard problem of how an agent can keep its internal state clean when the input stream itself is constantly being altered by unknown factors.

Rosa: The results show that across different visual benchmarks, this method actually boosts performance with both model-free and model-based control systems. They got a high recovery rate on those difficult changing conditions they simulated.

Dev: That's what we need to see—a method that provides reliable control performance even when the environment is actively misbehaving. It moves the focus from just pixel accuracy to actual, successful task execution under stress.

Taro: But they did flag something important, and it’s where my interest lies: the study says that while this works great on the data they trained on, if you throw a totally new kind of visual corruption at it in real life, especially one that messes up the basic structure or hides what is supposed to be foreground—that's still a tough spot.

Rosa: So they aren't claiming this is a guaranteed fix for every single possible real-world disaster right now, even though its performance under known changing conditions looks really solid.

Dev: Exactly, it’s robust against the specific types of noise and switching they trained on, but we have to keep an eye on those severe unseen visual challenges. That's where the next round of testing needs to go.

City University of Hong Kong · Tsinghua University

cs.RO

Submitted: 2026-04-27

Updated: 2026-11-19

Comments: 34 pages, including supplementary material. Zhengru Fang and Yu Guo contributed equally. Source code is available at https://github.com/fangzr/aco-moe-code

Code: https://github.com/fangzr/aco-moe-code

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 98/100

The gist: Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations introduces Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), a frozen…

Key concepts

Agent-Centric Observations
This refers to how an agent perceives its environment. The method focuses on adapting these observations specifically for robust visual control by ensuring that the perception layer is resilient to dynamic visual disturbances like noise or weather changes.
Mixture-of-Experts (MoE)
Instead of one fixed restoration method, ACO-MoE uses a bank of specialized 'experts' and a router to decide which expert is best for repairing a specific corrupted observation. This allows the system to handle diverse types of visual corruption effectively.
Foreground Mask Branch
This component helps the system identify and utilize the clean parts of an image, specifically focusing on what is in front of objects. By anchoring the representation to this clean foreground, it prevents corruption from contaminating critical task-relevant information.
Information Bottleneck
This theoretical concept suggests that a good representation should retain only the information relevant to the task and discard irrelevant noise. The paper uses this idea to prove that anchoring representations to clean foreground effectively filters out corruption-specific nuisance information.

Terminology

Summary

Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations introduces Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), a frozen, plug-and-play observation adapter that combines a routed bank of restoration experts with a foreground-mask branch to improve downstream visual control under dynamic perturbations. p. 2

Where the paper says it

To address these challenges, we introduce the Visual Degraded Control Suite (VDCS) to generate Markovswitching corruption sequences (Fig. 1a), and present Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), a frozen, plug-and-play observation adapter that pairs a routed bank of restoration experts with a foreground-mask branch.

The core finding is that faithfully reconstructing corrupted observations forces the latent state to encode corruption-specific nuisance information, which ACO-MoE counters by anchoring the representation to the clean foreground. p. 0
Where the paper says it

We further identify a fundamental failure mode of reconstruction-based representations: faithfully reconstructing corrupted observations forces the latent state to encode corruption-specific nuisance information, thereby contaminating downstream models. From an information-bottleneck perspective, anchoring the representation to the clean foreground eliminates this contamination.

The Problem Context

Real-world visual systems face time-varying perturbations, including weather, sensor noise, compression artifacts, and background distractions. Existing image restoration methods are typically designed for fixed corruption types and optimized for pixel-level fidelity, leaving open two questions: how restoration behaves under non-stationary corruption switching, and whether pixel-level fidelity preserves the task-relevant information needed by downstream models. p. 0

Where the paper says it

Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations Zhengru Fang1,2,∗ , Yu Guo1,∗ , Fei Liu1 , Yuang Zhang2 , Yihang Tao1 , Senkang Hu1 , Wenbo Ding2 , Yuguang Fang1 1 City University of Hong Kong, 2 Tsinghua University Real-world visual systems face time-varying perturbations, including weather, sensor noise, compression artifacts, and background distractions. Existing image restoration methods are typically designed for fixed corruption types and optimized for pixel-level fidelity, leaving open two questions: how restoration behaves under non-stationary corruption switching, and whether pixel-level fidelity preserves the task-relevant information needed by downstream models.

These limitations become most consequential when the downstream task is closed-loop visual control, where every restored frame feeds directly into a learned policy or world model. p. 0
Where the paper says it

Second, restoration is optimized for pixel-level fidelity against a clean target, with no notion of which pixels actually matter to the downstream system that consumes the restored output. These limitations become most consequential when the downstream task is closed-loop visual control, where every restored frame feeds directly into a learned policy or world model.

The ACO-MoE Mechanism

ACO-MoE is proposed as a frozen, plug-and-play observation adapter that pairs a routed bank of restoration experts with a foreground-mask branch. It is pretrained entirely offline on synthetic rendered data with automatically generated degradation pairs and simulation-derived foreground masks, requiring no manual annotation. p. 0

Where the paper says it

Motivated by this analysis, we propose Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), a frozen, plug-and-play observation adapter that combines a routed bank of restoration experts with a foreground-mask branch. ACO-MoE is pretrained entirely offline on synthetic rendered data with automatically generated degradation pairs and simulation-derived foreground masks, requiring no manual annotation.

At inference time, it takes only corrupted RGB as input without corruption labels, clean reference frames, or foreground masks. p. 0
Where the paper says it

ACO-MoE is pretrained entirely offline on synthetic rendered data with automatically generated degradation pairs and simulation-derived foreground masks, requiring no manual annotation. At inference time, it takes only corrupted RGB as input without corruption labels, clean reference frames, or foreground masks.

The module consists of a shared encoder, a compact router, and Ne repair experts. p. 4
Where the paper says it

During perturbation-time evaluation, ACO-MoE receives only corrupted RGB frames and the downstream backbone architecture and learning objective are left unchanged. ACO-MoE consists of a shared encoder, a compact router, and Ne repair experts.

Theoretical Justification

The authors formally show that reconstruction-based representations inevitably entangle corruption information in their latent state. Proposition B.1 demonstrates that accurate reconstruction forces the latent to retain corruption information, regardless of the encoder architecture. p. 0

Where the paper says it

To study this setting, we introduce the Visual Degraded Control Suite (VDCS), a benchmark that injects Markov-switching physical degradations into rendered scenes. We further identify a fundamental failure mode of reconstruction-based representations: faithfully reconstructing corrupted observations forces the latent state to encode corruption-specific nuisance information, thereby contaminating downstream models.

The Information-bottleneck perspective suggests anchoring the representation to the clean foreground eliminates this contamination. p. 0
Where the paper says it

We further identify a fundamental failure mode of reconstruction-based representations: faithfully reconstructing corrupted observations forces the latent state to encode corruption-specific nuisance information, thereby contaminating downstream models. From an information-bottleneck perspective, anchoring the representation to the clean foreground eliminates this contamination.

Corollary B.2 proves that setting Zt = Ft exactly eliminates corruption entanglement and meets the information-bottleneck task constraint I(Zt;Y) ≥ β with β = H(Y) − η. p. 2
Where the paper says it

(2) We formally show that reconstruction-based representations inevitably entangle corruption information in their latent state, and prove that foreground anchoring is a valid information-bottleneck surrogate that exactly eliminates this entanglement. (3) We propose ACO-MoE, a dual-stream Mixture-of-Experts architecture that achieves task-aware restoration by simultaneously repairing corrupted pixels and extracting task-relevant foreground, requiring no oracle corruption labels at training or inference.

Empirical Performance

Across VDCS, DMC-GB, and RoboSuite, ACO-MoE consistently improves downstream control with both model-free and model-based backbones. In the VDCS benchmark, ACO-MoE consistently recovers 95. p. 0

Where the paper says it

At inference time, it takes only corrupted RGB as input without corruption labels, clean reference frames, or foreground masks. Across VDCS, DMC-GB, and RoboSuite, ACO-MoE consistently improves downstream control with both model-free and model-based backbones, recovering 95.

3% of clean-input performance under challenging Markov-switching corruptions. p. 0
Where the paper says it

Across VDCS, DMC-GB, and RoboSuite, ACO-MoE consistently improves downstream control with both model-free and model-based backbones, recovering 95. 3% of clean-input performance under challenging Markov-switching corruptions.

The results show that ACO-MoE achieves state-of-the-art performance with both model-free and model-based downstream backbones across VDCS, DMC-GB (Color/Video Hard), and RoboSuite. p. 2
Where the paper says it

(4) Across VDCS, DMC-GB (Color/Video Hard), and RoboSuite, ACO-MoE achieves state-of-the-art performance with both model-free and model-based downstream backbones. Additional comparisons with frozen generic restorers further show that the gains arise from task-aware agent-centric adaptation rather than pixel restoration alone.

Ablation Insights

Ablations confirm the importance of input-adaptive expert capacity, dual-stream outputs, and frozen training. Removing RGB repair (MaskOnly) causes a smaller drop (10. p. 8

Where the paper says it

9%, suggesting that input-adaptive expert selection is more effective than adding comparable capacity to one decoder. Removing RGB repair (MaskOnly) causes a smaller drop (10.

6%), while removing foreground supervision (RGB-Only) is much more harmful (57. p. 8
Where the paper says it

Removing RGB repair (MaskOnly) causes a smaller drop (10. 6%), while removing foreground supervision (RGB-Only) is much more harmful (57.

1%), indicating that foreground composition is the dominant factor. p. 8
Where the paper says it

6%), while removing foreground supervision (RGB-Only) is much more harmful (57. 1%), indicating that foreground composition is the dominant factor and RGB repair provides complementary gains under pixel-level degradations.

The comparison with frozen generic restorers shows that ACO-MoE outperforms them by +220. p. 8
Where the paper says it

Appendix F. 6 further shows that ACO-MoE outperforms the strongest frozen generic restorer, AdaIR, by +220.

3 average return (+40. p. 8
Where the paper says it

6 further shows that ACO-MoE outperforms the strongest frozen generic restorer, AdaIR, by +220. 3 average return (+40.

3%).

Generalization and Limitations

ACO-MoE maintains strong zero-shot performance under unseen OOD visual perturbations excluded from adapter pretraining. However, the study notes that severe unseen perturbations that destroy spatial structure or occlude task-relevant foreground regions remain challenging. p. 0

Where the paper says it

3% of clean-input performance under challenging Markov-switching corruptions. It also generalizes zero-shot to unseen visual perturbations excluded from adapter pretraining.

The method is not inherently tied to simulator-only labels, although validating such a real-world data-generation pipeline on physical robotic platforms remains future work.

Future Directions

Future work includes open-set routing, real-world pseudo masks/pairs (Appendix D.2), physical-robot validation, failure monitoring, and conservative fallback mechanisms. p. 0

Where the paper says it

To study this setting, we introduce the Visual Degraded Control Suite (VDCS), a benchmark that injects Markov-switching physical degradations into rendered scenes. We further identify a fundamental failure mode of reconstruction-based representations: faithfully reconstructing corrupted observations forces the latent state to encode corruption-specific nuisance information, thereby contaminating downstream models.

The paper concludes that simulation robustness should not be viewed as a deployment-ready safety guarantee. p. 1
Where the paper says it

Figure 1: Dynamic perturbation examples and performance degradation on DMControl [55]. Left: Diverse perturbations including 7 VDCS physical degradations (red border) and 2 DMC-GB background distractions (green border); VDCS types switch dynamically via Markov chains during deployment.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Agent-Centric Observation Adaptation for Robust Visual Control under Dynamic Perturbations can be improved by decoupling perception from perturbation via ACO-MoE, which is a frozen, plug-and-play observation adapter that combines a routed bank of restoration experts with a foreground-mask branch. This allows the system to select an input-adaptive latent expert without requiring any oracle corruption label, ensuring the downstream backbone remains unchanged during adaptation.

  2. The improved system can achieve state-of-the-art performance under dynamic visual control by transforming corrupted observations toward a canonical agent-centric form: it repairs foreground pixels when they are degraded and removes background variation through foreground composition. This is achieved by predicting an RGB residual and a foreground mask, then composing an agent-centric observation using the formula x˜t = oˆt ⊙ mt + b ⊙ (1 − mt).

  3. The system gains robustness against non-stationary corruption switching because it anchors the representation to the clean foreground, eliminating contamination: anchoring the representation to the clean foreground eliminates this contamination. This approach is formally proven to be an information-bottleneck surrogate that exactly eliminates this entanglement, which means accurate reconstruction does not force latent states to encode corruption-specific nuisance information.

  4. The system can generalize zero-shot to unseen visual perturbations by pretraining ACO-MoE offline on synthetic data and maintaining a strong capacity: ACO-MoE also maintains strong zero-shot performance under unseen OOD visual perturbations excluded from adapter pretraining. This enables the controller to perform well even when encountering novel corruptions like defocus blur or frost, as seen in Table 13.

  5. The system can be made more efficient at inference by using a frozen, hard top-1 routing mechanism: At inference time, it takes only corrupted RGB as input without corruption labels, clean reference frames, or foreground masks, and utilizing hard top-1 routing for efficiency. This results in a total latency of 22.90 +/- 0.93 ms/step for the preprocessing module on an RTX 5090.

Abstract

Visual control maps visual streams to actions in reinforcement learning and vision-language-action (VLA) policies. Changing weather, noise, blur, and compression can obscure different task cues within one episode. Existing control benchmarks mainly test scene distractions, while generic restorers optimize whole-image fidelity. We introduce the Visual Degraded Control Suite (VDCS) to evaluate seven degradations under Markov switching. We propose Agent-Centric Observations with Mixture-of-Experts (ACO-MoE), which routes each frame to a repair expert, restores RGB content, and composes task-relevant foregrounds. On eight VDCS tasks, corruption lowers DreamerV3's mean return from 769 to 562. ACO-MoE raises it to 730, recovering 81.2% of the lost return. It also improves returns on DMC-GB and RoboSuite and retains 70% to 97% of clean-input mean return under six unseen perturbation families. We train ACO-MoE on robot demonstrations and deploy it with a 3B-parameter VLA policy on a Piper arm for cube placement in blue and white boxes and cup placement on a coaster. On the blue-box and cup tasks, motion blur and physical low light with added noise reduce success from 100% to at most 10%. ACO-MoE raises success to 75% to 95% over 20 trials per task and condition.

Sources

Related papers