CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts

arXiv:2607.03693 · cs.RO · Submitted 2026-07-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts".

Dev: The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts," and the main idea is that robots in the real world have all sorts of different sensors, some might fail, and some robots just don't have certain sensors built in.

Dev: Exactly. The core problem they're tackling is how to make a unified policy that can use those extra sensory inputs only when they're there, without breaking when they aren't available at deployment time.

Taro: So the authors are saying that existing VLA policies couple action generation to a fixed set of sensors through this shared dense computation path, which makes them brittle if those sensors go missing and limits their ability to handle different tasks well.

Rosa: That’s right. They propose CoRE-VLA as a way to fix that by formulating action generation as context-conditioned sparse computation instead of one fixed dense path for everything.

Dev: It uses task intent and sensor availability as two main pieces of context to guide this routing mechanism, which allows it to handle missing sensors gracefully without needing retraining.

Taro: So the framework splits the action representations into general experts and modality-specialized experts, and it routes them based on both what we want to do with a task and what sensors are actually present.

Rosa: That’s the key mechanism they focus on, using conditional routing guided by task intent and sensor availability to select the most relevant parts of the model.

Dev: And they use a specific process where intent conditions route action-side representations, and then those representations get sent to experts that are specialized for that particular modality or general patterns.

Taro: I think what’s interesting is how it prevents the model from just relying too much on one sensor input by using modality dropout and availability-based expert masking when a sensor isn't there.

Rosa: Right, so they treat the auxiliary sensor availability as a condition at the action generator level, which lets it exploit those depth cues when available while falling back to an RGB language path when depth is gone.

Dev: They train it using a flow matching objective over action chunks, and they even have this selection regularizer to make sure the average representation selection mass matches some target capacity ratio rho.

Taro: I see how that helps with efficiency too, because the paper shows that by controlling that capacity ratio rho, you can keep the number of computations low for smaller inputs while still keeping the expert computation scaled correctly.

Rosa: It really makes sense when you think about deployment; being able to rely on depth only when you have it but not crashing when it's missing is a big deal for real-world robotics.

Dev: So, to sum up this paper called "CoRE-VLA: Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts," they claim they are the first to explicitly formulate how to deploy unified VLA policies under sensor heterogeneity and missing auxiliary sensors as a joint architecture-and-training problem.

Taro: The implication for us is that scalable VLA policy learning shouldn't just be about scaling up the perception backbones, but it also has to make action generation adaptive to both the task intent and the physical sensory context.

Rosa: That’s a big picture idea. It suggests that if we want these models to work outside of a perfect lab setting, they need this kind of dynamic way of allocating their computational resources based on what's actually happening around them.

Conclusion: Rosa: So we're wrapping up this look at CoRE-VLA, which is titled "Towards Scalable and Robust Vision-Language-Action Modeling via Conditional Routing of Experts."

Dev: It’s a framework that handles sensors not by just stacking everything together, but by making the action generation sparse and context-conditioned.

Taro: Basically, instead of one big dense computation path for every single sensor setup, they route the computation based on two things: what the task is trying to do and what sensors are actually available.

Rosa: So it’s about being smart about when to use which brain parts depending on whether you have the right inputs or not.

Dev: The authors show that this approach lets the robot degrade gracefully when a sensor fails, meaning it doesn't just crash or give a bad action because one input is missing.

Taro: It changes things for autonomy because it means we don't have to build every single robot model from scratch for every possible sensor combination.

Rosa: Right, so if you’re looking at how this scales to different robots—like the LIBERO or those dual-arm setups they tested—it shows it can actually handle long-horizon tasks better than just a standard dense generator.

Dev: They show strong results on multi-task benchmarks, and when you look at real-world testing, it beats existing baseline models, even in zero-shot scenarios where the robot hasn't seen that specific setup before.

Taro: The numbers they shared suggest that this routing mechanism doesn't just give a slight bump; it really improves how the AI allocates its processing power across a long sequence of actions.

Rosa: It’s about making the policy more adaptive to both what you’re trying to achieve and what the physical world is actually showing you at any given moment.

Dev: So, this moves us past just building bigger models toward building models that are robust enough to operate in messy, real-world environments where things go wrong.

Taro: And it opens the door for future work on how we can make those task-intent signals even richer so the routing gets even smarter.

Zhejiang University · Shanghai Innovation Institute (Shanghai) · Fudan University

cs.RO

Submitted: 2026-07-04

Updated: 2026-10-09

Comments: Accepted at the 10th Conference on Robot Learning (CoRL 2026). Camera-ready version. 34 pages, including appendices

Code: https://github.com/NVIDIA/Isaac-GR00T

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and

Key concepts

Context-Conditioned Sparse Computation
Instead of using a single dense computation path for all actions, CoRE-VLA treats action generation as sparse computation guided by context. This means the model only activates the specific parts of its network that are relevant based on the current situation, such as which sensors are present and what the task intent is. This makes it efficient and adaptable.
Sensor Availability Gating
This mechanism uses sensor availability to control which specialized experts are used. If an auxiliary sensor, like depth, is present, its corresponding expert is activated. If the sensor is missing or unavailable during deployment, that expert is masked off. This allows the policy to gracefully degrade and rely only on available information without needing retraining.
Task Intent Routing
The framework routes action-side representations to experts based on task intent derived from the VLM's text embeddings. This ensures that the computation focuses on what is relevant for the current goal. By routing based on intent, the model can specialize its action generation for different long-horizon subgoals effectively.

Terminology

Summary

The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design.

CoRE-VLA Framework

CoRE-VLA is a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation via Conditional Routing of Experts, which enables graceful degradation under missing sensors without retraining. The key idea is to formulate action generation as context-conditioned sparse computation, rather than a fixed dense computation path shared by all tasks and sensor configurations. This framework uses two forms of context: first, sensor availability gates modality-specialized experts, allowing the policy to exploit auxiliary perceptual cues when they are present while masking the corresponding experts when they are absent, and second, task intent further routes action-side representations to task-relevant experts.

Conditional Routing Mechanism

The framework employs conditional routing guided by task intent and sensor availability. Specifically, the process involves several steps:

  1. Intent-conditioned routing is guided by a task-intent embedding g pooled from VLM-encoded text representations.

  2. Action-side representation selection occurs by assigning a selection score pi to each action-side representation, where only the TopK representations undergo cross-attention and expert routing.

  3. General and modality-specialized experts are divided to capture shared patterns and modality-dependent computation, respectively.

  4. Modality dropout and availability-based expert masking are used to prevent over-reliance on auxiliary sensor inputs, as the auxiliary modality branch is disabled when zmod = 0.

Training Objective and Inference

CoRE-VLA is trained with a flow matching objective over action chunks. The loss function is defined as Lact = E∥vˆθ − v∥2/2, where vˆθ = fθ,R(g,zmod)(xτ, τ, st, C). To stabilize the gradient path through continuous representation-selection scores, a selection regularizer Lsel is used to align the average representation-selection mass with the target capacity ratio ρ. The final training objective is L = Lact + λselLsel + λmoeLmoe, which includes a token-choice load-balancing loss Lmoe over available experts to encourage balanced expert utilization.

Experimental Results

Experiments on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation show that CoRE-VLA achieves strong results on long-horizon and multi-task benchmarks. In simulation benchmarks, CoRE-VLA achieves the best performance across all suites on LIBERO [21] and RoboCasa GR1 Tabletop [22, 4]. In real-world deployment, CoRE-VLA outperforms both a dense action-generator ablation and a strong pretrained VLA baseline, including in zero-shot generalization to unseen scenarios. Modality analysis shows that CoRE-VLA can exploit auxiliary depth when available while remaining robust when depth is unavailable during deployment.

Ablation Study Insights

The ablation study on LIBERO-Long reveals that task-intent routing with multiple general experts improves the dense baseline from 94.4% to 95.2%, suggesting better computation allocation across long-horizon subgoals. Furthermore, the comparison between Rows 5 and Row 6 in the ablation study shows that modality dropout alone may regularize the model toward a more conservative RGB-language-compatible representation and reduce the immediate benefit obtained from depth when there is no dedicated modality-specific capacity. The full CoRE-VLA configuration, combining task-intent routing, modality dropout, and modality-specialized experts, achieves the best success rate of 97.6% when depth is available at inference.

Routing Dynamics Analysis

The analysis of expert utilization shows that expert utilization changes over rollout time, indicating dynamic computation allocation during long-horizon execution. Video-aligned routing and token selection visualize how "different phases of the long-horizon rollout activate different expert mixtures and different subsets of action-side representations, suggesting task-progress-dependent conditional computation rather than a static routing pattern. Task-level heatmaps exhibit both shared and task-specific expert usage, suggesting that the router learns both reusable actionprocessing experts and task-adaptive expert allocation".

Efficiency Analysis

The analysis of CoRE efficiency shows that the FLOPs of a CoRE layer can be written as FlCoRE = kl Ccond(Lc, d) + Flroute, where kl ≈ ρLa. This demonstrates that the ideal saving is controlled by the capacity ratio ρ: smaller ρ activates fewer token-level conditional computations. The expert computation scales as Flexpert = klCexp ≈ ρLaCexp, rather than ELaCexp, which decouples model capacity from per-input activated computation.

Conclusion

CoRE-VLA can be viewed as a flow-matching action generator whose velocity field is parameterized by task- and modality-conditioned sparse computation. This design enables an RGB-pretrained VLA policy to be extended with auxiliary perceptual inputs through continued training, while preserving reliable execution when auxiliary sensors are absent or unavailable. Across LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation experiments, CoRE-VLA improves multi-task performance, long-horizon execution, and zero-shot generalization. This design suggests that scalable VLA policy learning should not only scale perception backbones, but also make action generation adaptive to both task intent and embodied sensory context. The findings suggest that CoRE-VLA remains robust when auxiliary depth is unavailable while benefiting from physical depth when available. The framework successfully exploits auxiliary geometric information while preserving a reliable non-depth computation path. The design enables the policy to exploit auxiliary perceptual modalities when they are present, fall back to an RGB-language computation path when they are absent, and allocate action-generation capacity adaptively across tasks and long-horizon subgoals. The final state is decoded as the predicted action chunk. The predicted chunk is then executed in a receding-horizon closed-loop manner.

The gist Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design.

How it works

CoRE-VLA is a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation. The key idea is to formulate action generation as context-conditioned sparse computation, rather than a fixed dense computation path shared by all tasks and sensor configurations. This framework uses two forms of context: first, sensor availability gates modality-specialized experts, allowing the policy to exploit auxiliary perceptual cues when they are present while masking the corresponding experts when they are absent, and second, task intent further routes action-side representations to task-relevant experts.

Conditional Routing Mechanism

The framework employs conditional routing guided by task intent and sensor availability. Specifically, the process involves several steps:

Improvements for AI systems

  1. Bold header: Robust deployment under sensor heterogeneity

The improved system can exploit auxiliary perceptual inputs when available while remaining reliable under sensor absence, as stated in A practical VLA policy should therefore exploit auxiliary perceptual inputs when they are available, while still acting reliably from RGB, proprioception, and language alone when they are absent. This enables graceful degradation without retraining.

  1. Bold header: Task-intent-conditioned computation allocation

The system can improve scalability across diverse tasks by using task intent to route action-side representations to task-relevant experts, which allows the action generator to allocate computation differently across diverse tasks and long-horizon subgoals. This addresses the limitation of a shared dense action generator that causes conflicting gradients, task interference, and negative transfer.

  1. Bold header: Modality-aware expert routing for robustness

The system can maintain reliable execution when auxiliary sensors are missing by employing availability-based expert masking, which masks modality-specialized experts when the sensor is absent (When zmod = 0, the auxiliary modality branch is disabled and modality-specialized experts are masked). This prevents the policy from overrely[ing] on them during deployment.

  1. Bold header: Capacity-aware computational efficiency

The system can achieve increased conditional computation capacity of the action generator without activating all added parameters for every input, as shown by the bound: FCoRE,total ≤ 1 − (1 − ρ)η + ¯ϵ. This provides a quantifiable trade-off between model size and activated compute.

  1. Bold header: Task-progress dependent dynamic computation

The system can perform complex maneuvers during long-horizon execution by exhibiting task-progress dependent computation allocation, as evidenced by the observation that routing patterns change over time, suggesting that CoRE-VLA dynamically reallocates action-side computation as the task progresses through different execution stages and subgoals.

Abstract

Vision-language-action (VLA) models have advanced generalist robotic manipulation, yet real-world deployment reveals a fundamental challenge: robots are equipped with diverse and heterogeneous sensor configurations, auxiliary sensors can fail unexpectedly during operation, and different robot embodiments often lack certain sensors by design. A unified policy that can exploit auxiliary perceptual inputs when available while remaining reliable under sensor absence, whether incidental or by design, is therefore essential for practical deployment. However, existing VLA policies couple action generation to a fixed sensor set through shared dense computation, making them brittle when sensors are missing and limiting their ability to specialize across diverse tasks and long-horizon behaviors. We propose CoRE-VLA, a scalable and robust VLA framework that formulates action generation as context-conditioned sparse computation. Sensor availability gates modality-specialized experts, enabling graceful degradation under missing sensors without retraining. Task intent further routes action-side representations to task-relevant experts, improving specialization across diverse tasks and long-horizon subgoals. While the framework is designed to accommodate different auxiliary sensors, we focus on depth as a representative and practically important auxiliary modality in our experiments. Experiments on LIBERO, RoboCasa GR1 Tabletop, and real-world dual-arm manipulation show that CoRE-VLA achieves strong results on long-horizon and multi-task benchmarks, and outperforms both a dense-action-generator ablation and a strong pretrained VLA baseline, including in zero-shot generalization to unseen scenarios. Modality analysis shows that CoRE-VLA can exploit auxiliary depth when available while remaining robust when depth is unavailable during deployment.

Sources

Related papers