Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies

arXiv:2609.15940 · cs.RO · Submitted 2026-09-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Beyond Single-Axis Testing".

Dev: The gist: Paired evaluation of compound robustness in vision-language-action policies reveals that aggregate success rates alone cannot characterize policy robustness because opposing outcome transitions can cancel out,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So what’s the actual setup here? This paper is introducing this six-axis benchmark, LIBERO-CTRL, to test vision language action policies in a way that looks at how they handle multiple things happening together.

Dev: That pairing means for every starting point, they test each single axis separately and then they test all those same axes simultaneously.

Rosa: It’s designed specifically to reveal two different kinds of outcome changes that just adding up success rates can’t tell apart.

Dev: They call these two things emergent failures, where everything works alone but the simultaneous rollout fails, and compensated successes, where at least one single axis fails but the combined rollout succeeds.

Taro: I mean, so if you look at aggregate success rates across those different conditions, one might just see a steady number and think the policy is robust.

Rosa: That’s exactly the problem they’re pointing out; these two transitions can happen in opposite directions and cancel each other out when you just add up the success percentages.

Dev: Because one transition decreases compound success while the other increases it, they can really mask substantial changes in how a policy behaves under joint pressure.

Taro: So, if we only look at the aggregate score on those individual tests, we might get a consistent result even when many initial states actually change outcome significantly.

Rosa: That’s why this paired evaluation protocol is important; it links the individual and simultaneous outcomes right at the level of matched initial states.

Dev: It separates two things that aggregate evaluation mixes up: whether vulnerabilities to different single axis perturbations happen on the same initial states, and whether they are independent.

The paper's summary: Rosa: Now, let’s talk about what this paper actually found when they ran this LIBERO-CTRL test across different policies and severity levels.

Dev: They applied it to six language conditioned VLA policies that span different parameter scales and three severity levels.

Rosa: The results showed substantial bidirectional outcome changes even when those emergent failures and compensated successes nearly cancel out in the aggregate success rate.

Dev: In fact, they found that as many as twenty-nine point zero percent of matched initial states still change outcome across all policy severity conditions.

Taro: That’s quite a number; it means that even if you think you have a stable policy, there are still significant points where the outcome shifts when you introduce this joint perturbation.

Rosa: They also noted that in the most affected condition, these changes reach thirty-four point five percent across all those severity levels.

Dev: The relative frequency of these two transitions varies depending on the specific policies and severities they are testing, but the transition rates themselves stay pretty similar even when you re-evaluate stochastic policies independently.

Taro: So it sounds like the pattern of these shifts isn't just random; it has a structure that depends on which policy you’re looking at.

Rosa: Right, and they gave us an equation to break down exactly what those changes are doing between simultaneous success and independence prediction.

The paper's improvements: Dev: They present this decomposition formula: Ssim minus Sindep equals (Sconj minus Sindep) times D plus (Ssim minus Sconj) times I.

Rosa: That equation is really interesting because it separates the cross axis dependence already present in the paired single axis outcomes, which is term D, from the remaining term I.

Dev: Term I equals Ssim minus Sconj shows that it’s just the signed difference in success rate between the simultaneous condition and the paired single axis conjunction.

Taro: So this identity means that emergent failures and compensated successes contribute with opposite signs to that aggregate difference, which is why they can cancel out.

Rosa: Because of this cancellation, neither marginal single axis success rates nor just term I alone fully characterizes compound robustness on its own.

Dev: The practical implication they suggest is that compound robustness needs to be evaluated using matched outcomes rather than just the aggregate success rates we've been used to relying on.

Taro: They suggest that a paired single axis sweep gives you both Sconj and D, while adding one more matched simultaneous condition helps identify Re, Rc, Rd, and I.

Rosa: Within those limits they found that neither the marginal single axis success rates nor just the signed difference Ssim minus Sconj fully characterize compound robustness when you’re near performance boundaries.

Conclusion: Dev: So to wrap up this paper on "Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies," the main point is that aggregate success rates alone are not enough to define policy robustness.

Rosa: They show that emergent failures and compensated successes can cancel out in aggregate, so you have to look at the directional changes separately.

Dev: The key finding is that because of this cancellation, neither marginal single axis success rates nor just the signed difference I alone fully characterizes compound robustness.

Taro: If we’re looking for actual shifts in behavior under combined stress, we need those directional transition rates reported separately instead of just the aggregate metrics.

Rosa: They suggest that a paired evaluation should report those two directional transition rates separately to get a clearer picture of what’s happening under simultaneous perturbation.

Dev: It means we need that additional matched simultaneous condition when doing these kinds of evaluations to properly characterize compound robustness.

Sony Computer Science Laboratory

cs.RO

Submitted: 2026-09-14

Updated: 2026-10-15

Code: https://github.com/hiroki-oist/LIBERO-ctrl

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: The gist: Paired evaluation of compound robustness in vision-language-action policies reveals that aggregate success rates alone cannot characterize policy robustness because opposing outcome

Key concepts

LIBERO-CTRL: A six-axis benchmark designed to test vision-language-action policies. It pairs every initial state tested under a single disturbance condition with a corresponding state under a simultaneous condition. This setup allows researchers to compare individual axis effects against combined effects.
Emergent Failures: A scenario where all individual tests on different axes succeed, but the policy fails when all conditions are applied simultaneously. These failures cannot be detected by looking at single-axis success rates alone because they are hidden in the joint evaluation.
Compensated Successes: A scenario where at least one single-axis test fails, but the policy succeeds when all conditions are applied simultaneously. This outcome can cancel out a failure from an emergent failure, making it indistinguishable from a successful aggregate performance based on marginal tests.

Terminology

Robustness — 1× in this paper · explained in 5 episodes

Summary

The gist: Paired evaluation of compound robustness in vision-language-action policies reveals that aggregate success rates alone cannot characterize policy robustness because opposing outcome transitions can cancel out, necessitating matched per-instance evaluation to reveal how joint perturbations alter behavior.

How it works

The paper introduces LIBERO-CTRL, a six-axis benchmark designed to pair each initial state across single-axis conditions and a matched simultaneous condition This design reveals two opposing outcome changes that aggregate success rates cannot distinguish: emergent failures, where all single-axis rollouts succeed but the simultaneous rollout fails, and compensated successes, where at least one single-axis rollout fails but the simultaneous rollout succeeds Because one transition decreases compound success while the other increases it, they can cancel, making aggregate compound performance appear consistent with single-axis measurements even when individual outcomes differ substantially

Key findings from the paired analysis

The study applies LIBERO-CTRL to six language-conditioned VLA policies spanning different parameter scales and three severity levels The resulting paired evaluations reveal substantial bidirectional outcome changes Even when emergent failures and compensated successes nearly cancel in aggregate, as many as 29.0% of matched initial states still change outcome Across all policy-severity conditions, these changes reach 34.5% in the most affected condition The relative prevalence of the two transitions varies across policies and severities, while the transition rates remain similar under independent re-evaluation of stochastic policies

Decomposition of outcome changes

The decomposition between simultaneous success and independence prediction is given by the equation Ssim − Sindep = (Sconj − Sindep) D + (Ssim − Sconj) I The term D captures cross-axis dependence already present in the paired single-axis outcomes, whereas the remaining term, I = Ssim − Sconj, is the signed difference in success rate between the simultaneous condition and the paired single-axis conjunction This identity I = Rc − Re shows that emergent failures and compensated successes contribute with opposite signs to the aggregate difference and can therefore cancel

Implications for robustness characterization

The practical implication is that compound robustness should be evaluated using matched outcomes rather than aggregate success rates alone A paired single-axis sweep provides Sconj and D, while one additional matched Simultaneous condition identifies Re, Rc, Rd, and I The directional rates are also constrained near performance boundaries Within these limits, neither marginal single-axis success rates nor the signed difference Ssim − Sconj fully characterises compound robustness

Limits of single-axis evaluation

Single-axis robustness is strongly anisotropic across policies and perturbation axes A low compound success rate may therefore be inherited from a dominant single-axis vulnerability rather than from an additional effect of combining perturbations Conversely, strong constituent performance does not guarantee compound reliability because emergent failures occur on configurations that succeed under every perturbation individually, and therefore cannot be identified from marginal single-axis success rates Compensated successes further show that compound outcomes are not simply a stricter version of the constituent conditions

Conclusion

Paired evaluation separates shared constituent vulnerability from outcome changes that emerge under simultaneous perturbation, and the directional decomposition reveals whether those changes reflect emergent failures, compensated successes, or both Given an already paired single-axis sweep, this requires only one additional matched Simultaneous condition The central finding is that the signed difference I = Ssim − Sconj can conceal substantial matched-outcome changes because emergent failures and compensated successes contribute with opposite signs Thus, neither marginal single-axis success rates nor I alone fully characterises compound robustness The paper concludes that paired evaluation should report the two directional transition rates separately

Limitations

Our study evaluates released checkpoints in simulation on a single embodiment and cannot attribute cross-policy differences to specific architectures or training procedures The Simultaneous condition does not identify which lower-order perturbation combinations produce each transition, and near-boundary conditions provide limited resolution in one transition direction The language axis is categorical rather than metrically calibrated, so conclusions about graded perturbation magnitude apply only to the metric axes

References

[1] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone “Libero: Benchmarking knowledge transfer for lifelong robot learning” Advances in Neural Information Processing Systems vol 36 pp 44 776–44 791 2023

[2] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard Calvin A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks IEEE Robotics and Automation Letters vol 7 no 3 pp 7327–7334 2022

[3] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He et al Libero-plus In-depth robustness analysis of vision-language-action models arXiv preprint arXiv:2510.13626 2025

[4] X. Zhou, Y. Xu, G. Tie, Y. Chen, G. Zhang et al Libero-pro Towards robust and fair evaluation of vision-language-action models beyond memorization arXiv preprint arXiv:2510.03827 2025

[5] J. Sun, W. Zhang, Z. Qi, S. Ren et al Vla-jepa Enhancing visionlanguage-action model with latent world model arXiv preprint arXiv:2602.10098 2026

[6] A. Brohan, N. Brown, J. Carbajal et al Rt-1 Robotics transformer for real-world control at scale arXiv preprint arXiv:2212.06817 2022

[7] A. Brohan, N. Brown, J. Carbajal et al Rt-2 Vision-language-action models transfer web knowledge to robotic control arXiv preprint arXiv:2307.15818 2023

[8] M. J. Kim, K. Pertsch, S. Karamcheti et al Openvla An open-source vision-language-action model arXiv preprint arXiv:2406.09246 2024

[9] M. J. Kim, C. Finn, and P. Liang Fine-tuning vision-language-action models Optimizing speed and success arXiv preprint arXiv:2502.19645 2025

[10] P. Intelligence, K. Black, N. Brown et al π0.5 A vision-language-action model with open-world generalization arXiv preprint arXiv:2504.16054 2025

[11] K. Black, N. Brown, D. Driess et al π0 A vision-language-action flow model for general robot control arXiv preprint arXiv:2410.24164 2024

[12] Q. Bu, Y. Yang, J. Cai et al Univla Learning to act anywhere with task-centric latent actions arXiv preprint arXiv:2505.06111 2025

[13] M.

Improvements for AI systems

System can distinguish between emergent failures and compensated successes by analyzing directional transitions (I = Rc - Re), rather than relying on aggregate success rates alone, which can conceal substantial matched-outcome changes.
The improved system will utilize a paired evaluation protocol, like LIBERO-CTRL, to explicitly measure the difference between the single-axis conjunction and simultaneous condition, thereby revealing whether vulnerabilities to different single-axis perturbations occur on the same initial states versus those that are independent.
The system can provide a nuanced understanding of robustness by reporting directional rates (Re and Rc separately) instead of just aggregate metrics, as this avoids losing substantial outcome changes through aggregation.
The system's evaluation will incorporate statistical controls, such as cluster bootstrap and independent stochastic-policy replications, to assess sensitivity to policy stochasticity while maintaining the integrity of the directional decomposition.