Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies

arXiv:2605.17204 · cs.RO, cs.AI · Submitted 2026-05-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies".

Dev: Here's the gist of the paper. Event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability The authors propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts This method is significant because it addresses the difficulty of transferring mechanistic interpretability tools from language and vision-language models to Vision-Language-Action (VLA) policies, where outputs are robot actions rather than human-readable tokens The pipeline scores every alive SAE feature against external behavioral events extracted from rollouts, achieving full coverage of the SAE basis rather than only the features that stand out in SAE activation statistics Pruning redundant reasoning tokens The core problem addressed is that mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs because outputs are robot actions instead of human-readable tokens, making interventions hard to read This paper proposes an event-grounded pipeline that links kinematic keyframes, task-local event clusters, and event-aligned ranking to score SAE features without per-feature manual labeling The authors utilize SAEs because they decompose superposed activations into a sparse basis whose features can be enumerated and scored individually, and its training is unsupervised How the training signal is built The pipeline involves several stages anchored by signals independent of the SAE activations being analyzed 1.

Rosa: SAE Training and Fidelity Evaluation involves training one BatchTopK SAE per (architecture stream, suite, layer) combination on per-token residual-stream activations from closed-loop rollouts Quality is assessed through offline reconstruction diagnostics (FVE, alive fraction, average L0) and behavioral fidelity under a reconstruction-only hook that replaces the hidden state at layer l with x′ = Dec(Enc(x)) 2.

Dev: Kinematic Keyframe Extraction uses Automatic Waypoint Extraction (AWE) to identify candidate event anchors, which are independent of SAE activations, semantic labels, and feature scores 3.

Rosa: Event Clustering groups keyframe bundles into task-local event clusters by concatenating a normalized visual embedding (v), a normalized robot state vector (s), and a normalized temporal-progress scalar (p) Where the gains come from The effectiveness of the pipeline is tested through four ranking strategies to select candidate features worth testing, including Event-aligned ranking, Window-mean ranking, Task-mean ranking, and a Random-alive control The authors find that event-aligned rankings yield the strongest causal effects on OpenVLA and transfer to the continuous action chunks of π0.5 Closed-loop intervention experiments stress-test which SAE features actually drive policy behavior, comparing hard zero-out against soft interventions using residual-preserving latent edits The results show that event-aligned rankings produce the largest SR drop for OpenVLA layer 31, well separated from the randomalive control Intervention behavior and limitations The study reveals architecture- and site-dependent intervention behavior, noting that individual event-aligned features measurably drop closed-loop success on OpenVLA while the PaliGemma VLA backbone barely responds to single-feature edits The authors observe that for the π0.

Paper summary: Rosa: So to summarize what this paper is doing with Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies, they are proposing a pipeline that anchors the analysis of sparse autoencoder features directly to specific behavioral events.

Dev: They’re trying to solve the problem where mechanistic interpretability tools from language and vision models don't translate cleanly to VLAs because the model outputs physical actions instead of human-readable tokens, making interventions really hard to read.

Taro: Their method involves several stages anchored by signals that are independent of the SAE activations being analyzed, which is a key part of how they build this ground.

Rosa: First, they train one batch top k autoencoder per architecture stream suite layer combination on residual stream activations from closed loop rollouts, and then they check the quality using offline reconstruction diagnostics like alive fraction and average L zero error.

Dev: Then they extract kinematic keyframes using automatic waypoint extraction to find candidate event anchors that don't depend on the SAE activations or semantic labels at all.

Taro: After that, they group these keyframe bundles into task local event clusters by combining a normalized visual embedding, a robot state vector, and a temporal progress scalar.

Rosa: The real innovation comes in how they score the features: instead of just relying on activation statistics to pick features, they use four ranking strategies—event aligned ranking, window mean ranking, task mean ranking, and a random alive control—to pick which SAE features are worth testing.

Dev: And their closed loop intervention experiments show that event aligned rankings give the strongest causal effects on OpenVLA and even transfer to the continuous action chunks of the pi zero point five policy.

Taro: The results stress-test which specific SAE features actually drive policy behavior, and they found that event aligned rankings produce the largest drop in success for OpenVLA layer thirty-one when using residual preserving latent edits, which is well separated from their random alive control.

Rosa: So the main point here is that this method achieves full coverage of the SAE basis by scoring every live feature against external behavioral events, not just the ones that stand out in activation statistics.

Dev: This moves the interpretability work into a place where behavior actually matters, which is pretty significant for understanding these complex policies.

Conclusion: Rosa: So looking at Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies by Jin and others, the authors are showing how to anchor SAE feature analysis to actual robot behavior rather than just abstract language contexts.

Dev: They’re addressing the difficulty of applying interpretability tools from vision language models to VLA policies because the outputs are actions, not tokens, which makes testing interventions difficult.

Taro: The implication is that by using event-grounded features, we can get a clearer picture of what specific parts of the SAE basis control robot behavior in a closed loop setting.

Rosa: It suggests that instead of just looking at what activations are high in the SAE, we should be linking those activations directly to observable physical events like keyframes and task clusters.

Dev: The authors found that event aligned rankings yield the strongest causal effects on OpenVLA behavior and even transfer those findings to the continuous action chunks of the pi zero point five policy.

Taro: What this means for autonomy is that we can start testing these internal representations by looking at what happens during real-world tasks, rather than relying solely on static feature importance scores from activation data.

Department of Computer Science, Purdue University

cs.RO, cs.AI

Submitted: 2026-05-17

Updated: 2026-11-05

Comments: NeurIPS 2026 camera-ready version

Code: https://github.com/xc-j/Event-SAE

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: Event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability The authors propose an event-grounded interpretability pipeline that anchors SAE feature…

Key concepts

Event-Grounded Interpretability
This is a pipeline that links Sparse Autoencoder features directly to real-world behavioral events, such as specific robot movements or task clusters. Instead of just looking at which SAE features are activated in text, this method scores every feature against external actions derived from robot rollouts.
Sparse Autoencoders (SAEs)
SAEs are used to decompose complex model activations into a sparse basis—a set of specific, enumerable features. These features can then be individually scored. The training for these SAEs is unsupervised, meaning the system learns the feature decomposition without needing pre-labeled data.
Event Clustering
This process groups extracted kinematic keyframes into task-local event clusters. This grouping uses a combination of visual embeddings, robot state vectors, and temporal progress to identify meaningful moments in the robot's behavior that serve as anchors for interpreting the model's decisions.

Terminology

Summary

Event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability The authors propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts This method is significant because it addresses the difficulty of transferring mechanistic interpretability tools from language and vision-language models to Vision-Language-Action (VLA) policies, where outputs are robot actions rather than human-readable tokens The pipeline scores every alive SAE feature against external behavioral events extracted from rollouts, achieving full coverage of the SAE basis rather than only the features that stand out in SAE activation statistics

Pruning redundant reasoning tokens

The core problem addressed is that mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs because outputs are robot actions instead of human-readable tokens, making interventions hard to read This paper proposes an event-grounded pipeline that links kinematic keyframes, task-local event clusters, and event-aligned ranking to score SAE features without per-feature manual labeling The authors utilize SAEs because they decompose superposed activations into a sparse basis whose features can be enumerated and scored individually, and its training is unsupervised

How the training signal is built

The pipeline involves several stages anchored by signals independent of the SAE activations being analyzed

  1. SAE Training and Fidelity Evaluation involves training one BatchTopK SAE per (architecture stream, suite, layer) combination on per-token residual-stream activations from closed-loop rollouts Quality is assessed through offline reconstruction diagnostics (FVE, alive fraction, average L0) and behavioral fidelity under a reconstruction-only hook that replaces the hidden state at layer l with x′ = Dec(Enc(x))

  2. Kinematic Keyframe Extraction uses Automatic Waypoint Extraction (AWE) to identify candidate event anchors, which are independent of SAE activations, semantic labels, and feature scores

  3. Event Clustering groups keyframe bundles into task-local event clusters by concatenating a normalized visual embedding (v), a normalized robot state vector (s), and a normalized temporal-progress scalar (p)

Where the gains come from

The effectiveness of the pipeline is tested through four ranking strategies to select candidate features worth testing, including Event-aligned ranking, Window-mean ranking, Task-mean ranking, and a Random-alive control The authors find that event-aligned rankings yield the strongest causal effects on OpenVLA and transfer to the continuous action chunks of π0.5 Closed-loop intervention experiments stress-test which SAE features actually drive policy behavior, comparing hard zero-out against soft interventions using residual-preserving latent edits The results show that event-aligned rankings produce the largest SR drop for OpenVLA layer 31, well separated from the randomalive control

Intervention behavior and limitations

The study reveals architecture- and site-dependent intervention behavior, noting that individual event-aligned features measurably drop closed-loop success on OpenVLA while the PaliGemma VLA backbone barely responds to single-feature edits The authors observe that for the π0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Event-grounded SAE pipeline for VLA analysis allows researchers to score every alive SAE feature against these events without per-feature manual labeling, enabling automated discovery of behaviorally relevant features across different simulation architectures and real-robot studies.

  2. Architecture and site dependent intervention behavior can be identified, revealing that individual event-aligned features measurably drop closed-loop success on OpenVLA while the PaliGemma backbone shows limited direct leverage under the residual-preserving zero-out protocol.

  3. Policy tuning can be guided by a controllable dial, as the magnitude of behavioral disruption can be selected directly rather than only chosen between identity and policy collapse when using soft intervention (alpha f).

  4. Real-robot deployment safety is enhanced by using conservative latent-space suppression because aggressive or poorly tuned interventions can collapse behavior in simulation and induce large real-robot motions.

  5. The system can perform causal sensitivity probes, such as the Target/Off-Target Zero-Out Specificity Probe, to determine if event cluster selected features are task specific or share manipulation roles across tasks within a suite.

Sources

Related papers