TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models".
Dev: Here's the gist of the paper. Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models.
Rosa: Pruning redundant reasoning tokens TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert.
Dev: The framework updates these components at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction.
Rosa: This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback.
Paper summary: Rosa: So we're looking at this paper called TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models. Basically, it tackles how to fine tune these vision language action models when you want them to actually perform physical tasks like robot manipulation.
Dev: Right. The main idea is that current methods often use one single update strategy for the whole model, which isn't great because the vision language part and the action part do different things, right? This paper proposes TEMPO as a framework with two separate loops for optimization.
Rosa: Exactly. They freeze the big vision language backbone to keep all that general semantic knowledge intact, and then they focus the reinforcement learning updates on just two parts: a semantic projection layer and a low-level action expert.
Dev: That decoupling is key because it lets them update these components at different rates, which is what they call a two-timescale strategy. They want to keep the latent action stable while rapidly incorporating feedback from real-time interaction.
Taro: But if you decouple them, how do you make sure the semantic information and the action execution actually talk to each other correctly during training?
Rosa: That's what they address by setting up two separate TD3 loops. The first loop optimizes the semantic projection layer, which maps those multimodal representations to latent actions.
Dev: And that loop uses a state space based on those multimodal semantic representations, calling them ht, and it tries to minimize a loss function LQs by comparing the predicted action zt with the actual desired action y at each time step t.
Taro: So for the semantic side, they are optimizing how well you can predict what latent action should be based on what you see and hear?
Rosa: Precisely. Now, for the second loop, they focus on the action expert which takes that latent action zt and turns it into a physical action chunk at each time step t.
Dev: That second loop has a different state space where the state is just that latent action zt itself, and it optimizes the action chunk parameters by minimizing LQa using an expectation over the reward function r.
Taro: I'm curious about how they manage these two loops running at different speeds. They mention a specific ratio for coordinating these updates.
Rosa: They introduce a module-specific update-frequency strategy where they compare the frequency of updating the action expert to the frequency of updating the semantic projection layer. This ratio is defined as fa over fs, which is Nads over Nsda, where Nads and Nsda are their respective update counts for actions and semantics.
Dev: The authors found that settings with a ratio greater than one generally perform better than when it's exactly one, but they also showed that different ratios give you different levels of improvement depending on what you're trying to achieve.
Taro: So, if I’m thinking about real-world deployment, does this frequency tuning help prevent the latent action from drifting too fast while still letting it learn the necessary control behaviors?
Rosa: That’s exactly why they do it. They update the semantic projection layer less frequently to keep that latent action stable, which prevents fast policy updates from destabilizing those high-level semantic representations.
Dev: But the action expert needs to be updated more often so it can quickly incorporate that control feedback you get from online interaction, which is crucial for learning execution.
Taro: I wonder about the practical side of this decoupling. When things go wrong in a complex environment, how does this structure handle unexpected world behavior?
Rosa: That's where Taro comes in. The system is designed so that even when the world misbehaves, the core semantic understanding stays relatively stable because that backbone is frozen during these RL updates.
Dev: And for the action expert update, they use a stop-gradient operation to ensure that gradients don't accidentally flow back into the frozen vision language backbone or affect those semantic projection parameters.
Taro: That sounds like a safety mechanism—making sure the learning process only modifies what it’s supposed to modify. It keeps things contained within those two focused loops.
Rosa: It does that, but their limitation is that they are still relying on TD3 loops for both modules, and they aren't exploring other update frequencies beyond what they tested in their experiments.
Dev: They did test different ratios of fa over fs, and the results showed improvement with settings greater than one, but you can't just pick a ratio without testing it for your specific task.
Taro: So, for someone actually running this outside the lab, what does this mean for how long the system can operate reliably before needing a re-tuning?
Rosa: It means they are aiming for better stability in the latent action over time, which should translate to more robust performance when deployed in the real world.
Dev: Overall, TEMPO is about making sure you don't destabilize the high-level understanding while still getting rapid learning on the low-level execution details through that frequency difference.
Taro: The authors are focusing heavily on how these distinct modules interact during reinforcement learning post-training for VLA models.
Rosa: And they are proposing this semantic-action decoupled framework to manage those interactions effectively using a two-timescale strategy.
Conclusion: Rosa: So, to wrap up this talk on TEMPO, we're looking at how they've taken these vision language action models and fine-tuned them using a two separate reinforcement learning loop to handle the semantic and the action parts differently.
Dev: That’s right. The authors are proposing this semantic-action decoupled approach for post-training RL that uses different update frequencies for the components, which is what they call a two timescale strategy.
Rosa: It essentially freezes the big vision language backbone to keep its general understanding stable, and then it focuses the reinforcement learning optimization on just two parts: the semantic projection layer and a low-level action expert.
Dev: The idea is to update those components at different rates—the semantic projection layer infrequently so the latent action stays steady, and the action expert more often to quickly catch feedback from online interaction.
Rosa: They found that this module specific update frequency strategy helps prevent that drift in the latent action while still letting the action expert learn execution details efficiently from real-time data.
Dev: And they show that setting a ratio between those frequencies, like one divided by one, actually doesn't always give you the best results; settings with a ratio greater than one generally do better.
Rosa: It means you can tune this frequency to get the right balance for your specific task without having to retrain the whole model constantly.
Dev: The authors are aiming for better stability in that latent action over time, which should mean more reliable performance when these models are actually used outside of a controlled lab setting.
Rosa: It's about making sure you don't mess up the high-level understanding while still getting rapid learning on the low-level control details through that frequency difference.
Dev: Now, we can talk about how this decoupling handles when the real world throws unexpected problems at it and what happens to its safety case during those kinds of tricky situations.
School of Computer Science and Technology, Zhejiang Gongshang University · KTH Royal Institute of Technology
cs.RO, cs.CV
Submitted: 2026-08-07
Updated: 2026-11-12
Project page: https://anonymous.4open.science/w/tempo-page
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose…
Key concepts
- Semantic Projection Layer ($\pi_s\theta$)
- This layer is responsible for mapping multimodal semantic representations ($h_t$) into latent actions ($z_t$). It is updated infrequently during training to ensure the underlying high-level semantic knowledge remains stable and consistent throughout the process.
- Action Expert ($\pi_a\phi$)
- This component learns how to generate specific action chunks from the latent actions ($z_t$). It is updated more frequently than the semantic layer to rapidly incorporate control feedback from online interactions, allowing for efficient policy learning.
- Two-Timescale RL
- TEMPO uses different update frequencies for the two components. The semantic layer is updated slowly (low frequency), while the action expert is updated quickly (high frequency). This decoupling strategy prevents rapid changes in control policy from destabilizing the general semantic representations.
- Decoupled RL Loops
- The framework optimizes two distinct RL loops: one for the semantic projection layer ($\pi_s\theta$) and one for the action expert ($\pi_a\phi$). These loops operate on different state spaces (semantic representations vs. latent actions) to manage adaptation dynamics effectively.
Terminology
Summary
Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. p. 0 TEMPO: Semantic-Action Decoupled RL Post-Training for
Vision-Language-Action Models
Ziheng Liu1
, Quantao Yang2
Abstract— Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL)
post-training. SFT is prone to distribution mismatch, and
existing RL approaches typically apply a single, uniform update
strategy to all model components, ignoring their distinct functional roles.Where the paper says it
Pruning redundant reasoning tokens
TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. The framework updates these components at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. p. 0 TEMPO freezes the pretrained vision-language backbone to
preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops:
the semantic projection layer and the low-level action expert. We update them at different rates—the semantic projection
layer infrequently, to keep the latent action stable, and the
action expert frequently, to rapidly incorporate control feedback from online interaction. We update them at different rates—the semantic projection
layer infrequently, to keep the latent action stable, and the
action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning
strategy prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action
expert to learn efficiently from online feedback.Where the paper says it
Where the paper says it
How the training signal is built
The system framework decomposes RL post-training into two dedicated RL loops optimizing the semantic projection layer πsθ and the action expert πaϕ, while the pretrained vision-language backbone remains frozen. For the semantic-level RL loop, the state space H consists of the multimodal semantic representations ht, while the action space Z consists of the latent actions zt = πsθ(ht). p. 2 θ and ϕ denote the trainable
parameters of π
s
θ
and π
a
ϕ
, and are collectively represented
as ω = (θ, ϕ). For the semantic-level RL loop, the state space H consists
of the multimodal semantic representations ht, while the
action space Z consists of the latent actions zt = π
s
θ
(ht). (4)
Using the target latent action, the semantic-level Bellman
target is defined as
y
S
t = rt + γ(1 − dt) min
i=1,2
Q¯s
i
(ht+1, zet+1). (5)
The twin semantic-level critics are optimized by minimizing the squared Temporal-Difference (TD) error:
LQs =
X
2
i=1
EDS
hQ
s
i
(ht, zt) − y
S
t
2
i
. Accordingly, the semantic-level state-action critic Qs
(ht, zt)
evaluates and updates π
s
θ
. For the action-level RL loop, the
state space is Z, with zt serving as the state, while the
action space A consists of the action chunks at = π
a
ϕ
(zt)
executed in the environment. (9)
Using the target latent action and action chunk, the actionlevel Bellman target is defined as
y
A
t = rt + γ(1 − dt) min
i=1,2
Q¯a
i
(˜zt+1, aet+1). (10)
The twin action-level critics are optimized by minimizing
the squared temporal-difference error:
LQa =
X
2
i=1
EDA
hQ
a
i
(zt, at) − y
A
t
2
i
.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The authors introduce a module-specific update-frequency strategy to coordinate the adaptation dynamics of semantic and action modules. TEMPO updates the semantic projection layer less frequently to mitigate latent-action drift, while updating the action expert more frequently to efficiently incorporate control feedback. p. 1 • We introduce a module-specific update-frequency strategy to coordinate the adaptation dynamics of semantic
and action modules. TEMPO updates the semantic
projection layer less frequently to mitigate latent-action
drift, while updating the action expert more frequently
to efficiently incorporate control feedback, achieving a
balance between semantic stability and policy improvement. Their effective update frequencies per post-training
round are therefore
fa =
Na
da
, fs =
Ns
ds
, (13)
where fa and fs denote the effective update frequencies
of the action expert and the semantic projection layer,
respectively. The action-to-semantic update-frequency ratio
is then defined as
ρ =
fa
fs
=
Nads
Nsda
. controls their relative update frequencies. We find that settings with ρ > 1 generally outperform ρ = 1, while different
frequency ratios yield different levels of improvement.Where the paper says it
Where the paper says it
Where the paper says it
The semantic-level TD3 updates πsθ using the first semantic-level critic: Ls = −Eht∼DS [Q s 1(ht, πsθ(ht))]. During this update, gradients are applied only to the semantic projection parameters θ and do not propagate into the frozen vision-language backbone or the action expert. p. 2 (7)
During this update, the replayed ht is treated as a fixed
input. Therefore, gradients are applied only to the semantic
projection parameters θ and do not propagate into the frozen
vision-language backbone or the action expert (10)
The twin action-level critics are optimized by minimizing
the squared temporal-difference error:
LQa =
X
2
i=1
EDA
hQ
a
i
(zt, at) − y
A
t
2
i
. (11)
For the action-expert update, the current latent action is
recomputed from the replayed ht as z
sg
t = sg(π
s
θ
(ht)), where
sg(·) denotes the stop-gradient operation. (12)
During this update, z
sg
t
is treated as a fixed input. Therefore, gradients are applied only to the action-expert parameters ϕ and do not propagate into the semantic projection
layer or the frozen vision-language backbone.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Decouple adaptation into two separate module-level TD3 loops optimizing semantic projection layer and action expert, allowing them to learn from distinct feedback streams. This prevents
fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback.
-
Implement a module-specific update-frequency strategy by updating the semantic projection layer less frequently than the action expert. This is achieved by setting an update-to-update frequency ratio rho greater than 1, which
mitigate latent-action drift
while enabling the action expert toincorporate control feedback more frequently.
-
The improved AI system can achieve better long-horizon performance on tasks like CALVIN and real-world manipulation by maintaining stable semantic representations while rapidly incorporating online interaction feedback for precise control.
Abstract
Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning(RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically optimize all trainable components within a single RL loop at a uniform update rate, although these components play distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone and adapts two components with dedicated RL loops: the semantic projection layer, which produces the latent action conditioning the action expert, and the low-level action expert itself. The semantic projection layer is updated infrequently and the action expert frequently, which aims to limit latent-action drift while allowing the action expert to learn from online feedback. Our analyses show that, compared with equal update frequencies, less frequent semantic updates reduce latent-action drift and improve policy performance. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on three real-world tasks.
Sources
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- RL Token: Bootstrapping Online RL with Vision-Language-Action Models
- Improving Vision-Language-Action Model with Online Reinforcement Learning
- ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flow
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Interactive Post-Training for Vision-Language-Action Models
- VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- Acting While Understanding: Asynchronous Semantic-Action Decoupling for Real-Time Vision-Language-Action Models
- DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model
- RL-VLA cubed: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
- Unified Vision-Language-Action Model
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
- Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation