TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

arXiv:2608.07314 · cs.RO, cs.CV · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models".

Dev: Here's the gist of the paper. Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models.

Rosa: Pruning redundant reasoning tokens TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert.

Dev: The framework updates these components at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction.

Rosa: This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback.

Paper summary: Rosa: So we're looking at this paper called TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models. Basically, it tackles how to fine tune these vision language action models when you want them to actually perform physical tasks like robot manipulation.

Dev: Right. The main idea is that current methods often use one single update strategy for the whole model, which isn't great because the vision language part and the action part do different things, right? This paper proposes TEMPO as a framework with two separate loops for optimization.

Rosa: Exactly. They freeze the big vision language backbone to keep all that general semantic knowledge intact, and then they focus the reinforcement learning updates on just two parts: a semantic projection layer and a low-level action expert.

Dev: That decoupling is key because it lets them update these components at different rates, which is what they call a two-timescale strategy. They want to keep the latent action stable while rapidly incorporating feedback from real-time interaction.

Taro: But if you decouple them, how do you make sure the semantic information and the action execution actually talk to each other correctly during training?

Rosa: That's what they address by setting up two separate TD3 loops. The first loop optimizes the semantic projection layer, which maps those multimodal representations to latent actions.

Dev: And that loop uses a state space based on those multimodal semantic representations, calling them ht, and it tries to minimize a loss function LQs by comparing the predicted action zt with the actual desired action y at each time step t.

Taro: So for the semantic side, they are optimizing how well you can predict what latent action should be based on what you see and hear?

Rosa: Precisely. Now, for the second loop, they focus on the action expert which takes that latent action zt and turns it into a physical action chunk at each time step t.

Dev: That second loop has a different state space where the state is just that latent action zt itself, and it optimizes the action chunk parameters by minimizing LQa using an expectation over the reward function r.

Taro: I'm curious about how they manage these two loops running at different speeds. They mention a specific ratio for coordinating these updates.

Rosa: They introduce a module-specific update-frequency strategy where they compare the frequency of updating the action expert to the frequency of updating the semantic projection layer. This ratio is defined as fa over fs, which is Nads over Nsda, where Nads and Nsda are their respective update counts for actions and semantics.

Dev: The authors found that settings with a ratio greater than one generally perform better than when it's exactly one, but they also showed that different ratios give you different levels of improvement depending on what you're trying to achieve.

Taro: So, if I’m thinking about real-world deployment, does this frequency tuning help prevent the latent action from drifting too fast while still letting it learn the necessary control behaviors?

Rosa: That’s exactly why they do it. They update the semantic projection layer less frequently to keep that latent action stable, which prevents fast policy updates from destabilizing those high-level semantic representations.

Dev: But the action expert needs to be updated more often so it can quickly incorporate that control feedback you get from online interaction, which is crucial for learning execution.

Taro: I wonder about the practical side of this decoupling. When things go wrong in a complex environment, how does this structure handle unexpected world behavior?

Rosa: That's where Taro comes in. The system is designed so that even when the world misbehaves, the core semantic understanding stays relatively stable because that backbone is frozen during these RL updates.

Dev: And for the action expert update, they use a stop-gradient operation to ensure that gradients don't accidentally flow back into the frozen vision language backbone or affect those semantic projection parameters.

Taro: That sounds like a safety mechanism—making sure the learning process only modifies what it’s supposed to modify. It keeps things contained within those two focused loops.

Rosa: It does that, but their limitation is that they are still relying on TD3 loops for both modules, and they aren't exploring other update frequencies beyond what they tested in their experiments.

Dev: They did test different ratios of fa over fs, and the results showed improvement with settings greater than one, but you can't just pick a ratio without testing it for your specific task.

Taro: So, for someone actually running this outside the lab, what does this mean for how long the system can operate reliably before needing a re-tuning?

Rosa: It means they are aiming for better stability in the latent action over time, which should translate to more robust performance when deployed in the real world.

Dev: Overall, TEMPO is about making sure you don't destabilize the high-level understanding while still getting rapid learning on the low-level execution details through that frequency difference.

Taro: The authors are focusing heavily on how these distinct modules interact during reinforcement learning post-training for VLA models.

Rosa: And they are proposing this semantic-action decoupled framework to manage those interactions effectively using a two-timescale strategy.

Conclusion: Rosa: So, to wrap up this talk on TEMPO, we're looking at how they've taken these vision language action models and fine-tuned them using a two separate reinforcement learning loop to handle the semantic and the action parts differently.

Dev: That’s right. The authors are proposing this semantic-action decoupled approach for post-training RL that uses different update frequencies for the components, which is what they call a two timescale strategy.

Rosa: It essentially freezes the big vision language backbone to keep its general understanding stable, and then it focuses the reinforcement learning optimization on just two parts: the semantic projection layer and a low-level action expert.

Dev: The idea is to update those components at different rates—the semantic projection layer infrequently so the latent action stays steady, and the action expert more often to quickly catch feedback from online interaction.

Rosa: They found that this module specific update frequency strategy helps prevent that drift in the latent action while still letting the action expert learn execution details efficiently from real-time data.

Dev: And they show that setting a ratio between those frequencies, like one divided by one, actually doesn't always give you the best results; settings with a ratio greater than one generally do better.

Rosa: It means you can tune this frequency to get the right balance for your specific task without having to retrain the whole model constantly.

Dev: The authors are aiming for better stability in that latent action over time, which should mean more reliable performance when these models are actually used outside of a controlled lab setting.

Rosa: It's about making sure you don't mess up the high-level understanding while still getting rapid learning on the low-level control details through that frequency difference.

Dev: Now, we can talk about how this decoupling handles when the real world throws unexpected problems at it and what happens to its safety case during those kinds of tricky situations.

School of Computer Science and Technology, Zhejiang Gongshang University · KTH Royal Institute of Technology

cs.RO, cs.CV

Submitted: 2026-08-07

Updated: 2026-11-12

Project page: https://anonymous.4open.science/w/tempo-page

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose…

Key concepts

Semantic Projection Layer ($\pi_s\theta$)
This layer is responsible for mapping multimodal semantic representations ($h_t$) into latent actions ($z_t$). It is updated infrequently during training to ensure the underlying high-level semantic knowledge remains stable and consistent throughout the process.
Action Expert ($\pi_a\phi$)
This component learns how to generate specific action chunks from the latent actions ($z_t$). It is updated more frequently than the semantic layer to rapidly incorporate control feedback from online interactions, allowing for efficient policy learning.
Two-Timescale RL
TEMPO uses different update frequencies for the two components. The semantic layer is updated slowly (low frequency), while the action expert is updated quickly (high frequency). This decoupling strategy prevents rapid changes in control policy from destabilizing the general semantic representations.
Decoupled RL Loops
The framework optimizes two distinct RL loops: one for the semantic projection layer ($\pi_s\theta$) and one for the action expert ($\pi_a\phi$). These loops operate on different state spaces (semantic representations vs. latent actions) to manage adaptation dynamics effectively.

Terminology

Summary

Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training The authors propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. p. 0

Where the paper says it

TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models Ziheng Liu1 , Quantao Yang2 Abstract— Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles.

Pruning redundant reasoning tokens

TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. The framework updates these components at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. p. 0

Where the paper says it

TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction.

This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback. p. 0
Where the paper says it

We update them at different rates—the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback.

How the training signal is built

The system framework decomposes RL post-training into two dedicated RL loops optimizing the semantic projection layer πsθ and the action expert πaϕ, while the pretrained vision-language backbone remains frozen. For the semantic-level RL loop, the state space H consists of the multimodal semantic representations ht, while the action space Z consists of the latent actions zt = πsθ(ht). p. 2

Where the paper says it

θ and ϕ denote the trainable parameters of π s θ and π a ϕ , and are collectively represented as ω = (θ, ϕ). For the semantic-level RL loop, the state space H consists of the multimodal semantic representations ht, while the action space Z consists of the latent actions zt = π s θ (ht).

The semantic-level TD3 optimizes πsθ by minimizing LQs = X2i=1 EDS hQ s i(ht, zt) − y S t2i. p. 2
Where the paper says it

(4) Using the target latent action, the semantic-level Bellman target is defined as y S t = rt + γ(1 − dt) min i=1,2 Q¯s i (ht+1, zet+1). (5) The twin semantic-level critics are optimized by minimizing the squared Temporal-Difference (TD) error: LQs = X 2 i=1 EDS hQ s i (ht, zt) − y S t 2 i .

For the action-level RL loop, the state space is Z, with zt serving as the state, while the action space A consists of the action chunks at = πaϕ(zt) executed in the environment. p. 2
Where the paper says it

Accordingly, the semantic-level state-action critic Qs (ht, zt) evaluates and updates π s θ . For the action-level RL loop, the state space is Z, with zt serving as the state, while the action space A consists of the action chunks at = π a ϕ (zt) executed in the environment.

The action-level TD3 optimizes πaϕ by minimizing LQa = X2i=1 EDA hQ a i(zt, at) − y A t2i. p. 3
Where the paper says it

(9) Using the target latent action and action chunk, the actionlevel Bellman target is defined as y A t = rt + γ(1 − dt) min i=1,2 Q¯a i (˜zt+1, aet+1). (10) The twin action-level critics are optimized by minimizing the squared temporal-difference error: LQa = X 2 i=1 EDA hQ a i (zt, at) − y A t 2 i .

Where the gains come from

The authors introduce a module-specific update-frequency strategy to coordinate the adaptation dynamics of semantic and action modules. TEMPO updates the semantic projection layer less frequently to mitigate latent-action drift, while updating the action expert more frequently to efficiently incorporate control feedback. p. 1

Where the paper says it

• We introduce a module-specific update-frequency strategy to coordinate the adaptation dynamics of semantic and action modules. TEMPO updates the semantic projection layer less frequently to mitigate latent-action drift, while updating the action expert more frequently to efficiently incorporate control feedback, achieving a balance between semantic stability and policy improvement.

The action-to-semantic update-frequency ratio is defined as ρ = fa/fs = Nads/Nsda. p. 3
Where the paper says it

Their effective update frequencies per post-training round are therefore fa = Na da , fs = Ns ds , (13) where fa and fs denote the effective update frequencies of the action expert and the semantic projection layer, respectively. The action-to-semantic update-frequency ratio is then defined as ρ = fa fs = Nads Nsda .

They find that settings with ρ > 1 generally outperform ρ = 1, while different frequency ratios yield different levels of improvement. p. 4
Where the paper says it

controls their relative update frequencies. We find that settings with ρ > 1 generally outperform ρ = 1, while different frequency ratios yield different levels of improvement.

The semantic-level TD3 updates πsθ using the first semantic-level critic: Ls = −Eht∼DS [Q s 1(ht, πsθ(ht))]. During this update, gradients are applied only to the semantic projection parameters θ and do not propagate into the frozen vision-language backbone or the action expert. p. 2

Where the paper says it

(7) During this update, the replayed ht is treated as a fixed input. Therefore, gradients are applied only to the semantic projection parameters θ and do not propagate into the frozen vision-language backbone or the action expert

Similarly, for the action-expert update, zsgt is computed as zsgt = sg(πsθ(ht)), where sg(·) denotes the stop-gradient operation. p. 3
Where the paper says it

(10) The twin action-level critics are optimized by minimizing the squared temporal-difference error: LQa = X 2 i=1 EDA hQ a i (zt, at) − y A t 2 i . (11) For the action-expert update, the current latent action is recomputed from the replayed ht as z sg t = sg(π s θ (ht)), where sg(·) denotes the stop-gradient operation.

During this update, gradients are applied only to the action-expert parameters ϕ and do not propagate into the semantic projection layer or the frozen vision-language backbone. p. 3
Where the paper says it

(12) During this update, z sg t is treated as a fixed input. Therefore, gradients are applied only to the action-expert parameters ϕ and do not propagate into the semantic projection layer or the frozen vision-language backbone.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decouple adaptation into two separate module-level TD3 loops optimizing semantic projection layer and action expert, allowing them to learn from distinct feedback streams. This prevents fast policy updates from destabilizing highlevel semantic representations while still allowing the action expert to learn efficiently from online feedback.

  2. Implement a module-specific update-frequency strategy by updating the semantic projection layer less frequently than the action expert. This is achieved by setting an update-to-update frequency ratio rho greater than 1, which mitigate latent-action drift while enabling the action expert to incorporate control feedback more frequently.

  3. The improved AI system can achieve better long-horizon performance on tasks like CALVIN and real-world manipulation by maintaining stable semantic representations while rapidly incorporating online interaction feedback for precise control.

Abstract

Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning(RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically optimize all trainable components within a single RL loop at a uniform update rate, although these components play distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone and adapts two components with dedicated RL loops: the semantic projection layer, which produces the latent action conditioning the action expert, and the low-level action expert itself. The semantic projection layer is updated infrequently and the action expert frequently, which aims to limit latent-action drift while allowing the action expert to learn from online feedback. Our analyses show that, compared with equal update frequencies, less frequent semantic updates reduce latent-action drift and improve policy performance. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on three real-world tasks.

Sources

Related papers