Real-Time Execution with Autoregressive Policies

summary

The gist

Real-time execution with autoregressive policies enables smooth action trajectories and fast reactivity critical for realistic Vision-Language-Action models The autoregressive policy consistently…

In short

The research investigates real-time execution for Vision-Language-Action models by adapting autoregressive policies. The method involves tailoring the policy with specific tokenization, constrained decoding, and multi-trajectory decoding to ensure fast action generation. Results show that a fine-tuned autoregressive policy significantly outperforms flow-matching counterparts in real-time settings.

Key concepts

Synchronous Inference
This is the standard way VLA models work where the robot waits for the entire policy to finish before taking an action. This causes delays because the policy computation time often exceeds how fast a robot controller needs to issue commands, leading to pauses in real-time operation.
Autoregressive Policy
This is a type of model that generates output sequentially, one piece after another (like text generation). The paper shows that this approach is superior for real-time execution because it allows the system to generate actions incrementally, enabling faster reactivity and smoother action trajectories compared to other methods.
Constrained Decoding
This technique limits how much the model can deviate from a desired action sequence during generation. By enforcing strict constraints, researchers can guarantee that the decoding process finishes within a specific time limit, which is crucial for meeting real-time latency requirements.
Multi-trajectory Decoding
Instead of generating just one path, this method generates several potential action sequences simultaneously. The system then selects the best sequence from these multiple options based on performance metrics, effectively using any available idle computation time to maximize overall task success.

Terminology used across episodes

This episode discusses

The paper

Real-Time Execution with Autoregressive Policies · Read on arXiv

Sangkyu Lee, Seohyeon Park, Tackgeun You, Avi Caciularu, Idan Szpektor, Hwasup Lim, Youngjae Yu

Korea Institute of Science and Technology · Seoul National University · Google Research

Real-time execution, enabled by asynchronous inference that ensures both smooth action trajectories and fast reactivity, is critical for realistic deployments of large-scale Vision-Language-Action models. However, recent work on real-time execution primarily focuses on variants of diffusion policies, even though it is more critical for autoregressive policies given their slower rollout speed in synchronous inference. In contrast, we demonstrate that autoregressive policies can achieve real-time execution by adjusting the tokenization horizon and applying constrained decoding, thereby guaranteeing strict latency bounds that enable multi-trajectory decoding to maximize performance. Across simulated and real-world environments, we find that the autoregressive policy consistently outperforms its equivalent-level flow-matching policy counterpart while achieving significantly improved task completion speeds from synchronous inference. Coupled with the inherent advantages of autoregressive policies, such as faster convergence and better generalizability in instruction-following, these results confirm that autoregressive policies can remain a competitive policy type supporting real-time execution.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Real-Time Execution with Autoregressive Policies".

Dev: Here's the gist of the paper. Real-time execution with autoregressive policies enables smooth action trajectories and fast reactivity critical for realistic Vision-Language-Action models The autoregressive policy consistently outperforms its equivalent-level flow-matching policy counterpart while achieving significantly improved task completion speeds from synchronous inference The era of foundation models has extended to large multimodal models such as Vision-Language models (VLMs), whose capabilities are now being transferred to robotics through Vision-Language-Action models (VLAs) The root cause of this pausing behavior lies in the synchronous inference widely adopted by default for VLAs, or the think-then-move approach, where the robot executes actions after policy inference because the policy latency of VLAs generally exceeds the expected command interval of the robot controller in realistic scenarios Therefore, investigating asynchronous inference methods that guarantee these constraints is the core motivation of recent work aiming to achieve real-time execution with VLA policies The approach involves tailoring the existing autoregressive policy for supporting real-time execution in four steps Specifically, the steps include: 1) selecting a sufficient action horizon H = 2m to minimize the number of tokens to be decoded for maintaining reasonable latency 2) applying tokenization on each m horizon rather than the full action horizon H to enable conditioning on action chunks to be executed in the action queue 3) adopting constrained decoding to guarantee decoding to be completed within the delay dm ≤ m 4) employing multi-trajectory decoding to maximize the performance by fully utilizing idle computation time arising from synchronization Key Findings on Policy Performance The experiments confirm that π0-REALFAST, simply fine-tuned from π0-FAST with our approach, sufficiently outperforms π0 with real-time action chunking (RTC) in real-time execution, which is the equivalent-level counterpart VLA policy based on flowmatching policy Furthermore, it achieves task success rates and rollout speeds that approach those of the successor VLA policy, π0.5 The original advantage of autoregressive policies observed under synchronous inference persists in real-time execution, underscoring the strong motivation for pretraining autoregressive policies to natively support real-time execution Impact of Constraints and Decoding Strategies The necessary condition for real-time execution reveals that we have a free choice of m, which is directly related to the tokenization horizon m · ∆t in autoregressive policies We can control δt,m by reducing the number of tokens to be decoded, which governs δt,m The factor that has a significant impact on performance is dm, which is dominated by the inference frequency δt,m in asynchronous inference This observation supports the claim that recent autoregressive VLA policies can still be the type of policy for successful real-time execution, since they may be restricted to follow dm ≤ m while still achieving a reasonable average δt,m Qualitative and Quantitative Results In Figure 4, we show the average task success and progress rate of π0-REALFAST, π0, and π0.5 in real-time execution on evaluation suites for zero-shot DROID deployment We observe that π0-REALFAST significantly surpasses the base model with synchronous inference, π0-FAST, while outperforming π0 and showing performance comparable to π0.

Title and authors: Rosa: So we're talking about this paper called "Real-Time Execution with Autoregressive Policies." It’s about making these big vision language action models actually work in real time when you put them on robots or in other physical systems.

Dev: Yeah, the main point here is that right now, most of the focus on real time execution is on diffusion policies, but this paper argues it's more important for autoregressive policies because they have a slower rollout speed when they run synchronously.

Rosa: Right. So what's the core idea? Basically, they show that you can make these autoregressive models work in real time by tweaking how you handle tokenization and adding some constrained decoding tricks to guarantee strict time limits are met.

Dev: It’s about solving the problem where the policy takes too long to decide what action to take before the robot needs it, which is a huge issue in synchronous inference.

Taro: From my side, I'm wondering how this actually translates when things go wrong in a real environment; does this method handle unexpected delays well?

Rosa: That's a good question, Taro. The paper suggests they select an action horizon H equal to two times m to keep the number of tokens being decoded down so latency stays reasonable.

Dev: And then they apply tokenization on those smaller chunks, not the whole horizon H, which lets them condition on those specific action chunks that are actually going into the robot's action queue.

Taro: So instead of waiting for a huge decision to finish before moving, it’s about feeding the system small pieces of information and making decisions in tiny time slots?

Rosa: Exactly. And they use constrained decoding to make sure that even with this smaller tokenization, the actual decoding step finishes within a certain delay limit dm less than or equal to m.

Dev: That constraint on dm is crucial because it’s dominated by how often you're inferring, which they call the inference frequency delta t sub m. So they have to keep that frequency high enough to meet their time requirements.

Taro: If the system has to repeat inference asynchronously whenever decoding finishes before a certain number of steps, how does that affect the actual execution flow?

Rosa: They use multi-trajectory decoding there, which is like picking the best out of several possible paths by sorting them based on how much they look good according to their token loglikelihood.

Title and authors: Dev: That sounds like they're using the idle computation time when things finish early to actually pick a better action path, which helps maximize performance.

Taro: So it's not just about speed; it’s about using the extra time smartly to improve the quality of what you're doing when you finally decide.

Rosa: Right. And on real-world tests, they found that their policy version, called pi zero real fast, beats the equivalent flow matching policy in terms of performance while achieving much faster task completion speeds from synchronous inference.

Dev: That’s interesting because it suggests that even though autoregressive policies have a slower rollout speed normally, this approach makes them competitive in VLA systems by offering better instruction following and faster deployment rollouts.

Taro: So the implication for me is that if we can get this kind of fast rollout speed, maybe we can move these complex models out of the lab and into things like remote deployments where latency is a real killer.

Rosa: That’s what they are aiming for. The paper shows that by tailoring the existing autoregressive policy with these steps—horizon selection, chunked tokenization, constrained decoding—they get a policy type that can actually handle real-time execution in deployment scenarios.

Dev: And they showed that the necessary condition for real-time execution involves choosing an m value related to the action chunking horizon m times the time step delta t sub m. They can control this delta t sub m by reducing how many tokens they decode, which gives them control over their latency bound dm less than or equal to m.

Taro: So they have some knobs there, like choosing that action horizon H and controlling the tokenization scheme, which lets them trade off between decoding speed and the quality of the decision.

Rosa: And what they found is that the factor with a significant impact on performance ends up being dm less than or equal to m, which is mainly controlled by that inference frequency delta t sub m in their asynchronous setup.

Dev: It really supports the idea that these autoregressive VLA policies can still be the right choice for successful real-time execution, as long as you respect that dm less than or equal to m constraint while still aiming for a reasonable average delta t sub m.

Title and authors: Taro: So it’s not just about hitting a hard latency number; it's about finding that sweet spot where the speed of inference meets the smoothness of the action stream, especially when you use multi-trajectory decoding to maximize that gap.

Rosa: And qualitatively, in their zero shot DROID deployment evaluations, pi zero real fast significantly surpassed the base model with synchronous inference and even outpaced pi zero and pi zero point five.

Dev: That comparison against pi zero and pi zero point five is what really shows the strength of this approach; it's not just beating one other model but getting close to the performance of their newer, more advanced policy versions.

Taro: It means this method isn't just a small tweak; it’s showing that you can leverage the inherent strengths of autoregressive models—their instruction following and generalizability—while forcing them to operate under real-time constraints.

Rosa: Exactly. The paper "Real-Time Execution with Autoregressive Policies" demonstrates that by using these specific techniques, we can get an autoregressive policy to be a competitive policy type in VLAs because it offers fast convergence and better generalizability in instruction following while maintaining a fast rollout speed on deployment.

Dev: So the practical implication is that for anyone building real time systems with vision language action models, you should look at how you’re tokenizing and decoding your autoregressive policies to ensure you meet those strict latency bounds.

Taro: I think the biggest thing here is realizing that reducing policy latency doesn't automatically mean better deployment; there's a trade off between policy robustness and inference latency that they highlight.

Rosa: Right. They showed that the autoregressive policy can outperform its equivalent-level counterpart with lower inference latency in real time execution, which is a finding worth keeping in mind for deploying these systems on the edge.

Dev: So we wrap up this look at "Real-Time Execution with Autoregressive Policies." It shows that by applying tokenization horizon adjustments and constrained decoding, autoregressive policies can achieve real time execution through asynchronous inference and multi-trajectory decoding.

Taro: It’s a solid piece of work because it tackles the core problem of making these powerful models practical for robots that need to react quickly in the real world.

Rosa: Indeed. We’ll keep an eye on how these methods evolve as we look at other papers on arXiv coming through.

The paper's summary: Rosa: So, to wrap up what we just talked about, this paper is looking at how to make these big vision language action models actually work in real time when you put them on robots or in other physical systems.

Dev: The core idea is that right now, most of the focus on real time execution is on diffusion policies, but this paper argues it's more important for autoregressive policies because they have a slower rollout speed when they run synchronously.

Rosa: Right. Basically, they show that you can make these autoregressive models work in real time by tweaking how you handle tokenization and adding some constrained decoding tricks to guarantee strict time limits are met.

Dev: It’s about solving the problem where the policy takes too long to decide what action to take before the robot needs it, which is a huge issue in synchronous inference.

Rosa: And they found that simply fine-tuning an existing autoregressive model with their method makes it perform better than other equivalent flow matching policies when we force it into this real time execution setup.

Dev: That’s a big claim, because the original advantage of autoregressive policies seen under standard synchronous inference still holds up even when you introduce these real time constraints.

Rosa: Exactly. They're saying that pretraining these autoregressive models to natively support real time execution is a really strong motivation for researchers working on vision language action systems.

Dev: The necessary condition for this real time execution involves picking a number, let's call it m, which is related to the tokenization horizon m times the time step delta t sub m. They can control that delta t sub m by reducing how many tokens they decode, which gives them control over their latency bound dm less than or equal to m.

Rosa: So they have these knobs there, like choosing that action horizon H and controlling the tokenization scheme to manage the trade off between decoding speed and the quality of the decision.

Dev: And what they found is that the factor with a significant impact on performance ends up being dm less than or equal to m, which is mainly controlled by that inference frequency delta t sub m in their asynchronous setup.

Rosa: It really supports the idea that these autoregressive vision language action policies can still be the right choice for successful real time execution, as long as you respect that constraint while still aiming for a reasonable average delta t sub m.

Dev: And they showed qualitatively on tests with zero shot DROID deployment that their policy version, pi zero real fast, significantly surpassed the base model with synchronous inference and even outpaced pi zero and pi zero point five.

Rosa: That comparison against pi zero and pi zero point five is what really shows the strength of this approach; it's not just beating one other model but getting close to the performance of their newer, more advanced policy versions.

Dev: It means that by using these specific techniques—adjusting the horizon, chunking tokens, constrained decoding—they can get an autoregressive policy to be a competitive type in vision language action systems because it offers fast convergence and better generalizability in instruction following while maintaining a fast rollout speed on deployment.

Rosa: So the practical implication for us is that if we can get this kind of fast rollout speed, maybe we can move these complex models out of the lab and into things like remote deployments where latency is a real killer.

Dev: That’s what they are aiming for. They're showing that by applying these specific techniques, we can get an autoregressive policy to be a competitive policy type in VLAs because it offers fast convergence and better generalizability in instruction following while maintaining a fast rollout speed on deployment.

Rosa: This whole thing boils down to how you structure the model's decision-making process so that it respects those strict timing requirements of the physical world.

Dev: And this finding suggests that reducing policy latency doesn't necessarily lead to better deployment; there's a trade off between policy robustness and inference latency that they highlight.

Rosa: Right. They showed that the autoregressive policy can outperform its equivalent-level counterpart with lower inference latency in real time execution, which is a finding worth keeping in mind for deploying these systems on the edge.

Dev: So we wrap up this look at "Real-Time Execution with Autoregressive Policies." It shows that by applying tokenization horizon adjustments and constrained decoding, autoregressive policies can achieve real time execution through asynchronous inference and multi-trajectory decoding.

Rosa: It’s a solid piece of work because it tackles the core problem of making these powerful models practical for robots that need to react quickly in the real world.

Dev: Indeed. We’ll keep an eye on how these methods evolve as we look at other papers on arXiv coming through.

The paper's improvements: Rosa: We just talked about how they managed to get these autoregressive models running in real time by using constrained decoding and chunking action horizons to keep latency low for deployment on robots or other physical systems.

Dev: Right, and now we're going to look at the specific fixes they suggest to make that even better, because the paper isn't just showing a result; it’s proposing a whole new way of doing things.

Rosa: The main improvement involves selecting an action horizon H equal to two times m so that you minimize the number of tokens being decoded while keeping latency reasonable.

Dev: And then they change how they do tokenization, instead of using the full horizon H, they apply tokenization on each m length action chunk instead.

Rosa: That means we condition the AI on those smaller chunks that are actually going into the robot's action queue, which helps make the connection between action steps much smoother.

Dev: They also adopt constrained decoding using a dynamic programming approach to guarantee that the decoding finishes within a certain delay dm less than or equal to m.

Rosa: That constraint on dm is important because it restricts how fast the time step delta t sub m can grow, which gives them control over the latency bound.

Dev: And when things are running asynchronously, they repeat inference whenever it finishes before a certain number of steps B, but they choose that number B carefully based on the expected error rate and that time step delta t sub m.

Rosa: This mechanism ensures that we can guarantee dm less than or equal to m with constrained decoding to restrict the upper bound of delta t sub m.

Dev: Then, for performance, they use multi-trajectory decoding by sorting sequences by how much they look good according to their token loglikelihood on the constrained support and picking the best one.

Rosa: That strategy is used to fully maximize performance by reducing that computation idle time we talked about earlier when things finish early.

Dev: So, in simple terms, they’re saying you can control the timing of your AI decisions by manipulating how you break down the action sequence and how strictly you limit the decoding step.

Rosa: This implies that autoregressive policies have a lot of room to be tailored for real time execution without completely losing their instruction following abilities or generalizability.

Dev: And this is important because it shows we don't always have to switch entirely to diffusion models if we can get these autoregressive ones tuned correctly for speed.

Rosa: It’s about finding that sweet spot where the inherent strength of the autoregressive model—its ability to follow complex instructions—meets the hard timing constraints of a physical robot.

Dev: This moves us toward a system where we maintain a continuous stream of actions at the controller's timing by introducing an action queue on the client side, satisfying that necessary condition you mentioned earlier.

Rosa: That means we are shifting focus from just getting one perfect answer to maintaining that continuous stream of actions at the controller's timing, making latency unnoticeable to the client.

Dev: This is a big shift from what we usually look for in control loops, which is often about deterministic response, toward something more continuous and adaptive.

Rosa: So it’s not just about hitting a hard latency number; it’s about finding that sweet spot where the speed of inference meets the smoothness of the action stream when you use these multi-trajectory decoding techniques.

Dev: This is a solid piece of work because it tackles the core problem of making these powerful models practical for robots that need to react quickly in a real world.

Conclusion: Tom: So, we've gone through how they tweak the tokenization and decoding to make autoregressive policies fit within real time constraints for vision language action systems.

Rosa: Exactly. It's about using those specific techniques—like constrained decoding and multi-trajectory decoding—to get a competitive policy type that actually runs fast on deployment.

Dev: And what this means for controls is that we can leverage the instruction following abilities of autoregressive models while keeping the rollout speed high enough for a robot to actually keep up with its movements.

Taro: When we think about real world failure modes, this suggests these models are more robust because they're designed to operate under those strict timing conditions right from the start.

Rosa: Right. The main implication is that we can use these powerful autoregressive models in applications where latency matters, like remote robot operations or real time control loops.

Dev: But there's a caveat, as the paper points out, that reducing policy latency doesn't automatically mean better deployment; there’s a trade off between policy robustness and inference latency that they highlight.

Taro: I guess it means we have to be careful about where we push these models; they work well when the timing constraints are respected, but if the world misbehaves unexpectedly, we need a good recovery plan for that.

Rosa: That's a fair point, Taro. The paper doesn't solve all failure modes, it just shows how to make the policy execution part of that robust system design.

Dev: So to wrap up this look at "Real-Time Execution with Autoregressive Policies," we see that by applying tokenization horizon adjustments and constrained decoding, autoregressive policies can achieve real time execution through asynchronous inference and multi-trajectory decoding.

Rosa: It’s a solid piece of work because it tackles the core problem of making these powerful models practical for robots that need to react quickly in the real world.

Dev: Indeed. We'll keep an eye on how these methods evolve as we look at other papers on arXiv coming through, especially those dealing with bandwidth limits like Kinematic Hypothesis Context and Optical Network Digital Twins.

More episodes

← Home