Retrieve in Time, Correct in Frequency

arXiv:2608.04527 · cs.RO · Submitted 2026-08-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Retrieve in Time, Correct in Frequency".

Dev: Here's the gist of the paper. Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages.

Rosa: This paper introduces Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.

Dev: Framework Overview RTCF separates memory reuse into progress-aware retrieval and frequency-domain correction.

Rosa: The framework decouples progress-aware memory retrieval from frequency-domain action correction.

Paper summary: Rosa: We’re wrapping up with this discussion on Retrieve in Time Correct in Frequency by Yuze Fan and Yue Cao et al. The authors are working at the Research Institute of Tsinghua University and Everwise-Tech Co., Ltd. Tongji University, and Fudan University.

Dev: What they’ve shown is a test-time correction framework that improves frozen VLA performance with low overhead by separating memory retrieval from frequency domain action correction.

Taro: The big picture here is that you can use powerful frozen policies for long horizon tasks, but this method gives you a way to fix the accumulated errors and visual aliasing during actual execution without needing any retraining.

Rosa: The key contribution is the Progressive Memory Alignment mechanism which causally aligns the history with successful trajectories to find an aligned memory position that indexes a future action chunk as a prior.

Dev: It transforms that memory chunk into a low-frequency motion prior and applies a bounded low-frequency residual transfer in coefficient space, using a correction mask for motion dimensions.

Taro: The implications are that this could be useful for deploying complex VLA systems where you need to adapt to the specific environment you're in at deployment time, not just during training.

Rosa: They showed aggregate success rising from eighty-six point four percent to eighty-eight point four percent, with the largest gain being seven point zero percentage points on LIBERO Long.

Dev: The paper uses a single policy invocation and adds about ten point nine nine milliseconds across the two measured stages per action chunk.

Taro: It’s definitely a test-time adaptation method for robot policies that reuses successful deployment experience to adapt the frozen generative VLA at test time, injecting an aggregated action prior into an intermediate state of the generative sampler.

Rosa: It gives us a practical way to get better performance when dealing with long horizon manipulation challenges where accumulated error is a big issue.

Conclusion: Rosa: So, to wrap up: this paper is about taking those frozen vision language action policies that we've been training and giving them a way to fix their mistakes during actual execution without retraining them at all.

Dev: Right, it’s called Retrieve in Time Correct in Frequency. The authors are from Tsinghua University and other places, they’re focused on improving how these models handle long sequences of actions.

Taro: It sounds like they're tackling that problem of accumulated error and visual aliasing when the robot is actually doing the task versus when it's just training in simulation.

Rosa: Exactly. They’ve figured out a way to separate how they retrieve information from how they correct those visual errors, which keeps the model overhead pretty low during testing.

Dev: The core idea is this Progressive Memory Alignment mechanism, where they track successful past executions to find an aligned memory position that points to a future action chunk as a prior.

Taro: So instead of just replaying old actions from scratch, it’s using the history to guide the current step by giving it a low-frequency motion prior.

Rosa: That prior is treated like a low-frequency signal, and they use it to transfer a bounded residual correction directly into the action coefficients.

Dev: They use this binary mask to specifically target motion dimensions, excluding higher frequencies or non-motion parts, which keeps the correction focused on what matters for movement.

Taro: It seems like a very practical way to handle those long-horizon issues where drift happens over many steps in the physical world.

Rosa: They showed that this test-time correction boosts the frozen policy's overall success rate, moving it from eighty-six point four percent up to eighty-eight point four percent across several benchmark tasks.

Dev: And they pointed out that this gain is real, especially on tasks like LIBERO Long, where they saw a seven point zero percentage point increase.

Taro: What’s interesting is that the analysis showed both the history-aware alignment and the frequency-selective correction parts were actually necessary to get those results.

Rosa: It means you can use these powerful frozen models for deployment and still get performance bumps without needing extra GPU power or retraining them on a new dataset.

Dev: It requires just one policy invocation during testing, adding about ten point nine nine milliseconds across the two correction stages per action chunk.

Taro: So it’s not some massive model overhaul; it’s a clever way to inject context into an already trained system at deployment time.

Rosa: This framework suggests we can get much better real-world performance from these frozen vision language action policies than we thought possible.

Yuze Fan, Yue Cao, Pengjie Gao, Haojia Gao, Guangqiu Guo, Ziyue Zhang, Junbo Tan, Bokui Chen, Ïa Zhuo Zou, Ïa Xueqian Wang

Research Institute of Tsinghua University in Shenzhen · Tongji University · Fudan University

cs.RO

Submitted: 2026-08-05

Updated: 2026-10-28

Comments: The current version of this manuscript is preliminary, and the experimental evidence is not yet sufficient to fully support its main conclusions. Substantial additional experiments and validation are required to establish the reliability and generalizability of the reported findings. We therefore request withdrawal of the current version

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 100/100

The gist: Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across…

Key concepts

Progressive Memory Alignment (PMA)
PMA tracks the current execution within successful past actions by maintaining a frontier of alignments between the growing visual history and completed memory. This allows the system to find relevant prior information for future steps, treating aligned memory as a reliable 'prior' instead of simply replaying old actions.
Frequency Correction Strategy
This strategy treats the retrieved prior action as a low-frequency motion guide. It uses a binary mask to select only motion dimensions and low frequencies, excluding noise and high frequencies. A residual transfer then applies this correction to the current action, refining it based on the retrieved guidance.
Retrieve in Time (RIT)
This involves locating relevant past actions by matching the current execution against stored memories using a cosine dissimilarity metric. The alignment frontier is updated causally, ensuring that each new action query contributes exactly one assignment to retrieve the most temporally relevant prior memory for correction.
Frozen VLA Policy
This refers to a vision-language-action policy that has been trained and cannot be updated further during testing. Because it is frozen, RTCF's test-time correction framework is necessary to overcome its inherent limitations regarding accumulated execution errors across long sequences.

Terminology

Summary

Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. p. 0

Where the paper says it

Retrieve in Time, Correct in Frequency Yuze Fan2,3 Yue Cao1,2,4 Pengjie Gao5 Haojia Gao1 Guangqiu Guo1,2,5 Ziyue Zhang1 Junbo Tan1 Bokui Chen5 Zhuo Zou4 Xueqian Wang5 1Research Institute of Tsinghua University in Shenzhen 2Everwise-Tech Co. 3Tongji University 4Fudan University 5Tsinghua Shenzhen International Graduate School, Tsinghua University Abstract Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages.

This paper introduces Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead. p. 0
Where the paper says it

Successful rollouts provide useful corrective evidence, yet currentframe retrieval can return progress-misaligned actions, while direct replay or time-domain fusion can overwrite the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.

Framework Overview

RTCF separates memory reuse into progress-aware retrieval and frequency-domain correction. The framework decouples progress-aware memory retrieval from frequency-domain action correction. p. 1

Where the paper says it

We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low overhead. RTCF decouples progress-aware memory retrieval from frequency-domain action correction.

It maintains a frontier over monotonic alignments between the growing visual execution history and each complete successful memory through Progressive Memory Alignment (PMA). p. 1
Where the paper says it

RTCF decouples progress-aware memory retrieval from frequency-domain action correction. Its online retrieval mechanism, Progressive Memory Alignment (PMA), maintains a frontier over monotonic alignments between the growing visual execution history and each complete successful memory.

This aligned memory position indexes a future action chunk, which is used as a prior rather than replayed directly. p. 1
Where the paper says it

Its online retrieval mechanism, Progressive Memory Alignment (PMA), maintains a frontier over monotonic alignments between the growing visual execution history and each complete successful memory. The resulting aligned memory position indexes a future action chunk, which is used as a prior rather than replayed directly.

Memory Retrieval Mechanism

PMA locates the current execution within successful trajectories by maintaining monotonic alignment frontiers. The local matching cost is defined as the cosine dissimilarity ci,t(q) = 1 − clipz⊤ t zi,q −1, 1. p. 3

Where the paper says it

Rather than greedily committing to a single progress estimate, the frontier preserves competing monotonic hypotheses, allowing later observations to resolve ambiguities introduced by earlier visually similar states. For an incoming descriptor zt, its local matching cost at position q of memory i is defined as the cosine dissimilarity ci,t(q) = 1 − clipz ⊤ t zi,q, −1, 1 , (5) Figure 3: Progressive Memory Alignment.

The alignment frontier is updated using an asymmetric causal transition pattern where every new policy invocation advances the query by exactly one step. p. 3
Where the paper says it

Let Ji,t(q) denote the minimum cumulative cost of aligning Qt to memory i while assigning its latest descriptor zt to memory position q. We use an asymmetric causal transition pattern: every new policy invocation advances the query by exactly one step, while the corresponding memory position may remain unchanged or advance by a

This process ensures that every query descriptor contributes exactly one alignment assignment, which is important for causal action retrieval. p. 4
Where the paper says it

Unlike a symmetric DTW transition rule, the recurrence contains no within-frontier transition: the alignment can update its progress only when a new observation arrives. Consequently, every query descriptor contributes exactly one alignment assignment, which is important for causal action retrieval.

Frequency Correction Strategy

The aligned prior is treated as a low-frequency motion prior and applied via a bounded low-frequency residual transfer. The binary correction mask Gf,d = I[0 < f < Fc]I[d ∈ Imotion] excludes the DC term, higher frequencies, and non-motion dimensions. p. 4

Where the paper says it

We define the binary correction mask Gf,d = I[0 < f < Fc]I[d ∈ Imotion], (10) where Fc is the temporal-frequency cutoff and Imotion contains the motion-channel indices. The mask excludes the DC term, higher frequencies, and non-motion dimensions.

The residual is transferred as ∆Ct = G ⊙ clip(Cmem t − Cπ t, −δmax, δmax), and the final corrected action is Ccorr t = Cπ t + λ0∆Ct. p. 4
Where the paper says it

In particular, event-like gripper commands remain unchanged to avoid blurring discrete open–close transitions. We transfer a bounded residual rather than replacing the selected coefficients: ∆Ct = G ⊙ clip(Cmem t − Cπ t , −δmax, δmax), Ccorr t = Cπ t + λ0∆Ct.

Performance and Analysis

Across four LIBERO suites, RTCF improves the frozen policy’s aggregate success from 86.4% to 88.4%, with its largest suite-level gain of 7. p. 1

Where the paper says it

Across four LIBERO suites, RTCF improves the frozen policy’s aggregate success from 86. 4%, with its largest suite-level gain of 7.

0 percentage points on LIBERO-Long. p. 1
Where the paper says it

4%, with its largest suite-level gain of 7. 0 percentage points on LIBERO-Long.

The gains require no parameter updates, repeated VLA inference, or additional GPU resources. p. 0
Where the paper says it

4% and improves LIBERO-Long from 61. These gains require no parameter updates, repeated VLA inference, or additional GPU resources: correction can be performed on the client CPU after a single policy invocation, and the median latencies sum to only 10.

Component ablations show that history-aware alignment and frequency-selective correction make complementary contributions. p. 6
Where the paper says it

These results show that history-aware alignment and frequency-selective correction make complementary contributions, with selective transfer becoming especially important in multi-stage execution. Figure 6: LIBERO-Long task-level success under the frozen policy, two component ablations, and full RTCF.

The largest change occurs on Task 5, where SR rises from 46. p. 5
Where the paper says it

RTCF improves Tasks 0, 2, 3, 5, 7, and 9, matches the frozen policy on Task 1, and regresses on Tasks 4, 6, and 8. The largest change occurs on Task 5, where SR rises from 46.

0% to 68.0% (+11 successes). p. 5
Where the paper says it

0 percentage points). On LIBEROLong, RTCF adds 35 successes and raises SR from 61.

Conclusion

RTCF raises aggregate success from 86.4% to 88.4%, with the largest observed gain on LIBERO-Long, from 61. p. 7

Where the paper says it

Across four LIBERO suites, RTCF raises aggregate success from 86. 4%, with the largest observed gain on LIBERO-Long, from 61.

6% to 68.6%. It uses a single policy invocation, requires no additional GPU resources, and adds 10. p. 7
Where the paper says it

4%, with the largest observed gain on LIBERO-Long, from 61. It uses a single policy invocation, requires no additional GPU resources, and adds 10.

99 ms across the two measured stages per action chunk. p. 7
Where the paper says it

It uses a single policy invocation, requires no additional GPU resources, and adds 10. 99 ms across the two measured stages per action chunk.

The ablations support both history-aware alignment and frequency-selective correction. p. 7
Where the paper says it

99 ms across the two measured stages per action chunk. The ablations support both history-aware alignment and frequency-selective correction.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Progressive Memory Alignment (PMA) enables causal alignment of growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and current aligned memory position without stage labels. This allows the system to identify the successful trajectory whose temporal evolution best explains the execution observed so far.

  2. Frequency-selective residual transfer applies a clipped low-frequency residual on motion channels derived from an aligned action chunk, while preserving higher-frequency components and gripper decisions inherited from the frozen policy proposal. This ensures that higher-frequency components and gripper decisions remain inherited from the frozen policy, maintaining reactive control structure while correcting global motion trends.

  3. The RTCF framework allows for low-overhead performance improvement for frozen VLAs by performing correction on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk. This enables deployment of sophisticated test-time memory correction without requiring parameter updates or additional GPU resources.

Abstract

Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. Successful rollouts provide useful corrective evidence, yet current frame retrieval can return progress-misaligned actions,while direct replay or time-domain fusion can overwrite the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.RTCF separates which experience to retrieve from which part of its action to transfer. Progressive Memory Alignment (PMA) causally aligns the growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and the current aligned memory position without stage labels. From the aligned action chunk,RTCF transfers a coefficient-wise-clipped low-frequency residual on motion channels. Higher-frequency components and gripper decisions remain inherited from the frozen policy. Across four LIBERO suites and 2,000 episodes per condition, RTCF raises aggregate success from 86.4% to 88.4% and improves LIBERO-Long from 61.6% to 68.6%.These gains require no parameter updates, repeated VLA inference, or additional GPU resources: correction can be performed on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk

Sources

Related papers