Retrieve in Time, Correct in Frequency
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Retrieve in Time, Correct in Frequency".
Dev: Here's the gist of the paper. Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages.
Rosa: This paper introduces Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.
Dev: Framework Overview RTCF separates memory reuse into progress-aware retrieval and frequency-domain correction.
Rosa: The framework decouples progress-aware memory retrieval from frequency-domain action correction.
Paper summary: Rosa: We’re wrapping up with this discussion on Retrieve in Time Correct in Frequency by Yuze Fan and Yue Cao et al. The authors are working at the Research Institute of Tsinghua University and Everwise-Tech Co., Ltd. Tongji University, and Fudan University.
Dev: What they’ve shown is a test-time correction framework that improves frozen VLA performance with low overhead by separating memory retrieval from frequency domain action correction.
Taro: The big picture here is that you can use powerful frozen policies for long horizon tasks, but this method gives you a way to fix the accumulated errors and visual aliasing during actual execution without needing any retraining.
Rosa: The key contribution is the Progressive Memory Alignment mechanism which causally aligns the history with successful trajectories to find an aligned memory position that indexes a future action chunk as a prior.
Dev: It transforms that memory chunk into a low-frequency motion prior and applies a bounded low-frequency residual transfer in coefficient space, using a correction mask for motion dimensions.
Taro: The implications are that this could be useful for deploying complex VLA systems where you need to adapt to the specific environment you're in at deployment time, not just during training.
Rosa: They showed aggregate success rising from eighty-six point four percent to eighty-eight point four percent, with the largest gain being seven point zero percentage points on LIBERO Long.
Dev: The paper uses a single policy invocation and adds about ten point nine nine milliseconds across the two measured stages per action chunk.
Taro: It’s definitely a test-time adaptation method for robot policies that reuses successful deployment experience to adapt the frozen generative VLA at test time, injecting an aggregated action prior into an intermediate state of the generative sampler.
Rosa: It gives us a practical way to get better performance when dealing with long horizon manipulation challenges where accumulated error is a big issue.
Conclusion: Rosa: So, to wrap up: this paper is about taking those frozen vision language action policies that we've been training and giving them a way to fix their mistakes during actual execution without retraining them at all.
Dev: Right, it’s called Retrieve in Time Correct in Frequency. The authors are from Tsinghua University and other places, they’re focused on improving how these models handle long sequences of actions.
Taro: It sounds like they're tackling that problem of accumulated error and visual aliasing when the robot is actually doing the task versus when it's just training in simulation.
Rosa: Exactly. They’ve figured out a way to separate how they retrieve information from how they correct those visual errors, which keeps the model overhead pretty low during testing.
Dev: The core idea is this Progressive Memory Alignment mechanism, where they track successful past executions to find an aligned memory position that points to a future action chunk as a prior.
Taro: So instead of just replaying old actions from scratch, it’s using the history to guide the current step by giving it a low-frequency motion prior.
Rosa: That prior is treated like a low-frequency signal, and they use it to transfer a bounded residual correction directly into the action coefficients.
Dev: They use this binary mask to specifically target motion dimensions, excluding higher frequencies or non-motion parts, which keeps the correction focused on what matters for movement.
Taro: It seems like a very practical way to handle those long-horizon issues where drift happens over many steps in the physical world.
Rosa: They showed that this test-time correction boosts the frozen policy's overall success rate, moving it from eighty-six point four percent up to eighty-eight point four percent across several benchmark tasks.
Dev: And they pointed out that this gain is real, especially on tasks like LIBERO Long, where they saw a seven point zero percentage point increase.
Taro: What’s interesting is that the analysis showed both the history-aware alignment and the frequency-selective correction parts were actually necessary to get those results.
Rosa: It means you can use these powerful frozen models for deployment and still get performance bumps without needing extra GPU power or retraining them on a new dataset.
Dev: It requires just one policy invocation during testing, adding about ten point nine nine milliseconds across the two correction stages per action chunk.
Taro: So it’s not some massive model overhaul; it’s a clever way to inject context into an already trained system at deployment time.
Rosa: This framework suggests we can get much better real-world performance from these frozen vision language action policies than we thought possible.
Yuze Fan, Yue Cao, Pengjie Gao, Haojia Gao, Guangqiu Guo, Ziyue Zhang, Junbo Tan, Bokui Chen, Ïa Zhuo Zou, Ïa Xueqian Wang
Research Institute of Tsinghua University in Shenzhen · Tongji University · Fudan University
cs.RO
Submitted: 2026-08-05
Updated: 2026-10-28
Comments: The current version of this manuscript is preliminary, and the experimental evidence is not yet sufficient to fully support its main conclusions. Substantial additional experiments and validation are required to establish the reliability and generalizability of the reported findings. We therefore request withdrawal of the current version
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across…
Key concepts
- Progressive Memory Alignment (PMA)
- PMA tracks the current execution within successful past actions by maintaining a frontier of alignments between the growing visual history and completed memory. This allows the system to find relevant prior information for future steps, treating aligned memory as a reliable 'prior' instead of simply replaying old actions.
- Frequency Correction Strategy
- This strategy treats the retrieved prior action as a low-frequency motion guide. It uses a binary mask to select only motion dimensions and low frequencies, excluding noise and high frequencies. A residual transfer then applies this correction to the current action, refining it based on the retrieved guidance.
- Retrieve in Time (RIT)
- This involves locating relevant past actions by matching the current execution against stored memories using a cosine dissimilarity metric. The alignment frontier is updated causally, ensuring that each new action query contributes exactly one assignment to retrieve the most temporally relevant prior memory for correction.
- Frozen VLA Policy
- This refers to a vision-language-action policy that has been trained and cannot be updated further during testing. Because it is frozen, RTCF's test-time correction framework is necessary to overcome its inherent limitations regarding accumulated execution errors across long sequences.
Terminology
Summary
Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. p. 0 Retrieve in Time, Correct in Frequency
Yuze Fan2,3 Yue Cao1,2,4 Pengjie Gao5 Haojia Gao1 Guangqiu Guo1,2,5
Ziyue Zhang1 Junbo Tan1 Bokui Chen5 Zhuo Zou4 Xueqian Wang5
1Research Institute of Tsinghua University in Shenzhen 2Everwise-Tech Co. 3Tongji University 4Fudan University
5Tsinghua Shenzhen International Graduate School, Tsinghua University
Abstract
Frozen vision-language-action (VLA) policies generate
temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution
error and visual aliasing across task stages. Successful
rollouts provide useful corrective evidence, yet currentframe retrieval can return progress-misaligned actions,
while direct replay or time-domain fusion can overwrite
the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a
training-free test-time correction framework that improves
frozen VLA performance with low model-side overhead.Where the paper says it
Where the paper says it
Framework Overview
RTCF separates memory reuse into progress-aware retrieval and frequency-domain correction. The framework decouples progress-aware memory retrieval from frequency-domain action correction. p. 1 We introduce Retrieve in Time, Correct in Frequency
(RTCF), a training-free test-time correction framework
that improves frozen VLA performance with low overhead. RTCF decouples progress-aware memory retrieval from
frequency-domain action correction. RTCF decouples progress-aware memory retrieval from
frequency-domain action correction. Its online retrieval
mechanism, Progressive Memory Alignment (PMA), maintains a frontier over monotonic alignments between the
growing visual execution history and each complete successful memory. Its online retrieval
mechanism, Progressive Memory Alignment (PMA), maintains a frontier over monotonic alignments between the
growing visual execution history and each complete successful memory. The resulting aligned memory position indexes a future action chunk, which is used as a prior rather
than replayed directly.Where the paper says it
Where the paper says it
Where the paper says it
Memory Retrieval Mechanism
PMA locates the current execution within successful trajectories by maintaining monotonic alignment frontiers. The local matching cost is defined as the cosine dissimilarity ci,t(q) = 1 − clipz⊤ t zi,q −1, 1. p. 3 Rather than greedily committing to a single
progress estimate, the frontier preserves competing monotonic hypotheses, allowing later observations to resolve
ambiguities introduced by earlier visually similar states. For an incoming descriptor zt, its local matching cost at
position q of memory i is defined as the cosine dissimilarity
ci,t(q) = 1 − clipz
⊤
t zi,q, −1, 1
, (5)
Figure 3: Progressive Memory Alignment. Let Ji,t(q) denote the minimum cumulative cost of aligning Qt to memory i while assigning its latest descriptor
zt to memory position q. We use an asymmetric causal
transition pattern: every new policy invocation advances
the query by exactly one step, while the corresponding
memory position may remain unchanged or advance by a Unlike a symmetric DTW transition
rule, the recurrence contains no within-frontier transition:
the alignment can update its progress only when a new
observation arrives. Consequently, every query descriptor
contributes exactly one alignment assignment, which is
important for causal action retrieval.Where the paper says it
Where the paper says it
Where the paper says it
Frequency Correction Strategy
The aligned prior is treated as a low-frequency motion prior and applied via a bounded low-frequency residual transfer. The binary correction mask Gf,d = I[0 < f < Fc]I[d ∈ Imotion] excludes the DC term, higher frequencies, and non-motion dimensions. p. 4 We
define the binary correction mask
Gf,d = I[0 < f < Fc]I[d ∈ Imotion], (10)
where Fc is the temporal-frequency cutoff and Imotion contains the motion-channel indices. The mask excludes the
DC term, higher frequencies, and non-motion dimensions. In particular, event-like gripper commands remain unchanged to avoid blurring discrete open–close transitions. We transfer a bounded residual rather than replacing
the selected coefficients:
∆Ct = G ⊙ clip(Cmem
t − Cπ
t
, −δmax, δmax),
Ccorr
t = Cπ
t + λ0∆Ct.Where the paper says it
Where the paper says it
Performance and Analysis
Across four LIBERO suites, RTCF improves the frozen policy’s aggregate success from 86.4% to 88.4%, with its largest suite-level gain of 7. p. 1 Across four LIBERO suites, RTCF improves the frozen
policy’s aggregate success from 86. 4%, with
its largest suite-level gain of 7. 4%, with
its largest suite-level gain of 7. 0 percentage points on
LIBERO-Long. 4% and improves LIBERO-Long from 61. These gains require no parameter updates, repeated VLA
inference, or additional GPU resources: correction can be
performed on the client CPU after a single policy invocation, and the median latencies sum to only 10. These results show that history-aware
alignment and frequency-selective correction make complementary contributions, with selective transfer becoming
especially important in multi-stage execution. Figure 6: LIBERO-Long task-level success under the frozen
policy, two component ablations, and full RTCF. RTCF improves Tasks 0, 2, 3, 5, 7, and 9, matches the
frozen policy on Task 1, and regresses on Tasks 4, 6, and 8. The largest change occurs on Task 5, where SR rises from
46. 0 percentage points). On LIBEROLong, RTCF adds 35 successes and raises SR from 61.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
RTCF raises aggregate success from 86.4% to 88.4%, with the largest observed gain on LIBERO-Long, from 61. p. 7 Across four LIBERO suites, RTCF raises aggregate success from 86. 4%, with the largest observed gain
on LIBERO-Long, from 61. 4%, with the largest observed gain
on LIBERO-Long, from 61. It uses a single
policy invocation, requires no additional GPU resources,
and adds 10. It uses a single
policy invocation, requires no additional GPU resources,
and adds 10. 99 ms across the two measured stages per
action chunk. 99 ms across the two measured stages per
action chunk. The ablations support both history-aware
alignment and frequency-selective correction.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Progressive Memory Alignment (PMA) enables causal alignment of growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and current aligned memory position without stage labels. This allows the system to identify
the successful trajectory whose temporal evolution best explains the execution observed so far.
-
Frequency-selective residual transfer applies a clipped low-frequency residual on motion channels derived from an aligned action chunk, while preserving higher-frequency components and gripper decisions inherited from the frozen policy proposal. This ensures that
higher-frequency components and gripper decisions remain inherited from the frozen policy,
maintaining reactive control structure while correcting global motion trends. -
The RTCF framework allows for low-overhead performance improvement for frozen VLAs by performing correction
on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk.
This enables deployment of sophisticated test-time memory correction without requiring parameter updates or additional GPU resources.
Abstract
Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. Successful rollouts provide useful corrective evidence, yet current frame retrieval can return progress-misaligned actions,while direct replay or time-domain fusion can overwrite the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.RTCF separates which experience to retrieve from which part of its action to transfer. Progressive Memory Alignment (PMA) causally aligns the growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and the current aligned memory position without stage labels. From the aligned action chunk,RTCF transfers a coefficient-wise-clipped low-frequency residual on motion channels. Higher-frequency components and gripper decisions remain inherited from the frozen policy. Across four LIBERO suites and 2,000 episodes per condition, RTCF raises aggregate success from 86.4% to 88.4% and improves LIBERO-Long from 61.6% to 68.6%.These gains require no parameter updates, repeated VLA inference, or additional GPU resources: correction can be performed on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk
Sources
- HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval
- Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
- VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models
- Hyper-DP3: Frequency-Aware Right-Sizing of 3D Diffusion Policies for Visuomotor Control
- ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration
- Retrieve-then-Steer: Online Success Memory for Test-Time Adaptation of Generative VLAs
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation