Retrieve in Time, Correct in Frequency
summary
The gist
Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across…
In short
Frozen vision-language-action policies struggle with long tasks due to errors accumulating over time and visual confusion between steps. This paper introduces Retrieve in Time, Correct in Frequency (RTCF), a training-free method that fixes these issues during testing. It uses memory alignment and frequency correction to improve frozen VLA performance without needing any model retraining or extra hardware.
Key concepts
- Progressive Memory Alignment (PMA)
- PMA tracks the current execution within successful past actions by maintaining a frontier of alignments between the growing visual history and completed memory. This allows the system to find relevant prior information for future steps, treating aligned memory as a reliable 'prior' instead of simply replaying old actions.
- Frequency Correction Strategy
- This strategy treats the retrieved prior action as a low-frequency motion guide. It uses a binary mask to select only motion dimensions and low frequencies, excluding noise and high frequencies. A residual transfer then applies this correction to the current action, refining it based on the retrieved guidance.
- Retrieve in Time (RIT)
- This involves locating relevant past actions by matching the current execution against stored memories using a cosine dissimilarity metric. The alignment frontier is updated causally, ensuring that each new action query contributes exactly one assignment to retrieve the most temporally relevant prior memory for correction.
- Frozen VLA Policy
- This refers to a vision-language-action policy that has been trained and cannot be updated further during testing. Because it is frozen, RTCF's test-time correction framework is necessary to overcome its inherent limitations regarding accumulated execution errors across long sequences.
Terminology used across episodes
This episode discusses
- Retrieve in Time, Correct in Frequency · Paper Radio
- HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval
- Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
- VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models · Paper Radio
- Hyper-DP3: Frequency-Aware Right-Sizing of 3D Diffusion Policies for Visuomotor Control
- ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration
- Retrieve-then-Steer: Online Success Memory for Test-Time Adaptation of Generative VLAs
The paper
Retrieve in Time, Correct in Frequency · Read on arXiv
Yuze Fan, Yue Cao, Pengjie Gao, Haojia Gao, Guangqiu Guo, Ziyue Zhang, Junbo Tan, Bokui Chen, Ïa Zhuo Zou, Ïa Xueqian Wang
Research Institute of Tsinghua University in Shenzhen · Tongji University · Fudan University
Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages. Successful rollouts provide useful corrective evidence, yet current frame retrieval can return progress-misaligned actions,while direct replay or time-domain fusion can overwrite the reactive structure of the policy proposal. We introduce Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.RTCF separates which experience to retrieve from which part of its action to transfer. Progressive Memory Alignment (PMA) causally aligns the growing visual execution history with complete successful trajectories through incrementally updated monotonic frontiers, jointly identifying a relevant memory and the current aligned memory position without stage labels. From the aligned action chunk,RTCF transfers a coefficient-wise-clipped low-frequency residual on motion channels. Higher-frequency components and gripper decisions remain inherited from the frozen policy. Across four LIBERO suites and 2,000 episodes per condition, RTCF raises aggregate success from 86.4% to 88.4% and improves LIBERO-Long from 61.6% to 68.6%.These gains require no parameter updates, repeated VLA inference, or additional GPU resources: correction can be performed on the client CPU after a single policy invocation, and the median latencies sum to only 10.99 ms per action chunk
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Retrieve in Time, Correct in Frequency".
Dev: Here's the gist of the paper. Frozen vision-language-action (VLA) policies generate temporally extended action chunks, but long-horizon manipulation remains vulnerable to accumulated execution error and visual aliasing across task stages.
Rosa: This paper introduces Retrieve in Time, Correct in Frequency (RTCF), a training-free test-time correction framework that improves frozen VLA performance with low model-side overhead.
Dev: Framework Overview RTCF separates memory reuse into progress-aware retrieval and frequency-domain correction.
Rosa: The framework decouples progress-aware memory retrieval from frequency-domain action correction.
Paper summary: Rosa: We’re wrapping up with this discussion on Retrieve in Time Correct in Frequency by Yuze Fan and Yue Cao et al. The authors are working at the Research Institute of Tsinghua University and Everwise-Tech Co., Ltd. Tongji University, and Fudan University.
Dev: What they’ve shown is a test-time correction framework that improves frozen VLA performance with low overhead by separating memory retrieval from frequency domain action correction.
Taro: The big picture here is that you can use powerful frozen policies for long horizon tasks, but this method gives you a way to fix the accumulated errors and visual aliasing during actual execution without needing any retraining.
Rosa: The key contribution is the Progressive Memory Alignment mechanism which causally aligns the history with successful trajectories to find an aligned memory position that indexes a future action chunk as a prior.
Dev: It transforms that memory chunk into a low-frequency motion prior and applies a bounded low-frequency residual transfer in coefficient space, using a correction mask for motion dimensions.
Taro: The implications are that this could be useful for deploying complex VLA systems where you need to adapt to the specific environment you're in at deployment time, not just during training.
Rosa: They showed aggregate success rising from eighty-six point four percent to eighty-eight point four percent, with the largest gain being seven point zero percentage points on LIBERO Long.
Dev: The paper uses a single policy invocation and adds about ten point nine nine milliseconds across the two measured stages per action chunk.
Taro: It’s definitely a test-time adaptation method for robot policies that reuses successful deployment experience to adapt the frozen generative VLA at test time, injecting an aggregated action prior into an intermediate state of the generative sampler.
Rosa: It gives us a practical way to get better performance when dealing with long horizon manipulation challenges where accumulated error is a big issue.
Conclusion: Rosa: So, to wrap up: this paper is about taking those frozen vision language action policies that we've been training and giving them a way to fix their mistakes during actual execution without retraining them at all.
Dev: Right, it’s called Retrieve in Time Correct in Frequency. The authors are from Tsinghua University and other places, they’re focused on improving how these models handle long sequences of actions.
Taro: It sounds like they're tackling that problem of accumulated error and visual aliasing when the robot is actually doing the task versus when it's just training in simulation.
Rosa: Exactly. They’ve figured out a way to separate how they retrieve information from how they correct those visual errors, which keeps the model overhead pretty low during testing.
Dev: The core idea is this Progressive Memory Alignment mechanism, where they track successful past executions to find an aligned memory position that points to a future action chunk as a prior.
Taro: So instead of just replaying old actions from scratch, it’s using the history to guide the current step by giving it a low-frequency motion prior.
Rosa: That prior is treated like a low-frequency signal, and they use it to transfer a bounded residual correction directly into the action coefficients.
Dev: They use this binary mask to specifically target motion dimensions, excluding higher frequencies or non-motion parts, which keeps the correction focused on what matters for movement.
Taro: It seems like a very practical way to handle those long-horizon issues where drift happens over many steps in the physical world.
Rosa: They showed that this test-time correction boosts the frozen policy's overall success rate, moving it from eighty-six point four percent up to eighty-eight point four percent across several benchmark tasks.
Dev: And they pointed out that this gain is real, especially on tasks like LIBERO Long, where they saw a seven point zero percentage point increase.
Taro: What’s interesting is that the analysis showed both the history-aware alignment and the frequency-selective correction parts were actually necessary to get those results.
Rosa: It means you can use these powerful frozen models for deployment and still get performance bumps without needing extra GPU power or retraining them on a new dataset.
Dev: It requires just one policy invocation during testing, adding about ten point nine nine milliseconds across the two correction stages per action chunk.
Taro: So it’s not some massive model overhaul; it’s a clever way to inject context into an already trained system at deployment time.
Rosa: This framework suggests we can get much better real-world performance from these frozen vision language action policies than we thought possible.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies