VLM-Guided Experience Replay
cs.LG, cs.AI
Submitted: 2026-02-02
Updated: 2026-09-27
Comments: Revised experimental results and expanded ablations and analysis
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample
Terminology
Abstract
Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 15-57% higher average success rates and improve sample efficiency by 35-55% compared to previous approaches. Project page: https://esharony.me/projects/vlm-rb/
Sources
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- TorchRL: A data-driven decision-making library for PyTorch
- Uncertainty Prioritized Experience Replay
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
- Investigating Human Priors for Playing Video Games
- FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning
- Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning
- Foundation Models for Semantic Novelty in Reinforcement Learning
- Soft Actor-Critic Algorithms and Applications
- Topological Experience Replay
- Large Batch Experience Replay
- ROER: Regularized Optimal Experience Replay
- Code as Policies: Language Model Programs for Embodied Control
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models
- Playing Atari with Deep Reinforcement Learning
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
- Prioritized Experience Replay
- The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
- VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks