Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents
cs.AI, cs.CL
Submitted: 2026-10-09
Updated: 2026-10-09
Code: https://github.com/whi497/ReTRAIL
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories.
Abstract
Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions. We introduce ReTRAIL, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration. The environment state moves backward while the agent's knowledge continues to move forward. ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO compares alternative continuations from revisited states to improve execution, while a branch-advancement reward guides reflection through the progress of subsequent exploration. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind primitive provides limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration. These results demonstrate the value of coupling environment rewinding with experience reuse within an ongoing task for long-horizon task solving. The code is available at https://github.com/whi497/ReTRAIL.
Sources
- Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- Language Models that Think, Chat Better
- Test-Time Adaptation for LLM Agents via Environment Interaction
- Learning to Self-Evolve
- Learning to Self-Verify Makes Language Models Better Reasoners
- VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
- ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning
- Fundamentals of Building Autonomous LLM Agents
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- WESE: Weak Exploration to Strong Exploitation for LLM Agents
- Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts
- VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
- Meta-RL Induces Exploration in Language Agents
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoning
- Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agents
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems