Do LLMs Learn from Rewards in Context?: Rethinking the role of reward in In-Context RL
cs.LG, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-14
Comments: NeurIPS 2026 Spotlight (Negative Results Track)
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters.
Terminology
Abstract
LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Filtering Learning Histories Enhances In-Context Reinforcement Learning
- The Llama 3 Herd of Models
- In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
- Training Language Models to Self-Correct via Reinforcement Learning
- In-context Reinforcement Learning with Algorithm Distillation
- In-Context Reinforcement Learning via Communicative World Models
- In-context Learning and Induction Heads
- Reward Is Enough: LLMs Are In-Context Reinforcement Learners
- Qwen3 Technical Report
- Negation: A Pink Elephant in the Large Language Models' Room?
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- Agent Workflow Memory
- Larger language models do in-context learning differently
- From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks