From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
cs.LG
Submitted: 2026-09-24
Updated: 2026-09-24
Comments: 41 pages, 6 figures. Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Existing in-context reinforcement learning methods mainly pretrain Transformers with supervised behavior-prediction objectives.
Terminology
Abstract
Existing in-context reinforcement learning methods mainly pretrain Transformers with supervised behavior-prediction objectives. This enables task inference from context, but makes the learned policy strongly depend on the quality of offline actions: when trajectories are weak or suboptimal, imitation itself becomes a biased learning signal. We propose Q-Target Pretrained Transformers (QTPT), which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective. QTPT therefore learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy. We theoretically analyze QTPT in stochastic linear bandits and finite-horizon MDPs, showing stronger robustness to data quality than supervised pretraining. Empirically, QTPT improves over supervised behavior prediction on controlled RL benchmarks with random or suboptimal data, and we examine extensions to D4RL Kitchen and AntMaze. Supplementary experiments evaluate backbone robustness, meta-RL comparisons, task-coherent context, and unsupported-action value overestimation. These comparisons distinguish the benefits of Q-target pretraining from the remaining limitations of offline coverage.
Sources
- What learning algorithm is in-context learning? Investigations with linear models
- Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
- Language Models are Few-Shot Learners
- Training Verifiers to Solve Math Word Problems
- PASTA: Pessimistic Assortment Optimization
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Q-value Regularized Transformer for Offline Reinforcement Learning
- Is Q-learning Provably Efficient?
- Is Pessimism Provably Efficient for Offline RL?
- MOReL : Model-Based Offline Reinforcement Learning
- Offline Reinforcement Learning with Implicit Q-Learning
- Conservative Q-Learning for Offline Reinforcement Learning
- In-context Reinforcement Learning with Algorithm Distillation
- Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
- Is Value Learning Really the Main Bottleneck in Offline RL?
- Recursive Introspection: Teaching Language Model Agents How to Self-Improve
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks