Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training
cs.AI, cs.LG
Submitted: 2026-10-28
Updated: 2026-11-10
Comments: 31 pages, 6 figures, 10 tables, including references and appendices
Code: https://github.com/EdisonScientific/prime-rl-adversarial-paper
License: http://creativecommons.org/licenses/by/4.0/
The gist: In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards.
Terminology
Abstract
In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. With sufficiently capable judges and enough training, RAL matches direct RLVR. We then apply RAL to scientific abstract generation in two settings. In the first setting, RAL guides the generator to produce abstracts in the scientific domain of the reference abstracts, even though the prompt leaves that domain unspecified. In the second setting, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers.
Sources
- Escaping the Verifier: Learning to Reason via Demonstrations
- Process Reinforcement through Implicit Rewards
- Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
- Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learning
- Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Continuous control with deep reinforcement learning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Toward Diverse Text Generation with Inverse Reinforcement Learning
- Solving math word problems with process- and outcome-based feedback
- Qwen3 Technical Report
- Black-Box On-Policy Distillation of Large Language Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents