Action Shaping: Policies Absorb What They Can Express
cs.AI, cs.LG
Submitted: 2026-09-26
Updated: 2026-10-06
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy.
Terminology
Abstract
Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.
Sources
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies
- An Agency-Transferring Model-Free Policy Enhancement Technique
- The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models
- Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correction
- SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training
- Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation
- Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers
- Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs
- Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation
- SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
- What Makes Value Learning Efficient in Residual Reinforcement Learning?
- ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback
- Constrained Exploration and Recovery from Experience Shaping
- Data-Free Pruning of Self-Attention Layers in LLMs
- Proximal Policy Optimization Algorithms
- PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning
- Residual Policy Learning
- DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies
- APEX: Action Priors Enable Efficient Exploration for Robust Motion Tracking on Legged Robots
- One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems