Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
summary
The gist
The gist The power distribution bridges sampling, self-reward KL-regularized RL, and self-distillation as the sampling target, closed-form RL optimum, and teacher distribution.
In short
The power distribution links sampling methods, KL-regularized reinforcement learning, and self-distillation. The paper shows that sequence-level information is needed to accurately reproduce power sampling approximations at inference time. Power self-distillation uses this distribution as a target for offline training, achieving self-reward sharpening and linking true reward gains to the covariance between true and self-reward.
Key concepts
- Power Distribution
- This distribution describes how token probabilities are sharpened using a factor alpha greater than one. It is structurally difficult to reproduce using simple per-token approximations without knowing the entire sequence, suggesting it requires sequence-level context.
- KL-Regularized RL Optimum
- The power distribution is identified as the optimal outcome of KL-regularized reinforcement learning when using a specific reward signal derived from the model's sequence log-probabilities. This provides a closed-form target for policy optimization.
- Power Self-Distillation
- This is an offline training technique that uses the power distribution as its target. It acts as a surrogate for expensive power sampling by allowing supervised training on teacher samples, effectively achieving self-reward sharpening at lower inference costs.
- Reward Covariance Relationship
- The improvement in the true downstream reward depends entirely on how much the true reward covaries with the self-reward under the power distribution. This covariance term precisely characterizes whether increasing a parameter improves performance.
Terminology used across episodes
This episode discusses
- Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation · Paper Radio
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Program Synthesis with Large Language Models
- Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning · Paper Radio
- Evaluating Large Language Models Trained on Code
- Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening · Paper Radio
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Decoupled Weight Decay Regularization
- WebGPT: Browser-assisted question-answering with human feedback
- Maximizing Confidence Alone Improves Reasoning
- Spurious Rewards: Rethinking Training Signals in RLVR
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Qwen2.5 Technical Report
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
The paper
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation · Read on arXiv
Akiyoshi Tomihari, Issei Sato
Department of Computer Science, The University of Tokyo
Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time sampling, including power sampling, have emerged as effective ways to improve LLM performance. However, the relationship among RL, distillation, and sampling remains unclear. In this study, we focus on the power distribution, the target distribution of power sampling, and show that the power distribution bridges sampling, self-reward KL-regularized RL, and self-distillation. From the sampling perspective, we show that inexpensive local approximations cannot reproduce sequence-level power without information about possible suffixes. From the RL perspective, the power distribution is the closed-form optimizer of KL-regularized RL when the model's sequence-level log-probabilities are used as the reward. This identification leads to power self-distillation, an offline distillation surrogate that shares the same target distribution and amortizes the cost of power sampling into supervised training on teacher samples. We further show that power self-distillation can achieve self-reward sharpening, that downstream true-reward gains are governed by the covariance between true reward and self-reward under the power distribution, and that this ideal power-teacher gain transfers to a distilled student when the student is close to the power teacher. Experiments on reasoning tasks support our analysis: power sampling raises self-reward, true-reward gains depend on alignment with self-reward, and power self-distillation can match or exceed the performance of power sampling at much lower inference cost. The code is available at https://github.com/tom4649/power-self-distillation.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation".
Jane: Here's the gist of the paper. The gist The power distribution bridges sampling, self-reward KL-regularized RL, and self-distillation as the sampling target, closed-form RL optimum, and teacher distribution.
Tom: Connection between Sampling and Power Distribution The power distribution is shown to be structurally difficult to reproduce by inexpensive inference-time approximations because per-token approximations cannot match it without sequence-level information.
Jane: Specifically, Proposition 1 shows that the ratio of next-token odds under power sampling versus temperature scaling is controlled by the R´enyi entropy of the suffix.
Tom: This implies that sequence-level power sharpening favors continuations whose suffix distributions under π are more peaked, i.e., have lower R´enyi entropy.
Title and authors: Jane: So, who are these authors, and what does that title actually mean in plain English?
Tom: The paper is by Akiyoshi Tomihari and Issei Sato. They are from the Department of Computer Science at The University of Tokyo. Their focus here is on figuring out the relationship between how we sample text during inference, how reinforcement learning can make models smarter, and how we use distillation to transfer that knowledge.
Meng: From my side, I’m thinking about the practical implication—if this bridge exists, it means there's a way to take something that costs a lot of computation at inference time and turn it into something we can train for cheaply.
Jane: That sounds like a big deal for deployment. So, what is the core contribution they are making here?
Lu: The core idea is identifying this specific power distribution, πα(y x), as the target distribution that power sampling aims for, and simultaneously showing it’s the mathematically exact optimum when you use KL-regularized reinforcement learning with a reward based on sequence-level log-probabilities >
The paper's summary: Tom: So, to summarize what they found: they show that if you try to approximate the power distribution using just per-token approximations during inference, it’s structurally difficult because you need information about the whole sequence suffix, not just one token at a time >
Jane: That makes sense. If you only look at the next token prediction, you can't capture how likely a whole continuation is without seeing what comes after that token >
Lu: And then they show that this power distribution is also exactly what you get when you solve the KL-regularized reinforcement learning objective if your reward is defined as those sequence-level log-probabilities, which they call self-reward >
The paper's improvements: Tom: The main improvement they propose comes from linking this distribution to power self-distillation. This is an offline distillation surrogate that uses teacher samples drawn from this same target distribution to train a student policy >
Jane: So, how does this actually help the model? What are the tangible benefits of using this power self-distillation approach over just using standard distillation methods?
Lu: It achieves sharpening. Proposition three gives a sharpening bound for the distilled model, meaning you get improvements in performance on downstream tasks > <ref:2605.04542#pg2>
Conclusion: Tom: So, to wrap up, they show that this power distribution connects sampling, self-reward RL, and self-distillation by identifying it as the target of all three. This means we can convert expensive inference procedures into offline training objectives through power self-distillation >
Jane: And what about the real world impact? How does this affect how we think about improving these models for actual tasks?
Lu: The improvement in a true reward depends directly on the covariance between the true reward and the self-reward under that power distribution, which they characterize in Proposition four > <ref:2605.04542#pg2>
Tom: So, to wrap up, the paper "Power Distribution Bridges Sampling Self-Reward RL and Self-Distillation" shows that this power distribution serves as a link between sampling methods, KL-regularized reinforcement learning objectives, and self-distillation techniques. It lets us turn costly inference procedures into offline training steps through power self-distillation >
Jane: It seems like a very concrete way to think about how we can get the model to improve its own reasoning abilities without needing constant online feedback.
Lu: Exactly. And by showing that sharpening is achieved, they give us a way to measure exactly how much true reward improvement we can expect based on that covariance term >
Meng: For deployment, if power self-distillation can match the performance of power sampling at a much lower inference cost, that’s something I can actually see in terms of running costs.
Lalam: I am confident in this direction; the paper shows that by targeting this distribution, we are aligning the model's behavior with its own internal reward structure, and it achieves self-reward sharpening >
More episodes
- 2512.01384-CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression
- 2512.06208-SparsePixels: Efficient Convolution for Sparse Data on FPGAs
- 2512.11838-D-STEER: Two-Way Control of DPO-Induced Safety Behavior in Activation Space
- 2512.12508-Generative Spatiotemporal Data Augmentation
- 2512.12605-Causal inference and model explainability analysis of a real world sales dataset
- 2512.24555-From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme
- 2601.07189-Standardization of Post-Publication Code Verification is Possible with the Support of the Community
- 2601.13542-Refined Gradient-Based Temperature Optimization for the Replica-Exchange Monte-Carlo Method
- 2601.13653-TimeART: Towards Agentic Time Series Reasoning via Tool-Augmentation
- 2601.13824-ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge