Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of- K Objective
Melveena Jolly, Midhun Xavier
cs.LG, stat.ME, stat.ML
Submitted: 2026-07-13
Code: https://github.com/midhunxavier/rank-conditioned-sample-reuse
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study the coupled objective J K WOR = E S PL-WOR K[max i in S R i]: the expected maximum reward of a size-K Plackett-Luce draw without replacement, the law of Gumbel-Top-K / Stochastic Beam Search
Terminology
Abstract
We study the coupled objective J K WOR = E S PL-WOR K[max i in S R i]: the expected maximum reward of a size-K Plackett-Luce draw without replacement, the law of Gumbel-Top-K / Stochastic Beam Search decoding. This estimand differs from the conventional i.i.d. objective J K iid = E[max i<=K R i] targeted by existing sample-reuse Max@K estimators, and reusing their i.i.d. weights under the coupled sampler is provably biased (a closed-form three-item instance gives E[g iid] = (4/5) grad J K WOR exactly; pass@K under the coupled sampler is the binary-reward special case). Generic joint-score REINFORCE is already unbiased for J K WOR; what it lacks is sample reuse. Our contribution is to instantiate standard rank-conditioned Horvitz-Thompson estimation for the J K WOR subset total: from one Gumbel-Top-n pool (n>K) and its observed priority threshold we build an estimator that reuses all C(n,K) embedded K-subsets, unbiased with an unbiased exact score-function surrogate gradient, plus a reward-sorted Max-specific dynamic program that collapses the C(n,K)-term subset sum (with K!-cost set probabilities) exactly to a one-dimensional integral. A fixed-Q quadrature evaluation costs O(n log n + nKQ) arithmetic and is numerically, not algebraically, exact; no epsilon-approximation rate is certified. Each nonzero degree-K Horvitz-Thompson term has finite second moment exactly when n >= 2K; under the same assumptions the full surrogate gradient has finite second moment whenever n >= 2K (sharpness there is open). At K=1 the construction recovers classical priority sampling. All quantities require only the values and differentiable computation graphs of the n+1 drawn items' probabilities, so finite structured sequence policies sampled by exact SBS are covered. A certified finite-Q quadrature bound and countably infinite support remain open. Validation code is included as ancillary files.
Sources
- Variational Best-of-N Alignment
- QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling
- Understanding R1-Zero-Like Training: A Critical Perspective
- Conditional Poisson Stochastic Beam Search
- Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
- Gradient Estimation with Stochastic Softmax Tricks
- Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients
- BOND: Aligning LLMs with Best-of-N Distillation
- Leader Reward for POMO-Based Neural Combinatorial Optimization
- Reparameterizable Subset Sampling via Continuous Relaxations
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks