Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing
PPOW reformulates drafter training as a reinforcement learning problem over speculative windows instead of individual tokens. It optimizes for inference-time acceptance behavior by using three rewards: cost-aware speedup, distribution proximity, and adaptive windowing. This approach leads to better performance in speculative decoding by aligning optimization with how the model actually accepts prefixes.