From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding
This study investigated optimizing speculative diffusion decoding (SDD) by introducing 'verifier skipping,' a lossy method to save verification costs. The research found that effective skipping depends on feasible prefixes and generation dynamics, not just token prediction. Raw confidence showed the most significant reduction in verifier calls, but no single signal dominated across quality or throughput.