Testing Processor Contracts in Speculative Decoding Engines
cs.SE, cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/ggml-org/llama
License: http://creativecommons.org/licenses/by/4.0/
The gist: Speculative sampling can accelerate decoding without changing the target distribution.
Terminology
Abstract
Speculative sampling can accelerate decoding without changing the target distribution. Its standard guarantee assumes that logits processors, which enforce constraints or modify token scores, receive the appropriate token prefix and state at each verification position. We formalize this processor-prefix contract and present BADC (branch-aware differential conformance testing). BADC combines enumerated references on tiny random models, checks of processor inputs and state, and simultaneous confidence bounds on output-event probabilities. Preregistered, source-informed experiments on selected Transformers and llama-cpp-python releases confirm two causes: processors read unverified drafts or advance state without rollback. For one pilot-selected prompt on a trained checkpoint, fresh samples give a 60.26-percentage-point lower bound on the difference in the probability of a predefined first-token event between ordinary and prompt-lookup decoding, with nominal simultaneous 95% coverage under within-arm i.i.d. sampling. Controlled comparisons establish no cause-recovery advantage of directed selection over random selection, while permutation testing detects each tested family at least as often as BADC. These findings connect processor-interface failures to observable decoding differences without establishing diagnostic superiority or workload prevalence.
Sources
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Accelerating Large Language Model Decoding with Speculative Sampling
- DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
- Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
- Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
- A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
- An Empirical Study of Speculative Decoding on Software Engineering Tasks
- A First Look at Bugs in LLM Inference Engines
- Total Variation Distance Estimation in Autoregressive Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Inference with Reference: Lossless Acceleration of Large Language Models
- Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters