The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "The Silent Hyperparameter".
Jane: Here's the gist of the paper. The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing.
Tom: Landscape Survey The authors first survey the inference landscape to identify 200 distinct engines across various categories.
Jane: They categorize these engines into three types: Self-Hosted Inference Libraries, Managed Inference Platforms, and Aggregators and Routers.
Tom: The landscape is dominated by self-hosted libraries, which account for around 61 % of the total ecosystem.
Paper summary: Tom: We're moving on to a deeper look at "The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility." The authors first survey the inference landscape, identifying two hundred distinct engines across various categories, which they categorize into self-hosted libraries, managed inference platforms, and aggregators and routers <ref:2605.19537#pg1,first survey the inference landscape, identifying 200 distinct engines>.
Jane: That survey gave them a pretty clear picture of where things are concentrated; they found that self-hosted libraries account for about sixty-one percent of the total ecosystem being surveyed.
Lu: They also analyzed over thirty-five thousand recent machine learning publications to see how often the inference stack is reported, and their finding was that while code sharing is becoming standard, the inference backend remains a largely undocumented source of experimental variance <ref:2605.19537#pg1>.
Meng: That means even if we're all using similar model weights, there’s a huge amount of hidden variance in how those models are being tested because people aren't reporting what engine they used.
Lalam: They then presented a systematic empirical study specifically testing how these inference backends affect LLM benchmark results across five widely used engines, showing that the choice of backend alone can shift scores by up to sixteen point six percentage points and induce high rates of output disagreement.
Tom: That part is really striking because they held model weights, decoding parameters, and hardware constant while testing across multiple open-weight models and established benchmarks.
Jane: And they found that this backend-induced variance can dethrone a model or falsely elevate a weaker one; the paper stresses that a superior performance might not come from better architecture but from the specific numerical properties of the inference engine used during testing.
Lu: To explain *why* this matters, they categorize the divergence into two sources: systematic, but correctable engine defaults and optimization-induced numerical drift.
Meng: So it’s like there are built-in settings in the software that you can fix, and then there’s the math itself drifting because of how the high-performance features execute.
Lalam: The paper points out that for example, hidden prompt mutation and default penalty parameters explain the most extreme score divergences observed in Table one <ref:2605.19537#pg3>.
Tom: So, we see these specific settings as culprits when we isolate the variance; they aren't just random noise everywhere.
Jane: This makes sense because if those defaults are wrong for a specific task or model, it directly impacts how that model is measured against its peers.
Lu: The authors also looked at the landscape of inference engines, showing a distribution of two hundred surveyed engines across open-source and proprietary systems and the programming languages used in the open-source ones <ref:2605.19537#pg1>.
Meng: Seeing that breakdown helps us understand which types of software are most prevalent in the ecosystem where this variance is hiding.
Lalam: They also detailed their evaluation metrics, showing things like Benchmark Accuracy, Disagreement Rate, Length Error, Token Divergence Index, and LogProb Error to measure macro-level agreement.
Tom: Those specific metrics give us a concrete way to measure how much the engine is messing with things—for instance, the Disagreement Rate often exceeds the absolute differences in accuracy because it indicates the backend alters the model’s decision boundary.
Jane: And they looked at length error too, noting that reasoning models often lie outside of two-five tokens on average, experiencing massive shifts in their generation length that change their whole thought process.
Lu: The study also showed that these effects persist even under batched generation and across different GPU architectures during the ablation studies.
Meng: It’s good to know that we can't just assume this variance disappears when we start running things in a real production environment with batching involved.
Lalam: So, in short, the paper is demonstrating that inference backends are an active and influential component of the LLM evaluation pipeline.
Conclusion: Tom: To wrap up "The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility," the authors really hammered home that backend-induced numerical differences can propagate into divergent generations and benchmark score shifts large enough to affect model rankings.
Jane: So, in simple terms, the paper is arguing that if you want reliable AI comparisons, you have to treat the inference engine choice as a primary variable in your experiment.
Lu: The authors point out that these differences are driven by a combination of hidden engine defaults and compounding numerical drift caused by essential high-performance features.
Meng: This means we can’t just hope that optimizing for speed will automatically translate into perfect, unbiased evaluation results without careful monitoring.
Lalam: They conclude by advocating for standardized reporting so researchers publish exact environment specifications or at minimum, the specific backend library and version used to ensure better reproducibility and interpretability.
Tom: It's a call to action for the research community: treat the inference backend as a first-class experimental parameter for scientific validation.
Jane: That seems like a fair way to frame it—it shifts the focus from just model weights to including the environment that runs them.
CISPA Helmholtz Center for Information Security
cs.LG
Submitted: 2026-05-19
Updated: 2026-11-18
Code: https://github.com/ggml-org/llama.cpp
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline…
Key concepts
- Inference Backend
- The specific software or library used to run a Large Language Model (LLM) during the generation process. Different backends use different internal optimizations, leading to variations in how the model processes input and produces output.
- Numerical Drift
- Subtle, compounding errors that accumulate during computation due to high-performance features like prefix caching or custom kernels. This drift causes the generated text or scores to slowly deviate from a reference version over time.
- Systematic Engine Defaults
- Predefined settings within an inference engine that are correct but can cause significant differences between backends. Examples include default penalty parameters and hidden prompt mutation, which explain some of the most extreme observed divergences.
- Disagreement Rate
- A metric measuring how often a specific backend's prediction differs from a reference prediction for the same input, regardless of whether the ground truth is correct. High disagreement suggests the backend fundamentally changes the model's decision boundary.
Terminology
Summary
The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. p. 0 By isolating backend optimizations and tracing the execution
pipeline, we find this divergence is driven by system-level optimizations like prefix
caching and CUDA graphs, custom kernels, and engine-specific defaults in logit
processing. Our findings identify the inference backend as a previously unreported
but consequential hyperparameter in the evaluation of LLM and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability
of benchmark comparisons.Where the paper says it
Landscape Survey
The authors first survey the inference landscape to identify 200 distinct engines across various categories. They categorize these engines into three types: Self-Hosted Inference Libraries, Managed Inference Platforms, and Aggregators and Routers. p. 2 Self-Hosted
Libraries
Managed
Platforms
Aggregators/
Routers
0
20
40
60
80
100
120
Number of Engines
121
51
28
Open Source
Proprietary
(a) Engine Categories
Python
(47. 0%)
C/C++
(26. Figure 1a illustrates
the distribution of these systems. We find that the landscape is dominated by self-hosted libraries,
which account for around 61 % of the total ecosystem.Where the paper says it
Where the paper says it
Controlled Evaluation
The study presents a systematic empirical study of how inference backends affect LLM benchmark results across five widely used engines. They show that the choice of backend alone can shift benchmark scores by up to 16. p. 0 cpp, across multiple open-weight
models and established benchmarks. We show that the choice of backend alone can
shift benchmark scores by up to 16. We show that the choice of backend alone can
shift benchmark scores by up to 16. 6 percentage points and induce high rates of
output disagreement. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference
engines, including vLLM, SGLang, and llama. cpp, across multiple open-weight
models and established benchmarks.Where the paper says it
Where the paper says it
Where the paper says it
Root Cause Analysis
The divergence is driven by two primary sources when isolating specific optimizations and tracing the execution pipeline. These causes are categorized into systematic, but correctable, engine defaults and optimization-induced numerical drift. p. 5 1 8B and DeepSeek R1 on GSM8K. We categorize root causes into two distinct
groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed
in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical
execution of the model (detailed ablation results are provided in Appendix D). We categorize root causes into two distinct
groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed
in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical
execution of the model (detailed ablation results are provided in Appendix D). 1 Systematic Engine Defaults
The most extreme divergences, such as DeepSeek R1 scoring only 61.Where the paper says it
Where the paper says it
Reproducibility in ML Research
The authors analyze over 35,000 recent publications from top machine learning venues to quantify how frequently the inference stack is reported. The survey reveals that while code sharing is becoming standard, the inference backend remains a largely undocumented source of experimental variance. p. 7 To ensure the reliability of our automated pipeline, we conducted a manual verification on a random subset of papers at each filtering and classification stage (details in
Appendix F. Ultimately, these findings confirm that while code sharing is becoming standard, the inference
backend remains a largely undocumented source of experimental variance. For papers without code artifacts, we analyzed how frequently the
inference stacks were disclosed. Among the 2,257 papers offering no code, 820 (36 %) explicitly
2Using the vLLM backend (version 0.Where the paper says it
Where the paper says it
Recommendations
The paper advocates for standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons. Researchers are advised to publish exact environment specifications or, at minimum, the specific backend library and version. p. 8 Reporting Standards: We advocate for
publishing exact environment specifications (e. , Docker containers), or at a minimum, the specific
backend library and version. We recommend averaging results across multiple seeds. Ensure
Fair Comparisons: If the inference backend of a state-of-the-art method cannot be determined due
to missing documentation, the experiments should be rerun in a comparable setting, using the same
inference backend for all experiments. Deterministic Modes: We encourage developers to implement “strict reproducibility” flags. While
high-performance non-deterministic kernels are essential for production, a slower, deterministic
execution path is necessary for scientific debugging and validation.Where the paper says it
Where the paper says it
Where the paper says it
Limitations
The controlled environment used in the study does not fully reflect production settings, meaning the observed variance may represent a lower bound. While greedy decoding was used to minimize sampling randomness, this strategy is not optimal for all architectures and may skew metrics for reasoning models. p. 8 We utilized greedy decoding to minimize sampling randomness. However, this strategy is not optimal
for all architectures, particularly reasoning models (e. However, in this paper, we were interested in the relative
differences of runs with varying inference backends. Finally, ablating every custom kernel is infeasible, and fully disabling these features to achieve perfect
determinism is often impossible without abandoning the engine entirely.Where the paper says it
Where the paper says it
Conclusion
Inference backends are an active and influential component of the LLM evaluation pipeline. Backend-induced numerical differences can propagate into divergent generations and benchmark score shifts large enough to affect model rankings. p. 8 9 Conclusion
Our results reveal that inference backends are not a benign implementation detail, but an active and
influential component of the LLM evaluation pipeline. Across models, benchmarks, and metrics, we
find that backend-induced numerical differences can propagate into divergent generations, altered
decision boundaries, and benchmark score shifts large enough to affect model rankings. These effects
are particularly pronounced for reasoning-oriented models, where early token-level divergence can
fundamentally reshape the generated chain of thought. We demonstrate that this behavior is driven
by a combination of hidden engine defaults and compounding numerical drift caused by essential
high-performance features. While the community has developed careful conventions around datasets, decoding strategies, and
random seeds, the inference stack itself remains largely invisible, even though it can dominate other
sources of variance. To address this gap, we advocate treating the inference backend as a first-class
experimental parameter: explicitly reporting backend implementations, repeated evaluations, and
supporting deterministic execution modes for scientific validation.Where the paper says it
Where the paper says it
Where the paper says it
Acknowledgments and Disclosure of Funding
This work was supported by the Helmholtz Association’s Initiative and Networking Fund on the HAICORE@FZJ partition and by the German Federal Ministry of Education and Research under the grant AIgenCY (16KIS2012) and SisWiss (16KIS2330). Moreover, this work was supported by the LCIS center VW-Vorab-2025, ZN4704 11-76251-2055, as well as the Daimler and Benz Foundation under the grant Ladenburger Kolleg, Project KonCheck. p. 9 Acknowledgments and Disclosure of Funding
This work was supported by the Helmholtz Association’s Initiative and Networking Fund on the
HAICORE@FZJ partition and by the German Federal Ministry of Education and Research under the
grant AIgenCY (16KIS2012) and SisWiss (16KIS2330). Moreover, this work was supported by the
LCIS center VW-Vorab-2025, ZN4704 11-76251-2055, as well as the Daimler and Benz Foundation
under the grant Ladenburger Kolleg, Project KonCheck.Where the paper says it
References
The paper cites a broad literature on LLM inference and reproducibility. It extends this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance. p. 1 Recent studies highlight varying LLM results due to floatingpoint non-associativity [8], ambiguous semantic benchmarks [9], and model versioning [10]. We
extend this literature by isolating the inference engine itself as a key, previously undocumented source
of evaluation variance. Recent studies highlight varying LLM results due to floatingpoint non-associativity [8], ambiguous semantic benchmarks [9], and model versioning [10]. We
extend this literature by isolating the inference engine itself as a key, previously undocumented source
of evaluation variance. Assessing inference performance and determinism. Prior work evaluating inference engines
primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency
across various platforms [11–13]. Assessing inference performance and determinism. Prior work evaluating inference engines
primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency
across various platforms [11–13]. Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent
randomness. We build on these performance-centric studies to quantify how bare backend design
choices fundamentally alter generation trajectories. Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent
randomness. We build on these performance-centric studies to quantify how bare backend design
choices fundamentally alter generation trajectories.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A Backend Versions
Table 2 details the specific versions of the inference backends and reference libraries utilized throughout all controlled experiments in Section 5. This section provides detailed visualizations and breakdowns for the evaluation metrics introduced in Section 5. p. 3 We selected a set of five inference engines based on the popularity metrics from
our landscape survey (cf. Section 3), namely: vLLM [1], SGLang [2], llama. 9 Conclusion
Our results reveal that inference backends are not a benign implementation detail, but an active and
influential component of the LLM evaluation pipeline. Across models, benchmarks, and metrics, we
find that backend-induced numerical differences can propagate into divergent generations, altered
decision boundaries, and benchmark score shifts large enough to affect model rankings.Where the paper says it
Where the paper says it
B Additional Metrics
This section provides detailed visualizations and breakdowns for the evaluation metrics introduced in Section 5. It includes metrics like Benchmark Accuracy, Disagreement Rate, and Length Error to measure macro-level agreement. p. 3 12) on a single
NVIDIA H100 GPU using fixed software versions for all backends (Appendix A). 1 Outcome Consistency
To measure macro-level agreement, we evaluate Benchmark Accuracy, Disagreement Rate, and
Length Error. Benchmark Accuracy (Table 1). Our results indicate that the inference engine is a significant
source of variance, with accuracy discrepancies often exceeding several percentage points, enough
to alter leaderboard ranking.Where the paper says it
Where the paper says it
B.1 Disagreement Rate
The Disagreement Rate is defined as the frequency with which a backend’s prediction y differs from the reference yref for the same input regardless of ground-truth correctness. In many cases, we observe that disagreement rates consistently exceed the absolute differences in accuracy, indicating that the backend alters the model’s decision boundary. p. 3 We define this as the frequency with which a
backend’s prediction y differs from the reference yref for the same input, regardless of ground-truth
correctness: D =
1
N
P1(y ̸= yref ). While in some settings the disagreement is small, in many cases
we observe that disagreement rates consistently exceed the absolute differences in accuracy, indicating
that the backend alters the model’s decision boundary.Where the paper says it
B.2 Length Error
The authors analyze structural deviations in generated responses by measuring Output Length consistency via Signed Difference and Absolute Difference. Reasoning models frequently lie outside the ideal zone of ±25 tokens, experiencing massive shifts in generation length that fundamentally alter their chain-of-thought process. p. 4 Even more distinct are the differences for
reasoning models where the length differs significantly across all the tested datasets. For instance, the
Ollama backend produces DeepSeek R1 outputs that are, on average, over 9,000 tokens shorter than
the reference on GPQA, fundamentally altering the chain-of-thought process.Where the paper says it
B.3 Token Divergence
The Token-Level Divergence Index is calculated to pinpoint when the generations begin to differ. Reasoning models consistently diverge from the reference generation much earlier than standard architectures, particularly on difficult benchmarks like GPQA. p. 4 3), we observe that reasoning models consistently diverge from the reference
generation, much earlier than standard architectures. This is amplified on difficult benchmarks like
GPQA, where, for example, Llama 3.Where the paper says it
B.4 LogProb Error
The LogProb RMSE quantifies the floating-point variance of the top1 token calculated on the matching prefix before divergence. Reasoning models systematically exhibit higher drift and specific configurations display distinct error spikes in this metric. p. 4 While most
backends maintain similar precision with the reference (RMSE<0. 01), we observe that reasoning
models systematically exhibit higher drift and specific configurations display distinct error spikes.Where the paper says it
B.5 Top-5 Token Jaccard Similarity
The Top-K Jaccard Similarity measures the overlap of the top-5 candidate tokens between the backend and the reference implementation. Lower values indicate that the backend’s numerical deviations fundamentally alter the model’s candidate pool, bringing entirely different tokens into the top-5 predictions. p. 6 Instead, slight numerical drifts caused by
custom kernels compound continuously across the model’s depth, eventually cascading into different
Top-1 token predictions. 7 Inference Reproducibility in ML Research
Given the substantial impact of backend choice on benchmark performance, we conducted a systematic survey of recent ML publications to contextualize this variance and measure the prevalence of
inference stack reporting.Where the paper says it
C Ablation Studies
The authors conducted four targeted ablation studies to validate robustness and address real-world implications. These studies include testing safety vulnerabilities, batching effects, hardware independence, and stochastic sampling. p. 5 Evaluating batched generation (batch size = 4) revealed that performance
differences persist, and slight numerical shifts within vLLM and SGLang verify that batching actively
influences generation. Hardware Independence (Section C. Evaluating on NVIDIA L40 GPUs yielded consistent,
slightly increased backend-induced variance (up to 17. 2% Max-Min difference), confirming divergence is rooted in software implementations rather than specific GPU architectures.Where the paper says it
Where the paper says it
D Root Cause Analysis
The massive divergences are the direct result of specific system-level design choices. These causes are broadly categorized into Systematic Engine Defaults, which are correctable configurations, and Optimization-Induced Numerical Variance, which is tied to underlying mathematical execution. p. 5 1 8B and DeepSeek R1 on GSM8K. We categorize root causes into two distinct
groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed
in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical
execution of the model (detailed ablation results are provided in Appendix D).Where the paper says it
E Paper Survey Methodology and Artifacts
The authors employed a multi-stage automated extraction pipeline utilizing an LLM-as-a-judge to survey 35,000 papers. This pipeline involved keyword pre-filtering, LLM prompting with Chain-of-Thought, and structured output parsing for programmatic routing. p. 6 Filtering and Extraction Pipeline. A keyword pre-filter first flagged potentially relevant papers. For papers without code artifacts, we analyzed how frequently the
inference stacks were disclosed. Among the 2,257 papers offering no code, 820 (36 %) explicitly
2Using the vLLM backend (version 0.Where the paper says it
Where the paper says it
F Manual Verification of Judge Results
A manual verification on 150 audited papers showed high agreement rates across all processing stages. Disagreements in engine extraction were primarily due to false positives flagging low-level kernel libraries as full inference engines. p. 6 For these
confirmed papers, we ran two parallel extraction processes using the LLM judge: a Text Analysis
scan to determine if specific inference engines were explicitly named, and a Code Analysis scan to
extract repository URLs. The full methodology, including the exact keyword list, pipeline logic, and
complete LLM prompts, is detailed in Appendix E.Where the paper says it
G Impact Statement
The primary positive impact of this work is to encourage more transparent reporting standards, ensuring that claims of This creates a dangerous “deployment gap” between research validation and production
reality, where backend-induced discrepancies can undermine not only performance claims but also
safety guarantees, potentially leading to harmful or non-compliant behavior in real-world use. While prior work has examined sources of variability such as prompt sensitivity [5], quantization [6],
and decoding strategies [7], the role of the inference backend itself has remained largely unexplored.State-of-the-Art
performance are statistically significant rather than artifacts of system optimizations. The work also has implications for AI safety and reliability, as optimization techniques can alter model behavior. p. 1Where the paper says it
H Model & Dataset Licenses
Table 8 lists the licenses for various models and datasets used in the paper, including Llama3.1 8B [22] and Qwen3-235B-A22B-Instruct-2507-AWQ [31]. p. 6 2 Results
Setup. We extracted text from PDF files using pymupdf and utilized Qwen3-235B-A22B-Instruct2507-AWQ [31] as the LLM judge. 2 Results
Setup. We extracted text from PDF files using pymupdf and utilized Qwen3-235B-A22B-Instruct2507-AWQ [31] as the LLM judge. Benchmarks. We employ four widely adopted datasets to evaluate distinct model capabilities:
GSM8K (Math) [25], GPQA Diamond (Science) [26], SimpleQA Verified (Factuality) [27], and
LiveCodeBench v6 (Code) [28]. Benchmarks. We employ four widely adopted datasets to evaluate distinct model capabilities:
GSM8K (Math) [25], GPQA Diamond (Science) [26], SimpleQA Verified (Factuality) [27], and
LiveCodeBench v6 (Code) [28].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The paper cites a broad literature on LLM inference and reproducibility. It extends this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance. Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms. The authors build on these performance-centric studies to quantify how bare backend design choices fundamentally alter generation trajectories. It also references works on prompt sensitivity, quantization, and decoding strategies as sources of variability. p. 1 This creates a dangerous “deployment gap” between research validation and production
reality, where backend-induced discrepancies can undermine not only performance claims but also
safety guarantees, potentially leading to harmful or non-compliant behavior in real-world use. While prior work has examined sources of variability such as prompt sensitivity [5], quantization [6],
and decoding strategies [7], the role of the inference backend itself has remained largely unexplored.Where the paper says it
Assessing inference performance and determinism
Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms. Other research analyzes the impact of general system design, such as caching and decoding strategies. p. 1 Prior work evaluating inference engines
primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency
across various platforms [11–13]. Other research analyzes the impact of general system design, such
as caching and decoding strategies [14], expanding earlier findings on CNNs and RNNs [15], or
evaluates differences between plain quantization formats [16]. Other research analyzes the impact of general system design, such
as caching and decoding strategies [14], expanding earlier findings on CNNs and RNNs [15], or
evaluates differences between plain quantization formats [16]. Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent
randomness.Where the paper says it
Where the paper says it
Model & Dataset Licenses
Table 8 lists the licenses for various models and datasets used in the paper, including Llama3.1 8B [22] and Qwen3-235B-A22B-Instruct-2507-AWQ [31]. The licenses for GSM8K and LiveCodeBench v6 are also noted.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Standardize reporting by explicitly stating backend implementations for every benchmark run to ensure scientific reproducibility, as
the choice of inference backend alone can shift benchmark scores by up to 16 percentage points.
-
Implement rigorous testing protocols that account for non-determinism, such as averaging results across twelve unique seeds to mitigate variance caused by
optimization-induced numerical drift,
which can cause a model’s output to change over time. -
Develop safety guarantees through backend awareness; since
identical model weights can exhibit noticeable distinct safety profiles when deployed on different inference engines,
deployment systems must verify the security profile of a model across all anticipated serving backends. -
Establish stricter leaderboard validation by requiring platforms to report an
Inference Stack
and quantifying uncertainty with abackend confidence interval
by testing models on both reference and high-throughput engines. -
Design deterministic execution modes for scientific validation, encouraging developers to implement mechanisms that provide
strict reproducibility,
even if it means sacrificing some high-throughput performance during debugging.
Abstract
Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying 200 distinct engines, and analyze 35,000 ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama.cpp, across multiple open-weight models spanning 4B-30B parameters and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations such as prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported yet consequential hyperparameter in LLM evaluations and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.
Sources
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
- A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency
- Hardware Acceleration for Neural Networks: A Comprehensive Survey
- Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements
- Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
- LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
- The Llama 3 Herd of Models
- Qwen3 Technical Report
- Training Verifiers to Solve Math Word Problems
- SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis