The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

arXiv:2605.19537 · cs.LG · Submitted 2026-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The Silent Hyperparameter".

Jane: Here's the gist of the paper. The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing.

Tom: Landscape Survey The authors first survey the inference landscape to identify 200 distinct engines across various categories.

Jane: They categorize these engines into three types: Self-Hosted Inference Libraries, Managed Inference Platforms, and Aggregators and Routers.

Tom: The landscape is dominated by self-hosted libraries, which account for around 61 % of the total ecosystem.

Paper summary: Tom: We're moving on to a deeper look at "The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility." The authors first survey the inference landscape, identifying two hundred distinct engines across various categories, which they categorize into self-hosted libraries, managed inference platforms, and aggregators and routers <ref:2605.19537#pg1,first survey the inference landscape, identifying 200 distinct engines>.

Jane: That survey gave them a pretty clear picture of where things are concentrated; they found that self-hosted libraries account for about sixty-one percent of the total ecosystem being surveyed.

Lu: They also analyzed over thirty-five thousand recent machine learning publications to see how often the inference stack is reported, and their finding was that while code sharing is becoming standard, the inference backend remains a largely undocumented source of experimental variance <ref:2605.19537#pg1>.

Meng: That means even if we're all using similar model weights, there’s a huge amount of hidden variance in how those models are being tested because people aren't reporting what engine they used.

Lalam: They then presented a systematic empirical study specifically testing how these inference backends affect LLM benchmark results across five widely used engines, showing that the choice of backend alone can shift scores by up to sixteen point six percentage points and induce high rates of output disagreement.

Tom: That part is really striking because they held model weights, decoding parameters, and hardware constant while testing across multiple open-weight models and established benchmarks.

Jane: And they found that this backend-induced variance can dethrone a model or falsely elevate a weaker one; the paper stresses that a superior performance might not come from better architecture but from the specific numerical properties of the inference engine used during testing.

Lu: To explain *why* this matters, they categorize the divergence into two sources: systematic, but correctable engine defaults and optimization-induced numerical drift.

Meng: So it’s like there are built-in settings in the software that you can fix, and then there’s the math itself drifting because of how the high-performance features execute.

Lalam: The paper points out that for example, hidden prompt mutation and default penalty parameters explain the most extreme score divergences observed in Table one <ref:2605.19537#pg3>.

Tom: So, we see these specific settings as culprits when we isolate the variance; they aren't just random noise everywhere.

Jane: This makes sense because if those defaults are wrong for a specific task or model, it directly impacts how that model is measured against its peers.

Lu: The authors also looked at the landscape of inference engines, showing a distribution of two hundred surveyed engines across open-source and proprietary systems and the programming languages used in the open-source ones <ref:2605.19537#pg1>.

Meng: Seeing that breakdown helps us understand which types of software are most prevalent in the ecosystem where this variance is hiding.

Lalam: They also detailed their evaluation metrics, showing things like Benchmark Accuracy, Disagreement Rate, Length Error, Token Divergence Index, and LogProb Error to measure macro-level agreement.

Tom: Those specific metrics give us a concrete way to measure how much the engine is messing with things—for instance, the Disagreement Rate often exceeds the absolute differences in accuracy because it indicates the backend alters the model’s decision boundary.

Jane: And they looked at length error too, noting that reasoning models often lie outside of two-five tokens on average, experiencing massive shifts in their generation length that change their whole thought process.

Lu: The study also showed that these effects persist even under batched generation and across different GPU architectures during the ablation studies.

Meng: It’s good to know that we can't just assume this variance disappears when we start running things in a real production environment with batching involved.

Lalam: So, in short, the paper is demonstrating that inference backends are an active and influential component of the LLM evaluation pipeline.

Conclusion: Tom: To wrap up "The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility," the authors really hammered home that backend-induced numerical differences can propagate into divergent generations and benchmark score shifts large enough to affect model rankings.

Jane: So, in simple terms, the paper is arguing that if you want reliable AI comparisons, you have to treat the inference engine choice as a primary variable in your experiment.

Lu: The authors point out that these differences are driven by a combination of hidden engine defaults and compounding numerical drift caused by essential high-performance features.

Meng: This means we can’t just hope that optimizing for speed will automatically translate into perfect, unbiased evaluation results without careful monitoring.

Lalam: They conclude by advocating for standardized reporting so researchers publish exact environment specifications or at minimum, the specific backend library and version used to ensure better reproducibility and interpretability.

Tom: It's a call to action for the research community: treat the inference backend as a first-class experimental parameter for scientific validation.

Jane: That seems like a fair way to frame it—it shifts the focus from just model weights to including the environment that runs them.

CISPA Helmholtz Center for Information Security

cs.LG

Submitted: 2026-05-19

Updated: 2026-11-18

Code: https://github.com/ggml-org/llama.cpp

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline…

Key concepts

Inference Backend
The specific software or library used to run a Large Language Model (LLM) during the generation process. Different backends use different internal optimizations, leading to variations in how the model processes input and produces output.
Numerical Drift
Subtle, compounding errors that accumulate during computation due to high-performance features like prefix caching or custom kernels. This drift causes the generated text or scores to slowly deviate from a reference version over time.
Systematic Engine Defaults
Predefined settings within an inference engine that are correct but can cause significant differences between backends. Examples include default penalty parameters and hidden prompt mutation, which explain some of the most extreme observed divergences.
Disagreement Rate
A metric measuring how often a specific backend's prediction differs from a reference prediction for the same input, regardless of whether the ground truth is correct. High disagreement suggests the backend fundamentally changes the model's decision boundary.

Terminology

Summary

The specific inference stack is rarely reported despite this widespread diversity By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. p. 0

Where the paper says it

By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported but consequential hyperparameter in the evaluation of LLM and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.

Landscape Survey

The authors first survey the inference landscape to identify 200 distinct engines across various categories. They categorize these engines into three types: Self-Hosted Inference Libraries, Managed Inference Platforms, and Aggregators and Routers. p. 2

Where the paper says it

Self-Hosted Libraries Managed Platforms Aggregators/ Routers 0 20 40 60 80 100 120 Number of Engines 121 51 28 Open Source Proprietary (a) Engine Categories Python (47. 0%) C/C++ (26.

The landscape is dominated by self-hosted libraries, which account for around 61 % of the total ecosystem. p. 2
Where the paper says it

Figure 1a illustrates the distribution of these systems. We find that the landscape is dominated by self-hosted libraries, which account for around 61 % of the total ecosystem.

Controlled Evaluation

The study presents a systematic empirical study of how inference backends affect LLM benchmark results across five widely used engines. They show that the choice of backend alone can shift benchmark scores by up to 16. p. 0

Where the paper says it

cpp, across multiple open-weight models and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.

6 percentage points and induce high rates of output disagreement. p. 0
Where the paper says it

We show that the choice of backend alone can shift benchmark scores by up to 16. 6 percentage points and induce high rates of output disagreement.

The evaluation held model weights, decoding parameters, and hardware constant while testing across multiple open-weight models and established benchmarks. p. 0
Where the paper says it

Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama. cpp, across multiple open-weight models and established benchmarks.

Root Cause Analysis

The divergence is driven by two primary sources when isolating specific optimizations and tracing the execution pipeline. These causes are categorized into systematic, but correctable, engine defaults and optimization-induced numerical drift. p. 5

Where the paper says it

1 8B and DeepSeek R1 on GSM8K. We categorize root causes into two distinct groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical execution of the model (detailed ablation results are provided in Appendix D).

For example, hidden prompt mutation and default penalty parameters explain the most extreme divergences observed in Table 1. p. 5
Where the paper says it

We categorize root causes into two distinct groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical execution of the model (detailed ablation results are provided in Appendix D). 1 Systematic Engine Defaults The most extreme divergences, such as DeepSeek R1 scoring only 61.

Reproducibility in ML Research

The authors analyze over 35,000 recent publications from top machine learning venues to quantify how frequently the inference stack is reported. The survey reveals that while code sharing is becoming standard, the inference backend remains a largely undocumented source of experimental variance. p. 7

Where the paper says it

To ensure the reliability of our automated pipeline, we conducted a manual verification on a random subset of papers at each filtering and classification stage (details in Appendix F. Ultimately, these findings confirm that while code sharing is becoming standard, the inference backend remains a largely undocumented source of experimental variance.

Only 820 papers without code artifacts explicitly named the backend, with transformers being the most common mention. p. 6
Where the paper says it

For papers without code artifacts, we analyzed how frequently the inference stacks were disclosed. Among the 2,257 papers offering no code, 820 (36 %) explicitly 2Using the vLLM backend (version 0.

Recommendations

The paper advocates for standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons. Researchers are advised to publish exact environment specifications or, at minimum, the specific backend library and version. p. 8

Where the paper says it

Reporting Standards: We advocate for publishing exact environment specifications (e. , Docker containers), or at a minimum, the specific backend library and version.

Furthermore, evaluators should ensure fair comparisons by using the same inference backend for all experiments if it cannot be determined otherwise. p. 8
Where the paper says it

We recommend averaging results across multiple seeds. Ensure Fair Comparisons: If the inference backend of a state-of-the-art method cannot be determined due to missing documentation, the experiments should be rerun in a comparable setting, using the same inference backend for all experiments.

The authors also recommend that providers expose system fingerprints and implement deterministic modes for scientific debugging. p. 8
Where the paper says it

Deterministic Modes: We encourage developers to implement “strict reproducibility” flags. While high-performance non-deterministic kernels are essential for production, a slower, deterministic execution path is necessary for scientific debugging and validation.

Limitations

The controlled environment used in the study does not fully reflect production settings, meaning the observed variance may represent a lower bound. While greedy decoding was used to minimize sampling randomness, this strategy is not optimal for all architectures and may skew metrics for reasoning models. p. 8

Where the paper says it

We utilized greedy decoding to minimize sampling randomness. However, this strategy is not optimal for all architectures, particularly reasoning models (e.

Finally, fully disabling custom kernels to achieve perfect determinism is often impossible without abandoning the engine entirely. p. 8
Where the paper says it

However, in this paper, we were interested in the relative differences of runs with varying inference backends. Finally, ablating every custom kernel is infeasible, and fully disabling these features to achieve perfect determinism is often impossible without abandoning the engine entirely.

Conclusion

Inference backends are an active and influential component of the LLM evaluation pipeline. Backend-induced numerical differences can propagate into divergent generations and benchmark score shifts large enough to affect model rankings. p. 8

Where the paper says it

9 Conclusion Our results reveal that inference backends are not a benign implementation detail, but an active and influential component of the LLM evaluation pipeline. Across models, benchmarks, and metrics, we find that backend-induced numerical differences can propagate into divergent generations, altered decision boundaries, and benchmark score shifts large enough to affect model rankings.

The work demonstrates that this behavior is driven by a combination of hidden engine defaults and compounding numerical drift caused by essential high-performance features. p. 8
Where the paper says it

These effects are particularly pronounced for reasoning-oriented models, where early token-level divergence can fundamentally reshape the generated chain of thought. We demonstrate that this behavior is driven by a combination of hidden engine defaults and compounding numerical drift caused by essential high-performance features.

The authors advocate treating the inference backend as a first-class experimental parameter for scientific validation. p. 8
Where the paper says it

While the community has developed careful conventions around datasets, decoding strategies, and random seeds, the inference stack itself remains largely invisible, even though it can dominate other sources of variance. To address this gap, we advocate treating the inference backend as a first-class experimental parameter: explicitly reporting backend implementations, repeated evaluations, and supporting deterministic execution modes for scientific validation.

Acknowledgments and Disclosure of Funding

This work was supported by the Helmholtz Association’s Initiative and Networking Fund on the HAICORE@FZJ partition and by the German Federal Ministry of Education and Research under the grant AIgenCY (16KIS2012) and SisWiss (16KIS2330). Moreover, this work was supported by the LCIS center VW-Vorab-2025, ZN4704 11-76251-2055, as well as the Daimler and Benz Foundation under the grant Ladenburger Kolleg, Project KonCheck. p. 9

Where the paper says it

Acknowledgments and Disclosure of Funding This work was supported by the Helmholtz Association’s Initiative and Networking Fund on the HAICORE@FZJ partition and by the German Federal Ministry of Education and Research under the grant AIgenCY (16KIS2012) and SisWiss (16KIS2330). Moreover, this work was supported by the LCIS center VW-Vorab-2025, ZN4704 11-76251-2055, as well as the Daimler and Benz Foundation under the grant Ladenburger Kolleg, Project KonCheck.

References

The paper cites a broad literature on LLM inference and reproducibility. It extends this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance. p. 1

Where the paper says it

Recent studies highlight varying LLM results due to floatingpoint non-associativity [8], ambiguous semantic benchmarks [9], and model versioning [10]. We extend this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance.

p. 1
Where the paper says it

Recent studies highlight varying LLM results due to floatingpoint non-associativity [8], ambiguous semantic benchmarks [9], and model versioning [10]. We extend this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance.

Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms. p. 1
Where the paper says it

Assessing inference performance and determinism. Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms [11–13].

p. 1
Where the paper says it

Assessing inference performance and determinism. Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms [11–13].

The authors build on these performance-centric studies to quantify how bare backend design choices fundamentally alter generation trajectories. p. 1
Where the paper says it

Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent randomness. We build on these performance-centric studies to quantify how bare backend design choices fundamentally alter generation trajectories.

p. 1
Where the paper says it

Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent randomness. We build on these performance-centric studies to quantify how bare backend design choices fundamentally alter generation trajectories.

A Backend Versions

Table 2 details the specific versions of the inference backends and reference libraries utilized throughout all controlled experiments in Section 5. This section provides detailed visualizations and breakdowns for the evaluation metrics introduced in Section 5. p. 3

Where the paper says it

We selected a set of five inference engines based on the popularity metrics from our landscape survey (cf. Section 3), namely: vLLM [1], SGLang [2], llama.

The table lists specific software versions for the inference backends and reference libraries used in our evaluation to ensure that any observed numerical variance is strictly attributable to the architectural differences between the backends. p. 8
Where the paper says it

9 Conclusion Our results reveal that inference backends are not a benign implementation detail, but an active and influential component of the LLM evaluation pipeline. Across models, benchmarks, and metrics, we find that backend-induced numerical differences can propagate into divergent generations, altered decision boundaries, and benchmark score shifts large enough to affect model rankings.

B Additional Metrics

This section provides detailed visualizations and breakdowns for the evaluation metrics introduced in Section 5. It includes metrics like Benchmark Accuracy, Disagreement Rate, and Length Error to measure macro-level agreement. p. 3

Where the paper says it

12) on a single NVIDIA H100 GPU using fixed software versions for all backends (Appendix A). 1 Outcome Consistency To measure macro-level agreement, we evaluate Benchmark Accuracy, Disagreement Rate, and Length Error.

The results indicate that the inference engine is a significant source of variance, with accuracy discrepancies often exceeding several percentage points. p. 3
Where the paper says it

Benchmark Accuracy (Table 1). Our results indicate that the inference engine is a significant source of variance, with accuracy discrepancies often exceeding several percentage points, enough to alter leaderboard ranking.

B.1 Disagreement Rate

The Disagreement Rate is defined as the frequency with which a backend’s prediction y differs from the reference yref for the same input regardless of ground-truth correctness. In many cases, we observe that disagreement rates consistently exceed the absolute differences in accuracy, indicating that the backend alters the model’s decision boundary. p. 3

Where the paper says it

We define this as the frequency with which a backend’s prediction y differs from the reference yref for the same input, regardless of ground-truth correctness: D = 1 N P1(y ̸= yref ). While in some settings the disagreement is small, in many cases we observe that disagreement rates consistently exceed the absolute differences in accuracy, indicating that the backend alters the model’s decision boundary.

B.2 Length Error

The authors analyze structural deviations in generated responses by measuring Output Length consistency via Signed Difference and Absolute Difference. Reasoning models frequently lie outside the ideal zone of ±25 tokens, experiencing massive shifts in generation length that fundamentally alter their chain-of-thought process. p. 4

Where the paper says it

Even more distinct are the differences for reasoning models where the length differs significantly across all the tested datasets. For instance, the Ollama backend produces DeepSeek R1 outputs that are, on average, over 9,000 tokens shorter than the reference on GPQA, fundamentally altering the chain-of-thought process.

B.3 Token Divergence

The Token-Level Divergence Index is calculated to pinpoint when the generations begin to differ. Reasoning models consistently diverge from the reference generation much earlier than standard architectures, particularly on difficult benchmarks like GPQA. p. 4

Where the paper says it

3), we observe that reasoning models consistently diverge from the reference generation, much earlier than standard architectures. This is amplified on difficult benchmarks like GPQA, where, for example, Llama 3.

B.4 LogProb Error

The LogProb RMSE quantifies the floating-point variance of the top1 token calculated on the matching prefix before divergence. Reasoning models systematically exhibit higher drift and specific configurations display distinct error spikes in this metric. p. 4

Where the paper says it

While most backends maintain similar precision with the reference (RMSE<0. 01), we observe that reasoning models systematically exhibit higher drift and specific configurations display distinct error spikes.

B.5 Top-5 Token Jaccard Similarity

The Top-K Jaccard Similarity measures the overlap of the top-5 candidate tokens between the backend and the reference implementation. Lower values indicate that the backend’s numerical deviations fundamentally alter the model’s candidate pool, bringing entirely different tokens into the top-5 predictions. p. 6

Where the paper says it

Instead, slight numerical drifts caused by custom kernels compound continuously across the model’s depth, eventually cascading into different Top-1 token predictions. 7 Inference Reproducibility in ML Research Given the substantial impact of backend choice on benchmark performance, we conducted a systematic survey of recent ML publications to contextualize this variance and measure the prevalence of inference stack reporting.

C Ablation Studies

The authors conducted four targeted ablation studies to validate robustness and address real-world implications. These studies include testing safety vulnerabilities, batching effects, hardware independence, and stochastic sampling. p. 5

Where the paper says it

Evaluating batched generation (batch size = 4) revealed that performance differences persist, and slight numerical shifts within vLLM and SGLang verify that batching actively influences generation. Hardware Independence (Section C.

The results confirm that backend variance persists under batched generation and across different GPU architectures. p. 5
Where the paper says it

Evaluating on NVIDIA L40 GPUs yielded consistent, slightly increased backend-induced variance (up to 17. 2% Max-Min difference), confirming divergence is rooted in software implementations rather than specific GPU architectures.

D Root Cause Analysis

The massive divergences are the direct result of specific system-level design choices. These causes are broadly categorized into Systematic Engine Defaults, which are correctable configurations, and Optimization-Induced Numerical Variance, which is tied to underlying mathematical execution. p. 5

Where the paper says it

1 8B and DeepSeek R1 on GSM8K. We categorize root causes into two distinct groups: (1) systematic, but correctable, engine defaults that explain the massive outliers observed in Table 1, and (2) optimization-induced numerical drift that inherently alters the mathematical execution of the model (detailed ablation results are provided in Appendix D).

E Paper Survey Methodology and Artifacts

The authors employed a multi-stage automated extraction pipeline utilizing an LLM-as-a-judge to survey 35,000 papers. This pipeline involved keyword pre-filtering, LLM prompting with Chain-of-Thought, and structured output parsing for programmatic routing. p. 6

Where the paper says it

Filtering and Extraction Pipeline. A keyword pre-filter first flagged potentially relevant papers.

The survey found that only 820 papers without code artifacts explicitly named the backend. p. 6
Where the paper says it

For papers without code artifacts, we analyzed how frequently the inference stacks were disclosed. Among the 2,257 papers offering no code, 820 (36 %) explicitly 2Using the vLLM backend (version 0.

F Manual Verification of Judge Results

A manual verification on 150 audited papers showed high agreement rates across all processing stages. Disagreements in engine extraction were primarily due to false positives flagging low-level kernel libraries as full inference engines. p. 6

Where the paper says it

For these confirmed papers, we ran two parallel extraction processes using the LLM judge: a Text Analysis scan to determine if specific inference engines were explicitly named, and a Code Analysis scan to extract repository URLs. The full methodology, including the exact keyword list, pipeline logic, and complete LLM prompts, is detailed in Appendix E.

G Impact Statement

The primary positive impact of this work is to encourage more transparent reporting standards, ensuring that claims of State-of-the-Art performance are statistically significant rather than artifacts of system optimizations. The work also has implications for AI safety and reliability, as optimization techniques can alter model behavior. p. 1

Where the paper says it

This creates a dangerous “deployment gap” between research validation and production reality, where backend-induced discrepancies can undermine not only performance claims but also safety guarantees, potentially leading to harmful or non-compliant behavior in real-world use. While prior work has examined sources of variability such as prompt sensitivity [5], quantization [6], and decoding strategies [7], the role of the inference backend itself has remained largely unexplored.

H Model & Dataset Licenses

Table 8 lists the licenses for various models and datasets used in the paper, including Llama3.1 8B [22] and Qwen3-235B-A22B-Instruct-2507-AWQ [31]. p. 6

Where the paper says it

2 Results Setup. We extracted text from PDF files using pymupdf and utilized Qwen3-235B-A22B-Instruct2507-AWQ [31] as the LLM judge.

p. 6
Where the paper says it

2 Results Setup. We extracted text from PDF files using pymupdf and utilized Qwen3-235B-A22B-Instruct2507-AWQ [31] as the LLM judge.

The licenses for GSM8K and LiveCodeBench v6 are also noted. p. 3
Where the paper says it

Benchmarks. We employ four widely adopted datasets to evaluate distinct model capabilities: GSM8K (Math) [25], GPQA Diamond (Science) [26], SimpleQA Verified (Factuality) [27], and LiveCodeBench v6 (Code) [28].

p. 3
Where the paper says it

Benchmarks. We employ four widely adopted datasets to evaluate distinct model capabilities: GSM8K (Math) [25], GPQA Diamond (Science) [26], SimpleQA Verified (Factuality) [27], and LiveCodeBench v6 (Code) [28].

The paper cites a broad literature on LLM inference and reproducibility. It extends this literature by isolating the inference engine itself as a key, previously undocumented source of evaluation variance. Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms. The authors build on these performance-centric studies to quantify how bare backend design choices fundamentally alter generation trajectories. It also references works on prompt sensitivity, quantization, and decoding strategies as sources of variability. p. 1

Where the paper says it

This creates a dangerous “deployment gap” between research validation and production reality, where backend-induced discrepancies can undermine not only performance claims but also safety guarantees, potentially leading to harmful or non-compliant behavior in real-world use. While prior work has examined sources of variability such as prompt sensitivity [5], quantization [6], and decoding strategies [7], the role of the inference backend itself has remained largely unexplored.

Assessing inference performance and determinism

Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms. Other research analyzes the impact of general system design, such as caching and decoding strategies. p. 1

Where the paper says it

Prior work evaluating inference engines primarily focuses on hardware-level optimizations, absolute inference speed, and energy efficiency across various platforms [11–13]. Other research analyzes the impact of general system design, such as caching and decoding strategies [14], expanding earlier findings on CNNs and RNNs [15], or evaluates differences between plain quantization formats [16].

Despite optimizations for determinism and guidelines recommending fixed seeds and low temperatures, LLMs retain inherent randomness. p. 1
Where the paper says it

Other research analyzes the impact of general system design, such as caching and decoding strategies [14], expanding earlier findings on CNNs and RNNs [15], or evaluates differences between plain quantization formats [16]. Despite optimizations for determinism [17] and guidelines recommending fixed seeds and low temperatures [18], LLMs retain inherent randomness.

Model & Dataset Licenses

Table 8 lists the licenses for various models and datasets used in the paper, including Llama3.1 8B [22] and Qwen3-235B-A22B-Instruct-2507-AWQ [31]. The licenses for GSM8K and LiveCodeBench v6 are also noted.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Standardize reporting by explicitly stating backend implementations for every benchmark run to ensure scientific reproducibility, as the choice of inference backend alone can shift benchmark scores by up to 16 percentage points.

  2. Implement rigorous testing protocols that account for non-determinism, such as averaging results across twelve unique seeds to mitigate variance caused by optimization-induced numerical drift, which can cause a model’s output to change over time.

  3. Develop safety guarantees through backend awareness; since identical model weights can exhibit noticeable distinct safety profiles when deployed on different inference engines, deployment systems must verify the security profile of a model across all anticipated serving backends.

  4. Establish stricter leaderboard validation by requiring platforms to report an Inference Stack and quantifying uncertainty with a backend confidence interval by testing models on both reference and high-throughput engines.

  5. Design deterministic execution modes for scientific validation, encouraging developers to implement mechanisms that provide strict reproducibility, even if it means sacrificing some high-throughput performance during debugging.

Abstract

Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying 200 distinct engines, and analyze 35,000 ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama.cpp, across multiple open-weight models spanning 4B-30B parameters and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations such as prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported yet consequential hyperparameter in LLM evaluations and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.

Sources

Related papers