Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

arXiv:2610.09778 · cs.PF, cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Reproducible LLM Inference Benchmarking".

Jane: The gist The Sequential Isolation Methodology is a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency for Large Language Model (LLM) inference…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper called "Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing." It’s about how to make testing these massive language models actually reliable when you run the same test twice and get different results.

Jane: That sounds really practical, Tom. When you're trying to compare different versions of an AI model or see if a new configuration is better, but the measurements keep jumping around because of things like how the system was loaded—that makes it impossible to know what you're actually seeing >

Lu: The authors are Arnold Olympio and Juan Manuel Servera Bondroit, with Guang Lu as a key contributor on this work >

Meng: It sounds like they are trying to solve that problem where you can't trust your benchmark numbers because the system state is changing every time you run it >

Lalam: I think what they’re aiming for is a way to get a stable reference measurement so that when we compare models, we aren't comparing noise instead of actual performance >

Tom: Exactly. They introduce this Sequential Isolation Methodology as a protocol to reduce that variation between runs. It’s about controlling the execution lifecycle rather than trying to fix every tiny piece of interference at once >

Jane: So, the core idea is isolating the test so that it's repeatable across different model versions or software setups >

Lu: They focus on controlling how the serving process starts and ends for each measurement, which they call sequential isolation >

Meng: It’s interesting they aren't trying to build a whole new inference engine; they are layering this methodology onto existing systems like vLLM zero point nine.one > <ref:2610.09778#pg1>

Lalam: That makes sense because building a whole new architecture is huge, and if you just control the measurement process, you might get solid results quickly >

The paper's summary: Tom: Okay, so they detail this Sequential Isolation Methodology as a four-stage process that progressively increases experimental control. They show a big drop in measurement variation, going from an average coefficient of variation of fifteen point two percent in the very first version, V1, down to just two point two percent by the fourth version, V4 > <ref:2610.09778#pg1>

Jane: That drop is huge. So they are showing that by adding more control—like making sure the serving environment restarts for every context size—they can really tame that noise >

Lu: They do this by running the benchmark matrix sequentially at the run level, and restarting the serving environment when you change things like context size >

Meng: They have five repetitions at each level, so that means forty runs against the same server instance for a specific context size, with a ten-second interval between them > <ref:2610.09778#pg2>

Lalam: It’s designed to reduce controllable between-run variability through this standardized execution lifecycle, which helps make the results more trustworthy >

Tom: They are testing three specific models—Mistral-7B-Instruct-v0 point 3, Llama-three point one-8B-Instruct, and Qwen2 point 5-7B—on an NVIDIA A100 80GB GPU using vLLM zero point nine.one >

Jane: And they test six different context sizes—from one thousand tokens all the way up to thirty-two thousand tokens >

Lu: They also test eight different concurrency levels, ranging from five users up to a thousand users >

Meng: The whole setup involves fifteen repetitions for each model times context size times concurrency combination, which is a lot of runs >

Lalam: So the results show that this final protocol achieves an average configuration-level coefficient of variation of two point two three percent across all the models they tested > <ref:2610.09778#pg1>

The paper's improvements: Tom: The main improvement is definitely that you get a stable reference measurement, which is what they call reducing between-run measurement variance so you can compare things fairly >

Jane: So, instead of getting wildly different numbers depending on when the test ran or what was running on the server, you get a consistent number to use for comparison >

Lu: They’re evaluating whether progressively increasing this experimental control actually reduces run-to-run variation rather than just claiming they've figured out exactly why that variation happens >

Meng: That distinction is important because it shows if the protocol is genuinely making the measurement stable, or if it's just hiding a different kind of instability >

Lalam: The improvements show that one hundred thirteen out of one hundred forty-four configurations had a coefficient of variation below a three percent reference threshold, which means most setups are pretty consistent now >

Tom: And they also found some interesting things about latency changes. Specifically, they saw a marked latency transition between two hundred and five hundred concurrent users on the tested stack >

Jane: That suggests that capacity planning isn't just a flat line; there’s a real shift in how slow the system gets when you hit those higher concurrency levels >

Lu: They also found descriptive differences in P99 latency across the three models, which gives us some insight into model behavior under load >

Meng: So for us shipping models, seeing that Qwen2 point 5-7B has the lowest overall average P99 time to first token is something we can actually use for deployment decisions >

Conclusion: Tom: So, to wrap up the Sequential Isolation Methodology, they’ve delivered a controlled protocol that achieves an average coefficient of variation of two point two three percent across all the evaluated models > <ref:2610.09778#pg1>

Jane: That’s a significant reduction from the starting point, showing that by controlling the execution lifecycle through this protocol, we get much more reliable benchmarks >

Lu: For those who are interested in deployment economics, they also calculated that break-even against API-based deployment lies between approximately three point seven and fourteen point nine billion tokens per month depending on the API pricing >

Meng: That cost analysis is useful because it relates to how much data we actually need to process before running our own infrastructure becomes cheaper than using an API >

Lalam: And they showed that under their stated assumptions, the measured single-instance throughput at seventy percent utilization is approximately two hundred eighty-eight million tokens per month, which is below the full break-even range they calculated >

Tom: So, while this protocol gives us a controlled reference measurement and has not been validated as a predictor of concurrent production behavior on real traffic, it’s a solid tool for regression testing >

Jane: It’s important to remember that these findings are controlled measurements; they shouldn't be interpreted as absolute performance under uncontrolled production workloads >

Lu: The Sequential Isolation Methodology is presented as a protocol for reproducible LLM inference benchmarking and regression comparison, and it sets the stage for more reliable comparisons in this area >

Meng: It’s a great reference measurement, but we still need to see how this transfers when you put it into a live production environment with unpredictable traffic >

Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu, João Carvalho

Lucerne University of Applied Sciences and Arts and Arts · Microsoft Switzerland · Uthereal AG Switzerland · ETH Zurich Switzerland

cs.PF, cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist The Sequential Isolation Methodology is a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload

Key concepts

Reproducibility Challenge: LLM inference results are hard to repeat because they change based on the system state, like scheduler interactions or residual execution. The goal is not to predict real-world behavior but to create a stable reference point so that comparisons between different models and configurations are fair and reliable.
Sequential Isolation Methodology: This protocol involves starting the serving process fresh for every test run, executing all concurrency levels in a specific order, and spacing runs out by 10 seconds. This standardized lifecycle aims to minimize variability caused by execution state or sustained device load during testing.
Between-Run Measurement Variance (CV): This measures how much the results change from one test run to the next. The methodology successfully reduced this variance significantly, dropping from 15.2% in an earlier version to 2.2% in the final version, demonstrating improved experimental control and measurement stability.
Cost-Performance Trade-off Analysis: This section calculates the financial break-even point for self-hosting versus using paid APIs based on measured throughput and pricing. It shows that the cost of running a single instance is compared against the volume of tokens processed to determine when one deployment method becomes economically favorable.

Terminology

Benchmarking — 1× in this paper · explained in 4 episodes

Summary

The gist The Sequential Isolation Methodology is a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency for Large Language Model (LLM) inference measurements.

Reproducibility Challenge

The challenge in LLM inference benchmarking stems from the fact that repeated measurements can vary with execution and system state, which makes results difficult to reproduce and capacity planning unreliable The objective of the present work is not to predict absolute production behavior from an isolated benchmark but rather to seek a stable reference measurement against which models, system versions and configurations can be compared fairly. Potential sources of between-run variation include scheduler interactions, residual execution state, initialization effects and sustained device load. The authors evaluate whether progressively increasing experimental control reduces observed run-to-run variation rather than claiming a causal decomposition of its sources.

Sequential Isolation Methodology

The Sequential Isolation Methodology is developed through four cumulative methodology stages, showing an average CV reduction from 15.2% in V1 to 2.2% in V4 as progressively greater experimental control was introduced. Sequential refers to execution of the benchmark matrix and control of the serving lifecycle, not to eliminating concurrency within a test run. For each model, the serving process is started fresh for a context size. All eight target concurrency levels are then executed in sequence, with five repetitions at each level, giving 40 runs against the same server instance for that context size. A 10-second interval separates successive runs. The protocol is designed to reduce controllable between-run variability through a standardized execution lifecycle.

Experimental Design and Models

The experimental design consists of six context sizes (1k, 2k, 4k, 8k, 16k and 32k tokens), eight concurrent-user levels (5, 10, 20, 50, 100, 200 and 500 and 1,000 users), and five repetitions for each model × context-size × concurrency configuration. The three models selected for this study were Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, and Qwen2.5-7B (all 7B parameter classes). All experiments were conducted on version-pinned infrastructure using vLLM 0.9.1 and an NVIDIA A100 80GB GPU.

Key Findings

The final Sequential Isolation protocol achieved an average configuration-level CV of 2.23% across the evaluated models. Of the 144 configurations, 113 (78.5%) had CV below the predefined 3% reference threshold. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. Qwen2.5-7B has the lowest overall average P99 TTFT at 8,022 ms, compared with 8,496 ms for Llama-3.1-8B and 9,124 ms for Mistral-7B. The cost analysis shows that break-even against API-based deployment lies between approximately 3.7 and 14.9 billion tokens per month depending on API pricing.

Limitations and Future Work

Several limitations constrain the external validity of the findings, including using a single NVIDIA A100 80GB GPU with vLLM 0.9.1. The study uses controlled synthetic prompts with a fixed 512-token output target. Furthermore, the V1–V4 evaluation is cumulative rather than factorial, so the independent causal effects of individual controls cannot be separated. Future work will validate the protocol across additional serving frameworks and newer GPU generations such as H100 and H200. A further priority is comparison of controlled reference measurements with concurrent production-like serving to determine when the former do and do not transfer to operational workloads. The protocol provides a controlled reference measurement and has not been validated as a predictor of concurrent production behavior.

Cost-Performance Trade-off Analysis

A continuously provisioned Azure Standard NC24ads A100 v4 instance costs approximately 2,241 per month using 730 billed hours. The break-even volume is calculated as V∗ = (c × h)/p, where p is the API cost per token. Measured single-instance throughput at 70% utilization is approximately 288 million tokens/month, which is below the full break-even range under the stated assumptions. The economic analysis complements Pan et al.’s (2025) deployment framework by relating cost comparison to empirically measured single-instance throughput.

Conclusion

We presented the Sequential Isolation Methodology as a controlled protocol for reproducible LLM inference benchmarking and regression comparison. Across 144 model × context-size × concurrency configurations, each repeated five times, the final protocol achieved an average CV of 2.23%, with 113 configurations (78.5%) below the predefined 3% reference threshold. On the tested A100/vLLM stack, we observed a marked latency transition between 200 and 500 concurrent users and descriptive differences in P99 latency across the three models. The cost analysis further shows that, under the stated assumptions, API/self-hosting break-even lies between approximately 3.7 and 14.9 billion tokens per month depending on API pricing. These findings provide controlled reference measurements; they should not be interpreted as predictions of absolute performance under uncontrolled production traffic.

Improvements for AI systems

This system can perform regression testing with a significantly lower measurement variance, as demonstrated by reducing average coefficient of variation (CV) from 15.2% in V1 to 2.2% in V4 under the final protocol, which helps distinguish between model and configuration differences.
The system can identify latency transitions under high concurrency; specifically, it can detect a marked latency transition within this interval between 200 and 500 concurrent users on the tested stack, allowing for better capacity planning based on empirical evidence.
The system can optimize model selection based on context length; for instance, it can determine that Qwen2.5-7B has the lowest observed P99 latency at short contexts (1k–4k tokens) and excels at the longest tested context (32k tokens) with a 29.4% advantage over Mistral.
The system can provide cost-performance guidance for deployment decisions; it can calculate a sensitivity range for break-even volume, showing that under the stated assumptions, API/self-hosting break-even lies between approximately 3.7 and 14.9 billion tokens per month depending on API pricing.
The system can support configuration comparison during regression testing by providing descriptive differences in tail latency; it can report that Qwen2.5-7B has the lowest overall average P99 TTFT at 8,022 ms, compared with 8,496 ms for Llama-3.1-8B and 9,124 ms for Mistral-7B.