Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
The Sequential Isolation Methodology is a controlled protocol designed to reduce measurement variance when benchmarking Large Language Models (LLMs). By starting fresh for each test, running workloads sequentially, and varying concurrency levels systematically, researchers achieved a low average configuration-level CV of 2.23%. This provides a stable reference measurement for comparing different models and system versions.