LLMRunner: Closed Workload Model Performance Testing Framework for LLM Servings

summary

The gist

Performance testing large language model (LLM) serving engines is a crucial requirement considering the criticality of the infrastructure for deploying conversational AI at scale Most of the…

In short

LLMRunner is a performance testing framework designed to generate realistic load for LLM serving engines by simulating multi-turn conversations with configurable user parameters like think time. It was used to test NVIDIA CUDA Unified Virtual Memory (UVM) as a replacement for the Key-Value cache memory allocator in vLLM, showing no measurable performance penalty.

Key concepts

Workload Modeling Capabilities
LLMRunner generates realistic load by simulating multi-turn conversations. It allows users to configure parameters such as the number of concurrent users, prompt length, and think time distributions to accurately model how real end-users interact with an AI system.
Multi-turn Conversations
This refers to a sequence of interactions where a user sends a prompt, receives a reply, pauses to 'think,' and then sends another follow-up question. LLMRunner models this by maintaining active conversation pools where each worker appends the new turn to the history before sending it back to the server.
KV Cache Management Study
This study investigates using UVM memory allocation for the Key-Value cache in vLLM. The goal was to see if UVM could replace existing pre-allocation methods without hurting performance. The results showed that using UVM incurs no measurable throughput or latency penalty compared to the standard method.
System Architecture Layers
The framework operates through four sequential layers: Plan (defining the experiment scope via YAML), Expansion (creating all possible run combinations), Execution (launching vLLM and the load generator), and a background Monitor (sampling GPU, CPU, and KV-cache metrics).

Terminology used across episodes

This episode discusses

The paper

LLMRunner: Closed Workload Model Performance Testing Framework for LLM Servings · Read on arXiv

Indian Institute of Technology Kanpur

Performance testing large language model (LLM) serving engines is a crucial requirement considering the criticality of the infrastructure for deploying conversational AI at scale. Most of the benchmarking tools used to carryout performance evaluation of LLM serving do not capture realistic workload orchestration unlike closed workload performance testing infrastructure used in other domains such as web and database engines. For example, state-of-the-art LLM benchmarks for performance testing model traffic as streams of independent single-turn and stand-alone requests omitting the end-user behavior such as think-time between conversational turns. We present LLMRunner, a performance tester for LLM serving that addresses this gap by providing a framework to generate realistic load consisting of multi-turn conversations with configurable end-user parameters such as number of users in the workload loop, think-time distributions etc. Moreover, LLMRunner provides declarative matrix-sweep experiment plans, integrated GPU and KV-cache monitoring, and an automated artifact collection and plotting subsystem. We use LLMRunner to comparatively analyze the performance of LLM serving when NVIDIA CUDA Unified Virtual Memory (UVM) is used as a drop-in replacement memory allocator for the Key-Value cache (KV-Cache). Our test demonstrates that UVM incurs no measurable throughput or latency penalty compared to the baseline memory allocation technique for non-oversubscription scenarios.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "LLMRunner: Closed Workload Model Performance Testing Framework for LLM Servings".

Jane: Here's the gist of the paper. Performance testing large language model (LLM) serving engines is a crucial requirement considering the criticality of the infrastructure for deploying conversational AI at scale Most of the benchmarking tools used to carry out performance evaluation of LLM serving do not capture realistic workload orchestration unlike closed workload performance testing infrastructure used in other domains such as web and database engines We present LLMRunner, a performance tester for LLM serving that addresses this gap by providing a framework to generate realistic load consisting of multiturn conversations with configurable end-user parameters such as number of users in the workload loop, think-time distributions etc We use LLMRunner to comparatively analyze the performance of LLM serving when NVIDIA CUDA Unified Virtual Memory (UVM) is used as a drop-in replacement memory allocator for the Key-Value cache (KV-Cache) Our test demonstrates that UVM incurs no measurable throughput or latency penalty compared to the baseline memory allocation technique for non-oversubscription scenarios.

Tom: Workload Modeling Capabilities LLMRunner provides a framework to generate realistic load consisting of multiturn conversations with configurable end-user parameters such as number of users in the workload loop, think-time distributions etc A crucial aspect of LLM workload is multi-turn conversations where a user sends a prompt, reads the reply, thinks for some interval, and then sends a follow-up prompt again LLMRunner proposes a workload generator where parameters such as number of users in the workload loop, prompt length and think time distribution can be specified It supports both single-turn open-loop traffic and full multi-turn conversations with configurations to define the workload mix In multi-turn mode, the harness maintains a pool of active conversations, where each worker appends the next user turn to the accumulated history, sends the full context to the server, receives the assistant reply, and appends it before issuing the next turn.

Jane: System Architecture and Experiment Design LLMRunner is organized into four layers that execute sequentially for each run in the expanded matrix The Plan layer defines the scope for an experiment through a single declarative YAML plan The Expansion layer computes the Cartesian product of all matrix dimensions and instantiates one frozen run specification per combination Each specification receives a 12-character SHA-1 identifier derived from its content, making run IDs stable across re-runs and sessions The Execution layer starts vLLM serve with the correct environment variables including VLLM USE UVM, polls the server’s health endpoints until it is ready, launches the load generator, and runs a background monitor that concurrently samples GPU, CPU, and KV-cache metrics.

Tom: Performance Analysis Features The framework includes several features for comprehensive analysis: Workload realism is achieved through configurable per-user thinktime distributions, enabling faithful modeling of end-user interaction and the KV-cache accumulation patterns it induces.: Systematic sweep is supported by a single declarative YAML plan that the harness expands into a Cartesian product of runs, launches each in isolation, and collects all artifacts without manual intervention.: Reproducibility is ensured because every run receives a stable, content-derived identifier so that runs from different sessions that share the same configuration can be matched for comparison without directory-encoded parameters.

Paper summary: Tom: So, we're talking about this paper called "LLMRunner: Closed Workload Model Performance Testing Framework for LLM Servings". Basically, they say that testing large language model serving engines needs to be much better than what we're currently doing.

Jane: That makes sense because most of the tools out there just test them with streams of independent single-turn requests, which doesn't really capture how people actually use AI in real conversation.

Tom: Exactly. The core idea here is that you need to test the whole user behavior, not just isolated prompts. This paper introduces LLMRunner as a framework to generate realistic load consisting of multi-turn conversations.

Lu: What they're claiming is that this framework lets you configure things like the number of users in a workload loop and even the distribution of how much time a user spends thinking between turns, which is crucial for modeling real interaction.

Meng: So, what’s the big takeaway for us on the ground? Can this thing actually run on our production traffic without causing massive headaches? What does it cost us to set up this kind of testing environment?

Lalam: Lalam has looked at the results and can tell you that they used LLMRunner to compare vLLM serving when using NVIDIA CUDA Unified Virtual Memory, or UVM, as a drop-in replacement for how the Key-Value cache is currently managed.

Tom: And what's the finding there? They are testing this UVM memory allocator against the baseline strategy. What did they find out about performance?

Lalam: The paper shows that using UVM for KV cache incurs no measurable throughput or latency penalty compared to the standard memory allocation technique when you're not oversubscribing resources.

Jane: That’s interesting because I know there's a lot of buzz around memory management techniques in these serving setups. So, if it doesn't hurt performance in those non-oversubscription scenarios, what does that imply for us building things at scale?

Lu: It implies that we can explore using UVM to look into future strategies, like demand-driven migration or capacity oversubscription, because it removes the hard coupling between where the allocation happens and when the data is placed.

Meng: Demand-driven migration sounds good in theory, but what are the actual overheads introduced during that first access from a GPU? Does that mean latency spikes for our users?

Lalam: The paper says that within normal GPU memory utilization bounds, where the working set fits within physical device memory, UVM-managed allocations behave similar to standard device allocations except for those initial access overheads when the GPU first touches a page residing in host memory.

Tom: So they're saying it’s fine if you don't hit those high-load or oversubscription limits, but we need to be careful about those initial accesses, right? That’s a key caveat.

Jane: And on the workload modeling side, how much control do we really get with this framework? They mentioned declarative matrix-sweep experiment plans.

Paper summary: Lu: You can define the scope of your experiment through a single YAML plan file, and that harness expands it into a Cartesian product of all those combinations automatically.

Tom: That’s good for systematic testing, I guess. It means you don't have to manually set up dozens of separate tests for every possible combination of settings.

Meng: But how do we ensure the load generator isn't just hitting the server with random traffic? What guarantees are there about how realistically it mimics human conversation flow?

Lalam: Realism is achieved through configurable per-user think-time distributions, which lets you model end-user interaction faithfully and see how that impacts KV cache accumulation patterns.

Jane: So, if we want to test something specific, say a very long context window usage pattern, can we configure the think time to reflect that?

Lu: Yes, you can specify prompt length and think time distribution when setting up the workload generator. In multi-turn mode specifically, each worker appends the next user turn to the accumulated history before sending it back to the server.

Tom: I’m curious about reproducibility here because they mentioned something about stable identifiers for each run. How do you make sure that if we rerun an experiment with the same settings, we get the exact same result?

Jane: They use a twelve-character SHA one identifier derived from the content of the plan, which makes those run IDs stable across different sessions or re-runs <ref:2611.15372#pg1>. That’s pretty robust for comparison.

Meng: That's helpful for debugging, I guess. But what about the automated artifact collection? Are we looking at a lot of manual work after it runs?

Lalam: The system includes an automated artifact collection and plotting subsystem that converts raw directories into presentation-ready figures using gnuplot without you having to do any data extraction manually.

Tom: So, the framework seems designed to handle the complexity of setting up these realistic, multi-turn conversations without requiring a whole team of researchers to build each test case from scratch.

Jane: It seems like they’ve built a tool specifically for closing that gap between testing and actual user experience modeling. This paper's title, "LLMRunner: Closed Workload Model Performance Testing Framework for LLM Servings," really sums up the practical utility of this system.

Lu: From my view, the implication is that we move away from just testing raw throughput numbers to actually testing how our KV cache management strategies behave under realistic conversational stress.

Meng: So, if I were deploying a model right now, what’s the first thing I’d look at regarding this paper? Is it the UVM part or the workload generation part? What breaks first in a real deployment?

Lalam: Given that they showed no penalty for non-oversubscription scenarios with UVM, I'd say start by looking at how you handle those oversubscription limits, because that's where the classical APIs fail.

Tom: So we’re looking at using this framework to stress test our KV cache policies and see exactly where the performance starts to degrade when traffic gets heavy. That sounds like a solid way to approach infrastructure hardening.

Conclusion: Tom: So we're wrapping up this look at LLMRunner, which is basically a framework for performance testing AI serving engines using realistic conversations.

Jane: Right, so it’s this tool that lets you simulate how users actually interact with a model over multiple turns, not just sending one quick question and getting one answer.

Lu: What the authors did is build this whole system to generate that real conversational load, controlling things like how many users are talking at once and how much time they take to think between their messages.

Meng: So for us in production, what does this mean? It means we can finally test our serving infrastructure with traffic that actually looks like real people chatting.

Lalam: Exactly. They showed you can generate these multi-turn conversations, and they tested it against using a specific memory allocator called UVM for the key value cache.

Tom: And the result there was pretty clean, they found that using UVM doesn't add any measurable penalty to throughput or latency when resources aren't fully saturated.

Jane: That’s a big deal because memory management is often where these kinds of performance issues hide, and they proved this specific change doesn't hurt us in standard scenarios.

Lu: It suggests that for many serving setups right now, you can probably stick with the baseline allocation without worrying about immediate performance regressions.

Meng: But what about when things get really busy? When we hit those oversubscription limits that we know cause problems? Does this framework even test those breaking points effectively?

Lalam: The paper tested a wide range of scenarios, from simple rate mode experiments to full closed loop conversations with different models and generation methods.

Tom: They even made sure the results were reproducible by giving every single test run a stable identifier based on its configuration.

Jane: That’s good for the science part, but what about the big picture? What does this tool mean for how we think about building AI systems at a much larger scale?

Lu: It means moving past just measuring raw speed and actually testing how your memory choices affect the entire conversation flow over time.

Meng: So if I'm shipping models now, I should be using this framework to stress test my KV cache policy, not just running it on some standard benchmark.

Tom: Yeah, it gives you the confidence that when you scale up the workload, your memory strategy is actually holding up under pressure.

More episodes

← Home