The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

arXiv:2610.15629 · cs.LG · Submitted 2026-10-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "The Price of Precision".

Jane: The gist: A cost model that accounts for binding resources supports latency prediction and mixed-precision allocation, which addresses gaps missed by byte-based proxies.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, we're looking at this paper now, "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving." The main idea here is that just using a byte count to guess how fast something will run isn't enough. They argue that serving speed can be limited by compute, or weight loading, or the key-value cache access, and this depends on the phase, batch size, and context length <ref:2610.15629#pg1>.

Jane: That sounds like a big problem for anyone trying to deploy large models efficiently. So what is their main claim? What are they actually saying about this cost model?

Tom: They build a cost model that accounts for the binding resource, which they evaluate across three hardware classes <ref:2610.15629#pg2>. They use this model in two ways: first as a predictor to estimate latency given a configuration and operating point, and second as an allocator to pick the best configuration under a latency or accuracy budget <ref:2610.15629#pg1>.

Lu: What they claim is that their model is better at capturing what limits execution than just using byte counts for speed proxies <ref:2610.15629#pg1>. They show this by evaluating it on Qwen3-4B, Qwen2 point 5-7B-Instruct, and Llama-three point two-1B-Instruct across three hardware classes <ref:2610.15629#pg3>.

Meng: I'm curious about how they handle the actual work on this cost model. It sounds complicated to build something that accounts for compute, weight loading, and KV cache access all at once. How do you separate those costs?

Tom: They use a roofline approximation to separate the arithmetic and weight-loading costs <ref:2610.15629#pg2>. The work time is approximated by the maximum of the compute time and weight-loading time, which sums up unit times, attention time, and a shared baseline for remaining work <ref:2610.15629#pg3>.

Jane: So it's not just one big number for the whole process; it breaks down into different parts based on what's happening in that specific phase <ref:2610.15629#pg3>. That sounds really practical when you're trying to tune a model for deployment.

Lu: They also talk about calibrating the model to a deployment, where they fit parameters like effective throughput, bandwidth, and overheads for each GPU and serving stack <ref:2610.15629#pg3>. This calibration supplies things like attention coefficients for prefill and decode <ref:2610.15629#pg3>.

Meng: Calibration is key because those numbers have to be real, right? If you don't calibrate to the actual hardware, the model's predictions are just guesses. What about how they handle things like batch size and context length in that calibration process?

Tom: They state that shapes, format bytes, and token expert counts are known inputs for calibration <ref:2610.15629#pg3>. And they jointly fit rates, bandwidth, blend scale, overheads for both prefill and decode phases <ref:2610.15629#pg3>.

Jane: So it's a shared baseline approach across the different phases of serving—prefill and decode—based on what they measure across those three hardware classes <ref:2610.15629#pg3>. That sounds like a lot of tuning to get right for real-world deployment.

Paper summary: Lu: And then they move into the allocator part, which is where they combine these modeled time costs with measured per-tensor accuracy costs <ref:2610.15629#pg1>. They use a dynamic program to select the best configuration under a latency or accuracy budget <ref:2610.15629#pg1>.

Meng: I'm interested in how this allocator compares to what people are already doing, like byte-aware allocation. Does this cost model actually give you a better choice?

Tom: They compare it against byte-aware and sensitivity-only allocation on Qwen3-Coder-30B-A3B-Instruct and find that the cost model allocator generates faster configurations <ref:2610.15629#pg1>. For example, one cost-based configuration takes sixty-eight point seven seven milliseconds with perplexity sixteen point three nine zero, versus seventy-five point seven four milliseconds and sixteen point four six seven for byte-aware <ref:2610.15629#pg1>.

Jane: That's a tangible difference in speed and accuracy, so it sounds like the model is actually capturing those hidden costs we talked about earlier <ref:2610.15629#pg3>. So what does this mean for someone who just wants to get a model running quickly?

Lu: It suggests that fewer bits don't always mean faster serving because you have to consider the compute, weight loading, or KV cache access depending on the phase and context length <ref:2610.15629#pg1>.

Meng: And what about when things get really long? If a user feeds in a very long context, how does this model handle that growing budget?

Tom: They can extend the allocator to include cache precision, denoted by kappa, where access time scales approximately with b(kappa)L in memory-bound decode <ref:2610.15629#pg3>. Growing context leaves less budget for weight operations, which prompts the model to switch weights to lower precision as context grows <ref:2610.15629#pg3>.

Jane: So if you have a long conversation going on, the system will automatically start using lower precision weights because it's running out of time budget for loading things <ref:2610.15629#pg3>. That’s a very smart way to manage resources dynamically.

Lu: The paper also mentions how they quantify accuracy change with a unit using A(u, p), which is the mean Kullback–Leibler divergence from reference to quantized next-token distributions <ref:2610.15629#pg1>. And time change, delta S(u, p), is normalized by the reference configuration’s predicted step time T0 at the same operating point <ref:2610.15629#pg3>.

Meng: I need to know what happens when you try to deploy this at scale. What's the actual cost of running this model with all these calibrations and dynamic scheduling? Is it feasible?

Tom: They mention that the calibration procedure minimizes equally weighted mean squared relative errors Jbase for prefill and decode phases <ref:2610.15629#pg3>. All platforms use the same cost equation and fitter, which helps standardize things across different hardware <ref:2610.15629#pg3>.

Jane: It sounds like they spent a lot of time on making sure their calibration is fair so that it works consistently regardless of the specific GPU you're using <ref:2610.15629#pg3>. So, what’s the overall message for us about this cost model?

Lu: The core message is that calibrated compute, weight-loading, and KV-cache costs support latency prediction and mixed-precision allocation <ref:2610.15629#pg1>. They capture differences that bytes alone miss and the increase in decode latency as context grows <ref:2610.15629#pg3>.

Paper summary: Tom: They also point out a limitation, which is that small timing errors can still make the slower configuration look faster <ref:2610.15629#pg4>. And they note that the 16K context comparison uses two WikiText passages on one device at fixed cache precision, so broader evaluation is needed to establish how well it generalizes <ref:2610.15629#pg4>.

Jane: So while this cost model is a big step forward in making better predictions and allocation decisions, the authors are still acknowledging that they need more work on improving timing reserves and modeling quality across longer passages <ref:2610.15629#pg4>.

Lu: The whole point is that by accounting for these specific costs—compute, loading, KV cache—we get a more realistic picture of serving speed than simpler proxies provide <ref:2610.15629#pg1>. This moves us toward better resource management in the AI serving stack.

Meng: For me, what matters is that when I deploy this, I have a tool that can actually tell me if I'm bottlenecked by compute or by loading weights during inference <ref:2610.15629#pg3>. That kind of diagnostic information is something we need for production environments.

Tom: Exactly. This cost model gives you a way to predict latency before you even deploy, based on the phase and batch size <ref:2610.15629#pg3>. It’s about making sure the hardware utilization matches what the software actually demands <ref:2610.15629#pg3>.

Jane: So, to sum up, "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving" gives us a way to move beyond simple byte counting by modeling the specific costs of compute, loading, and cache access <ref:2610.15629#pg1>. It shows how this model helps in both predicting latency and making smarter decisions about which precision to use at any given moment <ref:2610.15629#pg1>.

Lu: The implication is that for future work, we need to focus on making these models more robust when context grows, because they showed that the decode latency increases as context gets longer <ref:2610.15629#pg3>.

Meng: I think it's useful because it provides a framework. It’s not just one specific model; it’s a way to build better systems that can handle the complexity of running these large models efficiently <ref:2610.15629#pg3>.

Tom: Yeah, the focus is on making sure we don't make mistakes in our deployment choices based on incomplete information <ref:2610.15629#pg4>. It’s about having a more complete picture of what the hardware is actually doing during inference <ref:2610.15629#pg3>.

Jane: So that’s what we have here with "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving" <ref:2610.15629#pg1>. It’s a tool that helps us understand the trade-offs between speed and accuracy by looking at the real costs involved <ref:2610.15629#pg3>.

Lu: It’s about making resource decisions based on what limits execution in different stages of the model's operation, not just a single metric like memory size <ref:2610.15629#pg3>.

Meng: I think that helps because when I look at production, sometimes the bottleneck shifts from compute to loading weights depending on the specific model and batch size <ref:2610.15629#pg3>. This paper gives us a way to predict that shift <ref:2610.15629#pg3>.

Paper summary: Tom: So we're seeing how the cost model can help us choose a precision that balances those different bottlenecks, rather than just picking the smallest possible number <ref:2610.15629#pg1>. It’s about optimization across multiple factors, not just one <ref:2610.15629#pg3>.

Jane: That sounds like a really practical application for anyone working on deploying these large language models in the real world <ref:2610.15629#pg3>. It shows that understanding the mechanics behind serving, like how weight loading impacts things, is important <ref:2610.15629#pg3>.

Lu: And that's what they are pointing toward for future work—making these timing reserves and modeling quality better for context-dependent scheduling <ref:2610.15629#pg4>. It’s about the next level of detail in resource management <ref:2610.15629#pg4>.

Meng: Yeah, I think that’s where the real value is for scaling things up, understanding those dependencies on context length <ref:2610.15629#pg3>. It's not just about running a single prompt; it's about running long conversations efficiently <ref:2610.15629#pg3>.

Tom: So we’re looking at the paper, "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving" <ref:2610.15629#pg1>. It shows that by modeling compute, loading, and cache costs separately, we can get a more accurate picture of serving latency across different hardware <ref:2610.15629#pg3>.

Jane: It’s a model that helps us understand the trade-offs between speed and accuracy when we use mixed precision in LLM serving <ref:2610.15629#pg3>. It's about using calibrated costs to make smarter allocation choices <ref:2610.15629#pg1>.

Lu: The main idea is that a byte-based proxy doesn't capture all of these costs, so fewer bits don't always mean faster serving <ref:2610.15629#pg3>.

Meng: And the allocator part is what makes it actionable, combining those predicted time costs with measured accuracy costs to find the best configuration under a budget <ref:2610.15629#pg1>.

Tom: It’s about using that dynamic program to select the optimal setting based on whether we are prioritizing latency or accuracy <ref:2610.15629#pg3>.

Jane: So, for someone listening who doesn't read papers, this paper tells us that when deploying an AI model, you can't just look at its size or precision; you have to account for the actual work it does and how that work is limited by different parts of the system <ref:2610.15629#pg3>.

Lu: That’s the core idea behind "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving" <ref:2610.15629#pg1>. It shifts the focus from a simple size metric to a comprehensive cost estimation framework <ref:2610.15629#pg3>.

Meng: I think it’s important because it moves us closer to systems that can actually scale reliably in production, not just on benchmarks <ref:2610.15629#pg3>.

Tom: Yeah, the calibration part is what grounds this whole thing—making sure the model knows exactly what the hardware is capable of at different operating points <ref:2610.15629#pg3>.

Jane: So that’s it for this segment on "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving" <ref:2610.15629#pg1>. We'll be back next time to look at something else from arXiv.

Conclusion: Tom: So, we’re wrapping up this look at "The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving." Basically, they’ve built a system that looks at the real costs—the compute time, the weight loading time, and even how much memory is needed for the key-value cache—instead of just guessing based on how many bytes are in the model.

Jane: That seems like a really smart way to look at deployment because it moves past just looking at model size and tells you what actually limits your speed when you're serving these massive AI models.

Lu: They show that this cost model, which they call a predictor and an allocator, can be calibrated across different hardware classes—like the AMD Instinct chips and some consumer GPUs—and it gets within about ten percent of the actual measurement for models like Qwen3-4B <ref:2610.15629#pg3>.

Meng: But what’s really interesting is how this allocator works; it uses a dynamic program to choose a configuration that hits your latency or accuracy budget, and they found that their cost-based approach actually picked faster configurations than the byte-aware methods.

Tom: That means you can stop picking the smallest possible number of bits blindly; you’re optimizing across compute, loading, and cache access simultaneously.

Jane: It changes things for someone just listening to this show because it shows that serving speed isn't one single number; it depends on the phase of the model running and how long your context is.

Lu: They also extended this to include context length, showing that when a conversation gets really long, the system will automatically switch weights to lower precision because there’s less time budget left for weight operations.

Meng: That dynamic switching based on context length is something I’d actually deploy; it manages resources better than just setting a fixed precision.

Tom: Yeah, it’s about making sure the hardware utilization matches what the software demands in real-time, not just on paper benchmarks.

Jane: So to sum up, this paper gives us a framework that uses calibrated costs to predict latency and intelligently decide on mixed precision allocation across different hardware setups.

Lu: It’s a solid model for moving toward more realistic resource management in the AI serving stack.

Meng: The next question for me is how robust this whole calibration is when you move from short text to really long documents, because they flag that timing errors can still make a slower configuration look faster.

Tom: That’s a fair point, Meng; they admit that the model needs more work on improving modeling quality across complete passages, which means it might not be perfect for every single use case yet.

Mukund Athmaram, Manoj Rao, Liang Ma

Advanced Micro Devices Inc. · Cornell University

cs.LG

Submitted: 2026-10-11

Updated: 2026-10-11

Code: https://github.com/ggml-org/llama.cpp

Project page: https://jax-ml.github.io/scaling-book

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: A cost model that accounts for binding resources supports latency prediction and mixed-precision allocation, which addresses gaps missed by byte-based proxies.

Key concepts

Cost Model: This model estimates serving latency by calculating the time required for different operations, separating arithmetic work from weight loading using a roofline approximation. It sums up compute time, attention time, and a baseline for remaining work to predict total execution duration.
Predicting Served Speed: The model acts as a predictor for serving latency based on configuration and operating point. Testing shows it is within 10% of actual measurements across various models and hardware classes. It is particularly accurate in scenarios like Mixture-of-Experts (MoE) where it accounts for dynamic changes in active expert counts.
Allocator: The allocator uses a dynamic program to choose the best model configuration under constraints like a latency or accuracy budget. It minimizes the estimated cost while ensuring that accuracy loss and time increase remain within predefined limits, effectively balancing speed and quality.
Binding Resources: This refers to resources that limit performance, such as compute power, memory bandwidth for weight loading, and KV-cache access times. The model uses a roofline approximation to separate these costs accurately, which is a key improvement over simpler byte-based estimations.

Summary

The gist: A cost model that accounts for binding resources supports latency prediction and mixed-precision allocation, which addresses gaps missed by byte-based proxies.

The Cost Model

A cost model is built to estimate serving latency by accounting for the binding resource, evaluated as a predictor across three hardware classes The cost model accounts for the binding resource, evaluated as a predictor across three hardware classes Given a unit u at precision p, it estimates its time at the current phase, batch size, and sequence length GEMM work is estimated using the roofline approximation to separate arithmetic and weight-loading costs The work time t(g, p) is approximated by the maximum of compute time and weight-loading time Tstep(cfg; B, S, L) sums unit times, attention time, and a shared baseline for remaining work

Predicting Served Speed

The model is used as a predictor to estimate serving latency given a configuration and operating point Evaluation on Qwen3-4B, Qwen2.5-7B-Instruct, and Llama-3.2-1B-Instruct across three hardware classes shows that our model is within 10% of measurement Figure 3 compares prefill error with byte and constant estimates Our model is more accurate in some cases, such as the MoE example where it accounts for how expected active expert counts and weight traffic change with batch size

The Allocator

The allocator combines modeled time costs with measured per-tensor accuracy costs using a dynamic program to select a configuration under a latency or accuracy budget The optimization problem seeks to minimize the cost while satisfying constraints like ∆S(u, p) ≤ τ or ∆A(u, p) ≤ ϵ For a latency budget D, τ is set as D/T0 − 1

Measured Allocation Tradeoffs

Comparing our cost-based allocator against byte-aware and sensitivity-only allocation on Qwen3-Coder-30B-A3B-Instruct shows that the cost model allocator generates faster configurations For example, one cost-based configuration takes 68.77 ms with perplexity 16.390, versus 75.74 ms and 16.467 for byte-aware Both allocators choose identical decode configurations

Context Scheduling

The allocator can be extended to include cache precision κ, where access time scales approximately with b(κ)L in memory-bound decode Growing context leaves less budget for weight operations, prompting the model to switch weights to lower precision as context grows Figure 7 shows the resulting schedule through 16K at a 60 ms planning target, shifting toward lower precision as context grows and exceeding the target on one of 1,536 measured steps

Conclusion

Calibrated compute, weight-loading, and KV-cache costs support latency prediction and mixed-precision allocation The predictor captures differences that bytes alone miss and the increase in decode latency as context grows Improving timing reserves and modeling quality across complete passages remain future work for context-dependent scheduling.

How it works

The cost model uses a roofline approximation to separate arithmetic and weight-loading costs The allocator solves a dynamic program that tracks the lowest accuracy cost on a discretized time grid It selects the best entry within the latency or accuracy budget and backtracks to recover the choices

Accuracy Impact

The accuracy change for a unit is quantified by A(u, p) which is the mean Kullback–Leibler (KL) divergence from reference to quantized next-token distributions The time change ∆S(u, p) is normalized by the reference configuration’s predicted step time T0 at the same operating point

Binding Resources

The model uses the roofline approximation to separate arithmetic and weight-loading costs Expected active experts are approximated by spreading BSk assignments evenly across E+ active experts The maximum time is used to approximate steady-state time under ideal overlap, as T ≈ max 2WM/rz compute, W b/BW z weight load

Calibration Procedure

All platforms use the same cost equation and fitter Calibration minimizes equally weighted mean squared relative errors Jbase for prefill and decode phases The calibration procedure involves minimizing Jbase, their equally weighted average, so neither phase receives more weight simply because it has more measurements

Limitations

Small timing errors can still make the slower configuration look faster The 16K context comparison uses two WikiText passages on one device at fixed cache precision, so broader evaluation is needed to establish how well it generalizes

New Assets

The core code supplement includes a README, dependencies, input schemas, a CPU example, provenance, and validation records The scope and omitted experimental components are documented.

Declaration of LLM usage

LLM assistants supported experiment implementation, analysis scripts, code review, and manuscript editing The cost model, fitting objective, and allocator do not invoke an LLM.

Experimental Setting/Details

The calibration constants and workloads, held-out splits, precision formats, batch and sequence lengths are described in the costmodel section and experimental descriptions The evaluation involves the AMD Instinct MI355X GPU (BF16/MXFP4), AMD Instinct MI300A datacenter APU (BF16/FP8), and Radeon 780M (F16/Q8) The calibration procedure involves minimizing equally weighted mean squared relative errors Jbase for prefill and decode phases

Experiment Statistical Significance

Predictor and context plots report means, while MI300A allocation uses pooled mean request times and an output-length slope We do not claim population-level statistical significance.

Code of Ethics

The research conforms to the NeurIPS Code of Ethics It uses only publicly available models and text corpora and involves no human subjects or sensitive data

Broader Impacts

The paper studies serving efficiency and discusses technical limitations, but does not include a societal-impact analysis.

Licenses for existing assets

The evaluated model releases and WikiText are cited, and the supplement records code provenance without redistributing model weights or text corpora The paper does not explicitly list all asset licenses and terms of use.

New Assets

The core code supplement includes a README, dependencies, input schemas, a CPU example, provenance, and validation records Its scope and omitted experimental components are documented.

Improvements for AI systems

The AI system can dynamically select precision configurations to meet latency or accuracy budgets by solving min problems defined in Equation 7, allowing for a per-tensor allocator for latency or accuracy budgets.
The system can predict serving latency with a calibrated cost model, as the paper states: given a configuration and operating point, it estimates serving latency using constants calibrated for the model, GPU, and serving stack.
The system can implement context-dependent precision scheduling by choosing a precision assignment that minimizes estimated accuracy loss while keeping predicted latency within the budget across different context ranges.