The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving
A cost model was developed to predict latency and guide mixed-precision allocation for LLM serving by accounting for binding resources like compute, weight-loading, and KV-cache costs. This model outperforms byte-based proxies by accurately estimating time across three hardware classes. The resulting allocator selects configurations that balance speed and accuracy within set budgets.