GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference

arXiv:2610.17321 · cs.DC, cs.LG, cs.PF · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference".

Jane: The gist The GPU Goldmines framework provides a way to specify, execute,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: To summarize, this framework provides a way to define these tunable kernels and gives you tools to systematically run them and record how fast they perform across different setups. It’s about making sure those measurements are reliable, too.

Jane: They detail how a tunable kernel has parameters that define the execution context, which makes it flexible for different scenarios. Then they describe the kernel miner operating in three modes: exhaustive, heuristic, and warm-start heuristic to show how they explore the parameter space systematically.

Lu: The system uses those definitions to generate actual concrete kernel instances by either trying every valid setting or searching for good ones using heuristics based on those definitions. And to keep things solid, they include built-in support for warm-up executions, repeated timing iterations, and correctness validation during measurement.

Meng: I wonder about that exhaustive exploration part when you have so many dimensions involved in LLM operations. Does that mean you're just running everything and waiting for it to finish?

Tom: They filter the values using constraint functions called valid args and valid params, as well as the device limits, before instantiating and executing each configuration on the target device in two phases: warmup and measurement. During warmup, they run the kernel sixteen times to stabilize performance and validate outputs against a reference implementation.

Jane: Then during measurement, they run it one hundred twenty-eight times to get an average execution time recorded along with metrics like GFLOPs, which tells you the actual computational work being done. All these results go into that centralized database indexed by device, browser, operating system, kernel type, input dimensions and tuning parameters.

Lalam: So the system systematically explores valid configurations and stores all those performance measurements in a central database structured by context details to track how different setups perform against each other.

The paper's summary: Tom: The authors show that this method actually yields some significant performance gains when you compare it to existing WebGPU LLM kernels. This is where the real numbers come in.

Jane: Specifically, they show that these tuned configurations outperform existing WebGPU LLM kernels by seven point zero times over ONNX Runtime and one point four one times over llama.cpp specifically for matrix multiplication operations.

Lu: On top of that, they report average end-to-end prefill speedups of one point four eight times across these workloads when using llama.cpp. That’s a substantial gain in how fast the model can generate text from scratch.

Meng: That one point four eight times prefill speedup is pretty solid, but I gotta ask about the setup itself. Does this whole tuning process add a lot of overhead when you first set up these new configurations on hardware?

Tom: The framework helps derive simple parameter selection policies that map execution contexts and input shapes to high-performing configurations, reaching an average of ninety percent of hyper-tuned performance for fixed input sizes. This means you get close to the best possible speed without doing all the manual work yourself.

Jane: And here’s something interesting for new hardware: for previously unseen devices, the database allows for strong warm starts during heuristic search, enabling high-performing parameters to be identified fifty two point four percent faster than tuning from scratch.

Lalam: They also use a Bayesian approach that uses prior observations to guide the search when looking at new inputs; they select the nearest training size in Euclidean log2 space and reuse its Min-Avg portable configuration for unseen shapes, achieving eighty four percent retained performance for those cases.

The paper's improvements: Tom: So, to wrap up this paper on GPU Goldmines, it demonstrates that you can derive portable configurations that recover ninety percent of device-optimal performance when you know the input sizes.

Jane: And it also shows that for unknown devices or shapes, the system can warm-start tuning fifty two point four percent faster than starting from scratch. It’s a big win for how we manage performance across different WebGPU targets.

Lu: The authors conclude that their data-driven methods allow them to characterize portability across WebGPU’s broad target space and construct decision procedures that map execution contexts to high-performing parameter settings, which could become a standard way to approach performance portability in the WebGPU ecosystem for LLMs.

Meng: From a deployment standpoint, the ability to reuse prior measurements is key because it cuts down on the time needed for iterative testing on new hardware when you're deploying something new.

Lalam: I think the success of GPU Goldmines comes from its extensible design, letting both the database and kernel families grow alongside the WebGPU ecosystem as things change.

Tom: So they’ve shown how a centralized database can accelerate autotuning and configuration selection for matrix multiplication, matrix-vector multiplication, and flash attention by giving us these concrete performance metrics.

Jane: It proves that even with diverse hardware, you can get substantial speedups by having a system that learns from past performance on different devices through this framework.

Lu: The implication is that this data-driven approach to configuration selection could become a standard way to approach performance portability in the WebGPU ecosystem for LLMs.

Meng: For us in production, it means less manual optimization and more reliance on these automated selection policies when we deploy new model variants.

Conclusion: ---: Conclusion ---

Tom: So, we've seen how GPU Goldmines gives us a way to define tunable WebGPU kernels for edge LLM inference and build this centralized database to speed up all that autotuning later on.

Jane: Exactly. It shows you can get these tuned configurations, which might only hit seventy-seven percent of their potential performance on another device, but the database helps you figure out what’s best based on what you already measured.

Lu: The core idea is using those measurements to derive simple parameter selection policies that map execution contexts and input shapes to high-performing configurations for fixed input sizes.

Meng: For us in production, that means less manual optimization when we deploy a new model variant because we can rely on these automated selection policies.

Lalam: And for previously unseen devices, the warm-start heuristic lets you find high-performing parameters fifty two point four percent faster than tuning from scratch.

Tom: So they’ve successfully demonstrated how to use a centralized database to accelerate autotuning and configuration selection for matrix multiplication, matrix-vector multiplication, and flash attention.

Jane: It shows that even with diverse hardware, you can get substantial speedups by having a system that learns from past performance on different devices.

Lu: The implication is that this data-driven approach to configuration selection could become a standard way to approach performance portability in the WebGPU ecosystem for LLMs.

Meng: For scaling up, it means less time spent on iterative testing and more time actually building the product.

Lalam: I think the success of GPU Goldmines comes from its extensible design, letting both the database and kernel families grow alongside the WebGPU ecosystem.

Tom: That's a solid way to handle performance portability across different hardware targets in LLM inference.

Jane: It’s pretty fascinating how much information you can extract just by systematically executing and recording measurements on every device.

Lu: Next up, we have another paper looking at how embedding bias affects conditional independence testing in AI models.

Rithik Sharma, Reese Levine, Zheyuan Chen, Ananthajit Srikanth, Yanwen Xu, Tyler Sorensen

University of California Santa Cruz · EPFL · Microsoft Research

cs.DC, cs.LG, cs.PF

Submitted: 2026-10-12

Updated: 2026-10-12

Comments: 11 pages, 5 figures, 5 tables

Code: https://github.com/jansel/opentuner

Project page: https://microsoft.github.io/DirectXSpecs

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The GPU Goldmines framework provides a way to specify, execute, and tune WebGPU kernels for LLM inference by creating a centralized performance database that accelerates subsequent

Key concepts

Tunable WebGPU Kernel Families: The framework creates parameterized implementations of critical LLM operations like matrix multiplication and flash attention. These kernels are tunable by performance choices such as workgroup size and tile dimensions, allowing the system to test different configurations that suit specific hardware capabilities.
Centralized Performance Database: This database stores all kernel measurements indexed by device, browser, OS, and tuning parameters. It accumulates data across multiple runs and devices. This centralized knowledge is crucial for deriving policies that map execution contexts to high-performing parameter settings.
Warm-start Heuristic Tuning: When tuning on a new device or input shape, the system uses prior measurements from the database to guide the search. This warm start allows the miner to find optimal parameters significantly faster—up to 52.4% quicker than starting from scratch—by reusing previously successful configurations.
Portability Gaps and Recovery: The framework reveals that performance tuned for one device often yields only 77% of its potential on another. The database is used to derive policies that recover up to 90% of device-optimal performance for known input sizes, demonstrating how to achieve better cross-platform portability.

Summary

The gist The GPU Goldmines framework provides a way to specify, execute, and tune WebGPU kernels for LLM inference by creating a centralized performance database that accelerates subsequent autotuning and configuration selection.

How it works

The GPU Goldmines framework addresses the difficulty of achieving high performance portability across diverse hardware by making kernels tunable, which exposes a semantics-preserving family of implementations parameterized by performance-relevant choices like workgroup size and tile dimensions The framework instantiates kernels for key LLM building blocks: matrix multiplication, matrix-vector multiplication, and flash attention. It performs tuning in a specific execution context while retaining measurements in a centralized performance database

The toolsuite includes several components for systematic exploration and data management:

** Specifying, Executing, and Recording The framework provides an interface for defining tunable kernels, along with tools for systematically executing them and recording their performance. A tunable kernel is defined by a parameterized implementation, exposed tuning parameters, validity constraints, and input properties defining the execution context. The system can generate and execute concrete kernel instances by either exhaustively exploring the valid parameter space or searching it heuristically. All measurements are written back to a centralized database, allowing tuning results to accumulate across runs and devices. To ensure that these measurements are reliable, GPU Goldmines includes built-in support for warm-up executions, repeated timing iterations, and correctness validation. The kernel miner operates in three modes: Exhaustive, Heuristic, and Warm-start heuristic. In each case, the miner filters values using the constraint functions valid args and valid params and the device limits. For each valid configuration, the kernel template is instantiated, GPU buffers and pipeline resources are allocated, and the kernel is executed on the target device in two phases: warmup and measurement. During warmup, the kernel is run 16 times to stabilize performance and outputs are validated against the reference implementation. During measurement, the kernel is executed 128 times and the average execution time is recorded along with kernel-specific performance metrics such as GFLOPs. All performance results from the kernel miner are stored in a centralized database indexed by device, browser, operating system, kernel, input dimensions, and tuning parameters. The database grows continuously as new devices visit the site and contribute results. To address performance portability gaps in matrix multiplication across different devices and browsers, the framework shows that a configuration tuned for one device achieves only 77% of its potential performance on another device on average. The resulting data reveals substantial portability gaps. The database is used to derive simple parameter selection policies that map execution contexts and input shapes to high-performing configurations, reaching an average of 90% of hyper-tuned performance for fixed input sizes. For previously unseen devices, database-derived configurations provide strong warm starts for heuristic search, enabling high-performing parameters to be identified 52.4% faster than tuning from scratch. The system uses a Bayesian approach that uses prior observations to guide search. For previously unseen input shapes, the method selects the nearest training size in Euclidean log2 space and reuses its Min-Avg portable configuration to avoid overfitting. This distance-based rule transfers well to unseen matrix sizes, achieving retained performance of 84%. The framework achieves an average end-to-end prefill speedup of 1.48× across these workloads. The framework's contributions include GPU Goldmines, a browser-based framework for specifying tunable WebGPU kernels, systematically executing valid configurations, and recording performance measurements, Tunable WebGPU kernel families for three performance critical LLM routines: matrix multiplication, matrix-vector multiplication, and flash attention, and Data-driven methods that reuse the database to characterize portability, derive configuration selection policies, and accelerate tuning in new contexts. The evaluation shows that our tuned configurations outperform existing WebGPU LLM kernels, including 7.0× over ONNX Runtime and 1.41× over llama.cpp. The application-level evaluation observes an average end-to-end prefill speedup of 1.48× across these workloads. The paper concludes that GPU Goldmines’ extensible design allows the database and kernel families to grow alongside the WebGPU ecosystem. The final result is that From the database, we derive portable configurations recovering 90% of device-optimal performance, size prediction achieving 84% on unseen sizes, and seeded autotuning converging 52.4% faster. The framework's success is demonstrated by the observation that Matrix multiplication is the most tuning-sensitive kernel, with smaller chat-oriented sizes being the least portable. The overall performance gain observed in llama.cpp is a geometric mean speedup of 1.41× over llama.cpp kernels and 7.0× over ONNX Runtime kernels. The framework's ability to adapt to new contexts is supported by the fact that "the database provides the basis for three capabilities: (1) deep characterization across WebGPU’s broad target space, (2) accelerating subsequent tuning runs by reusing prior measurements, and (3) constructing decision procedures that map execution contexts to high-performing parameter settings. The framework's ability to select portable configurations for known input sizes achieves a higher average with a worst case of 60% in our finegrained results. The framework's ability to warm-start tuning for unknown devices converges 52% faster on average (iteration 14 vs 29) and finds configurations with 4.9% higher performance. The framework's ability to predict portable configurations for unseen sizes achieves a retained performance of 80%, 77%, and 96% for M = 32, 64, and 4096, respectively. The framework's ability to select portable configurations for known input sizes achieves 90% retained performance on average. The framework's ability to warm-start tuning for unknown devices converges 52.4% faster. The framework's ability to predict portable configurations for unseen sizes achieves 84% retained performance. The framework's ability to select portable configurations for known input sizes achieves 90% retained performance on average. The framework's ability to warm-start tuning for unknown devices converges 52% faster on average. The framework's ability to predict portable configurations for unseen sizes achieves 84% retained performance. The framework's ability to select portable configurations for known input sizes achieves 90% retained performance on average. The framework's ability to warm-start tuning for unknown devices converges 52% faster on average. The framework's ability to predict portable configurations for unseen sizes achieves 84% retained performance. The framework's ability to select portable configurations for known input sizes achieves 90% retained performance on average. The framework's ability to warm-start tuning for unknown devices converges 52% faster on average"

Improvements for AI systems

textbf Tunable WebGPU Kernel Framework (GPU Goldmines) for Edge LLMs: Enables cross-platform performance portability by providing a framework for specifying, executing, and tuning WebGPU kernels. This allows systems to leverage pre-existing high-performance implementations while systematically exploring tuning spaces across diverse devices, shapes, and browsers.
textbf Data-Driven Configuration Selection: Accelerates autotuning and configuration selection by using a centralized performance database to derive policies. The paper states that for devices already represented in the database, we derive simple parameter selection policies that map execution contexts and input shapes to high-performing configurations, reaching an average of 90% of hyper-tuned performance for fixed input sizes.
textbf Accelerated Tuning Initialization: Reduces the time required to find optimal kernel parameters on new hardware by utilizing prior measurements. The framework allows for warm-start tuning on unseen devices, enabling them to identify high-performing parameters 52.4% faster than tuning from scratch.
textbf Enhanced Matrix Multiplication Performance: Achieves significant speedups over existing WebGPU kernels for critical LLM components. The tuned configurations deliver geometric mean speedups of 1.41× over llama.cpp kernels and 7.0× over ONNX Runtime kernels, and end-to-end improvements of 1.48× in the prefill stage of llama.cpp for chat workloads.
textbf Context-Aware Kernel Optimization: Provides specialized kernel implementations tailored to specific hardware capabilities, such as leveraging hardware matrix units where available. For example, a tuned matmul-mma kernel shows a 7.0× geometric mean improvement across all device-size combinations when hardware matrix units are present.

Abstract

As demand for LLM inference grows and consumer GPUs grow more capable, edge deployment is becoming increasingly attractive. WebGPU is compelling in this setting because it offers cross-platform portability and browser accessibility. However, these same strengths also make high performance difficult to achieve, as an implementation that performs well in one context may not perform well in another. To address this, we present GPU Goldmines, a framework for specifying, executing, and tuning WebGPU kernels. We instantiate GPU Goldmines with kernels for key LLM building blocks: matrix multiplication, matrix-vector multiplication, and flash attention. We characterize performance across diverse devices, shapes, and browsers, and use the resulting database to accelerate subsequent autotuning and configuration selection. On shapes drawn from Llama and Qwen models, our matrix multiplication kernels outperform ONNX Runtime and llama.cpp WebGPU kernels by 7.0x and 1.41x geometric mean, respectively, and deliver end-to-end improvements of 1.48x in the prefill stage of llama.cpp for chat workloads.