GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU
cs.CY
Submitted: 2026-10-29
Updated: 2026-10-29
Comments: 11 pages, 7 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: WebGPU enables private, offline LLM inference inside web pages, but browser dispatch, inter-process communication…
Terminology
Abstract
WebGPU enables private, offline LLM inference inside web pages, but browser dispatch, inter-process communication, synchronous readback and single-token kernels slow decoding and make conventional speculative decoding inefficient. We present a WebGPU runtime that addresses these browser-specific costs. Deterministic small-batch integer kernels accelerate multi-token verification while preserving exact agreement with greedy decoding in the same quantized runtime. A GPU-resident loop performs drafting, verification, acceptance and KV-cache maintenance without a per-token CPU decision. A draft-free context-mixing drafter reuses page content, conversation history and session memory, while a device-aware controller adapts the speculation width and the submission window to measured costs. Across seven models from 0.5B to 8B parameters, 11 tasks and three client GPUs, our runtime decodes 3.5 to 5.4 times faster than WebLLM on a laptop GPU, 4.8 to 6.2 times faster on an integrated GPU and 4.8 to 8.6 times faster on a workstation GPU. The GPU-resident loop and speculation contribute a further 1.8 to 3.2 times over the optimized kernels on the workstation. Outputs match the quantized greedy decoding of the runtime exactly, and the mean GSM8K accuracy matches that of WebLLM. Optional draft heads raise the peak speedup to 10.2 times, while session memory provides additional gains without downloading extra weights.
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance