Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators

Name: Vectorizing the Trie: Efficient Constrained Decoding for LLM-based Generative Retrieval on Accelerators
Uploaded: 2026-10-09T00:46:42.669981+00:00
Duration: 345 s
Description: STATIC transforms trie-based constrained decoding into vectorized sparse matrix operations for hardware accelerators like TPUs and GPUs. It replaces slow pointer-chasing with static CSR matrices and branch-free kernels, achieving massive speedups over CPU methods while maintaining low latency for production LLM retrieval systems.

STATIC transforms trie-based constrained decoding into vectorized sparse matrix operations for hardware accelerators like TPUs and GPUs. It replaces slow pointer-chasing with static CSR matrices and branch-free kernels, achieving massive speedups over CPU methods while maintaining low latency for production LLM retrieval systems.

Open the full episode