KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation
cs.LG, cs.SE
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 59 pages, 7 figures
Code: https://github.com/ByteDance-Seed/cudaLLM
License: http://creativecommons.org/licenses/by/4.0/
The gist: High-performance GPU kernels are essential to modern machine learning systems, yet automatically generating kernels that are both correct and efficient remains challenging.
Terminology
Abstract
High-performance GPU kernels are essential to modern machine learning systems, yet automatically generating kernels that are both correct and efficient remains challenging. Existing LLM-based approaches face two major limitations: the scarcity of high-quality training data aligned with the model's current capabilities, and the inherent trade-off between kernel correctness and performance. To address these challenges, we propose KernelZero, a co-evolution framework that continuously improves GPU kernel generation through two specialized models: a Proposer that generates Torch modules from API sets and a Coder that translates them into CUDA or Triton kernels. KernelZero uses a frontier-driven module generation mechanism to continuously produce capability-aligned training modules based on the Coder's current weaknesses. It further introduces Correctness-Aware Group Relative Policy Optimization (CA-GRPO), which optimizes performance only after correctness becomes sufficiently reliable. By alternating the optimization of the Proposer and Coder, KernelZero forms an automatic curriculum that enables targeted and training-efficient capability improvement. Empirically, KernelZero-7B surpasses Claude-4.5-Sonnet on CUDA and DeepSeek-V4-Pro on Triton. On KernelBench Level 1 and 2, it achieves CUDA pass@1 scores of 75.8% and 69.6%, respectively, with pass@10 reaching 100% and 97%. On Triton, it achieves pass@1 scores of 77.2% and 72.5%, respectively.
Sources
- Kevin: Multi-Turn RL for Generating CUDA Kernels
- K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
- CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- STARK: Strategic Team of Agents for Refining Kernels
- GLM-5: from Vibe Coding to Agentic Engineering
- R-Zero: Self-Evolving Reasoning LLM from Zero Data
- Qwen2.5-Coder Technical Report
- TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators
- AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMs
- Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations
- gpt-oss-120b & gpt-oss-20b Model Card
- KernelBench: Can LLMs Write Efficient GPU Kernels?
- Astra: A Multi-Agent System for GPU Kernel Performance Optimization
- FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
- Qwen3 Technical Report
- QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks