Correct First, Fast Second: What a Local 7B Code Model Can and Cannot Do for GPU Kernel Optimization
cs.LG
Submitted: 2026-11-09
Updated: 2026-11-11
Code: https://github.com/pango0/KernelBench-Agentic
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study how much the interaction strategy around a fixed, locally hosted 7B-parameter code model (Qwen2.5-Coder-7B-Instruct) affects its ability to replace PyTorch operators with custom CUDA kernels.
Terminology
Abstract
We study how much the interaction strategy around a fixed, locally hosted 7B-parameter code model (Qwen2.5-Coder-7B-Instruct) affects its ability to replace PyTorch operators with custom CUDA kernels. On 30 KernelBench tasks (levels 1-3, first 10 each, evaluated on a V100), we compare four regimes of increasing structure: zero-shot prompting, profiling-guided prompting, iterative refinement on execution feedback, and a multi-agent loop (analysis, retrieval, generation, evaluation, feedback triage) in which the same model fills four roles around a non-LLM evaluator. Three findings stand out. (1) Correctness, not speed, is the binding constraint: even correct generated kernels average 0.97-0.99x the speed of eager PyTorch, because a 7B model cannot out-write cuBLAS/cuDNN on dense GEMM and convolution; the only real wins are fused memory-bound epilogues. (2) The best strategy depends on difficulty: profiling-guided prompting is strongest overall (43% correct, 27% faster than baseline), but the agent loop is the only method that solves any level-2 fusion problem (3/10 vs. 0/10 for all others). (3) Naive agentic scaffolding is actively harmful: our first agent scored 3% correct, below zero-shot's 30%, and the decisive repairs were matching the agent's ambition to the model's competence and spending compute on best-of-n sampling under a real execution oracle (3% to 33% correct from n=1 to n=8), not on more reasoning steps.
Sources
- KernelBench: Can LLMs Write Efficient GPU Kernels?
- Qwen2.5-Coder Technical Report
- cuDNN: Efficient Primitives for Deep Learning
- Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization
- Kevin: Multi-Turn RL for Generating CUDA Kernels
- CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learning
- CUDA-LLM: LLMs Can Write Efficient CUDA Kernels
- Astra: A Multi-Agent System for GPU Kernel Performance Optimization
- TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators
- AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMs
- Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks
- Evaluating Large Language Models Trained on Code
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis