CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs
cs.LG
Submitted: 2026-05-08
Updated: 2026-05-08
Code: https://github.com/NVIDIA/compute-eval
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware
Terminology
Abstract
Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence.
Sources
- Kevin: Multi-Turn RL for Generating CUDA Kernels
- AVO: Agentic Variation Operators for Autonomous Evolutionary Search
- CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations
- KernelBench: Can LLMs Write Efficient GPU Kernels?
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
- CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimization
- CUDABench: Benchmarking LLMs for Text-to-CUDA Generation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks