HTAM: Hierarchical Transition-Attended Memory for Operator Optimization
cs.CL
Submitted: 2026-05-28
Updated: 2026-11-03
Comments: 25 pages, 5 figures. Published at EMNLP 2026 (Oral)
Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 12248-12272, 2026
DOI: 10.18653/2026.emnlp-main.531
License: http://creativecommons.org/licenses/by/4.0/
The gist: High-performance GPU kernels are essential for efficient LLM deployment, yet optimizing them remains expertise-intensive.
Terminology
Abstract
High-performance GPU kernels are essential for efficient LLM deployment, yet optimizing them remains expertise-intensive. Recent LLM-based code generation makes automatic GPU operator generation promising, but operator optimization remains a hardware-aware search problem. Existing LLM-based methods face a granularity mismatch: coarse hints are reusable but hard to execute, whereas detailed memories are actionable but enlarge the search space and obscure optimization bottlenecks. The key challenge is therefore to organize optimization experience at an appropriate granularity. To address this issue, this paper proposes HTAM (Hierarchical Transition-Attended Memory), a coarse-to-fine framework for LLM-based operator optimization. HTAM builds a two-level Hierarchical Transition Graph (HTG) to organize coarse global directions, detailed local strategies, and transition experience between optimization steps. During each evolution step, HTAM selects a global direction from the current state and recent optimization history, retrieves the corresponding local strategy memory, and uses it to guide concrete CUDA code generation. Experiments on KernelBench demonstrate that HTAM improves correctness, fast-solution rate, and speedup over LLM-based baselines, while backend and Robust-KBench studies indicate transferable benefits from structured memory.
Sources
- GPT-4 Technical Report
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Evaluating Large Language Models Trained on Code
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment
- DeepSeek-V3 Technical Report
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- TileLang: A Composable Tiled Programming Model for AI Systems
- Astra: A Multi-Agent System for GPU Kernel Performance Optimization
- Ansor: Generating High-Performance Tensor Programs for Deep Learning
- QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data