TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout
cs.AR
Submitted: 2026-07-04
Updated: 2026-11-04
Comments: Accepted to HPCA 2027. Jae Hyung Ju and Euijun Chung contributed equally
Code: https://github.com/deepseek-ai/DeepGEMM
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM).
Terminology
Abstract
Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade LLM inference performance by up to an order of magnitude. The root cause is read amplification during tiled matrix multiplication, in which each memory request fetches far more data than the compute tile needs. Such waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, causing each request to spill well beyond tile boundaries. To eliminate read amplification in LGMS, we propose to use tile-major layout. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major by only changing the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level GPU simulator using Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with our adaptive degree prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5 us NAND read latency, reducing the geomean slowdown from 1.62-6.69x with conventional layouts to within 10% of an HBM-only baseline.
Sources
- GPT-4 Technical Report
- gpt-oss-120b & gpt-oss-20b Model Card
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- The Llama 3 Herd of Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Acceleration
- MiniMax-01: Scaling Foundation Models with Lightning Attention
- DeepSeek-V3 Technical Report
- Dissecting the NVIDIA Hopper Architecture through Microbenchmarking and Multiple Level Analysis
- Challenges and Research Directions for Large Language Model Inference Hardware
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- ThunderKittens: Simple, Fast, and Adorable AI Kernels
- Qwen3 Technical Report
Related papers
- Embedded Arena: Iterative Optimization via Hardware Feedback
- WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
- SparsePixels: Efficient Convolution for Sparse Data on FPGAs
- Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats
- AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
- RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design