LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
cs.LG, cs.NA, math.NA
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/sbudzinskiy/lamp-attention
Sources
- Analysis of Floating-Point Matrix Multiplication Computed via Integer Arithmetic
- Longformer: The Long-Document Transformer
- Language Models are Few-Shot Learners
- Numerical stability analysis of large language models
- LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models
- PaLM: Scaling Language Modeling with Pathways
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Log-Linear Attention
- Deep Learning with Limited Numerical Precision
- A Study of BFLOAT16 for Deep Learning Training
- Gemma 3 Technical Report
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Reformer: The Efficient Transformer
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- DeepSeek-V3 Technical Report
- Mixed Precision Training
- FP8 Formats for Deep Learning
- Attention Is All You Need
- SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks