FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/yandex-research/flamia-blackwell-atten
Terminology
Sources
- FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
- Hardware-Aware FP4 FlashAttention-4
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Microscaling Data Formats for Deep Learning
- Pretraining Large Language Models with NVFP4
- Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
- Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
- SageAttention2++: A More Efficient Implementation of SageAttention2
- SageBwd: A Trainable Low-bit Attention
- A Fix for Softmax and Cross-Entropy
- EFQ-Softmax: Exp-Free Quantization for Softmax
- The Llama 3 Herd of Models
- TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning