Fast and Exact Relative-Position Attention via Structure-Aware Kernels
cs.LG, cs.DC
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/gravitee-io/GLiNER4j
Terminology
Sources
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Seamless: Multilingual Expressive and Streaming Speech Translation
- AttentionEngine: A Versatile Framework for Efficient Attention Mechanisms on Diverse Hardware Platforms
- FLASHSWIN: Unlocking Large Windows and Dense Tokens in Swin Vision Transformers with Memory Efficient Attention
- NeMo: a toolkit for building AI applications using Neural Modules
- Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention
- Online normalizer calculation for softmax
- Self-attention Does Not Need $O(n^2)$ Memory
- Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
- Flashlight: PyTorch Compiler Extensions to Accelerate Attention Variants
- FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis