Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions
David R. Wessels, Farhad Ramezanghorbani, Alireza Moradzadeh, David W. Romero, Olivia Viessmann, Maksim Zhdanov, John St. John, Ken Janik, David M Knigge, Yucheng Tang, Erik J Bekkers, Saee Gopal Paliwal
cs.LG, cs.CV, stat.ML
Submitted: 2026-07-01
Code: https://github.com/NVIDIA/cutlass
License: http://creativecommons.org/licenses/by/4.0/
The gist: Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models
Terminology
Abstract
Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc 1 D scan order that violates their spatial structure. We introduce HyenaND, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, nSubQ, fuses the FFT-convolution path to turn HyenaND's O(L L) scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.
Sources
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- BioNeMo Framework: a modular, high-performance library for AI model development in drug discovery
- Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
- VMamba: Visual State Space Model
- HyenaPixel: Global Image Context with Convolutions
- Retentive Network: A Successor to Transformer for Large Language Models
- Spike No More: Stabilizing the Pre-training of Large Language Models
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
- ViT-5: Vision Transformers for The Mid-2020s
- FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks