Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Tucker Bottleneck Attention (TuBA) addresses quadratic cost limitations of self-attention for long, multidimensional sequences by exploiting low-rank tensor structure. TuBA projects hidden tensors into compact Tucker cores, performs attention on these cores, and maps the result back. This enables subquadratic computation and achieves superior accuracy-efficiency trade-offs in tasks like video prediction and weather forecasting.