Geometry Matters in Packed 3D Attention
cs.CV, cs.AI, cs.LG
Submitted: 2026-10-28
Updated: 2026-11-12
Comments: Accepted to NeurIPS 2026 (Poster)
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Packing a 3D volume into a token sequence is not merely long-sequence modeling.
Abstract
Packing a 3D volume into a token sequence is not merely long-sequence modeling. Once the volume is packed, sequence order no longer encodes the geometry of the sample---so a positional mechanism indexed by sequence position can learn the memory layout rather than the object. This is a general failure mode for attention over packed geometric data. We introduce PACK3D, a local-global attention operator that separates memory layout from attention geometry. PACK3D keeps local neighborhood attention exact, compresses only distant context into contiguous 3D block summaries, and indexes every query-key interaction with axial RoPE on true token-grid coordinates rather than packed positions. Because the rotation is applied before a block is summarized, the query-to-summary logit remains a weighted sum of query-relative 3D displacements; compressed global context therefore stays spatially interpretable after packing, instead of collapsing to anonymous pooled content. We measure the failure mode directly. On a controlled synthetic task---train on one packing, change only the memory order of the tokens, hold the physical labeling rule fixed---learned positional embeddings and packed-sequence RoPE solve the training layout perfectly yet collapse under Morton, block-contiguous, and random repacking, while axial grid-coordinate RoPE stays exact throughout. The same coordinate system carries to real data: on Allen WTC-11 hiPSC microscopy, PACK3D improves over a parameter-matched 3D U-Net on the 15-structure joint task and over a matched learned-position local-global baseline, with controls isolating the coordinate system, not encoder capacity, as the source of the gain. A fused Triton implementation matches windowed-attention latency at 65k tokens while supplying the long-range context windowing cannot---so the operator is not only correct but trainable at scale.
Sources
- Longformer: The Long-Document Transformer
- Rethinking Attention with Performers
- Scaling Vision Transformers to 22 Billion Parameters
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- MethaneMapper: Spectral Absorption aware Hyperspectral Transformer for Methane Detection
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Linformer: Self-Attention with Linear Complexity
- Big Bird: Transformers for Longer Sequences
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs