Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction
cs.LG, cond-mat.mtrl-sci, cs.AI
Submitted: 2026-09-26
Updated: 2026-09-26
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by
Terminology
Abstract
We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a z-transform of joint order (0,0), whereas the induction discriminator requires key-side order at least 1. Applying the symplectic upper shear M γ:(q,p) (q+γp,p) to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit T 4-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at r=0.9876 with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the T 3 channel moves γ c at d k=64 from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is 6d FLOPs per token per layer, INT8 headroom grows by at most 2(1+2γ) bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at γ=0.80 in a mean of 717 steps, against 2/3 seeds and 2700 steps at γ=0. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.
Sources
- On the Power of Convolution Augmented Transformer
- Test-time regression: a unifying framework for designing sequence models with associative memory
- Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
- One-layer transformers fail to solve the induction heads task
- Differential Transformer
- KV Shifting Attention Enhances Language Modeling
- Scaling Laws for Neural Language Models
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
- Critical attention scaling in long-context transformers
- Invertible Query-Key Coupling Composes with Attention Mechanisms
- SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds
- DoPE: Denoising Rotary Position Embedding
- Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks