Isotropic Yet Undecodable: The Sequential Content-Sufficiency Gap in Latent-Predictive Text Representations
cs.AI, cs.CL, cs.LG
Submitted: 2026-10-06
Updated: 2026-10-06
License: http://creativecommons.org/licenses/by/4.0/
The gist: We study sequential content sufficiency by investigating whether a representation retains the ordered target information available in its input.
Abstract
We study sequential content sufficiency by investigating whether a representation retains the ordered target information available in its input. An information-theoretic decomposition separates input ambiguity, representation loss, and readout mismatch. We construct recoverable views where perfect agreement and joint isotropic Gaussianity coexist with zero target information, and establish limits imposed by deterministic canonical anchors. Token log-loss provides a one-sided information-loss bound; a fixed-penalty ridge analysis shows why rank alone cannot determine prediction risk. These results motivate CANOPE, a nonautoregressive framework with ordered latent canvases, canonical-token supervision, and geometric regularization. On 40,000 validation sequences, latent-agreement (PL0) and token-grounded (PL2) have nearly identical pooled ranks but reach 13.5% and 98.8% positional Recall@1, respectively, under strong natural corruption when the correct target length is provided. On 3,930 LJSpeech validation utterances, frozen PL2 with a trained MatchaTTS readout yields 21.54% word error rate (WER) on corrupted text, versus 99.22% for frozen PL0, while end-to-end MatchaTTS reaches 10.93%. These results show that geometric regularity alone does not guarantee recoverable sequential content or effective downstream access in the text settings studied here.
Sources
- Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction
- This is not correct! Negation-aware Evaluation of Language Generation Systems
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
- The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
- Learning Optimal Representations with the Decodable Information Bottleneck
- Non-Autoregressive Neural Machine Translation
- LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- Matcha-TTS: A fast TTS architecture with conditional flow matching
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- What Makes for Good Views for Contrastive Learning?
- Attention Is All You Need
- Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
- Rethinking Minimal Sufficient Representation in Contrastive Learning
- VISReg: Variance-Invariance-Sketching Regularization for JEPA training
- A Theory of Usable Information Under Computational Constraints
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems