Learn from your own latents and not from tokens: A sample-complexity theory
summary
The gist
Generative models achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require The authors investigate whether predicting one's own…
In short
The study investigates whether predicting one's own latent representations is more sample-efficient than predicting raw tokens for generative models. Using a hierarchical grammar model, the authors show that training a network to predict its own latents achieves necessary scaling with fewer samples, suggesting learning should occur in abstract latent spaces rather than just on tokens.
Key concepts
- Latent Representations
- These are the abstract, compressed internal states of a generative model. The paper suggests that these internal representations are more meaningful for learning than the raw sequences of input tokens.
- Sample Complexity
- This measures how many training examples (tokens or data points) a model needs to learn effectively. The authors demonstrate that predicting latents requires significantly fewer samples than traditional token-level objectives.
- Random Hierarchy Model (RHM)
- This is a probabilistic grammar used to generate text by recursively applying production rules across a tree of hidden symbols. It serves as the mathematical framework for modeling the structure of language generation.
Terminology used across episodes
This episode discusses
- Learn from your own latents and not from tokens: A sample-complexity theory · Paper Radio
- GPT-4 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- LLaMA: Open and Efficient Foundation Language Models
- LLM Pretraining with Continuous Concepts
- Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
- Predictive Coding: a Theoretical and Experimental Review
- Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
- HiT-JEPA: A Hierarchical Self-supervised Trajectory Embedding Framework for Similarity Computation
- Hierarchical JEPA Meets Predictive Remote Control in Beyond 5G Networks
- Deriving Neural Scaling Laws from the statistics of natural language
- Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models · Paper Radio
- U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models
- How transformers learn structured data: insights from hierarchical filtering
- Deep Learning and Hierarchal Generative Models
- A Provably Correct Algorithm for Deep Learning that Actually Works
- A Cookbook of Self-Supervised Learning
- Representation Learning with Contrastive Predictive Coding
- Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining
The paper
Learn from your own latents and not from tokens: A sample-complexity theory · Read on arXiv
Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart
Institute of Physics EPFL · Department of Applied Maths and Theoretical Physics University of Cambridge · Johns Hopkins University
Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their own latent representations of related views or masked regions, as in data2vec and JEPA -- an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L. For such data, supervised or token-level SSL require a number of samples exponential in L to recover the latent tree; we prove that latent prediction achieves this with a number of samples constant in L, up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Learn from your own latents and not from tokens".
Jane: Here's the gist of the paper. Generative models achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require The authors investigate whether predicting one's own latent representations can yield huge sample-complexity gains over token-level objectives The core problem addressed Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require Frontier LLMs are trained on 1013–1014 tokens
7, 8: , more than five orders of magnitude beyond what a child encounters before reaching adult-level competence
9–11: One hypothesis pursued is that learning may not be most efficient at the level of raw tokens, but instead in a more abstract latent space An alternative strategy to input reconstruction involves training a network to predict its own latent representations of related views or masked regions This idea relates to computational neuroscience, where predictive coding posits that the cortex seeks to predict its own future activity Theoretical framework and model setup The work addresses these questions using a tractable probabilistic context-free grammar (PCFG) called the Random Hierarchy Model (RHM) The RHM generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L Prior work shows that supervised end-to-end learning requires an order of mL samples for deep architectures, and token-level self-supervised objectives require of order mL+1 The authors establish three results against this baseline Key findings on sample complexity The study establishes that latent prediction achieves the required scaling with a number of samples constant in L, up to logarithmic factors Specifically, a hierarchical clustering algorithm recovers the full non-root latent tree from order m3 samples – independent of the depth L The iterative latent-clustering algorithm (ILC) recovers the non-root hierarchy with a sample complexity of order vm3 Furthermore, data2vec implicitly performs hierarchical latent prediction, leading to a sample complexity of order m3 This suggests that explicit stacking such as H-JEPA is largely redundant Validation and biological plausibility The results are confirmed by (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec The Stacked Latent-Clustering (SLC) network also saturates the vm3 statistical bound of Section 3 In a biologically plausible setting, the predictor-clusterer modules can reach the m3 scaling using only quasilocal learning rules Implications and limitations The main lesson is that latents at the same level of the hierarchy are far more correlated with each other than they are with raw tokens, so predicting from one’s own latents amplifies a signal that token-level prediction dilutes Practical implications suggest controlled comparisons between data2vec and next-token-prediction baselines to test the central prediction Limitations include extending the analysis to variable tree topologies and recursive production rules The authors also note that explicit hierarchical supervision, like H-JEPA, may be redundant Proof of recovery mechanism The proof of the iterative latent-clustering algorithm relies on two assumptions: a balanced and separated grammar and a stable clustering module Theorem 1 shows that if the sample size P is sufficiently large, the algorithm recovers h(1),...
Paper summary: Tom: So, to wrap up this discussion on "Learn from your own latents and not from tokens: A sample-complexity theory", the core message is that predicting a model's own latent representations offers a path to significantly better data efficiency <ref:2605.27734#pg1>. Jane It challenges the idea that raw token prediction is the only way to make these systems scale, suggesting instead we should look deeper into what the representation itself is trying to predict <ref:2605.27734#pg1>.
Lu: The authors establish that this latent prediction approach achieves a sample complexity of order m cubed, which is better than the order m plus one associated with token-level self-supervised objectives <ref:2605.27734#pg3>. Meng This means theoretically, we can build models that require far less training data to reach high performance levels compared to what we've been using for supervised learning or token-level SSL <ref:2605.27734#pg1>.
Tom: The implication is that the way we structure our self-supervision needs a serious rethink, moving toward these internal latent predictions instead of just matching raw text tokens <ref:2605.27734#pg1>. Jane It connects the abstract mathematical concepts of context-free grammars to real-world scaling observations in deep neural networks through this RHM framework <ref:2605.27734#pg3>.
Lu: The work confirms that latent prediction amplifies a signal that token-level prediction tends to dilute, and it shows how this works across different learning objectives like supervised classification <ref:2605.27734#pg1>. Meng From a deployment view, if we can achieve better sample complexity, it means smaller models might be sufficient for the same task performance <ref:2605.27734#pg1>.
Tom: It’s a solid theoretical argument showing that learning in an abstract latent space is where the real efficiency lies for these kinds of generative architectures <ref:2605.27734#pg2>. Jane It’s a very concrete way to think about how predictive coding principles might apply directly to training large language models <ref:2605.27734#pg1>.
Conclusion: Tom: So, this paper is about taking generative models and figuring out how to train them with way less data than we currently do <ref:2605.27734#pg1>. Jane It’s focused on this idea that instead of just looking at the raw words a model produces, it should be looking at its internal representations, or its latent space <ref:2605.27734#pg1>.
Lu: The core research here is testing whether predicting these own latent representations can give us huge gains in sample complexity over just trying to predict the next token <ref:2605.27734#pg1>. Meng So, if we can do this with fewer samples, it means models could actually be smaller and cheaper to run at scale <ref:2605.27734#pg1>.
Jane: The authors are using a model called the Random Hierarchy Model to test this idea <ref:2605.27734#pg3>. Tom And the main finding is that predicting these latent representations actually works really well for scaling <ref:2605.27734#pg1>. Lu Specifically, they show that the required number of samples stays constant as the model gets deeper, up to some logarithmic factors <ref:2605.27734#pg1>.
Jane: What this really means for us is that we might not need those massive datasets we’ve been feeding these models <ref:2605.27734#pg1>. Tom It shifts the focus from just matching tokens to understanding how the model organizes its own knowledge internally <ref:2605.27734#pg1>. Lu It suggests that latents at the same level are much more correlated with each other than the raw tokens are, so predicting one helps you predict all of them <ref:2605.27734#pg1>.
Meng: From a deployment view, if we can scale down the training data requirement this much, it makes sense for smaller models to be deployable in more constrained environments <ref:2605.27734#pg1>. Jane It’s a really interesting direction because it connects abstract theory about prediction coding to the actual data needs of large language models <ref:2605.27734#pg1>.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound