Learn from your own latents and not from tokens: A sample-complexity theory
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Learn from your own latents and not from tokens".
Jane: Here's the gist of the paper. Generative models achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require The authors investigate whether predicting one's own latent representations can yield huge sample-complexity gains over token-level objectives The core problem addressed Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require Frontier LLMs are trained on 1013–1014 tokens
7, 8: , more than five orders of magnitude beyond what a child encounters before reaching adult-level competence
9–11: One hypothesis pursued is that learning may not be most efficient at the level of raw tokens, but instead in a more abstract latent space An alternative strategy to input reconstruction involves training a network to predict its own latent representations of related views or masked regions This idea relates to computational neuroscience, where predictive coding posits that the cortex seeks to predict its own future activity Theoretical framework and model setup The work addresses these questions using a tractable probabilistic context-free grammar (PCFG) called the Random Hierarchy Model (RHM) The RHM generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L Prior work shows that supervised end-to-end learning requires an order of mL samples for deep architectures, and token-level self-supervised objectives require of order mL+1 The authors establish three results against this baseline Key findings on sample complexity The study establishes that latent prediction achieves the required scaling with a number of samples constant in L, up to logarithmic factors Specifically, a hierarchical clustering algorithm recovers the full non-root latent tree from order m3 samples – independent of the depth L The iterative latent-clustering algorithm (ILC) recovers the non-root hierarchy with a sample complexity of order vm3 Furthermore, data2vec implicitly performs hierarchical latent prediction, leading to a sample complexity of order m3 This suggests that explicit stacking such as H-JEPA is largely redundant Validation and biological plausibility The results are confirmed by (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec The Stacked Latent-Clustering (SLC) network also saturates the vm3 statistical bound of Section 3 In a biologically plausible setting, the predictor-clusterer modules can reach the m3 scaling using only quasilocal learning rules Implications and limitations The main lesson is that latents at the same level of the hierarchy are far more correlated with each other than they are with raw tokens, so predicting from one’s own latents amplifies a signal that token-level prediction dilutes Practical implications suggest controlled comparisons between data2vec and next-token-prediction baselines to test the central prediction Limitations include extending the analysis to variable tree topologies and recursive production rules The authors also note that explicit hierarchical supervision, like H-JEPA, may be redundant Proof of recovery mechanism The proof of the iterative latent-clustering algorithm relies on two assumptions: a balanced and separated grammar and a stable clustering module Theorem 1 shows that if the sample size P is sufficiently large, the algorithm recovers h(1),...
Paper summary: Tom: So, to wrap up this discussion on "Learn from your own latents and not from tokens: A sample-complexity theory", the core message is that predicting a model's own latent representations offers a path to significantly better data efficiency <ref:2605.27734#pg1>. Jane It challenges the idea that raw token prediction is the only way to make these systems scale, suggesting instead we should look deeper into what the representation itself is trying to predict <ref:2605.27734#pg1>.
Lu: The authors establish that this latent prediction approach achieves a sample complexity of order m cubed, which is better than the order m plus one associated with token-level self-supervised objectives <ref:2605.27734#pg3>. Meng This means theoretically, we can build models that require far less training data to reach high performance levels compared to what we've been using for supervised learning or token-level SSL <ref:2605.27734#pg1>.
Tom: The implication is that the way we structure our self-supervision needs a serious rethink, moving toward these internal latent predictions instead of just matching raw text tokens <ref:2605.27734#pg1>. Jane It connects the abstract mathematical concepts of context-free grammars to real-world scaling observations in deep neural networks through this RHM framework <ref:2605.27734#pg3>.
Lu: The work confirms that latent prediction amplifies a signal that token-level prediction tends to dilute, and it shows how this works across different learning objectives like supervised classification <ref:2605.27734#pg1>. Meng From a deployment view, if we can achieve better sample complexity, it means smaller models might be sufficient for the same task performance <ref:2605.27734#pg1>.
Tom: It’s a solid theoretical argument showing that learning in an abstract latent space is where the real efficiency lies for these kinds of generative architectures <ref:2605.27734#pg2>. Jane It’s a very concrete way to think about how predictive coding principles might apply directly to training large language models <ref:2605.27734#pg1>.
Conclusion: Tom: So, this paper is about taking generative models and figuring out how to train them with way less data than we currently do <ref:2605.27734#pg1>. Jane It’s focused on this idea that instead of just looking at the raw words a model produces, it should be looking at its internal representations, or its latent space <ref:2605.27734#pg1>.
Lu: The core research here is testing whether predicting these own latent representations can give us huge gains in sample complexity over just trying to predict the next token <ref:2605.27734#pg1>. Meng So, if we can do this with fewer samples, it means models could actually be smaller and cheaper to run at scale <ref:2605.27734#pg1>.
Jane: The authors are using a model called the Random Hierarchy Model to test this idea <ref:2605.27734#pg3>. Tom And the main finding is that predicting these latent representations actually works really well for scaling <ref:2605.27734#pg1>. Lu Specifically, they show that the required number of samples stays constant as the model gets deeper, up to some logarithmic factors <ref:2605.27734#pg1>.
Jane: What this really means for us is that we might not need those massive datasets we’ve been feeding these models <ref:2605.27734#pg1>. Tom It shifts the focus from just matching tokens to understanding how the model organizes its own knowledge internally <ref:2605.27734#pg1>. Lu It suggests that latents at the same level are much more correlated with each other than the raw tokens are, so predicting one helps you predict all of them <ref:2605.27734#pg1>.
Meng: From a deployment view, if we can scale down the training data requirement this much, it makes sense for smaller models to be deployable in more constrained environments <ref:2605.27734#pg1>. Jane It’s a really interesting direction because it connects abstract theory about prediction coding to the actual data needs of large language models <ref:2605.27734#pg1>.
Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart
Institute of Physics EPFL · Department of Applied Maths and Theoretical Physics University of Cambridge · Johns Hopkins University
cs.LG
Submitted: 2026-05-26
Updated: 2026-11-09
Comments: 10 pages, 5 figures in main. 31 pages, 16 figures, 1 table in all
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Generative models achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require The authors investigate whether predicting one's own…
Key concepts
- Latent Representations
- These are the abstract, compressed internal states of a generative model. The paper suggests that these internal representations are more meaningful for learning than the raw sequences of input tokens.
- Sample Complexity
- This measures how many training examples (tokens or data points) a model needs to learn effectively. The authors demonstrate that predicting latents requires significantly fewer samples than traditional token-level objectives.
- Random Hierarchy Model (RHM)
- This is a probabilistic grammar used to generate text by recursively applying production rules across a tree of hidden symbols. It serves as the mathematical framework for modeling the structure of language generation.
Terminology
Summary
Generative models achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require The authors investigate whether predicting one's own latent representations can yield huge sample-complexity gains over token-level objectives
The core problem addressed
Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require Frontier LLMs are trained on 1013–1014 tokens [7, 8], more than five orders of magnitude beyond what a child encounters before reaching adult-level competence [9–11] One hypothesis pursued is that learning may not be most efficient at the level of raw tokens, but instead in a more abstract latent space An alternative strategy to input reconstruction involves training a network to predict its own latent representations of related views or masked regions This idea relates to computational neuroscience, where predictive coding posits that the cortex seeks to predict its own future activity
Theoretical framework and model setup
The work addresses these questions using a tractable probabilistic context-free grammar (PCFG) called the Random Hierarchy Model (RHM) The RHM generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L Prior work shows that supervised end-to-end learning requires an order of mL samples for deep architectures, and token-level self-supervised objectives require of order mL+1 The authors establish three results against this baseline
Key findings on sample complexity
The study establishes that latent prediction achieves the required scaling with a number of samples constant in L, up to logarithmic factors Specifically, a hierarchical clustering algorithm recovers the full non-root latent tree from order m3 samples – independent of the depth L The iterative latent-clustering algorithm (ILC) recovers the non-root hierarchy with a sample complexity of order vm3 Furthermore, data2vec implicitly performs hierarchical latent prediction, leading to a sample complexity of order m3 This suggests that explicit stacking such as H-JEPA is largely redundant
Validation and biological plausibility
The results are confirmed by (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec The Stacked Latent-Clustering (SLC) network also saturates the vm3 statistical bound of Section 3 In a biologically plausible setting, the predictor-clusterer modules can reach the m3 scaling using only quasilocal learning rules
Implications and limitations
The main lesson is that latents at the same level of the hierarchy are far more correlated with each other than they are with raw tokens, so predicting from one’s own latents amplifies a signal that token-level prediction dilutes Practical implications suggest controlled comparisons between data2vec and next-token-prediction baselines to test the central prediction Limitations include extending the analysis to variable tree topologies and recursive production rules The authors also note that explicit hierarchical supervision, like H-JEPA, may be redundant
Proof of recovery mechanism
The proof of the iterative latent-clustering algorithm relies on two assumptions: a balanced and separated grammar and a stable clustering module Theorem 1 shows that if the sample size P is sufficiently large, the algorithm recovers h(1),...
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Latent prediction improves data efficiency by learning in a more abstract latent space rather than at raw token levels, as
predicting semantic concepts or continuous latents has been shown to improve data efficiency.
This allows models to achieve required performance with fewer training samples, scaling the sample complexity to order m3 instead of mL+1. -
Implement an iterative latent clustering algorithm (ILC) where each level of the hierarchy is recovered by clustering empirical context vectors, which
explicitly constructs the next-level latents.
This process allows for the recovery of the full non-root latent tree from a sample complexity scaling as vm3, which is exponentially fewer than that required by token-level self-supervised objectives. -
Develop a stacked architecture where each level has its own predictor and clusterer modules to achieve the same scaling, as
an explicit hierarchical architecture, with one predictor-clusterer module per latent level, can saturate the vm3 statistical bound of Section 3.
This system allows for astrictly local learning rule,
which is conceptually similar to contrastive predictive coding. -
Data2vec's student network can be adapted to implicitly perform hierarchical supervision by using its teacher representations as targets, where "Phase 0 reduces data2vec training to the token-level masked-prediction problem of Section 2 and recovers the level-1 latents at P >= vm3." This mechanism lifts the prediction problem onto higher latent levels through successive phases.
-
The SLC network architecture can be used to learn hierarchical structures by having
high-level representations are grounded in predicted lower-level representations,
allowing for atop→low prediction path
that is present in SLC but missing in H-JEPA.
Abstract
Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their own latent representations of related views or masked regions, as in data2vec and JEPA -- an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L. For such data, supervised or token-level SSL require a number of samples exponential in L to recover the latent tree; we prove that latent prediction achieves this with a number of samples constant in L, up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant.
Sources
- GPT-4 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- LLaMA: Open and Efficient Foundation Language Models
- LLM Pretraining with Continuous Concepts
- Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
- Predictive Coding: a Theoretical and Experimental Review
- Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
- HiT-JEPA: A Hierarchical Self-supervised Trajectory Embedding Framework for Similarity Computation
- Hierarchical JEPA Meets Predictive Remote Control in Beyond 5G Networks
- Deriving Neural Scaling Laws from the statistics of natural language
- Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
- U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models
- How transformers learn structured data: insights from hierarchical filtering
- Deep Learning and Hierarchal Generative Models
- A Provably Correct Algorithm for Deep Learning that Actually Works
- A Cookbook of Self-Supervised Learning
- Representation Learning with Contrastive Predictive Coding
- Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis