Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
summary
The gist
Simulations were distributed over a SLURM cluster across three main experimental setups: Optimal Representation Size, Autoencoder experiment, and LLM probe experiment.
In short
Simulations were conducted across three setups: Optimal Representation Size, Autoencoder experiment, and LLM probe experiment. The analysis explored how representation size impacts performance by testing various configurations and training regimes. Results suggest that representation size should be dynamically tuned based on data availability and task structure to maximize generalization error reduction.
Key concepts
- Optimal Representation Size
- This concept refers to finding the ideal dimensionality ($\alpha^*$) for a model's internal representations. The paper investigates how this optimal size changes depending on factors like the amount of unlabelled versus labelled data, the signal-to-noise ratio (SNR), and task alignment, aiming to minimize generalization error.
- Generalization Error
- This measures how well a model trained on one set of data performs on unseen data. The research seeks to find representation sizes that result in the lowest generalization error. This is crucial because it determines if the learned representations are truly useful for new, related tasks beyond the training set.
- Pretraining vs. Supervision Trade-off
- This examines how much knowledge a model gains from large amounts of unlabelled pretraining data versus specific labelled supervision. The study quantifies regimes where using extensive pretraining helps more than adding labelled samples, explaining why large models are effective for fine-tuning.
Terminology used across episodes
This episode discusses
- Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing · Paper Radio
- GPT-4 Technical Report
- The Llama 3 Herd of Models · Paper Radio
- Optimal Regularization for Performative Learning
- Generalization error of min-norm interpolators in transfer learning
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Understanding intermediate layers using linear classifier probes
- A mathematical theory for understanding when abstract representations emerge in neural networks
- A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
- Datasheet for the Pile
- From Principal Subspaces to Principal Components with Linear Autoencoders
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
The paper
Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing · Read on arXiv
Valentina Njaradi, Clémentine Dominé
University College London · Institute of Science and Technology Austria · Sainsbury Wellcome Centre
Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable structure from abundant unlabelled data, enabling efficient adaptation to new tasks from limited labelled data. This two-stage paradigm is now standard in modern training pipelines, where pretraining is followed by fine-tuning or linear probing. We provide an analytical model of this process: structure extraction is formalized as principal component analysis on unlabelled data, and downstream learning as linear regression on a separate labelled dataset. In the high-dimensional regime, we derive exact expressions for training and generalisation error showcasing their dependence on representation dimensionality, unlabelled and labelled sample sizes, and task alignment. Our results show that pretrained representations strongly influence downstream generalisation, and we characterize the optimal representation size as a function of task parameters: with abundant pretraining data but scarce downstream data, maximally compressed representations are optimal, whereas with limited pretraining data, higher-dimensional representations generalise better. Furthermore, we establish an exact trade-off between pretraining and supervision, quantifying how much unlabelled data is required to replace a single labelled sample. Beyond our idealised model, we observe similar phenomenology in autoencoders and pretrained LLMs. Altogether, we highlight that optimising representation size is critical, giving conditions for when compression during pretraining improves generalisation.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Optimal Representation Size".
Jane: Here's the gist of the paper. Simulations were distributed over a SLURM cluster across three main experimental setups: Optimal Representation Size, Autoencoder experiment, and LLM probe experiment.
Tom: For the Optimal Representation Size analysis, simulations were run on a SLURM cluster where each job was allocated 4 CPUs and 30 GB of RAM.
Jane: The full grid configuration involved 6 SNR times 4 lambda 1 times 4 lambda 2 times 30 n b and produced 2,880 SLURM jobs.
Tom: Each job processed 30 values of alpha with 100 independent replications at a matrix dimension of p=500.
Title and authors: Tom: Alright, so we're going through "Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing." The title itself tells you this isn't just about picking a number; it’s an analysis of the high-dimensional regime when you have pretraining followed by linear probing.
Jane: Right, so before we get into the details, what's the main takeaway for someone who doesn't read math? What is this paper actually saying about how to set up these two stages of learning?
Lu: It’s formalizing the idea that pretraining extracts structure from unlabelled data using something like principal component analysis, and then we learn the downstream task by doing linear regression on a separate labelled dataset. This whole pipeline is what they are analyzing.
Meng: So when you look at the practical side, is this paper suggesting we just use a fixed size, say two hundred dimensions, or does it give us a dynamic rule? Because for me, if I’m deploying something, I need to know what’s stable.
Lalam: It suggests that the optimal size depends on how the pretraining data and downstream labelled data are set up. They are deriving exact expressions for training in this high-dimensional regime >
Tom: Exactly. And they use a framework called spiked covariance models, which is a tool from statistics that helps them understand when those principal components actually align with the real underlying structure of the data, versus just being noise.
Jane: That’s where it gets interesting. The idea is that if the representation dimension is too high or too low relative to how much data you have, the quality of your downstream learning changes drastically.
Lu: They focus on a concept called the BBP phase transition, which tells you whether those principal components are useful or not based on the signal strength versus the noise level in your data >
The paper's summary: Tom: So summarizing what they did, they built an analytical model to characterize exactly how big that optimal retained dimension is going to be, depending on pretraining and downstream data. It’s a closed-form characterization >
Jane: When you unpack that for a listener, it means they are trying to solve the question: "Given I have lots of unlabelled text but very little labelled text for my specific task, how should I set the size of the AI's internal knowledge base to get the best results?"
Lu: They show that this optimal dimension isn't just a guess; it’s derived from analyzing how much signal is in your pretraining data compared to how much signal you have in your downstream labelled data >
Meng: Okay, so if I run my model and see that the required dimension for good performance is, say, five hundred instead of two hundred, what does that imply about the computational cost? Is it worth the extra memory allocation?
Lalam: It implies that using a smaller representation size would be detrimental because it wouldn't capture enough structure to transfer well to the labelled task >
Tom: It’s about capturing enough signal. And they use this analysis to show how representations support computation and transfer, or if they just cause catastrophic interference when things get complicated >
Jane: It connects the idea of representation quality directly to how well a model adapts, which is a big step because we often treat the two stages—pretraining and probing—as separate problems.
Lu: And by using this analytical theory, they can tell us if we are in the regime where high-dimensional structure is helpful or if we should stick to lower dimensional representations >
The paper's improvements: Tom: So the paper suggests a few ways to use this analysis. First, it gives a principled way to tune that optimal representation size based on the data availability—specifically when pretraining data is abundant but downstream data is scarce, in that case, maximally compressed representations are actually better for generalization >
Jane: That sounds counterintuitive at first. If we have lots of pretraining but little labelled data, why would we want to compress? What's the logic there?
Lu: Because if you have too much capacity from the pretraining but not enough specific guidance from the labelled data, a smaller representation prevents overfitting on that tiny labelled set >
Meng: So it’s about managing the trade-off between capturing general knowledge and focusing on task specifics. That sounds like something we deal with when we try to fine-tune models >
Lalam: They also show how you can quantify the trade-off between pretraining and supervision by identifying specific regimes where using unlabelled data actually reduces generalization error more than using labelled samples >
Tom: That’s a huge thing for understanding why large models trained on massive amounts of data are so effective for fine-tuning, because it gives us a mathematical reason for that effectiveness >
Jane: And then they get into adapting the representation size based on task structure itself. It suggests we can determine if low dimensional representations are actually useful or if specific tasks need those detailed high dimensional representations >
Conclusion: Tom: So to wrap up, the main implication of "Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing" is that we now have a formal, analytical way to determine the best representation dimension by looking at the data you have and the task structure.
Jane: It moves us past just guessing sizes; it gives us a principled way to tune those representations for better generalization when we move from massive pretraining to specific downstream tasks.
Lu: The key is recognizing that there's a phase transition where representation quality changes dramatically based on the signal strength in your data, and this paper explains how to find that point >
Meng: From an engineering standpoint, it helps us understand the cost of running different sizes. If we can predict the optimal size before we train everything, we save time and compute resources >
Lalam: I think it really solidifies the idea that pretraining on large datasets is powerful because it creates a structure that can be optimally refined for any subsequent learning step, as described in this paper >
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI