Factor Augmented High-Dimensional SGD

arXiv:2605.19291 · stat.ML, cs.LG, math.ST, stat.TH · Submitted 2026-05-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Factor Augmented High-Dimensional SGD".

Jane: Here's the gist of the paper. Factor Augmented SGD (FSGD) is a new optimization method that leverages latent factor representations in high-dimensional learning tasks.

Tom: This work establishes the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, providing a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.

Jane: Core Methodology The proposed Factor Augmented Stochastic Gradient Descent (FSGD) augments standard SGD with a factor model to leverage latent structure in high-dimensional predictors.

Tom: The key insight is that the effective information content of HD predictors often lies in a much lower-dimensional latent space, which can be captured by the “factors”.

Title and authors: Tom: So, we're looking at this paper today called Factor Augmented High-Dimensional SGD. It’s by Shubo Li and Yuefeng Han from The Pennsylvania State University and the University of Notre Dame.

Jane: It sounds pretty technical, Tom. What’s the main idea here in a nutshell? How does augmenting SGD with factors actually help when we're dealing with really high-dimensional learning tasks?

Tom: Well, basically, they propose this new optimization method called Factor Augmented SGD or FSGD. The core idea is that instead of trying to optimize directly in that huge high-dimensional feature space—which gets really messy—they leverage latent factor representations.

Lu: What I find interesting about the authors' approach is that it’s not a two-stage process where you learn the factors offline and then use them for training, which is what some other works do. This paper focuses on something different.

Meng: So, if I'm thinking about this from a deployment standpoint, what does "leveraging latent factor representations" actually mean in terms of cost or complexity when we're running this at scale?

Lalam: The paper suggests that the effective information content of these high-dimensional predictors often lives in a much lower-dimensional latent space that can be captured by these factors. This lets them operate purely on streaming data without needing to store massive amounts of data for offline representation learning, which is a big deal for scalability.

Jane: So, if I'm driving and I have all this information streaming in, how does this factor model fit into my car's system? Is it just more math?

Tom: It’s not just more math; they are doing something practical here. They establish the first theoretical framework that explicitly incorporates latent factor estimation error right into the analysis of standard SGD. That’s a new foundation for how we can use SGD reliably in these high-dimensional systems.

The paper's summary: Tom: So, to summarize what they actually did with Factor Augmented High-Dimensional SGD, they took standard Stochastic Gradient Descent and augmented it with a factor model to exploit the inherent structure in those high-dimensional predictors.

Jane: That means they are trying to find a simpler way to optimize the problem by looking at these hidden factors instead of just wrestling with all those huge parameters directly. What’s the big implication of this structural insight?

Lu: The main thing is that it changes the analytical route entirely. Existing studies on high-dimensional SGD have been limited, but this paper provides a rigorous characterization of convergence by explicitly accounting for data dimensionality through these factor models, which is a completely different way to look at how things converge > Factors are an explicit way to account for data dimensionality in the analysis of SGD >

Meng: That sounds good theoretically, but what about the practical implementation? If we have to estimate these factors online from streaming data, how does that affect our runtime or our ability to deploy this without massive pre-computation?

Lalam: They describe a two-stage process. Stage one is an Online PCA warm-up using Oja’s algorithm on the streaming data to get an initial subspace estimate. Then Stage two is joint online PCA and mini-batch SGD, where they update the model parameters in that induced factor space while simultaneously refining those factors from the stream through a dimension reduction procedure >

Tom: So it's not just one step; it's an online learning procedure designed for a fully streaming setting. It allows the factors to be continuously refined during model training instead of being fixed beforehand >

Jane: That sounds like they are trying to solve that problem of having a fixed, possibly bad, initial guess for the low-dimensional structure, which is something I’ve seen happen in other systems when you start with a poor setup.

The paper's improvements: Tom: They put forward several specific improvements this method offers over just using vanilla SGD or even some of the fixed factor-augmented methods out there. One improvement they highlight is adaptive representation learning for high-dimensional prediction >

Lu: What I mean by that is that the system can learn its low-dimensional latent factor representation online while simultaneously updating it with stochastic gradient descent on streaming data, which makes it more efficient than those older two-stage offline methods I mentioned earlier >

Meng: But if we're deploying this, are we talking about a significant reduction in computational overhead compared to just running vanilla SGD on the raw high-dimensional features? What’s the cost trade-off?

Tom: The paper suggests that for tasks like NCEP forecasting, real-world experiments show that FSGD achieves test R square scores comparable to or even better than more complex methods like PPCA, but it does it while requiring substantially less memory and keeping the runtime competitive with just Vanilla SGD >

Jane: That’s a strong claim if true. It means we can get high accuracy without needing the massive computational resources that come with those other advanced dimensionality reduction techniques.

Lalam: They also focus on theoretical guarantees under noise, which is crucial because real-world data is never perfect. Theorem three point six establishes the first q-th moment convergence analysis of SGD that accounts for both latent factor estimation error and idiosyncratic noise in these high-dimensional settings >

Tom: So, they aren't just proving it works on clean data; they are proving how much the noise and the error from estimating those factors actually hurt our convergence rate, which is a rigorous understanding beyond classical fixed-covariate theory >

Conclusion: Jane: So we’ve talked about FSGD, this Factor Augmented High-Dimensional SGD. To wrap up, what are the main things we should remember about this paper and what does it mean for us in the AI world?

Tom: The core message is that by augmenting SGD with a factor model online, we get a scalable way to handle high-dimensional learning tasks purely on streaming data. The key result they provide is that the moment convergence analysis explicitly accounts for both latent factor estimation error and idiosyncratic noise >

Lu: I think the most important implication is shifting the analytical perspective from optimization-side methods to a data-side approach that utilizes latent structure, which is a fresh theoretical route for understanding SGD with high-dimensional predictors >

Meng: From my side, what matters is that it’s robust enough. They noted that empirical performance stays stable even when you change the warm-up design choices or the Oja hyperparameters in Stage two, which suggests this online updating actually corrects poor initial subspace estimates >

Lalam: And for me, I think they confirm their direction with the real-world data. Synthetic experiments show scaling matches what they predicted in Theorem three point six, and on global atmospheric reanalysis datasets, FSGD outperformed Vanilla SGD and Random Projection in three out of four locations >

Jane: So it sounds like we have a method that’s both theoretically sound regarding noise and practically competitive with existing techniques for high-dimensional prediction, all while being built for streaming data. That gives us a lot to think about as we look at how we deploy complex models on the fly >

Shubo Li, Éuefeng Han, žžXiufan Yu

The Pennsylvania State University · University of Notre Dame

stat.ML, cs.LG, math.ST, stat.TH

Submitted: 2026-05-19

Updated: 2026-11-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 68/100

The gist: Factor Augmented SGD (FSGD) is a new optimization method that leverages latent factor representations in high-dimensional learning tasks.

Key concepts

Factor Augmented Stochastic Gradient Descent (FSGD)
A method that modifies standard SGD by incorporating a factor model to exploit latent structure in high-dimensional data. It learns a low-dimensional representation of the input data online while updating the prediction parameters, making it efficient for complex, high-dimensional problems.
Online PCA Warm-up
The initial stage of FSGD where Principal Component Analysis (PCA) is performed online using Oja's algorithm on streaming data. This step learns an initial low-dimensional subspace estimate (Q) from the incoming data stream before the main SGD optimization begins.
q-th Moment Convergence Analysis
A rigorous theoretical framework that analyzes how the iterates of FSGD converge to a target solution. Crucially, this analysis accounts for two sources of error: errors in estimating the latent factors and noise inherent in high-dimensional settings.

Terminology

Summary

Factor Augmented SGD (FSGD) is a new optimization method that leverages latent factor representations in high-dimensional learning tasks. p. 0

Where the paper says it

edu May 20, 2026 ABSTRACT Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks.

This work establishes the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, providing a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems. p. 0
Where the paper says it

Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in ℓ s norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.

Core Methodology

The proposed Factor Augmented Stochastic Gradient Descent (FSGD) augments standard SGD with a factor model to leverage latent structure in high-dimensional predictors. The key insight is that the effective information content of HD predictors often lies in a much lower-dimensional latent space, which can be captured by the “factors”. p. 1

Where the paper says it

To address the challenges induced by HD predictors, instead of directly optimizing HD-SGD in the high-dimensional feature space, we propose a new strategy, named Factor-Augmented Stochastic Gradient Descent (FSGD), that augments SGD with a factor model. The key insight is that the effective information content of HD predictors often lies in a much lower-dimensional latent space, which can be captured by the “factors”.

This approach differs from existing factor-augmented learning approaches by adopting an online factor-learning procedure designed for a fully streaming setting and seamlessly integrating it with SGD updates. p. 1
Where the paper says it

Such two-stage approaches rely on full access to the dataset and require storing or repeatedly scanning all samples, which is inefficient for iterative SGD updates and may even be infeasible in large-scale learning settings where data arrive sequentially and memory resources are limited. To tackle this challenge, we propose an online factor-learning procedure designed for a fully streaming setting and seamlessly integrate it with SGD updates, enabling memory-efficient and scalable learning.

Factor Estimation and Online Learning

FSGD involves two main stages: Stage I, which is an Online PCA Warm-up, and Stage II, which is Joint Online PCA and Mini-batch SGD. In Stage I, online PCA updates the subspace estimate Q using Oja’s algorithm on streaming data. p. 4

Where the paper says it

Since the true factors ft,i are not directly observed, they must be estimated online from the streaming data via a dimension reduction procedure. Algorithm 1 Factor Augmented SGD (FSGD) Require: Step sizes {ηt, ηoja,(0), η oja t }, factor dimension k, mini-batch size m Stage I: Online PCA Warm-up 1: Initialize Z0 ∈ R d×k of full column rank and set Q (0) 0 = QR[Z0].

During Stage II, the model parameters are updated using SGD on the induced factor space, and factors are estimated online from the streaming data via a dimension reduction procedure. p. 4
Where the paper says it

Since the true factors ft,i are not directly observed, they must be estimated online from the streaming data via a dimension reduction procedure. Algorithm 1 Factor Augmented SGD (FSGD) Require: Step sizes {ηt, ηoja,(0), η oja t }, factor dimension k, mini-batch size m Stage I: Online PCA Warm-up 1: Initialize Z0 ∈ R d×k of full column rank and set Q (0) 0 = QR[Z0].

This allows factor estimates to be continuously refined during model training rather than being fixed a priori. p. 1
Where the paper says it

Unlike classical two-stage representation learning approaches that require an offline PCA step with full data access, FSGD operates on streaming data and does not store the entire dataset, crucial for practical large-scale learning scenarios. As a result, factor estimates can be continuously refined during model training rather than being fixed a priori, causing the population minimizer to shift over time.

Theoretical Guarantees

The paper provides the first q-th moment convergence analysis of SGD that accounts for both latent factor estimation error and idiosyncratic noise in high-dimensional settings. Theorem 3.6 establishes the moment convergence of FSGD iterates θt to the rotated oracle θ∗t−1. p. 6

Where the paper says it

We now state the first main result of this paper. 6 establishes the moment convergence of FSGD iterates θt to the rotated oracle θ ∗ t−1 .

This analysis rigorously quantifies how representation estimation error and idiosyncratic noise propagate into the convergence rate of SGD. p. 1
Where the paper says it

6 provides the first q-th moment convergence analysis of SGD that accounts for both latent factor estimation error and idiosyncratic noise in high-dimensional settings. Unlike existing approaches that treat the learned representation as fixed or known and ignore the idiosyncratic component in the factor structure, we rigorously quantify how the representation estimation error and idiosyncratic noise propagate into the convergence rate of SGD.

Empirical Validation

Synthetic experiments on a linear model verify the scaling behavior stated by Theorem 3.6, showing that the rotated parameter error decreases as d grows, matching the d−1/2 scaling predicted by the first term in Theorem 3. p. 8

Where the paper says it

6 and vary d ∈ {10, 20, 40, 80, 160, 320}. Figure 1a shows that the rotated parameter error |θT − θ ∗ T −1 |s decreases as d grows, matching the d −1/2 scaling predicted by the first term in Theorem 3.

6. Real-data experiments on a global atmospheric reanalysis dataset confirm that FSGD achieves the best performance in three of four selected locations, outperforming both Vanilla SGD and Random Projection. p. 9
Where the paper says it

We compare FSGD with several baselines under different NN structures and projection updating strategies: • Factor-NN: the proposed FSGD method, which updates the projection matrix online via Oja’s algorithm throughout training; • Factor-NN-F0: the same architecture as Factor-NN, but the projection matrix is frozen immediately after warm-up; • Factor-NN-F300: the same architecture as Factor-NN, but the projection matrix is frozen after 300 epochs; • Oracle-NN: uses the true latent factors as inputs and serves as an ideal benchmark without factor estimation error; • Vanilla-NN: trains the neural network directly on the raw covariates x without representation learning; • PCA-A-NN: augments the raw input with the online factor projection, i. , z = d −1/2Q⊤ t−1x, x ; • NN-Joint: jointly learns the projection matrix and network weights from data, initialized from the warm-up phase; • Random Proj: replaces Oja’s online update with a random projection matrix sampled once at initialization and kept fixed thereafter; • PPCA: periodically recomputes the projection matrix via offline PCA using the most recent W mini-batches, and performs this refresh every M mini-batches.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive representation learning for high-dimensional prediction: The system can learn a low-dimensional latent factor representation online by updating it simultaneously with stochastic gradient descent on streaming data, which is more efficient than two-stage offline methods.

  2. Theoretical convergence guarantees under noise: The system can provide a first theoretical guarantee for SGD that explicitly incorporates both latent factor estimation error and idiosyncratic noise in the analysis of convergence rate, offering a rigorous understanding beyond classical fixed-covariate theory.

  3. Robustness to initialization and parameter drift: The system is robust to warm-up design choices and Stage II Oja hyperparameters, as empirical performance remains stable regardless of these settings, suggesting that online updates can correct poor initial subspace estimates.

  4. Improved predictive accuracy in real-world scenarios: For tasks like NCEP forecasting, the system achieves test R2 scores comparable to or better than complex methods like PPCA while requiring substantially less memory and maintaining competitive runtime compared to Vanilla SGD.

  5. Dynamic trade-off management: The system can balance factor estimation accuracy, optimization error, and minimizer shift by employing strategies like freezing the projection matrix after a certain number of iterations to mitigate the instability induced by time-varying target minimizers.

Abstract

Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks. Unlike standard two-stage dimension reduction approaches that rely on offline representation learning and full data storage, a key novelty of FSGD is that it operates purely on streaming data, making it scalable to large-scale and high-dimensional problems. Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in s norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems.

Sources

Related papers