Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning

The gist Learning from demonstration has enabled impressive robot behaviors, but a gap exists between generative policies and direct action regression due to residual modeling mismatches, which this paper addresses by introducing heteroscedastic Student-t action regression (HT-Policies) to close this performance gap

How it works

The core problem stems from the fact that MSE regression corresponds to a fixed-scale Gaussian residual model, which mismatches the real data where residuals exhibit input-dependent scales and heavy tails This mismatch causes MSE to allocate more gradient magnitude to observations with large action residuals, hurting optimization Conversely, Flow matching exhibits a similar concentration at high noise levels but distributes gradient magnitude more evenly, especially in low-noise regimes due to its denoising training objective

The paper introduces HT-Policies, which explicitly model input-dependent scales and heavy tails by incorporating the observed residual structure directly into the regression objective The heteroscedastic Student-t loss is defined as maximizing the likelihood of a multivariate Student-t distribution centered at zero, where the scale matrix σθ(o) is predicted from the input The gradient of this loss with respect to the predicted action depends on both the predicted scale and the magnitude of the prediction error relative to that scale, allowing it to downweight gradients from samples with large residuals

Key Findings on Residuals and Gradients

The analysis revealed substantial state-dependent variation in residual scales and heavier-than-Gaussian tails in real-world robot demonstration data Both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, which are caused by variation in residual scale across observations Normalizing these residuals by their coordinate RMS does not fully explain the observed heavy tails, suggesting that differences in scale alone do not account for them

The gradient analysis showed that under MSE, high-residual samples account for a dominating share of gradient magnitude, which can leave other action samples underfit The HT loss produces a more balanced allocation of gradients across observations because large-residual observations no longer dominate the gradients as they do under MSE This reduced tail influence is visible early and persists across the evaluated training stages

Performance and Efficiency Gains

HT-Policies achieve success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models In simulation, HT-Policies are on par with generative policies and substantially outperform MSE-based policies On real robots, HT-Policies largely close the performance gap between MSE-based and generative policies on all three tasks

Furthermore, HT-Policies offer greater training and inference efficiency than generative policies, such as flow-matching–based policies, as shown in Section 5.4 They require only a single feed-forward pass at inference, leading to higher end8 throughput than flow on all three models

Ablation and Robustness

Ablation studies confirmed that learning an observation-dependent scale improves success for every tested objective, with heteroscedastic Student-t achieving the highest success The selection of degrees of freedom ν is parameterized as ν = cd, where c = ν/d, and performance peaks at an intermediate value of ν, favoring moderate downweighting of large residuals over either strong downweighting or the Gaussian limit Overall, HT-Policies achieve the best performance when compared against alternative training objectives

Conclusion

In conclusion, the empirical performance gap between direct action-regression policies and generative policies is better explained by differences in how these methods handle network prediction residuals rather than just multimodal action distributions The findings motivate HT-Policies, which are performant without iterative computation, can be trained from scratch or fine-tuned from pretrained models, and offer improved training and inference efficiency

The paper demonstrates that modeling input-dependent scales and heavy tails with a heteroscedastic Student-t objective yields superior policy performance across simulation and real-world tasks compared to standard MSE regression The introduction of HT-Policies provides an efficient direct-regression alternative for a range of architectures and tasks This approach is particularly valuable because it offers closed-form likelihood evaluation and efficient action sampling, making it well-suited for reinforcement learning–based fine-tuning In summary, the analysis shows that improved residual modeling and more balanced gradient allocation yield better policy performance across various settings The paper suggests that generative objectives are not necessarily superior due to better mode modeling but rather because they mitigate the gradient imbalance caused by fixed-scale residual models This work provides a strong case for using robust regression objectives in robot learning from demonstrations The HT-Policies are shown to be competitive with generative baselines while enabling more efficient training and inference The paper concludes that the performance gap is better explained by differences in how these methods handle network prediction residuals rather than just multimodal action distributions

--- Page 1 ---

RESIDUAL MODELING CLOSES THE REGRESSION AND GENERATIVE POLICY GAP IN ROBOT LEARNING Yuchen Zhou1,∗ Jiacheng You2,∗ Weikang Wan3 Weijun Dong3 Yang Gao2 Jiayuan Mao1 1 University of Pennsylvania 2 Tsinghua University 3 UC San Diego ABSTRACT Learning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and FlowPolicies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization Motivated by these findings, we introduce heteroscedastic Student-t action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails HTPolicies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks

--- Page 2 ---

!0 2 4 6 Heavy-tailed residuals MSE Flow / DP HT Heavy Tail Multi-Modal Probability density Residual magnitude GR1 SIMPLER RoboMimic What explains the gap? Residual-model mismatch HT regression closes the gap (a) What explains the gap? (b) Residual-model mismatch (c) HT regression closes the gap MSE-Policy Figure 1: (a) We attribute the performance gap between MSE- and Flow-Policies to different statistical modeling of prediction residuals. (b) We found observation-dependent and heavy-tailed residuals for MSE-Policies on multiple benchmarks. (c) Benchmark success rates (%) for different policy classes. We use flow-matching for RoboCasa-GR1 and SIMPLER, and diffusion for RoboMimic

--- Page 3 ---

Pick up a bottle GR1 (Humanoid) Bridge (Tabletop) Fractal (Tabletop) Tool-Hang Tabletop Grasp the bottle Close the cabinet Reach for the eggplant Grasp the eggplant Approach the apple Move apple to sponge Reach for the frame Insert the frame Figure 2: Example tasks. We show 4 tasks from RoboCasa-GR1, Bridge, Fractal, and Tool-Hang

--- Page 4 ---

MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals. One source of these tails is variation in residual scale across observations: for example, reaching toward a hole and making fine adjustments during insertion may involve different prediction-error scales Yet scale variation alone does not fully account for the observed tails, which remain heavier than Gaussian after perobservation scale normalization, as illustrated in Figure 1b This structure matters for optimization because the gradient of MSE with respect to the predicted action is proportional to the residual We show that high-residual samples account for a dominating share of gradient magnitude under MSE Flow matching exhibits a similar concentration at high noise levels, but distributes gradient magnitude more evenly, especially at low noise levels due to its denoising training objective Motivated by these findings, we introduce heteroscedastic Student-t action regression (HTPolicies) (Lange et al., 1989)

--- Page 5 ---

Figure 3 Heavy-tailed residuals. Both MSE-Policies (a) and Flow-Policies (b) exhibit heavy-tailed action-prediction residuals.

1. textbf Heteroscedastic Student-t Action Regression (HT-Policies) Implementation for Direct Regression Policies (DRP): The improved system can predict action chunks with a single feed-forward pass by jointly learning an input-dependent residual scale and predicts action chunks with a single feed-forward pass, which is faster than iterative flow matching.

2. textbf Gradient Allocation Balancing: The system will exhibit reduced concentration of gradient magnitude on high-residual observations by incorporating the heteroscedastic Student-t loss, thereby preventing MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization.

3. textbf Robustness to Multimodality: The system can be trained from scratch or fine-tuned from pretrained models while achieving success rates competitive with generative policy baselines, overcoming the limitations of MSE-Policies which tend to average across modes, potentially producing actions that do not correspond to any demonstrated behavior.

4. textbf Training Efficiency Improvement: The HT-Policy offers a significant advantage in efficiency because it is described as being faster in training and inference than generative baselines, requiring only a single feed-forward pass at inference.

5. textbf Exploration Capability Enhancement: Stochastic exploration can be supported by HT-Policies; specifically, the system shows that modest noise gives HT-Policies pass@k comparable to Flow-Policies, demonstrating that direct action prediction can support successful stochastic exploration in this setting.

Learning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization. Motivated by these findings, we introduce heteroscedastic Student-t action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails. HT-Policies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone. Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference. Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks. Project page: https://the-labone.github.io/regression-policy-project/