Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos

arXiv:2611.13685 · cs.RO, cs.AI, cs.CV · Submitted 2026-11-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos".

Title and authors: Rosa: To summarize what we just covered about the technical architecture, this paper is basically saying that we can train robot policies using human videos without needing those messy action labels during training at all.

Dev: Right, it’s about using future visual predictions from human egocentric videos to give the robot learning a signal that doesn't require you to restore the exact actions taken in the video. It uses this Shared Prediction on Action and Representation architecture, or SPAR.

Taro: The core idea is setting up two separate prediction experts, an action expert and a generation expert, both running off the same visual input from those human videos.

Rosa: Exactly. The action expert tries to predict the next chunk of movement using some flow matching loss based on the internal states of the vision model.

Dev: And then you have this generation expert which predicts what the future visual state looks like, and they both use those same internal states from the main model backbone to do their jobs.

Taro: The training is set up so that robot data supervises both of those experts, while the egocentric videos only supervise the generation expert. That lets you mix different types of data in a controlled way.

Rosa: That mixing is key because it means we can use the egocentric videos to shape the main model's representations in a way that directly helps robot tasks later on, even when things are new or out of distribution.

Dev: And they found that this works best when you use high quality data. They had to filter out clips where the camera was moving too much and also make sure there was at least one hand clearly visible in a specific area <ref:2611.13685#pg4>.

Taro: That filtering process, using things like corner and optical flow checks for stable motion, ended up being really important. It made the results way better on both their initial co-training and when they only used robot data to fine-tune.

Rosa: And when they look at scaling up the video hours, they see that success on in-domain tasks doesn't change much, but success on out of distribution tasks goes up steadily as you give the system more video time to learn from.

Dev: That suggests that just dumping more data isn't enough; what matters is how well those egocentric videos improve the underlying shared vision language model backbone itself so it can generalize better.

Taro: They also looked at different ways to predict that future visual state, and they found that using regression supervision, which is just predicting the visual features directly, gives useful feedback comparable to diffusion methods for several targets <ref:2611.13685#pg5>.

Rosa: So you don't need a fancy diffusion objective if you want good results from this visual prediction approach. Regression works just as well as flow matching under this specific SPAR design.

Dev: And on the hardware side, they used a Qwen three point five two billion parameter model, with twenty four layers and observations processed at two hundred fifty six by two hundred fifty six pixels per image during training <ref:2611.13685#pg12>. That gives us a concrete picture of the model size involved in this whole setup.

Taro: At inference time, they just turn off the generation expert and get the action chunks by integrating that action expert's predicted velocity field from some Gaussian noise with four Euler steps <ref:2611.13685#pg3>.

Rosa: The implication here is that learned visual representations are better targets for prediction than reconstruction latents when you have a distribution shift problem.

Dev: And they also showed that even on real physical platforms, the model gets around forty percent success on average compared to forty four percent for pi zero point five, which is comparable despite using a smaller robot pretraining set <ref:2611.13685#pg7>.

Taro: They also found that the model shows improved robustness under visual changes compared to their baseline, meaning those representations learned from diverse human videos are less sensitive to changes in appearance <ref:2611.13685#pg8>.

Rosa: So the overall message is that egocentric video has real potential as a useful source of transferable visual supervision for learning robot policies.

The paper's summary: Rosa: To wrap up how this work suggests we can improve things further beyond what they’ve already achieved with their core system, we can look at how different prediction targets change the results.

Dev: Basically, they found that using different prediction targets really changes how well the robot generalizes when things are a bit out of distribution, like seeing a different lighting condition.

Taro: Yeah, they tested different representation targets, and SigLIP two achieved the highest success rate reported across six different encoders <ref:2611.13685#pg5>.

Rosa: That means we have to be careful about what we're trying to predict; choosing the right target directly dictates how well the robot policy handles visual changes like background shifts or lighting variations.

Dev: They also found that using regression supervision, which is just predicting the visual features directly, gives results that are comparable to diffusion for several targets.

Taro: So it's not just about picking one method; it’s about having flexibility in choosing the prediction objective and seeing how different ones perform on specific tasks.

Rosa: That flexibility is important because it means we don't have to be locked into one way of setting up the loss function if that method isn't working perfectly for our robot task.

Dev: They also emphasized that regression supervision is a viable alternative to flow matching when you design the system this way, which is good news for real-time deployment because it might be easier to run.

Taro: It’s about finding a way to get good visual supervision without needing the heavy computational lifting of a diffusion model running alongside the policy learning loop.

Rosa: And they pointed out that using more egocentric video data consistently raises the success rate on those out-of-distribution tasks, which suggests that scaling up the video source is still a strong path forward.

Dev: So, to summarize these improvements, we can pick better prediction targets and use flexible loss functions like regression for better generalization.

Taro: But they also made a point about data quality again; even with this new system, you still need to curate those egocentric videos well to get the best performance.

Rosa: Exactly. So the implication is that we can now build policies that are more robust to visual shifts by smartly selecting our prediction targets and loss functions while still keeping a careful eye on getting high-quality input data.

The paper's improvements: Dev: To wrap up this paper on Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos, the main point is that we can train robot policies using human videos without needing those explicit action labels during training at all.

Taro: Right, it means we leverage future visual state prediction from those videos to give the robot learning a signal that bypasses the need for direct action supervision. It’s building on this Shared Prediction on Action and Representation architecture.

Rosa: And they found that this lets you use heterogeneous data—robot data for one expert, egocentric videos for the other—to optimize the main vision language model backbone in a way that helps downstream performance on out-of-distribution robot tasks.

Dev: The implication is that learned visual representations are better targets for prediction than reconstruction latents when there's a distribution shift problem, which is pretty significant for real-world deployment.

Taro: And they proved that using regression supervision gives results comparable to diffusion for several targets, meaning you don't need a diffusion objective if you want good visual prediction feedback.

Rosa: That flexibility is important because it means we don't have to be locked into one way of setting up the loss function if that method isn't working perfectly for our specific robot task.

Dev: They also pointed out that scaling up the video hours consistently raises the success rate on those out-of-distribution tasks, suggesting that using more egocentric data is still a strong path forward.

Taro: But they did stress again how much you need to curate those videos well, filtering them for stable motion and visible hands really improved performance on both raw-video co-training and robot-only baselines.

Rosa: So the overall message of this paper on Ego Guidance is that egocentric video has real potential as a useful source of transferable visual supervision for learning robot policies.

Dev: It gives us a way to get better generalization by shaping the model's understanding of the world with human activity, even when we're testing it outside the lab.

Taro: It’s about moving towards systems that can learn robust representations just from observing humans, which is a step toward more flexible autonomy.

Conclusion: Rosa: So we're wrapping up Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos by saying that we can train robot policies using human videos without needing those explicit action labels during training at all.

Dev: Right, it means leveraging future visual state prediction from those videos to give the robot learning a signal that doesn't require direct action supervision. It uses this Shared Prediction on Action and Representation architecture, or SPAR.

Taro: The authors showed that this works by setting up two experts, an action expert and a generation expert, both running off the same visual input from those human videos.

Rosa: And they found that this lets you use heterogeneous data—robot data for one expert, egocentric videos for the other—to optimize the main vision language model backbone in a way that helps downstream performance on out-of-distribution robot tasks.

Dev: The implication is that learned visual representations are better targets for prediction than reconstruction latents when there's a distribution shift problem, which is pretty significant for real-world deployment.

Taro: And they proved that using regression supervision gives results comparable to diffusion for several targets, meaning you don't need a diffusion objective if you want good visual prediction feedback.

Rosa: That flexibility is important because it means we don't have to be locked into one way of setting up the loss function if that method isn't working perfectly for our specific robot task.

Dev: They also pointed out that scaling up the video hours consistently raises the success rate on those out-of-distribution tasks, suggesting that using more egocentric data is still a strong path forward.

Taro: But they did stress again how much you need to curate those videos well, filtering them for stable motion and visible hands really improved performance on both raw-video co-training and robot-only baselines.

Rosa: So the overall message of this paper on Ego Guidance is that egocentric video has real potential as a useful source of transferable visual supervision for learning robot policies.

Dev: It gives us a way to get better generalization by shaping the model's understanding of the world with human activity, even when we're testing it outside the lab.

Taro: It’s about moving towards systems that can learn robust representations just from observing humans, which is a step toward more flexible autonomy.

Rosa: That’s what they did with Ego Guidance: Enhancing Robot Policy Learning with Action-Free Human Egocentric Videos.

Dev: I'm looking forward to seeing how this kind of visual supervision meshes with the optimization strategies we discussed for power distribution networks next week.

Lin Long, Tianren Ma, Zhong Chu, Di Wu, Maosong Cao, Qixiang Ye, Junbo Zhao, Shanghang Zhang

Zhejiang University · University of Chinese Academy of Sciences

cs.RO, cs.AI, cs.CV

Submitted: 2026-11-10

Updated: 2026-11-10

Comments: Project Page: https://llong-cs.github.io/EgoGuidance

Code: https://github.com/OpenDriveLab/AgiBot-World

Project page: https://llong-cs.github.io/EgoGuidance

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: In this paper, human egocentric videos can benefit robot policy learning without action supervision by leveraging future visual state prediction through a Shared Prediction on Action and…

Key concepts

Shared Prediction on Action and Representation (SPAR)
This is an architecture that uses one main model backbone shared by two specialized experts: one predicting actions and another predicting future visual states. This allows the system to learn from action-free human videos by using these predictions as learning signals, helping the robot understand how to act based on what humans do visually.
Action Expert
This part of the model is designed to predict what action a person will take next by looking at the video's hidden states. It uses a specific loss function called flow matching to predict velocity, which helps guide the robot's policy learning based on human movement patterns.
Generation Expert
This expert predicts a future visual representation of the scene from the model's hidden states. This prediction acts as a rich supervision signal for the robot. The paper shows that choosing a specific visual target, like SigLIP2, helps the robot generalize better to different environments.
Data Curation Gates
The researchers filtered egocentric videos using two quality checks: one to ensure limited camera motion and reliable flow estimates, and another to make sure there was at least one clearly visible hand in a specific area. This careful filtering of data is crucial for getting good results from the egocentric videos.

Terminology

Summary

In this paper, human egocentric videos can benefit robot policy learning without action supervision by leveraging future visual state prediction through a Shared Prediction on Action and Representation (SPAR) architecture<ref:2611. p. 0

Where the paper says it

io/EgoGuidance Abstract In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction.

13685#pg3>. This work investigates how action-free data from human egocentric videos can provide independent and complementary learning signals alongside action prediction<ref:2611. p. 0
Where the paper says it

io/EgoGuidance Abstract In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction.

13685#pg5>.

Model Architecture

The proposed architecture is an MoT that extends a pretrained VLM backbone with two independently parameterized transformer stacks, an action expert and a generation expert<ref:2611.13685#pg3>. The action expert predicts an action chunk by using the per-layer hidden states from the VLM to predict velocity via flow matching loss, defined as Laction=Es,ϵ 1/Hda fψ(As t, s; H) − (At − ϵ) 2/2<ref:2611. p. 1

Where the paper says it

fψ cross-attends to these states and predicts an action chunk At = at:t+H ∈ R H×da with flow matching [13]: For a clean sample x, Gaussian noise ϵ ∼ N (0, I), and flow time s ∈ [0, 1], we define x s = (1 − s)ϵ + sx and predict the velocity x − ϵ. Applied to action chunks, this gives a common vision-language-action (VLA) [8, 14] loss: Laction = Es,ϵ 1 Hda fψ(As t , s; H) − (At − ϵ) 2 2 .

13685#pg3>. The generation expert predicts a future visual representation Zt from the same hidden states to provide a high-capacity supervision signal, optimized via Lgen=Es,ϵ,t 1/Nd gω(Z s t, s; H) − (Z t − ϵ) 2/2 or Lreg gen=Et 1/Nd gω(H) − Z¯ t 2/2<ref:2611. p. 1
Where the paper says it

Action labels provide a relatively low-dimensional, embodiment-specific training signal. To provide high-capacity and generalizable supervision for the shared VLM, we add a generation expert gω that predicts a future visual representation from the same hidden states H through layer-wise cross-attention.

13685#pg4>.

Training Strategy

The training involves co-training where each data source supervises the losses it can provide, specifically Lrobot = Laction + λLgen and Lego = λLgen, with both losses backpropagating through the cross-attention into the LLM<ref:2611.13685#pg4>. The training strategy defaults to a two-stage approach where the model is first co-trained on a broader collection of robot demonstrations and egocentric videos, and then fine-tuned on downstream-aligned demonstrations alone<ref:2611. p. 2

Where the paper says it

In two-stage training, we first co-train on a broader collection of robot demonstrations and egocentric videos, then fine-tune on downstream-aligned demonstrations alone. The single-stage strategy consistently yields poorer generalization, with little variation across experimental settings.

13685#pg5>. The action expert is updated only by robot data, while the effect of egocentric videos is mediated by the backbone representation they help shape<ref:2611. p. 1
Where the paper says it

Both losses backpropagate through the cross-attention into the LLM, while the action expert is updated only by robot data. The effect of egocentric videos on the policy is therefore entirely mediated by the backbone representation they help shape.

13685#pg4>.

Data Curation and Scaling

The egocentric data, such as Ego4D, undergoes rigorous curation involving two quality gates based on EgoSmith [30], which retain clips with limited camera motion and at least one sufficiently visible hand. This curation is consequential for downstream learning because training with raw Ego4D clips underperforms the robot-only baseline, whereas the clips retained after both gates surpass it at 16k and 20k fine-tuning steps<ref:2611. p. 3

Where the paper says it

The curation is consequential for downstream learning (Fig. Training with raw Ego4D clips underperforms the robot-only baseline, whereas the clips retained after both gates surpass it at 16k and 20k fine-tuning steps.

13685#pg4>. Data scaling experiments show that in-domain LIBERO success changes little across video hours, but LIBERO-Plus success rises steadily as video hours increase, suggesting the OOD trend is consistent with their visual prediction objective improving the shared VLM backbone<ref:2611. p. 3
Where the paper says it

By contrast, LIBERO-Plus success rises steadily as video hours increase. Since these videos provide no action labels, the OOD trend is consistent with their visual prediction objective improving the shared VLM backbone and, in turn, the generalization of the downstream policy.

13685#pg5>.

Performance and Generalization

Representational targets significantly affect generalization under distribution shift, with SigLIP2 achieving the highest reported average success rate (76.2%) on Libero-Plus across states encoded by six different encoders<ref:2611. p. 2

Where the paper says it

3 Preliminary Findings Representational target helps generalization. Figure 2(a) compares the per-dimension success rates on Libero-Plus across states encoded by six different encoders [12, 25–29].

13685#pg5>. Furthermore, regression supervision provides useful feedback comparable to diffusion for several targets, indicating that the benefits of visual prediction do not depend on a diffusion objective<ref:2611. p. 2
Where the paper says it

Interestingly, figure 2(b) shows that direct regression of future visual features also provides useful supervision. Its per-dimension results are broadly comparable to diffusion for several targets, particularly SigLIP2 and DINOv2, suggesting that the benefits of visual prediction in our model do not depend on a diffusion objective.

13685#pg5>. In real-world evaluation on a physical bimanual platform, the model achieves an average success rate of 40% compared to 44% for π0. p. 4
Where the paper says it

Together with the continuing gains from adding more hours, this suggests that egocentric video has not saturated the model over the tested range: expanding the amount of data across scenes, objects, and interactions appears more promising than maximizing any one diversity measure within a fixed video budget. 5 Real-World Robot Evaluation We evaluate our model on a physical bimanual platform and verify: (1) Can egocentric pretraining enable our model to perform comparably to a strong VLA baseline on real-world manipulation tasks?

5, performing comparably despite using a smaller robot pretraining corpus<ref:2611. p. 5
Where the paper says it

Notably, our model performs comparably to π0. 5 despite using a smaller robot pretraining corpus and training its action expert from scratch.

13685#pg7>. The model also shows improved robustness under visual perturbations compared to the baseline, suggesting that visual-state prediction on diverse human videos yields representations that are less sensitive to appearance changes<ref:2611. p. 6
Where the paper says it

4), this suggests that visual-state prediction on diverse human videos yields representations that are less sensitive to appearance changes. On Flower Handover, our model matches π0.

13685#pg8>.

Conclusion

The investigation concludes that learned visual representations are more effective prediction targets than reconstruction latents under distribution shift, and regression is a viable alternative to flow matching under the SPAR design<ref:2611.13685#pg5>. Data quality is important, as filtering Ego4D for stable motion and visible hands improves performance on both raw-video co-training and the robot-only baseline<ref:2611. p. 6

Where the paper says it

During model design exploration, we find that learned visual representations are more effective prediction targets than reconstruction latents under distribution shift, and regression is a viable alternative to flow matching under the Shared Prediction on Action and Representation (SPAR) design. Data quality also matters: filtering Ego4D for stable motion and visible hands improves on both raw-video co-training and the robot-only baseline.

13685#pg4>. As the amount of action-free video increases, OOD success rises consistently<ref:2611. p. 6
Where the paper says it

Data quality also matters: filtering Ego4D for stable motion and visible hands improves on both raw-video co-training and the robot-only baseline. As the amount of action-free video increases, OOD success rises consistently.

13685#pg5>. The findings show that egocentric video has potential as a useful source of transferable visual supervision<ref:2611. p. 6
Where the paper says it

On real-world manipulation tasks, our policy performs comparably to the powerful VLA baseline, with task-dependent results under visual perturbations. Together, these findings provide fresh insight for the utilization of actionfree egocentric video, showing its potential as a useful source of transferable visual supervision.

13685#pg8>.

Implementation Details

The backbone is Qwen3.5-2B with L=24 layers of width D=2048, and observations are processed at 256×256 pixels per image<ref:2611. p. 3

Where the paper says it

We then investigate whether predicting a single frame works, as it significantly reduce the training cost. For the high-dimensional SigLIP2 target (256 tokens of width 1,152; App.

13685#pg12>. The action expert input includes 32 learnable tokens followed by H action tokens, and the regression generation expert uses a three-layer MLP to map outputs to Zˆ t<ref:2611. p. 1
Where the paper says it

The noised target Z s t is linearly projected to the expert’s width and given learned positional embeddings, the blocks are conditioned on s through AdaLN, and an output projection predicts the velocity, optimized by L flow gen . For the regression version, we set the timestep to constant s = 0, and use a three-layer MLP to map the expert’s output to the prediction.

13685#pg10>

Inference Procedures

At inference, the generation expert is removed following Fast-WAM [16], and actions are obtained by integrating the action expert’s velocity field from Gaussian noise with a few Euler steps<ref:2611.13685#pg3>. The deployed policy thus has the same architecture and cost as a standard VLA<ref:2611. p. 1

Where the paper says it

At inference, the generation expert is removed following Fast-WAM [16], and actions are obtained by integrating the action expert’s velocity field from Gaussian noise with a few Euler steps. The deployed policy thus has the same architecture and cost as a standard VLA.

13685#pg3>. Starting from Gaussian noise, the action chunk is obtained with M=4 Euler steps, A sk+1 t = A sk t + 1/M fψ(A sk t, sk; H) with sk = k/M<ref:2611. p. 1
Where the paper says it

The effect of egocentric videos on the policy is therefore entirely mediated by the backbone representation they help shape. At inference, the generation expert is removed following Fast-WAM [16], and actions are obtained by integrating the action expert’s velocity field from Gaussian noise with a few Euler steps.

13685#pg10>. The generation expert is not executed<ref:2611. p. 0
Where the paper says it

An action expert Preprint. arXiv:2611.

13685#pg10>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved Policy Learning via Action-Free Video Supervision

The system can learn robot policies by leveraging human egocentric videos without requiring expensive action restoration, as this approach improve[s] downstream performance on out-of-distribution robot tasks, with further gains as video data scales. This allows robots to generalize better to novel situations by learning representations shaped by human activity.

  1. Enhanced Representation Learning Through Target Selection

The system can select the most effective visual representation target for prediction, as SigLIP2 achieves the highest reported average success rate (76.2%), followed by Qwen-ViT (75.6%) and DINOv2 (74.9%). This selection directly dictates how well the policy generalizes to different visual perturbations like lighting or background changes.

  1. Robust Data Curation for Effective Pretraining

The system can improve data quality by applying specific filters to egocentric videos, as Gate 1 uses corner and optical-flow checks to retain clips with limited camera motion and reliable flow estimates; Gate 2 retains clips with at least one sufficiently visible hand in a prescribed image region. This curation leads to better results, as requiring visible hands in addition to stable, reliable optical flow gives the strongest result.

  1. Scalable Learning from Large-Scale Data

The system can effectively scale its learning capacity by using vast amounts of egocentric video data, as LIBERO-Plus success rises steadily as video hours increase when co-training with robot data. This demonstrates that the OOD trend is consistent with their visual prediction objective improving the shared VLM backbone.

  1. Flexible Loss Function for Visual Prediction

The system can utilize either flow matching or regression supervision for predicting future visual states, as both methods are viable: Regression supervision also works. Interestingly, figure 2(b) shows that direct regression of future visual features also provides useful supervision. This flexibility allows the model to benefit from different predictive objectives.

Abstract

In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction. We find that representations learned from egocentric videos improve downstream performance on out-of-distribution robot tasks, with further gains as video data scales. We also study the effects of objective choices and data curation, providing insights into how to leverage action-free videos for scalable robot learning.

Sources

Related papers