HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model

The gist: HARP proposes a human-robot aligned representation learning framework to improve Vision-Language-Action (VLA) pretraining from human videos by jointly aligning visual representations and latent actions using paired demonstrations and scalable unpaired data.

How it works

HARP is a three-stage framework designed for transferring human demonstration knowledge to robot policy learning through aligned visual representations and latent actions The first stage jointly learns a robot-adapted visual encoder and a latent action model (LAM) from paired and unpaired videos Stage 1 training combines three objectives: the latent action prediction loss, the auxiliary loss, and the alignment loss The latent action prediction loss learns discrete latent actions through self- and cross-prediction, while the auxiliary loss injects manipulation-centric shared cues The alignment loss consists of a source-relative term and a pair-discriminative regularizer to adapt robot representations toward human semantics while maintaining pair-level discrimination.

Joint Visual and Latent-Action Alignment

The framework employs embodiment-aware visual encoding where frames are encoded into patch-level visual tokens using an embodiment-aware encoder composed of a frozen original visual encoder F and a robot-adapted encoder Tθ This design anchors human videos to the pretrained visual space while adapting robot representations toward the same semantic space. The latent action module is instantiated as a VQ-VAE-style inverse-and-forward dynamics model, where an encoder infers latent actions from visual transitions, and a decoder predicts future visual representations conditioned on the current observation and the quantized latent action. For paired videos (H, R), HARP performs cross-embodiment prediction where the latent action is inferred from the source transition, while the decoder is conditioned on the current representation of the target embodiment.

Auxiliary Cues and Alignment Losses

To encourage task-relevant dynamics, HARP regularizes latent-action learning with object-centric point tracks and wrist/end-effector trajectories The auxiliary loss is defined as Laux = λKLK + λELE, where LK and LE are Huber losses over visible object keypoints and wrist/end-effector positions, respectively. For paired demonstrations, the alignment loss consists of a source-relative term (LSR) and a pair-discriminative term (LPD) The source-relative term requires the adapted robot feature to improve over the frozen robot feature with respect to the paired human feature. To preserve pair-level discrimination, a pair-discriminative term is added, which uses non-matching pairs in the batch as negatives.

VLA Pretraining and Finetuning

Leveraging the learned latent action model and aligned vision encoder trained in stage 1, the framework labels video frame x t ∈ X with unified latent action q t X = Qθ(Eθ(z t X, zt+∆t X, lX)) This latent action is then used to pretrain a VLM into a VLA, where the model πθ is optimized to minimize the cross-entropy loss Lpretrain. Finally, HARP-VLA is finetuned with a small amount of real-action robot data using an action head that maps from the latent action embedding to normalized real action.

Experimental Validation

Experiments validate HARP across three perspectives: representation alignment, downstream policy learning, and generalist manipulation performance In terms of human-robot representation and latent-action alignment, HARP-SRPD improves bidirectional cross-embodiment retrieval Recall@1 from 43.55 to 78.50 over the unadapted encoder. In terms of robot policy learning, HARP-SRPD achieves the best policy performance, improving the average success rate from 37.56 to 46.59 over the unadapted encoder. For HARP-VLA, it improves the Calvin average length to 4.481 and realworld mean success rate to 76.3% compared to baselines. Ablation studies show that freezing the HARP-initialized vision encoder during Stage-2 pretraining further improves CALVIN average length from 4.250 to 4.481 and realworld success rate from 73.3% to 76.3%.

Limitations

HARP-VLA uses task-paired human-robot videos and auxiliary cues as cross-embodiment bridges, so its performance may depend on the diversity of the bridge data and the robustness of temporal alignment and shared cues Our current evaluation focuses on tabletop manipulation with a single robot platform and a limited set of human-video sources. Future work will scale to larger and more diverse human video collections, improve robustness to noisy auxiliary cues, and validate HARPVLA across more embodiments, longer-horizon tasks and bimanual manipulation.

References

[1] S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W Chao, B.-Y Lin, Liden et al., Latent action pretraining from videos

[2] Q Bu et al., Univla: Learning to act anywhere with task-centric latent actions

[3] H Kim et al., Uniskill: Imitating human videos via cross-embodiment skill representations

[4] X Chen et al., Igor: Image-goal representations are the atomic control units for foundation models in embodied ai

[5] S Kareer et al., Egomimic: Scaling imitation learning via egocentric video

[6] H Li et al., Mimicdreamer: Aligning human and robot demonstrations for scalable vla training

[7] M Xu et al., Dexumi: Using human hand as the universal manipulation interface for dexterous manipulation

[8] J Zhou et al., Mitigating the human-robot domain discrepancy in visual pre-training for robotic manipulation

[9] M J Kim et al., Openvla: An open-source vision-language-action model

[10] A Brohan et al., Rt-1: Robotics transformer for real-world control at scale

[11] A Brohan et al., Rt-2: Visionlanguage-action models transfer web knowledge to robotic control

[12] O M Team et al., Octo: An open-source generalist robot policy

[13] S Liu et al., Rdt-1b: a diffusion foundation model for bimanual manipulation

[14] K Black et al., π0: A vision-language-action flow model for general robot control

[15] P Intelligence et al., π0.5: a vision-language-action model with open-world generalization

[16] J Gu et al., Rt-trajectory: Robotic task generalization via hindsight trajectory sketches

[17] C Wang et al., Mimicplay: Long-horizon imitation learning by watching human play

[18] H Xiong et al., Learning by watching: Physical imitation of manipulation skills from human videos

[19] M Xu et al.

1. textbfString Alignment for Visual-Language-Action Models (HARP-VLA): The system can learn a unified vision and action representation by training a robot-adapted visual encoder and a latent action model jointly, which allows human videos to effectively boost VLA by mapping them into the same space as robot demonstrations.

2. textbfCross-Embodiment Dynamics Modeling: The system can generate aligned latent actions by using paired demonstrations for cross-prediction, such that the latent action is inferred from one embodiment to explain future dynamics of the other.

3. textbfEnhanced Latent Action Learning with Structural Cues: The system can learn more task-relevant dynamics by incorporating a manipulation-centric auxiliary cues loss, specifically using object-centric point tracks and wrist/end-effector trajectories to bias latent actions toward object motion and coarse agent motion.

4. textbfRobust Representation Alignment via Pair Discrimination: The system can ensure that the adapted robot representations are useful for policy learning by employing a source-relative pair-discriminative alignment loss which align[s] robot representations toward human semantics while maintaining pair-level discriminability.

5. textbfImproved Generalist Policy Performance: The resulting HARP-VLA policy can achieve state-of-the-art manipulation performance, demonstrated by achieving a 4.481 average length on CALVIN ABC→D and a 7.1% realworld success rate gain over the strongest baseline.