HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

arXiv:2608.12860 · cs.RO · Submitted 2026-08-13 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments".

Dev: Here's the gist of the paper. Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics.

Rosa: This paper introduces HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments.

Dev: Addressing Key Gaps The work targets three critical gaps in existing VLN research.

Rosa: These gaps include Humanoid-Aware Simulation, Navigability-Curated Scene Environments, and Realistic, Grounded Instruction Data.

Paper summary: Rosa: So, we’re talking about this paper called HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments. Basically, they're tackling the fact that current vision language navigation systems just don't work well when you put them on a real humanoid robot.

Dev: Right, so the core problem they identify is that bipedal locomotion has these physical constraints that wheeled robots or idealized agents just don't have, and humanoids are super varied in shape and how they move, which really messes with things.

Taro: I mean, it’s not just about moving a box; it’s about the physics of walking and keeping your balance when you're navigating a room using instructions.

Rosa: Exactly, and what they claim is that existing benchmarks fail to address these physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. They’re building this HumanoidVLN platform to fix that.

Dev: Their solution involves developing an Isaac Sim platform with a hierarchical control architecture that combines a reinforcement learning policy for low-level locomotion with interchangeable proportional derivative or model predictive control controllers for high-level path tracking.

Taro: That makes sense because it grounds the navigation in actual bipedal dynamics across four different humanoid morphologies, which is key because those differences in lower body degrees of freedom, from ten to twelve and heights between one point one seven meters and one point eight meters, directly impact gait stability and reachability.

Rosa: And they’re not just simulating it; they are curating the scene environments by manually verifying traversable areas that exceed one hundred square meters. They also use a real-to-sim pipeline based on three dimensional Gaussian Splatting to build simulation ready environments without needing fully artist authored assets.

Dev: The data generation part is also pretty interesting, because they propose this Multi Agent Annotation framework to collect nine hundred thirty three episodes each with one spatially grounded fine grained instruction and three coarse grained stylistic variants.

Taro: So they have a dual generator, a reviewer agent that checks against scene graph and trajectory priors, and a paraphraser that makes three styles formal, natural, or casual, which are then corrected by human annotators to produce the final benchmark dataset.

Rosa: The results show that JanusVLN achieves the highest mean success rate of forty-three point five five percent and a normalized dynamic time warping of forty-eight point thirty eight across all models and embodiments. DualVLN is also mentioned as being the most stable model across all embodiments, achieving the lowest fall rate.

Dev: And they confirmed this simulation to reality link with a pearson correlation of zero point nine three five for navigation error in their pilot scale sim to real study. That suggests the results aren't just confined to the simulator.

Taro: It’s important to note that they used a standardized json schema for every episode, which includes landmarks, a route summary, one spatially grounded instruction, and those three stylistic variants. The median path length across nine hundred thirty three episodes was nine point nine seven meters.

Rosa: So what does this mean for the world? It means we have a new way to test vision language navigation that actually respects the physical limitations of bipedal robots instead of just teleporting them around in a virtual space.

Dev: It provides a rigorous benchmark where you can plug and play new robots into this system using a single observation action interface, which is pretty useful for controlling diverse hardware.

Taro: It addresses the issue that existing simulators treat the humanoid as just one of several interchangeable robot types, but this work grounds navigation in actual bipedal dynamics across those different body types.

Rosa: What they did with that three dee Gaussian Splatting reconstruction pipeline is also pretty significant because it produces substantially smoother and more geometrically consistent surfaces compared to vanilla gsplat, which means better collision meshes for Isaac Sim without needing artists to do all the heavy lifting <ref:2608.12860#pg2>.

Dev: They also pointed out a limitation, though, and they mentioned that the modest path instruction correlation is zero point two nine, which indicates that instruction length isn't determined solely by trajectory length.

Taro: And they flagged scene diversity as a current limitation, along with human verification being a scaling bottleneck for annotation. Future work will look at integrating three dimensional foundation models for stronger instruction grounding and developing more efficient three dimensional gsplatting reconstruction methods.

Rosa: So, to sum up, HumanoidVLN gives us a physics grounded simulator and benchmark that handles diverse humanoid embodiments, generates realistic instruction data through their multi agent annotation framework, and gives us solid performance numbers like JanusVLN's forty-three point five five percent success rate.

Dev: The implication is that for anyone building autonomous navigation systems for humanoids, this platform offers a way to test those systems in a way that accounts for the real physics they will encounter when they step out of the lab.

Conclusion: Rosa: So we’re wrapping up HumanoidVLN, this paper by Rosa and Dev about building a simulator and benchmark for navigating humanoid robots with vision language instructions that actually respects physics across different robot shapes.

Dev: Yeah, it sets up this Isaac Sim platform that lets you test navigation models on four different humanoids with various lower body degrees of freedom and heights between one point one seven meters to one point eight meters.

Rosa: The main thing they did was create a way to handle all those physical differences by mixing a reinforcement learning policy for walking with interchangeable path trackers like PD or MPC controllers so every tested trajectory is actually executable by the specific robot.

Dev: And they didn't just do simulations; they built this data pipeline using a multi agent annotation framework to generate nine hundred thirty three episodes each with one instruction and three styles, formally, naturally, and casually.

Rosa: The results show that JanusVLN got a mean success rate of forty-three point five five percent across all models and robot shapes, which is pretty solid for this kind of setup.

Dev: DualVLN was the most stable model they tested because it had the lowest fall rate across every humanoid configuration, which tells us something about how robust the control loop needs to be when things go wrong.

Rosa: They also showed a simulation to reality study with a correlation of zero point nine three five for navigation error, meaning their simulated results match what you actually see in the real world quite well.

Dev: But they did flag that their instruction length and actual path length aren't perfectly correlated, with a correlation of zero point two nine, which means just having a longer route doesn't always mean you need a much longer instruction.

Rosa: So they’ve given us this way to test vision language navigation on diverse humanoids with realistic physics and data that they generated themselves.

Dev: It makes it easier for people building real-world humanoid systems to plug in new robot designs and see how their models perform under those specific physical constraints.

Quan-Dung Pham, Anh Dao, The-Anh Nguyen, Minh Nguyen-Dinh, Phuong Nam Dang, Tri Pham, Hung Tran, Bach Dao, Tuyen P. Le, Truong Nguyen

VinMotion, Inc. · University of Southern California

cs.RO

Submitted: 2026-08-13

Updated: 2026-08-13

Project page: https://humanoid-vln.github.io

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address…

Key concepts

HumanoidVLN
A physics-grounded simulator and benchmark designed for Vision-Language Navigation (VLN) across many different humanoid robot bodies. It tests how robots can follow spoken or written instructions in realistic physical environments, overcoming limitations of existing benchmarks.
Hierarchical Control Architecture
A system combining a Reinforcement Learning policy for low-level movement with interchangeable PD or MPC path trackers for high-level navigation. This setup ensures that the robot's actual physical capabilities and the specific humanoid configuration are respected during testing.
3D Gaussian Splatting (3DGS)
A technique used to reconstruct complex 3D environments from visual data without needing pre-made 3D models. This method produces smoother, more geometrically consistent surfaces than older methods, allowing for the creation of simulation environments directly from real-world scans.
Multi-Agent Annotation (MAA)
A framework used to generate high-quality instruction data. It involves multiple AI agents creating initial routes, a reviewer verifying them against scene data, and a paraphraser adding stylistic variants, ensuring the final instructions are accurate and varied.

Terminology

Summary

Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. p. 0

Where the paper says it

1: Overview of the HumanoidVLN physics-grounded benchmark pipeline, from embodiment-diverse humanoid platforms and hierarchical control to scene curation, episode construction, multimodal dataset generation, and plug-andplay VLN evaluation. Abstract— Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics.

This paper introduces HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. p. 0
Where the paper says it

Abstract— Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments.

Addressing Key Gaps

The work targets three critical gaps in existing VLN research. These gaps include Humanoid-Aware Simulation, Navigability-Curated Scene Environments, and Realistic, Grounded Instruction Data. p. 5

Where the paper says it

The 87 scenes span 17 indoor classes across six application domains, with a median navigable area of 266 m2 . CONCLUSION We presented HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments, addressing three critical gaps: embodimentdiverse simulation, navigability-curated scenes, and reliable grounded instruction generation.

The authors address these by developing an Isaac Sim platform with a hierarchical control architecture combining a Reinforcement Learning locomotion policy with interchangeable PD or MPC path trackers to handle embodiment diversity. p. 0
Where the paper says it

Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations — demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10–12 lower-body DoF and heights from 1. 80 m — via a hierarchical control stack combining a Reinforcement Learning locomotion policy with interchangeable PD or MPC path trackers.

Furthermore, they curate scenes with manually verified traversable areas exceeding 100 m2 and employ a Real2Sim pipeline based on 3D Gaussian Splatting (3DGS) to construct simulation-ready environments without requiring fully artist-authored assets. p. 1
Where the paper says it

Humanoid robots require large, obstacle-sparse traversable areas; scenes lacking this criterion introduce topological bottlenecks infeasible for bipedal navigation. We curate scenes with manually verified traversable areas of at least 100 m2 , a practical threshold used to exclude small or highly constrained environments, and employ a Real2Sim pipeline based on 3D Gaussian Splatting (3DGS) [8] to construct simulation-ready environments without requiring fully artist-authored assets [7].

Finally, they propose the Multi-Agent Annotation (MAA) framework to collect 933 episodes each with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants via a Dual Generator + Reviewer + Paraphraser MAA. p. 1
Where the paper says it

• Navigability-Curated Real2Sim Scene Suite: 87 highfidelity 3D environments (≥100 m2 navigable area) spanning 17 indoor classes across 6 domains, drawn from artist-designed and 3DGS reconstructed sources. • MAA-Augmented Instruction Dataset: 933 episodes each with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants, generated by a Dual Generator-Reviewer + Paraphraser MAA and verified by human annotators.

Platform and Control Architecture

The HumanoidVLN platform is built on NVIDIA Isaac Sim and supports four humanoid configurations spanning 10–12 lower-body DoF and heights from 1.17 m to 1.80 m. The control stack utilizes a hierarchical RL-plus-path-tracker architecture where an RL policy governs low-level locomotion while interchangeable PD or MPC controllers serve as high-level path trackers, ensuring every evaluated trajectory is physically executable by the specific robot under test. p. 1

Where the paper says it

8 m — and these differences directly impact gait stability, step reachability, and CoM dynamics. We develop an Isaac Sim platform with a hierarchical control architecture: an RL policy governs low-level locomotion while interchangeable Proportional-Derivative (PD) or Model Predictive Control (MPC) controllers serve as high-level path trackers, grounding navigation in actual bipedal dynamics across four humanoid morphologies.

This setup allows for plug-and-play integration of new robots and VLN models through a single observation-action interface. p. 1
Where the paper says it

61 m), and Internal-B (12 DoF, 1. 17 m) via a hierarchical RL-plus-path-tracker control stack, with a plug-and-play interface for additional robots and VLN models.

Data Generation Pipeline

The instruction dataset is generated using the Multi-Agent Annotation (MAA) framework, which involves two generator agents independently deriving route graphs from egocentric keyframe sequences, followed by a reviewer agent that verifies against scene-graph and trajectory priors. This verified instruction is then paraphrased into three coarse-grained stylistic variants—Formal, Natural, and Casual—and corrected by human annotators to produce the final benchmark dataset. p. 3

Where the paper says it

Two generator agents independently derive a route graph from the egocentric keyframe sequence alone; their graphs are reconciled, and residual contradictions are forwarded to a reviewer agent that verifies against scene-graph, trajectory, and occupancy priors. The verified instruction is paraphrased into three stylistic variants and corrected by human annotators to produce the benchmark dataset.

Evaluation and Results

The experiments evaluate four VLN models (NaVILA, StreamVLN, DualVLN, and JanusVLN) across four humanoid embodiments under physics-based execution. JanusVLN achieves the highest mean Success Rate (SR) of 43. p. 0

Where the paper says it

, Vietnam 2University of Southern California, USA ∗Equal contribution instruction and three coarse-grained stylistic variants (Formal, Natural, Casual). Across four models and four humanoid embodiments, JanusVLN achieves the highest mean SR of 43.

55% and Normalized Dynamic Time Warping (nDTW) of 48. p. 4
Where the paper says it

Following standard VLN protocol we report Success Rate (SR; stop within 3. 0 m of goal), Oracle Success Rate (OS), Navigation Error (NE, m), Success weighted by Path Length (SPL), and Normalized Dynamic Time Warping TABLE III: Zero-shot VLN model performance on HumanoidVLN evaluation set across four humanoid embodiments.

38 across all models and embodiments. p. 0
Where the paper says it

, Vietnam 2University of Southern California, USA ∗Equal contribution instruction and three coarse-grained stylistic variants (Formal, Natural, Casual). Across four models and four humanoid embodiments, JanusVLN achieves the highest mean SR of 43.

DualVLN is noted as the most stable model across all embodiments, achieving the lowest Fall Rate (FR). p. 5
Where the paper says it

3: Fall rate (% of n = 933 episodes) across four humanoid embodiments and four VLN models. Heatmap confirming DualVLN is the most stable model across all embodiments.

The pilot-scale sim-to-real study confirms strong correlation between simulation and reality for navigation error, with a Pearson r of 0. p. 5
Where the paper says it

Q4 — Real-World Correlation A benchmark built on reconstructed scenes is useful only if agent behavior remains consistent with the corresponding real environment. We therefore conduct a pilot-scale sim-toreal study using paired episode and trajectory analysis.

935.

Scene Quality and Realism

The 3DGS reconstruction pipeline is shown to produce substantially smoother and more geometrically consistent surfaces compared to vanilla gsplat, as evidenced by the qualitative ablation study. This improved geometry translates directly to higher-quality TSDF fusion results and more reliable collision meshes for Isaac Sim, enabling the use of reconstructed environments without requiring artist-authored assets. p. 5

Where the paper says it

Our full pipeline, combining unbiased depth rendering [30] with depth–normal consistency, produces substantially smoother and more geometrically consistent surfaces while preserving scene details. These improvements translate directly to higher-quality TSDF fusion results and more reliable collision meshes for Isaac Sim, enabling the use of reconstructed environments without requiring artistauthored assets.

The pilot study also indicated that reconstructed scenes largely preserve both navigation difficulty and executed trajectory structure when compared to real environments. p. 5
Where the paper says it

These results indicate that the reconstructed scenes largely preserve both navigation difficulty and executed trajectory structure. This pilot study covers two scenes and one VLN checkpoint, so it does not establish scene-level generalization.

Benchmark Composition

HumanoidVLN comprises 933 episodes across 87 scenes, each episode stored in a standardized JSON schema including landmarks, a route summary, one spatially grounded instruction, and three stylistic variants. The benchmark is used solely for zero-shot evaluation where all scenes and episodes form a single evaluation set without any model training or fine-tuning on HumanoidVLN. p. 4

Where the paper says it

Each episode is stored in a standardized JSON schema that includes landmarks, a route summary, one spatially grounded, fine-grained instruction, and three stylistic variants (Formal, Natural, and Casual). The benchmark is used solely for zero-shot evaluation: all scenes and episodes form a single evaluation set, and no evaluated model is trained or fine-tuned on HumanoidVLN.

The median path length across 933 episodes is 9. p. 5
Where the paper says it

7 summarize the benchmark. Across 933 episodes, paths have a median length of 9.

97 m, and the median instruction length is 50 words. p. 5
Where the paper says it

Across 933 episodes, paths have a median length of 9. 97 m and instructions contain a median of 50 words.

Limitations and Future Work

Current limitations include scene diversity, human verification as a scaling bottleneck, and the computational cost of full physics simulation. Future work will focus on integrating 3D foundation models for stronger instruction grounding, developing more efficient 3DGS reconstruction methods, and expanding sim-real validation on physical humanoid platforms. p. 6

Where the paper says it

limitations include scene diversity, human verification as a scaling bottleneck, and the computational cost of full physics simulation. Future work will integrate 3D foundation models for stronger instruction grounding, develop more efficient 3DGS reconstruction, and expand sim-real validation on physical humanoid platforms.

The authors also noted that the modest path–instruction correlation (r = 0. p. 5
Where the paper says it

97 m and instructions contain a median of 50 words. The modest path– instruction correlation (r = 0.

29) indicates that instruction length is not determined solely by trajectory length. p. 5
Where the paper says it

The modest path– instruction correlation (r = 0. 29) indicates that instruction length is not determined solely by trajectory length.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Embodiment-Diverse Humanoid Simulation Platform: The platform supports Unitree G1, Unitree H1, Internal-A, and Internal-B robots via a hierarchical RL-plus-path-tracker control stack with a plug-and-play interface for additional robots and VLN models. This allows for controlled cross-embodiment evaluation under identical environmental conditions.

  2. Navigability Curation: Environments are curated by filtering scenes to ensure navigability areas exceed 100 m2, which addresses the need for large, obstacle-sparse traversable areas required by bipedal navigation.

  3. MAA-Augmented Instruction Dataset: A Dual Generator-Reviewer + Paraphraser MultiAgent Annotation (MAA) framework generates 933 episodes each with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants, verified by human annotators to produce a collision-aware reference dataset.

  4. Hierarchical Control Architecture: The system employs a two-level control hierarchy where an RL policy governs low-level locomotion while interchangeable PD or MPC controllers serve as high-level path trackers, ensuring Grounding path following in this hierarchy rather than kinematic teleportation ensures that every evaluated trajectory is subject to the true physical constraints of the specific morphology under test.

  5. Physics-Aware Episode Sampling: Episodes are sampled by slicing 3D collision meshes at robot body height to form a 2D occupancy map and connecting start-goal pairs via A∗ paths on this map, ensuring every episode’s video contains genuine locomotion-induced camera dynamics.

  6. Instruction Grounding and Route Generation: The system uses heterogeneous generators (like Qwen3-VL-30BA3B and Gemma-4-31B-it) to independently derive route graphs from egocentric keyframes, which are then reconciled by a reviewer agent that verifies the reconciled route using trajectory metadata, the A∗ route over the occupancy map, and a spatial–semantic scene graph.

  7. Cross-Embodiment Performance Evaluation: The platform enables a matched zero-shot evaluation of four representative VLN models across four humanoid configurations under physics-based execution, measuring navigation accuracy, path fidelity (nDTW), and Fall Rate (FR).

  8. 3DGS Reconstruction Quality: A customized pipeline combining unbiased depth rendering with depth-normal consistency produces substantially smoother and more geometrically consistent surfaces while preserving scene details, leading to higher-quality TSDF fusion results for collision meshes.

  9. Real-World Correlation Pilot: A sim-to-real study shows that per-episode navigation error is strongly correlated between simulation and reality (Pearson r = 0.935), indicating little systematic endpoint bias in reconstructed scenes compared to real environments.

Abstract

Vision-Language Navigation (VLN) for humanoid robots poses challenges existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled agents, humanoid morphologies vary across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations, demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10-12 lower-body DoF and heights from 1.17m to 1.80m, via a hierarchical control stack combining a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models integrate with minimal effort; we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigable areas exceeding 100 square meters. Instructions are generated by a dual generator-reviewer plus paraphraser multi-agent pipeline with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one fine-grained instruction and three coarse-grained stylistic variants (formal, natural, casual). Across four models and four embodiments, JanusVLN achieves the highest mean success rate of 43.55% and nDTW of 48.38. In a 20-episode sim-to-real pilot with DualVLN and the Unitree G1, navigation errors correlate strongly (r=0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 (+/-0.188) nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance at https://humanoid-vln.github.io/.

Sources

Related papers