HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments".
Dev: Here's the gist of the paper. Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics.
Rosa: This paper introduces HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments.
Dev: Addressing Key Gaps The work targets three critical gaps in existing VLN research.
Rosa: These gaps include Humanoid-Aware Simulation, Navigability-Curated Scene Environments, and Realistic, Grounded Instruction Data.
Paper summary: Rosa: So, we’re talking about this paper called HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments. Basically, they're tackling the fact that current vision language navigation systems just don't work well when you put them on a real humanoid robot.
Dev: Right, so the core problem they identify is that bipedal locomotion has these physical constraints that wheeled robots or idealized agents just don't have, and humanoids are super varied in shape and how they move, which really messes with things.
Taro: I mean, it’s not just about moving a box; it’s about the physics of walking and keeping your balance when you're navigating a room using instructions.
Rosa: Exactly, and what they claim is that existing benchmarks fail to address these physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. They’re building this HumanoidVLN platform to fix that.
Dev: Their solution involves developing an Isaac Sim platform with a hierarchical control architecture that combines a reinforcement learning policy for low-level locomotion with interchangeable proportional derivative or model predictive control controllers for high-level path tracking.
Taro: That makes sense because it grounds the navigation in actual bipedal dynamics across four different humanoid morphologies, which is key because those differences in lower body degrees of freedom, from ten to twelve and heights between one point one seven meters and one point eight meters, directly impact gait stability and reachability.
Rosa: And they’re not just simulating it; they are curating the scene environments by manually verifying traversable areas that exceed one hundred square meters. They also use a real-to-sim pipeline based on three dimensional Gaussian Splatting to build simulation ready environments without needing fully artist authored assets.
Dev: The data generation part is also pretty interesting, because they propose this Multi Agent Annotation framework to collect nine hundred thirty three episodes each with one spatially grounded fine grained instruction and three coarse grained stylistic variants.
Taro: So they have a dual generator, a reviewer agent that checks against scene graph and trajectory priors, and a paraphraser that makes three styles formal, natural, or casual, which are then corrected by human annotators to produce the final benchmark dataset.
Rosa: The results show that JanusVLN achieves the highest mean success rate of forty-three point five five percent and a normalized dynamic time warping of forty-eight point thirty eight across all models and embodiments. DualVLN is also mentioned as being the most stable model across all embodiments, achieving the lowest fall rate.
Dev: And they confirmed this simulation to reality link with a pearson correlation of zero point nine three five for navigation error in their pilot scale sim to real study. That suggests the results aren't just confined to the simulator.
Taro: It’s important to note that they used a standardized json schema for every episode, which includes landmarks, a route summary, one spatially grounded instruction, and those three stylistic variants. The median path length across nine hundred thirty three episodes was nine point nine seven meters.
Rosa: So what does this mean for the world? It means we have a new way to test vision language navigation that actually respects the physical limitations of bipedal robots instead of just teleporting them around in a virtual space.
Dev: It provides a rigorous benchmark where you can plug and play new robots into this system using a single observation action interface, which is pretty useful for controlling diverse hardware.
Taro: It addresses the issue that existing simulators treat the humanoid as just one of several interchangeable robot types, but this work grounds navigation in actual bipedal dynamics across those different body types.
Rosa: What they did with that three dee Gaussian Splatting reconstruction pipeline is also pretty significant because it produces substantially smoother and more geometrically consistent surfaces compared to vanilla gsplat, which means better collision meshes for Isaac Sim without needing artists to do all the heavy lifting <ref:2608.12860#pg2>.
Dev: They also pointed out a limitation, though, and they mentioned that the modest path instruction correlation is zero point two nine, which indicates that instruction length isn't determined solely by trajectory length.
Taro: And they flagged scene diversity as a current limitation, along with human verification being a scaling bottleneck for annotation. Future work will look at integrating three dimensional foundation models for stronger instruction grounding and developing more efficient three dimensional gsplatting reconstruction methods.
Rosa: So, to sum up, HumanoidVLN gives us a physics grounded simulator and benchmark that handles diverse humanoid embodiments, generates realistic instruction data through their multi agent annotation framework, and gives us solid performance numbers like JanusVLN's forty-three point five five percent success rate.
Dev: The implication is that for anyone building autonomous navigation systems for humanoids, this platform offers a way to test those systems in a way that accounts for the real physics they will encounter when they step out of the lab.
Conclusion: Rosa: So we’re wrapping up HumanoidVLN, this paper by Rosa and Dev about building a simulator and benchmark for navigating humanoid robots with vision language instructions that actually respects physics across different robot shapes.
Dev: Yeah, it sets up this Isaac Sim platform that lets you test navigation models on four different humanoids with various lower body degrees of freedom and heights between one point one seven meters to one point eight meters.
Rosa: The main thing they did was create a way to handle all those physical differences by mixing a reinforcement learning policy for walking with interchangeable path trackers like PD or MPC controllers so every tested trajectory is actually executable by the specific robot.
Dev: And they didn't just do simulations; they built this data pipeline using a multi agent annotation framework to generate nine hundred thirty three episodes each with one instruction and three styles, formally, naturally, and casually.
Rosa: The results show that JanusVLN got a mean success rate of forty-three point five five percent across all models and robot shapes, which is pretty solid for this kind of setup.
Dev: DualVLN was the most stable model they tested because it had the lowest fall rate across every humanoid configuration, which tells us something about how robust the control loop needs to be when things go wrong.
Rosa: They also showed a simulation to reality study with a correlation of zero point nine three five for navigation error, meaning their simulated results match what you actually see in the real world quite well.
Dev: But they did flag that their instruction length and actual path length aren't perfectly correlated, with a correlation of zero point two nine, which means just having a longer route doesn't always mean you need a much longer instruction.
Rosa: So they’ve given us this way to test vision language navigation on diverse humanoids with realistic physics and data that they generated themselves.
Dev: It makes it easier for people building real-world humanoid systems to plug in new robot designs and see how their models perform under those specific physical constraints.
Quan-Dung Pham, Anh Dao, The-Anh Nguyen, Minh Nguyen-Dinh, Phuong Nam Dang, Tri Pham, Hung Tran, Bach Dao, Tuyen P. Le, Truong Nguyen
VinMotion, Inc. · University of Southern California
cs.RO
Submitted: 2026-08-13
Updated: 2026-08-13
Project page: https://humanoid-vln.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 88/100
The gist: Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address…
Key concepts
- HumanoidVLN
- A physics-grounded simulator and benchmark designed for Vision-Language Navigation (VLN) across many different humanoid robot bodies. It tests how robots can follow spoken or written instructions in realistic physical environments, overcoming limitations of existing benchmarks.
- Hierarchical Control Architecture
- A system combining a Reinforcement Learning policy for low-level movement with interchangeable PD or MPC path trackers for high-level navigation. This setup ensures that the robot's actual physical capabilities and the specific humanoid configuration are respected during testing.
- 3D Gaussian Splatting (3DGS)
- A technique used to reconstruct complex 3D environments from visual data without needing pre-made 3D models. This method produces smoother, more geometrically consistent surfaces than older methods, allowing for the creation of simulation environments directly from real-world scans.
- Multi-Agent Annotation (MAA)
- A framework used to generate high-quality instruction data. It involves multiple AI agents creating initial routes, a reviewer verifying them against scene data, and a paraphraser adding stylistic variants, ensuring the final instructions are accurate and varied.
Terminology
Summary
Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. p. 0 1: Overview of the HumanoidVLN physics-grounded benchmark pipeline, from embodiment-diverse humanoid
platforms and hierarchical control to scene curation, episode construction, multimodal dataset generation, and plug-andplay VLN evaluation. Abstract— Vision-Language Navigation (VLN) for humanoid
robots presents unique challenges that existing benchmarks fail
to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies
vary significantly across platforms, and egocentric observations
are distorted by locomotion-induced camera dynamics. Abstract— Vision-Language Navigation (VLN) for humanoid
robots presents unique challenges that existing benchmarks fail
to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies
vary significantly across platforms, and egocentric observations
are distorted by locomotion-induced camera dynamics. We
present HumanoidVLN, a physics-grounded simulator and
benchmark for VLN across diverse humanoid embodiments.Where the paper says it
Where the paper says it
Addressing Key Gaps
The work targets three critical gaps in existing VLN research. These gaps include Humanoid-Aware Simulation, Navigability-Curated Scene Environments, and Realistic, Grounded Instruction Data. p. 5 The 87
scenes span 17 indoor classes across six application domains,
with a median navigable area of 266 m2
. CONCLUSION
We presented HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid
embodiments, addressing three critical gaps: embodimentdiverse simulation, navigability-curated scenes, and reliable
grounded instruction generation. Built on NVIDIA Isaac Sim, our platform supports an extensible
set of humanoid configurations — demonstrated on four robots
(Unitree G1, Unitree H1, Internal-A, Internal-B) spanning
10–12 lower-body DoF and heights from 1. 80 m — via
a hierarchical control stack combining a Reinforcement Learning locomotion policy with interchangeable PD or MPC path
trackers. Humanoid robots require large, obstacle-sparse traversable areas; scenes lacking
this criterion introduce topological bottlenecks infeasible for
bipedal navigation. We curate scenes with manually verified
traversable areas of at least 100 m2
, a practical threshold used
to exclude small or highly constrained environments, and
employ a Real2Sim pipeline based on 3D Gaussian Splatting (3DGS) [8] to construct simulation-ready environments
without requiring fully artist-authored assets [7]. • Navigability-Curated Real2Sim Scene Suite: 87 highfidelity 3D environments (≥100 m2 navigable area)
spanning 17 indoor classes across 6 domains, drawn
from artist-designed and 3DGS reconstructed sources. • MAA-Augmented Instruction Dataset: 933 episodes
each with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants, generated by a Dual Generator-Reviewer + Paraphraser
MAA and verified by human annotators.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Platform and Control Architecture
The HumanoidVLN platform is built on NVIDIA Isaac Sim and supports four humanoid configurations spanning 10–12 lower-body DoF and heights from 1.17 m to 1.80 m. The control stack utilizes a hierarchical RL-plus-path-tracker architecture where an RL policy governs low-level locomotion while interchangeable PD or MPC controllers serve as high-level path trackers, ensuring every evaluated trajectory is physically executable by the specific robot under test. p. 1 8 m —
and these differences directly impact gait stability, step
reachability, and CoM dynamics. We develop an Isaac Sim
platform with a hierarchical control architecture: an RL
policy governs low-level locomotion while interchangeable
Proportional-Derivative (PD) or Model Predictive Control
(MPC) controllers serve as high-level path trackers, grounding navigation in actual bipedal dynamics across four humanoid morphologies. 61 m), and Internal-B (12 DoF,
1. 17 m) via a hierarchical RL-plus-path-tracker control
stack, with a plug-and-play interface for additional
robots and VLN models.Where the paper says it
Where the paper says it
Data Generation Pipeline
The instruction dataset is generated using the Multi-Agent Annotation (MAA) framework, which involves two generator agents independently deriving route graphs from egocentric keyframe sequences, followed by a reviewer agent that verifies against scene-graph and trajectory priors. This verified instruction is then paraphrased into three coarse-grained stylistic variants—Formal, Natural, and Casual—and corrected by human annotators to produce the final benchmark dataset. p. 3 Two generator agents independently derive a route graph
from the egocentric keyframe sequence alone; their graphs are reconciled, and residual contradictions are forwarded to a
reviewer agent that verifies against scene-graph, trajectory, and occupancy priors. The verified instruction is paraphrased into
three stylistic variants and corrected by human annotators to produce the benchmark dataset.Where the paper says it
Evaluation and Results
The experiments evaluate four VLN models (NaVILA, StreamVLN, DualVLN, and JanusVLN) across four humanoid embodiments under physics-based execution. JanusVLN achieves the highest mean Success Rate (SR) of 43. p. 0 , Vietnam
2University of Southern California, USA
∗Equal contribution
instruction and three coarse-grained stylistic variants (Formal,
Natural, Casual). Across four models and four humanoid
embodiments, JanusVLN achieves the highest mean SR of
43. Following standard VLN protocol we report Success Rate (SR; stop within 3. 0 m of goal), Oracle Success
Rate (OS), Navigation Error (NE, m), Success weighted by
Path Length (SPL), and Normalized Dynamic Time Warping
TABLE III: Zero-shot VLN model performance on HumanoidVLN evaluation set across four humanoid embodiments. , Vietnam
2University of Southern California, USA
∗Equal contribution
instruction and three coarse-grained stylistic variants (Formal,
Natural, Casual). Across four models and four humanoid
embodiments, JanusVLN achieves the highest mean SR of
43. 3: Fall rate (% of n = 933 episodes) across four
humanoid embodiments and four VLN models. Heatmap
confirming DualVLN is the most stable model across all
embodiments. Q4 — Real-World Correlation
A benchmark built on reconstructed scenes is useful only
if agent behavior remains consistent with the corresponding
real environment. We therefore conduct a pilot-scale sim-toreal study using paired episode and trajectory analysis.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Scene Quality and Realism
The 3DGS reconstruction pipeline is shown to produce substantially smoother and more geometrically consistent surfaces compared to vanilla gsplat, as evidenced by the qualitative ablation study. This improved geometry translates directly to higher-quality TSDF fusion results and more reliable collision meshes for Isaac Sim, enabling the use of reconstructed environments without requiring artist-authored assets. p. 5 Our full pipeline, combining unbiased depth
rendering [30] with depth–normal consistency, produces
substantially smoother and more geometrically consistent
surfaces while preserving scene details. These improvements
translate directly to higher-quality TSDF fusion results and
more reliable collision meshes for Isaac Sim, enabling the
use of reconstructed environments without requiring artistauthored assets. These results indicate that the reconstructed scenes
largely preserve both navigation difficulty and executed trajectory structure. This pilot study covers two scenes and one VLN
checkpoint, so it does not establish scene-level generalization.Where the paper says it
Where the paper says it
Benchmark Composition
HumanoidVLN comprises 933 episodes across 87 scenes, each episode stored in a standardized JSON schema including landmarks, a route summary, one spatially grounded instruction, and three stylistic variants. The benchmark is used solely for zero-shot evaluation where all scenes and episodes form a single evaluation set without any model training or fine-tuning on HumanoidVLN. p. 4 Each episode is stored in a standardized JSON schema
that includes landmarks, a route summary, one spatially
grounded, fine-grained instruction, and three stylistic variants
(Formal, Natural, and Casual). The benchmark is used solely
for zero-shot evaluation: all scenes and episodes form a
single evaluation set, and no evaluated model is trained or
fine-tuned on HumanoidVLN. 7 summarize the benchmark. Across
933 episodes, paths have a median length of 9. Across
933 episodes, paths have a median length of 9. 97 m and
instructions contain a median of 50 words.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Work
Current limitations include scene diversity, human verification as a scaling bottleneck, and the computational cost of full physics simulation. Future work will focus on integrating 3D foundation models for stronger instruction grounding, developing more efficient 3DGS reconstruction methods, and expanding sim-real validation on physical humanoid platforms. p. 6 limitations include scene diversity, human verification as a
scaling bottleneck, and the computational cost of full physics
simulation. Future work will integrate 3D foundation models
for stronger instruction grounding, develop more efficient
3DGS reconstruction, and expand sim-real validation on
physical humanoid platforms. 97 m and
instructions contain a median of 50 words. The modest path–
instruction correlation (r = 0. The modest path–
instruction correlation (r = 0. 29) indicates that instruction
length is not determined solely by trajectory length.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Embodiment-Diverse Humanoid Simulation Platform: The platform supports Unitree G1, Unitree H1, Internal-A, and Internal-B robots via a hierarchical RL-plus-path-tracker control stack with a plug-and-play interface for additional robots and VLN models. This allows for controlled cross-embodiment evaluation under identical environmental conditions.
-
Navigability Curation: Environments are curated by filtering scenes to ensure navigability areas exceed 100 m2, which addresses the need for large, obstacle-sparse traversable areas required by bipedal navigation.
-
MAA-Augmented Instruction Dataset: A Dual Generator-Reviewer + Paraphraser MultiAgent Annotation (MAA) framework generates 933 episodes each with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants, verified by human annotators to produce a collision-aware reference dataset.
-
Hierarchical Control Architecture: The system employs a two-level control hierarchy where an RL policy governs low-level locomotion while interchangeable PD or MPC controllers serve as high-level path trackers, ensuring
Grounding path following in this hierarchy rather than kinematic teleportation ensures that every evaluated trajectory is subject to the true physical constraints of the specific morphology under test.
-
Physics-Aware Episode Sampling: Episodes are sampled by slicing 3D collision meshes at robot body height to form a 2D occupancy map and connecting start-goal pairs via A∗ paths on this map, ensuring
every episode’s video contains genuine locomotion-induced camera dynamics.
-
Instruction Grounding and Route Generation: The system uses heterogeneous generators (like Qwen3-VL-30BA3B and Gemma-4-31B-it) to independently derive route graphs from egocentric keyframes, which are then reconciled by a reviewer agent that
verifies the reconciled route using trajectory metadata, the A∗ route over the occupancy map, and a spatial–semantic scene graph.
-
Cross-Embodiment Performance Evaluation: The platform enables a matched zero-shot evaluation of four representative VLN models across four humanoid configurations under physics-based execution, measuring navigation accuracy, path fidelity (nDTW), and Fall Rate (FR).
-
3DGS Reconstruction Quality: A customized pipeline combining unbiased depth rendering with depth-normal consistency produces
substantially smoother and more geometrically consistent surfaces while preserving scene details,
leading to higher-quality TSDF fusion results for collision meshes. -
Real-World Correlation Pilot: A sim-to-real study shows that per-episode navigation error is strongly correlated between simulation and reality (Pearson r = 0.935), indicating
little systematic endpoint bias
in reconstructed scenes compared to real environments.
Abstract
Vision-Language Navigation (VLN) for humanoid robots poses challenges existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled agents, humanoid morphologies vary across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations, demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10-12 lower-body DoF and heights from 1.17m to 1.80m, via a hierarchical control stack combining a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models integrate with minimal effort; we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigable areas exceeding 100 square meters. Instructions are generated by a dual generator-reviewer plus paraphraser multi-agent pipeline with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one fine-grained instruction and three coarse-grained stylistic variants (formal, natural, casual). Across four models and four embodiments, JanusVLN achieves the highest mean success rate of 43.55% and nDTW of 48.38. In a 20-episode sim-to-real pilot with DualVLN and the Unitree G1, navigation errors correlate strongly (r=0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 (+/-0.188) nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance at https://humanoid-vln.github.io/.
Sources
- Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- Towards Physically Executable 3D Gaussian for Embodied Navigation
- VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
- AI2-THOR: An Interactive 3D Environment for Visual AI
- General Scene Adaptation for Vision-and-Language Navigation
- Qwen3-VL Technical Report
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- NaVILA: Legged Robot Vision-Language-Action Model for Navigation
- StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
- Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation
- JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation