Energy-Structured Latent World Models with Neural Time Fields for Physically Constistent Open-World Motion Planning

Affiliations: PDL Lab, College of Computer Science and Technology, National University of Defense Technology; State Key Laboratory of Complex & Critical Software Environment; EmPACT Lab, Nanyang Technological University, Singapore

arXiv: 2608.09876v1 [cs.RO] 10 Aug 2026

---

The paper addresses the challenge of physically consistent motion planning in embodied AI, where generated trajectories must strictly conform to real-world execution dynamics. The authors propose a novel Energy-Structured Latent World Model (ELWM) whose key idea is to structure the ELWM latent state to explicitly carry energy and momentum, ensuring strictly causal transitions via dissipation and control ports. Trained on multimodal RGB-D and inertial interaction histories, the model guarantees physically consistent predictions. The authors further implement this for motion planning by constructing Physics-Conditioned Neural Time Fields (PC-NTF), which integrates ELWM into an arrival time field via the Eikonal equation to yield a physically-informed navigation policy.

Key quantitative results across held-out scenes include:

• Reducing 0.8-s motion-prediction NRMSE from 0.36 to 0.29 compared to generic latent models

• Improving navigation success from 81.3% to 89.7% and SPL from 0.64 to 0.73 against Active Neural Time Fields

• Cutting physical collision rate from 12.1% to 5.8%

• Reducing Eikonal residual from 0.083 to 0.031

The paper concludes that embedding explicit physical structures into latent spaces intrinsically bridges the gap between predictive world models and safe, dynamically feasible motion planning.

The paper identifies that physically consistent motion planning is a fundamental challenge in embodied AI, the trajectory is useful only when the robot can execute it under its current dynamics. The authors note that "the collision-free geometric path may become unsafe or inefficient when dissipation payload, actuation limits, or the robot's dynamic state changes, and these factors often have weak geometric signatures and become observable through interaction."

The paper critiques existing latent world models: latent prediction does not imply reusable physical knowledge or formal physical consistency. Existing methods mostly optimize unconstrained future representations, leaving dissipation, interaction response, and other physical regularities entangled in task-, env- and embodiment-specific transition weights. They expose neither a causal physical structure nor a predicted quantity that motion planner can generally consume.

The authors also identify a gap in Neural Time Fields: ANTF derives this guidance primarily from static geometry map, it induces the same preference even when ELWM predicts different traversal responses under changes in payload or actuation.

The paper's stated contributions are:

1. ELWM Construction: an energy-structured latent world model that organizes action-conditioned dynamics through a latent phase state with explicit stored-energy, dissipation, and control structure

2. PC-NTF System Implementation: the motion planning system that utilize ELWM predicted execution consequences with NTF geometric time-to-go through specific terminal states and physically dimensioned traversal delays

3. Efficient motion planning results: reducing 0.8-s motion-prediction NRMSE from 0.36 to 0.29, improving navigation success from 81.3% to 89.7% and SPL from 0.64 to 0.73

The paper reviews three relevant research areas:

Physically Consistent Latent World Models: The authors note that while latent world models compress observation action-conditioned histories and future dynamics into meaningful representations, predictive sufficiency does not make robot–environment interaction mechanics reasonable. They cite recent works including AC-HGN (which embeds actions as external forces in an abstract Hamiltonian phase space), Phys-JEPA (which structures physical and residual components of latent prediction), and PH-Dreamer (which combines action-controlled energy flow and dissipation into recurrent world-model dynamics). However, these mainly evaluate with in the learned model or policy, whereas ELWM targets on the motion planning, extracting the navigation regularities and organize action-conditioned prediction through dissipation.

Neural Time Fields for Motion Planning: The paper notes that efficient motion planning can be characterized by Hamilton–Jacobi and Eikonal equations, and that neural fields recently have emerged as suitable environment representations due to their compactness and continuity property. NTFields applies Eikonal formulations to robot configuration spaces, and Active NTFields further learns arrival-time maps online in open-world environments. However, these methods define physical consistency primarily through the Eikonal PDE and geometry-derived propagation speed, but they did not take the dynamic online interaction consequences in consideration.

Open-World Motion Planning Under Changing Dynamics: The authors note that classical sampling-based planners search for collision-free paths in a given configuration-space representation, while learning-based planners accelerate this search using learned trajectory proposals and classical feasibility checks. Experience-driven navigation methods like BADGR and WayFAST learn navigational affordances from self-supervised experience and online traction estimates, but their planner-facing quantities are typically expressed as collision and do not explicitly predict how each candidate command sequence will execute under the robot–environment setting.

The paper considers a differential-drive robot that navigates in a planar workspace omega ⊂ R2 toward the goal g in Eikonal form. The robot state is expressed as:

• rt = (xt, ψt, ξt), where ξt = [vt, ωt]

• Executed control is ut = [vt cmd, ωt cmd]

• Actions at = (νt, ∆t) are velocity commands

• Ht phy collects recent proprioceptive and navigation interaction history

The hypothesis is that the physical quantity delivered to the planner is isotropic traversal-speed field: within one replanning window, the maximum speed the robot can stably sustain at each location is represented by scalar.

Latent physical state: The ELWM encoder Eθ maps interaction history to latent physical state: (Mt, Qt, Vt) = Eθ(Ht phy), where Mt captures the slow physical regularity dynamics (payload, ground response, actuation delay), and Qt, Vt ∈ R(dz/2) are latent generalized coordinates and velocities. The latent inertia G is parameterized by Cholesky factor ct with robot meta-info µr: Gt = Gθ(Mt, µr, ct) ≻ 0, with Pt = GtVt. The port-Hamiltonian latent phase state is Zt = [Qt, Pt].

Latent stored energy: The latent stored energy is formalized as:

Eθ(Qt, Pt; Mt, µr) = ½ PtT Gθ−1 Pt + Uθ(ρ(Qt), Mt, µr)

with kinetic energy Kθ and potential energy Uθ, where ρ(·) is a feature map restricted to relative quantities, which makes Uθ invariant to global translations of Qt.

Port-Hamiltonian transition: The latent transition follows the port-Hamiltonian formalism:

Żt = (J − Rθ(Zt, Mt, µr)) ∇Z Eθ + Bθ(Zt, Mt, µr) ut

with skew-symmetric operator J, dissipation Rθ ⪰ 0, control port Bθ, and latent control input ut = Eu(at, ot, µr). The energy balance property follows directly:

dEθ/dt = −∇Z EθT Rθ ∇Z Eθ + utT yt

This means latent energy changes is the injected system latent work minus a non-negative dissipated power, and it is conserved when ut = 0 and Rθ = 0.

Training: The world model training objective is future prediction:

LELWM = (1/K) Σ (wk/2dz) ‖Ẑt+k − Zt+k+‖2 + λsig RSIGReg

where RSIGReg is the isotropic-Gaussian embedding regularizer of LeJEPA applied to current, target, and rolled-out latents, which prevents collapse. The authors note that gradients propagate through the structure-preserving rollout into Eθ, Rθ, Bθ, Eu, we control the port-Hamiltonian mechanism constraints make effects that the model can reduce prediction error only after the physics.

PC-NTF implements ELWM priors with neural time fields in three steps:

Physics latent decoder: Maps latent predictions into traversal speed, trained by Huber loss from physics-speed envelopes. Given interaction history Ht and candidate command sequence Ut(i), the decoder exposes:

Γt(i) = AWM(Ht, Ut(i)) = (τ̂t(i), x̂t+H(i), ∆TWM(i), ρt(i))

where τ̂t(i) is the decoded observable motion, x̂t+H(i) is its terminal state, ∆TWM(i) ≥ 0 is traversal delay, and ρt(i) summarizes physical predicted tracking, safety, and control consequences.

NTF Construction and Physics Condition: The paper generalizes the geometry Eikonal to a physics-conditioned Eikonal:

sgeo,t(x) ‖∇x T‖2 = 1 → seff,t(x; ζt(x), µr) ‖∇x T‖2 = 1

where ζt(x) = Emap(bt, x) denotes map construction at the motion planning.

Motion-planning and navigation system: For candidate i, PC-NTF evaluates:

T(i) = T(x̂t+H(i), xg) + ∆Tphys(i)

where T(x̂t+H(i), xg) is the geometry time remaining from the predicted terminal location and ELWM provide the Tphys after decoder. The system follows: "query the NTF field, generate shared feasible candidates, predict their motion with the world model, and evaluate them using Eq. (11). Candidates with poor tracking, speed-limit violations, high uncertainty, or abrupt control changes are penalized, and the first action of the lowest-cost feasible candidate is executed."

The motion generation follows:

Sexec t = ΠCt (−seff,t(x) ∇T(x; xg)/‖∇T(x; xg)‖2 + ε)

where ΠCt is a deterministic projection enforcing occupancy boundaries, unknown-space rules, velocity and acceleration clipping, and emergency stops.

Environments: (i) iGibson indoor scenes with cluttered multi-room layouts; (ii) held-out environments sampled from HM3D for topological complexity; (iii) a terrain-variation suite with heterogeneous ground surfaces (carpet, tile, gravel) and payload changes. Train/test splits are by scene, surface, and robot meta-configuration.

Baselines: Active Neural Time Fields, Generic world models (capacity-matched generative action-consistent world model) with MPC budget, and the complete ELWM+PC-NTF system.

Metrics: World-model axis uses 0.8-s motion-prediction NRMSE, pose error, and twist error. Planning axis uses success rate, SPL, physical collision rate, and Eikonal residual.

Table 1 shows ELWM reduces observable-motion NRMSE from 0.36 to 0.29 relative to Generic WM, a 19.4% reduction. Pose and twist errors respectively decrease by 10.8% and 1.8%. A larger action-intervention ratio (1.193 to 1.009) shows the ELWM rollout depends more strongly on the executed control.

Table 2 shows ELWM substantially improves prediction on held-out axes: motion NRMSE decreases from 7.753 to 0.705 on unseen physical regimes and from 7.699 to 0.697 on unseen scenes, corresponding to relative reductions of 90.9% and 91.0%. Latent MSE is also reduced by 49.5% and 49.6%.

The trained ELWM checkpoint attains an energy-balance residual of 1.88 × 10−7, and its predicted dissipated power is nonnegative on all evaluated rollout steps.

Table 3 shows PC-NTF improves SR from 81.3% to 89.7%, SPL from 0.64 to 0.73, and reduces Eikonal residual from 0.083 to 0.031 compared with ANTF. ELWM physics prior bring more latency than origin ANTF, while we expose that PC-NTF latency is better than MPC control port (371.7 to 193.2).

Scalability: ELWM and Generic WM are trained under matched protocols with varying learning rate, weight decay, and SIGReg coefficient. ELWM consistently reaches a lower-loss region within the first few epochs and maintains lower training loss, validation loss, and future-latent consistency error across the tested configurations.

Ablations: Table 4 isolates where PC-NTF gain comes from. The full ELWM + PC-NTF system achieves NRMSE of 1.012, SPL of 0.661, physical collision rate of 5.79%, and latency of 278 ms. Ablations include:

• ANTF alone: NRMSE 1.269, SPL 0.597, collision 14.35%, latency 40 ms

• PC-NTF without physical condition: NRMSE 2.347, SPL 0.483, collision 18.75%, latency 279 ms

• PC-NTF with shuffled Mt: NRMSE 1.036, SPL 0.580, collision 22.11%, latency 266 ms

• PC-NTF with generic WM: NRMSE 1.015, SPL 0.654, collision 6.54%, latency 479 ms

The paper proposes the Energy-Structured Latent World Model (ELWM) that structures action-conditioned prediction through port-Hamiltonian dynamics with explicit dissipation and control ports and Physics-Conditioned Neural Time Fields (PC-NTF) as the navigating motion system realization. PC-NTF connects predicted motion consequences to the global arrival-time representation of Neural Time Fields through a physically interpretable time-to-go objective.

The two-level evaluation analyses physically consistent world-model prediction and closed-loop motion planning, providing evaluation for ELWM physical consistent latent prediction quality and system-level planning benefit.

Future Work: The authors state they are delighted to extend PC-NTF from planar navigation to anisotropic, state-space, and contact-rich motion planning, and to evaluate transfer across broader robot configurations and real-world environments.

Improvements to AI Systems:

1. Physics-Constrained Latent Dynamics for World Models

• Replace unconstrained latent transitions with port-Hamiltonian structure (skew-symmetric coupling, dissipation matrix, control port).

• Enforce energy balance during training (dE/dt = input power − dissipated power), guaranteeing non-negative dissipation and causal rollouts.

• Improved system: Predicts future states that respect physical invariants (energy conservation, momentum transfer), reducing prediction error by 19.4% on 0.8-s motion NRMSE and achieving energy-balance residual of 1.88×10−7.

2. Physics-Conditioned Neural Time Fields for Motion Planning

• Inject learned traversal-speed fields (from ELWM) into the Eikonal equation: s eff(x; ζ(x), μ r)‖∇T‖ = 1, replacing static geometric speed.

• Combine geometric time-to-go with predicted physical traversal delays (ΔT phys) from world-model rollouts.

• Improved system: Generates navigation policies that adapt to payload, terrain, and actuation limits in real time—improving success rate from 81.3% to 89.7%, SPL from 0.64 to 0.73, and cutting physical collisions from 12.1% to 5.8%.

3. Generalizable Physical-Regime Adaptation via Meta-Learning Latent Inertia

• Learn latent inertia matrix G t = G θ(M t, μ r, c t) parameterized by Cholesky factors, conditioned on slow physical regularities (payload, ground response).

• Train on diverse scenes and physical regimes with explicit train/test splits by scene, surface, and robot meta-configuration.

• Improved system: Transfers to unseen physical regimes and scenes with 90.9% and 91.0% reduction in motion NRMSE, respectively, without retraining.

4. Collision-Aware Trajectory Evaluation with Physics Summaries

• Decode latent rollouts into physics summaries ρ t (tracking error, speed-limit violations, control abruptness) and penalize candidates with poor feasibility.

• Use deterministic projection Π Ct for occupancy, unknown-space, velocity, and acceleration constraints.

• Improved system: Reduces physical collision rate by 52% (12.1% → 5.8%) while maintaining low latency (278 ms vs. 479 ms for MPC-based control).

5. Energy-Balanced Regularization for Latent Collapse Prevention

• Combine port-Hamiltonian structure with LeJEPA-style isotropic-Gaussian embedding regularizer (RSIGReg) to prevent latent collapse.

• Improved system: Achieves stable training across hyperparameter configurations, consistently lower loss, and better future-latent consistency than generic world models.

What the Improved AI System Can Do:

• Physically consistent motion prediction: Given RGB-D + inertial history and candidate commands, it predicts future states that obey energy conservation and dissipation, with quantifiable physical validity (non-negative dissipated power, energy-balance residual < 2×10−7).

• Adaptive open-world navigation: In unknown environments with changing terrain (carpet, tile, gravel) or payload, it re-plans trajectories that respect current dynamic limits, avoiding collisions and maintaining stability.

• Fast replanning under dynamic changes: It updates traversal-speed fields online via ELWM rollouts, enabling real-time adaptation to actuation degradation or payload shifts (latency 278 ms vs. 371.7 ms for MPC).

• Interpretable planning objectives: It provides physically dimensioned time-to-go values (geometric + physical delay) that are directly usable by downstream planners, not just abstract latent features.

• Robust generalization: It maintains high prediction accuracy and planning performance on held-out scenes and physical regimes, reducing NRMSE by >90% compared to generic models on unseen conditions.