Why Latent Actions Fail, and How to Prevent It
Latent action models (LAMs) often fail in real videos because they learn to encode noise from background clutter rather than actual actions. The analysis shows that standard training causes latent actions to leak future information and become inconsistent with different background states. Mitigating this requires using auxiliary objectives, like cross-exogenous reconstruction, which force the model to focus only on the action-related components.