LIME: Learning Intent-aware Camera Motion from Egocentric Video

The gist LIME learns intent-aware camera motion from passive human video and transfers it to robots by predicting relative target camera poses conditioned on a natural-language intent, enabling reusable active perception primitives for embodied tasks.

How it works

LIME formulates language-conditioned camera motion generation by predicting a relative SE(3) target camera pose and an observation-gain description from a current RGB observation and a free-form language intent given the current RGB observation Is and a free-form intent x, predict a relative camera motion Tgs that moves the camera from the current pose Ps to a goal pose Pg, producing a next observation Ig that provides visual evidence relevant to the intent In addition to motion, we predict an observation-gain description g, a natural-language summary of what the next view is expected to reveal beyond the current view The model learns pθ(g Is, x), pθ,ϕ(Tgs Is, x, g), where Tgs is represented by 3D translation and a continuous 6D rotation parameterization

Vision-Language Camera-Motion Generator

LIME implements the query with two coupled interfaces over a shared vision-language representation: a language interface for g and an action interface for Tgs The observation-gain interface keeps part of the task in the VLM’s native output space by describing the intended visual outcome of the movement The continuous flow-matching head then models the conditional distribution of relative SE(3) target poses from this fused representation Formally, given (Is, x), we construct a multimodal prompt and encode the hidden sequence Hx = VLMθ(Is, x) over visual and language tokens with the VLM backbone before generating the observation-gain description The language interface uses the VLM’s autoregressive decoder to generate the observation-gain description pθ(g Is, x) = Yk pθ(gk g<k, Hx) After autoregressive gain generation, we form the gain-conditioned hidden sequence Hx,g = [Hx; Hgx], where Hgx denotes the hidden states of the observation-gain tokens conditioned on the image-intent prompt Instead of representing actions or spatial coordinates as language tokens [11, 10, 54], we attach a separate continuous flow-matching head, which preserves geometric supervision in SE(3) and models the multimodal distribution of plausible target transforms We parameterize the target transform Tgs as y = ψ(Tgs) ∈ R9 using 3D translation and the first two columns of the rotation matrix For a flow time t ∼ U(0, 1) and Gaussian noise ϵ ∼ N (0, I), we construct zt = (1 − t)ϵ + ty The pose head Fϕ takes (Hx,g, zt, t) and predicts the clean target yˆϕ = Fϕ(Hx,g, zt, t), with loss Lpose = ∥yˆϕ − y∥2 We adopt the x-prediction parameterization [56]: instead of directly regressing the velocity field, the network predicts the clean target pose parameter vector yˆϕ = Fϕ(Hx,g, zt, t)

Mining Active Camera-Motion Supervision from Passive Egocentric Video

To obtain supervision without teleoperated active-perception demonstrations, we mine egocentric video by pairing temporally separated frames The relative camera transform provides a motion target, while a labeling module produces plausible intents and observation-gain descriptions from the image pair We sample temporally ordered start–goal frame pairs (Is, Ig) and retain local transitions with available RGB frames and valid camera geometry We then label each retained transition with a structured hindsight VLM prompt The labeller receives the current frame Is, the goal frame Ig, and a compact summary of Tgs, which gives explicit motion cues including translation direction, distance, and rotation angle Instead of asking for a free-form caption, the prompt asks for contrastive fields: a motion type, newly visible objects or regions, improved views of content already present in Is, spatial anchors between the two views, an observation-gain description g, and a set of plausible intents Xs,g = m i=1 We unroll each labeled transition into m examples (Is, xi, g, Tgs), one per intent The resulting training set contains approximately 3 M intent-conditioned examples from RoomTour3D [63] and Nymeria [58], covering room-scale walkthroughs and body-scale egocentric interactions

Training and Inference

For each training tuple (Is, xi, g, Tgs), we apply teacher-forced next-token prediction to g and cleantarget flow matching to Tgs, optimizing L = Lgain + λposeLpose Together, the losses keep the backbone aligned with language generation while training the continuous head for the same intent-conditioned motion We instantiate Qwen3-VL-4B-Instruct [54], freeze the vision encoder, and train the multimodal projector, language model, and flow-matching head The pose loss uses detached VLM hidden states, updating only the flow head and VLM-to-pose projection while the backbone is updated by the observation-gain loss At inference, we autoregressively generate gˆ and reuse cached gain-token hidden states as the flow-head condition, avoiding a second full VLM forward We train for one bf16 epoch on 16 NVIDIA GH200 GPUs, taking approximately 30 hours

Benchmark

To evaluate intent-conditioned camera motion at the granularity we study, the benchmark needs three properties: (R1) free-viewpoint photorealistic rendering over continuous SE(3) poses; (R2) diverse intent coverage across spatial scales; and (R3) outcome-level success measures under a shared motion budget We instantiate the benchmark on InteriorGS [64], a 1K-scene real-world indoor 3DGS dataset that supports photorealistic rendering from arbitrary camera poses and intrinsics Each start–reference pair contains (Is, Xs,g, Ig) plus the underlying camera poses and intrinsics The benchmark contains 425 examples across three intent families: 152 Target-approaching, 142 Exploration, and 131 Perspective-shift

Evaluation Protocol

All methods are evaluated through the same budgeted multi-step protocol.

1. Bold header: Intent-aware camera motion generation

LIME can predict a relative target camera pose for the next observation given an RGB observation and a natural-language intent, allowing robots to proactively move their cameras to acquire intent-relevant visual evidence.

2. Bold header: Continuous flow-matching pose head

The model utilizes a continuous flow-matching pose head that models the conditional distribution of relative SE(3) target poses from the fused representation, enabling it to represent multiple hypothesis target views instead of just a single deterministic next-pose regression.

3. Bold header: Multi-intent supervision mining

The system mines supervision by pairing plausible intents and observation-gain descriptions with relative SE(3) target poses from passive egocentric video, which allows the model to learn from ordinary human recordings into supervision for a reusable active-perception primitive.

4. Bold header: Downstream robotic task transfer

The learned intent-aware model can be used as a reusable active-perception primitive that supports manipulation, embodied question answering, and multi-step robot behaviors, enabling the robot to perform tasks like object scanning or preconditioning views for manipulation.

5. Bold header: Collision-aware success metric

Evaluation utilizes a collision-aware success rate (CA-SR) which requires the trajectory prefix to remain collision-free under a specified clearance threshold, ensuring that learned motions are not only visually successful but also physically traversable in the real world.

6. Bold header: Intent and spatial scale generalization

The method demonstrates effectiveness across tasks with diverse granularities by achieving high success rates across Target-approaching, Exploration, and Perspective-Shift intent families, indicating a model that generalizes beyond specific navigation tasks.