Object Concepts Emerge from Motion

arXiv:2609.04348 · cs.CV · Submitted 2026-09-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Object Concepts Emerge from Motion".

Jane: The gist The framework proposes a biologically inspired method to learn object-centric visual representations from raw videos by using motion boundaries as pseudo-instance supervision,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: , Jane, Lu, Meng, Lalam: Welcome back. We're diving into a new paper today called "Object Concepts Emerge from Motion." This research tackles how we teach AI to see objects in the real world by looking at motion cues instead of just static pictures.

Tom: So, what’s the big idea here? Basically, they argue that object concepts—understanding what an object is—come from watching things move, not just seeing a single frame. They want to build a system that learns these object-centric representations directly from raw videos.

Jane: That makes sense. It’s inspired by how babies learn to recognize objects through motion before they even grasp abstract categories like "chair" or "dog." The paper suggests that motion boundaries are a strong signal for grouping things into separate objects.

Lu: The core insight is that motion boundaries provide a strong signal for object-level grouping and can be used to derive pseudo-instance supervision, which they call it ><ref:2609.04348#pg1>. They’re taking the dynamic nature of video and using it to teach a static image encoder about the objects within those videos.

Meng: From an engineering standpoint, that sounds ambitious because you want to learn object identity without any human labels or camera calibration. How do they actually get these instance masks from just a video?

Tom: That’s where their methodology comes in. They use off-the-shelf optical flow and clustering to construct category-agnostic pseudo-instance masks at scale, which then supervise a single image encoder ><ref:2609.04348#pg3>. It separates the video motion source from the encoder that only needs a static image.

Jane: And they don't stop there; they use a two-cycle training process to expand that supervision significantly for better learning. They start with one hundred ninety-five million pseudo-labeled frames and then use something called Motion-Verified Self-Training to get up to four hundred twenty-one million frames ><ref:2609.04348#pg2>.

Lu: That second cycle is interesting because it uses low-confidence motion cues that the first cycle might have discarded, having the encoder propose candidate masks that are then refined by motion evidence ><ref:2609.04348#pg5>. It’s a way to leverage more subtle movement information.

Meng: So, what does this actually mean for practical AI? The abstract mentions applications like monocular depth estimation, three dee object detection, and end-to-end planning ><ref:2609.04348#pg1>. Is the performance competitive with existing methods?

Tom: The results show that the resulting models achieve competitive or superior performance relative to supervised and self-supervised pretraining baselines across several tasks ><ref:2609.04348#pg1>. They even have particularly strong transfer on geometry- and instance-sensitive tasks.

Jane: It sounds like they are showing that this motion-derived supervision can actually teach static image encoders how to represent visual instances, which is a really useful direction for scalable visual pretraining ><ref:2609.04348#pg1>. It’s providing a complementary way to train models without needing massive datasets of human annotations.

Paper summary: Lu: They trained their encoders up to Swin-H and then distilled those representations into a family of Swin backbones ><ref:2609.04348#pg2>. This distillation process helps make the learned representations more efficient for deployment in things like monocular depth estimation ><ref:2609.04348#pg9>.

Meng: So, if I’m looking at this from a development perspective, the fact that they don't need camera calibration or human annotations is huge for real-world deployment ><ref:2609.04348#pg1>. That removes a massive bottleneck in training models for physical tasks.

Tom: Exactly! The paper shows that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining ><ref:2609.04348#pg1>. It moves us toward systems that learn from the real world naturally as they see it.

Jane: Thinking about the titles and authors, Lu, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang and Naiyan Wang are looking at this from a perspective where motion is a fundamental component of visual understanding ><ref:2609.04348#pg1>. It’s less about just pattern matching and more about capturing the underlying structure of physical interaction.

Lu: We think what they are doing here is mimicking that developmental trajectory we talked about earlier, where infants learn object unity through motion before moving to abstract appearance-based concepts ><ref:2609.04348#pg2>. It’s a biologically inspired way to approach representation learning.

Meng: For me, I’m looking at how this could impact the engineering of systems that need to understand three dee space quickly, like autonomous vehicles or robotics, because they don't have time for slow human labeling ><ref:2609.04348#pg1>. If we can get good instance separation just from video flow, that’s a huge win for real-time perception.

Tom: That’s right; the paper shows that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining ><ref:2609.04348#pg1>. It's showing us a path forward where we use the physics of movement to build smarter vision systems.

Jane: So, looking at the title and authors of "Object Concepts Emerge from Motion," it really highlights how foundational motion cues are for visual cognition ><ref:2609.04348#pg1>. It’s suggesting that our computational models should incorporate dynamic interaction more explicitly to build robust object understanding.

Lu: The implication is that we don't have to wait for massive, perfectly labeled datasets of every possible scene to teach an AI what an object is ><ref:2609.04348#pg1>. We can start with the motion and let the system figure out the objects from there ><ref:2609.04348#pg2>.

Meng: From a practical standpoint, I see this as a way to improve how we handle scene understanding in complex environments where static images alone fail to capture object relationships accurately ><ref:2609.04348#pg1>. It’s about learning structure from interaction, not just appearance.

Paper summary: Tom: Absolutely. The paper shows that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining ><ref:2609.04348#pg1>. It opens up a new way to pretrain models on massive video data efficiently.

Jane: So, when we look at the conclusion of "Object Concepts Emerge from Motion," it really underscores how motion is a primary signal for object grouping ><ref:2609.04348#pg1>. It’s a reminder that the physical experience of moving through space is key to what we see.

Lu: The implication for the future is that we could design models that evolve their understanding from purely motion-based interactions toward abstract, appearance-based concepts as they mature ><ref:2609.04348#pg2>. It suggests a more organic way for AI to learn the world.

Meng: And for me, the practical implication is that future perception systems will likely integrate motion cues directly into their core learning loop, not just as a post-processing step ><ref:2609.04348#pg1>. It’s about making the model inherently aware of movement when it learns an object.

Tom: So, to wrap up this discussion on "Object Concepts Emerge from Motion," we see a framework that uses motion boundaries for supervision to teach static image encoders about individual objects ><ref:2609.04348#pg1>. This shows that learning can happen through dynamic experience before abstract concepts take over.

Jane: It really suggests that for scalable visual pretraining, focusing on the physical process of object grouping might be a more powerful starting point than just relying on static images ><ref:2609.04348#pg1>. It’s a nice shift in focus for how we think about training models.

Lu: We think this approach gives us a strong foundation, moving toward models that understand geometry and instance sensitivity better across various vision tasks ><ref:2609.04348#pg1>. It’s the bridge from seeing pixels to truly understanding objects in the physical world.

Meng: From my side, I just see how this could translate into more robust scene understanding for things that aren't perfectly clear in a single snapshot, like crowded or moving scenes ><ref:2609.04348#pg1>. It’s about handling the messiness of real video data better.

Tom: That’s right; the paper shows that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining ><ref:2609.04348#pg1>. It's a compelling look at how we can build more intuitive and instance-aware AI systems.

Jane: So that’s what we had with "Object Concepts Emerge from Motion," an interesting paper showing us how motion cues are key to teaching AI about objects ><ref:2609.04348#pg1>. We'll be right back after the break with more tech news.

Conclusion: Tom: So we’re wrapping up this look at "Object Concepts Emerge from Motion." Basically, they found that using motion cues in videos can teach an AI to actually understand what an object *is*, not just what it looks like in a single picture.

Jane: That’s the core idea, Tom. Think about it—instead of humans having to draw boxes around every single thing in a photo, the AI learns object separation just by watching things move around.

Lu: It’s biologically inspired, you know? They link it to how infants learn unity through movement before they grasp abstract concepts like 'chair' or 'ball.' The paper suggests motion boundaries are a strong signal for where one object ends and another begins.

Meng: But how does that translate into real-world use for systems that need precision? Like, if you’re building a self-driving car, what does this mean in terms of reliability when the motion data isn't perfect?

Lalam: From my side, Lalam sees this as a massive cultural shift. If AI can learn object structure from raw video motion without needing thousands of labeled images, it makes building culture and knowledge systems way more scalable.

Tom: Exactly, Meng. And the authors used optical flow and clustering to create these initial masks without any human labels or camera calibration needed at all. That’s a huge win for deployment.

Jane: They did get really rigorous with their training cycles too, expanding the supervision from one hundred and ninety-five million frames up to four hundred twenty-one million using that motion-verified self-training method.

Lu: And they distilled the complex Swin models into smaller versions, which means these object representations aren't just sitting in a giant brain; they can be used efficiently in actual applications like three dee occupancy prediction.

Meng: So, what does this mean for someone just listening to the show? It means future perception systems might start learning from movement directly instead of relying solely on static inputs.

Tom: Right. And the implication is that we might move toward AI that is inherently more aware of physical interaction, learning object identity through dynamic experience first. Next up, we’re going to look at some specific results on depth estimation and how these new models stack up against the current leaders in three dee sensing.

cs.CV

Submitted: 2026-09-03

Updated: 2026-10-09

Code: https://github.com/autonomousvision/navsim

Project page: https://tj12342.github.io/object-concepts-from-motion

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist The framework proposes a biologically inspired method to learn object-centric visual representations from raw videos by using motion boundaries as pseudo-instance supervision, which enables

Key concepts

Motion Boundaries: The paper suggests that the edges or boundaries where motion changes significantly often align with the actual boundaries of physical objects in a scene. By using these motion cues, the system can derive initial, category-agnostic pseudo-instance masks without needing manual annotations or camera calibration.
Pseudo-Instance Masks: These are temporary object segmentation labels generated automatically by combining optical flow and clustering algorithms applied to video frames. They serve as supervision for the encoder to learn how to separate individual objects within a single image, focusing on instance separation rather than semantic category identification.
Motion-Verified Self-Training: This is a second training cycle that refines the initial object proposals. It uses low-confidence motion evidence to verify and refine the candidate masks proposed by an earlier model stage. This process expands the usable supervision significantly, allowing the model to learn more robust instance representations.
Pixel-Level Metric Learning: This is a training objective applied directly to individual pixels of an image. It forces the encoder's output representation to place pixels belonging to the same object close together in a learned space, effectively encouraging the model to capture object unity and distinctness without relying on external category labels.

Summary

The gist The framework proposes a biologically inspired method to learn object-centric visual representations from raw videos by using motion boundaries as pseudo-instance supervision, which enables geometry- and instance-sensitive scene understanding.

Object Concepts Emerge from Motion

The paper introduces a framework that leverages motion cues to derive pseudo-instance supervision for training a single-image encoder without requiring human annotations or camera calibration (Page 2). The core insight is that motion boundaries often align with object boundaries, echoing findings in neuroscience where common motion is crucial to the early development of object unity (Page 2). This approach aims to mimic the developmental trajectory where infants acquire object understanding through motion-based interactions before evolving toward abstract, appearance-based concepts (Page 2).

Data Processing and Two-Cycle Training

The method utilizes a two-cycle training paradigm to scale the supervision effectively (Page 4).

Cycle 1 generates pseudo-instance masks by using an off-the-shelf optical flow estimation algorithm followed by clustering to create category-agnostic pseudo-instance labels (Page 5).
A unified processing pipeline scales the supervision across heterogeneous video sources to produce 195M pseudo-labeled frames (Page 2).
Cycle 2 employs Motion-Verified Self-Training to utilize low-confidence motion cues discarded in Cycle 1 by having the Cycle1 encoder propose candidate masks, which are then verified and refined by motion cues (Page 5). This expands usable supervision to 421M pseudo-labeled frames (Page 2).

Representation Learning and Distillation

The learned object representation is optimized through a pixel-level metric-learning objective supervised by the pseudo-instance masks (Page 7). The objective encourages capturing object unity and instance separation without explicitly supervising category identity (Page 7).

The model is scaled up to Swin-H in Cycle 1, and then this representation is distilled into a family of Swin backbones (Page 2). The distillation process involves using the Cycle2 Swin-H encoder as a frozen teacher to match its dense representation with newly initialized student backbones (Page 2).

**- Monocular Depth Estimation: The Swin-L model outperforms DINOv3 ViT-L in Abs Rel (0.042 vs. 0.044), RMSE (1.715 vs. 1.787), and δ1 (0.988 vs. 0.986) on the KITTI Eigen split (Page 9). The Swin-H model achieves the best performance among our models on the official KITTI online benchmark with 6.36 Abs Rel (Page 9). The model better preserves coherent surfaces compared to DINOv2 ViT-L (Page 10). The complete Cycle-2 pipeline improves all primary metrics, such as SILog decreasing from 5.90 to 5.69 on KITTI depth estimation (Page 11). The control experiment shows that a longer Cycle-1 run or a short Cycle-2 pretraining improves every non-saturated depth metric (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation (Page 11). The complete Cycle-2 pipeline improves SILog from 5.90 to 5.69 on KITTI depth estimation <ref:2609.

Improvements for AI systems

No-Annotation Object Concept Learning: The system can learn object-centric visual representations from raw videos by converting optical flow into pseudo-instance supervision, eliminating the need for human annotations or camera calibration. This enables deployment on large, unstructured video datasets without prior labeling overhead.
Instance-Sensitive Scene Understanding: The learned representations are explicitly structured to capture within-object coherence while separating distinct instances, which directly supports geometry and instance-sensitive scene understanding in downstream tasks like 3D object detection and planning.
Scalable Video Pretraining Pipeline: The framework scales efficiently through a unified pipeline, generating 195M pseudo-labeled frames from 7,163 hours of heterogeneous driving and self-collected web videos, demonstrating robustness across vast and diverse data sources.
Robust Self-Training via Motion Verification: The Motion-Verified Self-Training paradigm expands supervision to 421M pseudo-labeled frames by using motion cues to refine model proposals, ensuring that the training set incorporates both appearance and motion evidence for higher label coverage.
Geometry-Grounding in Depth Estimation: The representation transfers effectively to low-level geometric prediction, as demonstrated by achieving competitive or superior performance in monocular depth estimation metrics like Abs Rel and RMSE compared to baselines like DINOv3.
End-to-End Planning Integration: The object-centric features are successfully integrated into planning frameworks (DriveSuprim), yielding competitive Extended Predictive Driver Model Score (EPDMS) results, showing the representation's utility for complex decision-making tasks.