CanViT: Toward Active-Vision Foundation Models

summary

The gist

Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses <ref:2603.22570#pg2> but lacks scalable general-purpose architectures and…

In short

CanViT introduces Active-Vision Foundation Models (AVFMs) by combining a Vision Transformer backbone with a persistent scene-wide latent workspace called 'canvas'. It uses sequential, localized glimpses to perceive scenes, allowing for efficient, biologically plausible perception without needing specific tasks or policies defined beforehand.

Key concepts

Active-Vision Foundation Models (AVFMs)
These are models designed for active vision that learn to perceive scenes by taking sequential 'glimpses'—short, localized views. Unlike standard models, AVFMs are general-purpose and can adapt to various tasks without needing retraining or specific policies.
Canvas
The canvas is a persistent, scene-wide latent workspace that acts as a memory for the entire environment. It stores spatio-semantic information gathered from glimpses, functioning like a cognitive map that helps the model understand the whole scene.
Canvas Attention
This is a novel cross-attention mechanism used to interact between the localized glimpse stream and the persistent canvas. It alternates between reading and writing operations to condition how the backbone processes new visual data while simultaneously updating the memory.

Terminology used across episodes

This episode discusses

The paper

CanViT: Toward Active-Vision Foundation Models · Read on arXiv

McGill University · Mila - Quebec AI Institute · Université Laval

Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce the Canvas Vision Transformer (CanViT), the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "CanViT: Toward Active-Vision Foundation Models".

Jane: Here's the gist of the paper. Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses <ref:2603.22570#pg2> but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored <ref:2603.22570#pg2>.

Tom: CanViT introduces the first task- and policy-agnostic AVFM which uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace called the canvas <ref:2603.22570#pg2>.

Jane: Architectural Components CanViT utilizes a dual-stream structure consisting of a ViT backbone that processes localized glimpses and a persistent canvas that serves as fine-grained scene-wide spatio-semantic memory <ref:2603.22570#pg2> The glimpse stream is largely ephemeral, while the canvas stream is fully persistent and can function as a cognitive map <ref:2603.22570#pg2>.

Tom: The interaction between these streams occurs through Canvas Attention, a novel asymmetric cross-attention mechanism that alternates between read (R) and write (W) operations to condition the backbone’s processing on the canvas and populate it <ref:2603.22570#pg2>.

Paper summary: Tom: So, wrapping up CanViT: Toward Active-Vision Foundation Models, the authors are proposing a way to make active computer vision more scalable and general purpose by introducing this task and policy agnostic model. The main claim is that this architecture uses scene relative RoPE to bind a vision transformer backbone with a spatiotopic scene wide latent workspace called the canvas >

Jane: It matters because it addresses the gap where active vision promises efficient perception through sequential glimpses but lacks scalable general purpose architectures and pretraining pipelines, leaving active vision foundation models underexplored >

Lu: The authors tackle this by proposing CanViT, which uses Canvas Attention to create a dual stream structure that separates thinking from memory, leading to faster sequential inference and scalability to high output resolutions >

Meng: For deployment, the focus shifts from needing huge static training sets to having a mechanism for continuous scene-wide context building through that canvas memory >

Lalam: They also propose this label free active vision pretraining scheme called passive to active dense latent distillation which reconstructs scene wide dinov three embeddings from sequences of low resolution glimpses with randomized locations, zoom levels, and lengths >

Tom: The implication for the world is that we can move toward more biologically plausible perception systems that can operate in real time by being less dependent on massive, monolithic models trained once >

Jane: It suggests a future where perception isn't just about one big snapshot but about a continuous, localized interaction with the environment, conditioned by a persistent understanding of the scene >

Conclusion: Tom: So, CanViT is proposing this new way to build active vision models that aren't tied to just one specific task or policy, aiming for something more general and foundation-like.

Jane: Right, so instead of training a model just for one thing, like segmentation or classification, they’re building a backbone that can do many things by using this idea of localized glimpses and scene memory.

Lu: The core idea is using this dual stream structure: you've got the vision transformer processing those quick, localized glimpses while you have this persistent canvas that acts like a long-term cognitive map for the whole scene.

Meng: From an engineering standpoint, I’m interested in how they manage that canvas memory. It sounds like a lot of state to maintain if we want real-time performance on video, so the efficiency here is key.

Lalam: The paper shows they developed this using a label free pretraining method called passive to active dense latent distillation. They are essentially taking low resolution glimpses and reconstructing high resolution scene information by matching it against a powerful teacher model like DINOv3.

Tom: And the results show that CanViT-B sets new records on both ADE20K segmentation and ImageNet classification, even transferring its knowledge across different policies and temporal horizons without needing any retraining.

Jane: That transferability is really interesting for people who want to build versatile vision systems, because it means you don't have to retrain everything from scratch just to switch from one active task to another.

Lu: The performance gains, like that forty five point nine percent mean IoU on ADE20K after adding extra glimpses compared to the baseline, show how much benefit those sequential inputs provide for fine-tuning the perception.

Meng: I saw something about their Coarse to Fine policy; they say it outperforms others in both accuracy and speed on ADE20K. That’s important for deployment because efficiency is everything when you're running this stuff at scale.

Tom: Exactly, so we’re moving past models that only work well in a very specific setup, toward something that can adapt its perception based on what it sees sequentially within a scene context.

Jane: It really boils down to making active vision systems more flexible and capable of learning from their immediate environment without needing a massive, rigid training pipeline for every single application.

Lu: The authors also pointed out some limitations, like the fact that they trained and tested on static scenes initially, but they see the constant memory design as a big step toward real-time video processing.

Tom: So they’re pointing toward RL based policy learning and active to active self distillation as next steps for making these models even more goal directed and efficient.

More episodes

← Home