CanViT: Toward Active-Vision Foundation Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "CanViT: Toward Active-Vision Foundation Models".
Jane: Here's the gist of the paper. Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses <ref:2603.22570#pg2> but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored <ref:2603.22570#pg2>.
Tom: CanViT introduces the first task- and policy-agnostic AVFM which uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace called the canvas <ref:2603.22570#pg2>.
Jane: Architectural Components CanViT utilizes a dual-stream structure consisting of a ViT backbone that processes localized glimpses and a persistent canvas that serves as fine-grained scene-wide spatio-semantic memory <ref:2603.22570#pg2> The glimpse stream is largely ephemeral, while the canvas stream is fully persistent and can function as a cognitive map <ref:2603.22570#pg2>.
Tom: The interaction between these streams occurs through Canvas Attention, a novel asymmetric cross-attention mechanism that alternates between read (R) and write (W) operations to condition the backbone’s processing on the canvas and populate it <ref:2603.22570#pg2>.
Paper summary: Tom: So, wrapping up CanViT: Toward Active-Vision Foundation Models, the authors are proposing a way to make active computer vision more scalable and general purpose by introducing this task and policy agnostic model. The main claim is that this architecture uses scene relative RoPE to bind a vision transformer backbone with a spatiotopic scene wide latent workspace called the canvas >
Jane: It matters because it addresses the gap where active vision promises efficient perception through sequential glimpses but lacks scalable general purpose architectures and pretraining pipelines, leaving active vision foundation models underexplored >
Lu: The authors tackle this by proposing CanViT, which uses Canvas Attention to create a dual stream structure that separates thinking from memory, leading to faster sequential inference and scalability to high output resolutions >
Meng: For deployment, the focus shifts from needing huge static training sets to having a mechanism for continuous scene-wide context building through that canvas memory >
Lalam: They also propose this label free active vision pretraining scheme called passive to active dense latent distillation which reconstructs scene wide dinov three embeddings from sequences of low resolution glimpses with randomized locations, zoom levels, and lengths >
Tom: The implication for the world is that we can move toward more biologically plausible perception systems that can operate in real time by being less dependent on massive, monolithic models trained once >
Jane: It suggests a future where perception isn't just about one big snapshot but about a continuous, localized interaction with the environment, conditioned by a persistent understanding of the scene >
Conclusion: Tom: So, CanViT is proposing this new way to build active vision models that aren't tied to just one specific task or policy, aiming for something more general and foundation-like.
Jane: Right, so instead of training a model just for one thing, like segmentation or classification, they’re building a backbone that can do many things by using this idea of localized glimpses and scene memory.
Lu: The core idea is using this dual stream structure: you've got the vision transformer processing those quick, localized glimpses while you have this persistent canvas that acts like a long-term cognitive map for the whole scene.
Meng: From an engineering standpoint, I’m interested in how they manage that canvas memory. It sounds like a lot of state to maintain if we want real-time performance on video, so the efficiency here is key.
Lalam: The paper shows they developed this using a label free pretraining method called passive to active dense latent distillation. They are essentially taking low resolution glimpses and reconstructing high resolution scene information by matching it against a powerful teacher model like DINOv3.
Tom: And the results show that CanViT-B sets new records on both ADE20K segmentation and ImageNet classification, even transferring its knowledge across different policies and temporal horizons without needing any retraining.
Jane: That transferability is really interesting for people who want to build versatile vision systems, because it means you don't have to retrain everything from scratch just to switch from one active task to another.
Lu: The performance gains, like that forty five point nine percent mean IoU on ADE20K after adding extra glimpses compared to the baseline, show how much benefit those sequential inputs provide for fine-tuning the perception.
Meng: I saw something about their Coarse to Fine policy; they say it outperforms others in both accuracy and speed on ADE20K. That’s important for deployment because efficiency is everything when you're running this stuff at scale.
Tom: Exactly, so we’re moving past models that only work well in a very specific setup, toward something that can adapt its perception based on what it sees sequentially within a scene context.
Jane: It really boils down to making active vision systems more flexible and capable of learning from their immediate environment without needing a massive, rigid training pipeline for every single application.
Lu: The authors also pointed out some limitations, like the fact that they trained and tested on static scenes initially, but they see the constant memory design as a big step toward real-time video processing.
Tom: So they’re pointing toward RL based policy learning and active to active self distillation as next steps for making these models even more goal directed and efficient.
McGill University · Mila - Quebec AI Institute · Université Laval
cs.CV
Submitted: 2026-03-23
Updated: 2026-11-11
Comments: NeurIPS 2026. Code and weights: https://github.com/m2b3/CanViT
Code: https://github.com/m2b3/CanViT-PyTorch
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses <ref:2603.22570#pg2> but lacks scalable general-purpose architectures and…
Key concepts
- Active-Vision Foundation Models (AVFMs)
- These are models designed for active vision that learn to perceive scenes by taking sequential 'glimpses'—short, localized views. Unlike standard models, AVFMs are general-purpose and can adapt to various tasks without needing retraining or specific policies.
- Canvas
- The canvas is a persistent, scene-wide latent workspace that acts as a memory for the entire environment. It stores spatio-semantic information gathered from glimpses, functioning like a cognitive map that helps the model understand the whole scene.
- Canvas Attention
- This is a novel cross-attention mechanism used to interact between the localized glimpse stream and the persistent canvas. It alternates between reading and writing operations to condition how the backbone processes new visual data while simultaneously updating the memory.
Terminology
Summary
Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses <ref:2603. p. 0 krishna@mcgill. ca
Abstract
Active computer vision promises efficient, biologically plausible perception through
sequential, localized glimpses, but lacks scalable general-purpose architectures and
pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. krishna@mcgill. ca
Abstract
Active computer vision promises efficient, biologically plausible perception through
sequential, localized glimpses, but lacks scalable general-purpose architectures and
pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT
uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a
spatiotopic scene-wide latent workspace, the canvas.Where the paper says it
Where the paper says it
Where the paper says it
Architectural Components
CanViT utilizes a dual-stream structure consisting of a ViT backbone that processes localized glimpses and a persistent canvas that serves as fine-grained scene-wide spatio-semantic memory <ref:2603.22570#pg2> The glimpse stream is largely ephemeral, while the canvas stream is fully persistent and can function as a cognitive map <ref:2603. p. 4 The canvas stream, made up of Dcan-dimensional canvas tokens, is fully persistent. The canvas acts
as a scene-wide spatio-semantic memory, which can function as a cognitive map54 of the scene. The canvas regularly interacts with the glimpse stream via Canvas Attention (Figure 3), alternating
between read (R) and write (W) operations to, respectively, condition the backbone’s processing on
the canvas and populate the canvas. Both streams are equipped with register tokens (gray)
52
.Where the paper says it
Where the paper says it
Pretraining Methodology
The authors propose a label-free active vision pretraining scheme called passive-to-active dense latent distillation, which reconstructs scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths <ref:2603.22570#pg2>. This scheme involves a teacher model like DINOv3 ViT-B to produce high-resolution scene-wide spatio-semantic features which CanViT attempts to match using cheap, partial glimpses <ref:2603. p. 5 We use DINOv3 ViT-B as a high-resolution scene-wide spatio-semantic
teacher, which produces patch tokens (dense features) and a global CLS token. These highly informative reference embeddings represent an idealized scene understanding for CanViT to match using
sequences of cheap, partial glimpses. normalization60, then decode into DINOv3-space reconstructions via token-wise linear projections:
Zˆ
t = WC · LayerNorm(Ct), zˆt = Wh · LayerNorm(ht) (2)
Loss. Our loss combines patch-level and CLS-level reconstruction, averaged across space and time:
L =
1
T
T
X−1
t=0
1
HW
∥Zˆ
t − Z
∗
∥
2
F + ∥zˆt − z
∗
∥
2
(3)
5.Where the paper says it
Where the paper says it
Performance and Generalization
CanViT-B sets a new state of the art among active vision models on both ADE20K segmentation and ImageNet-1k classification while transferring across policies, temporal horizons, and canvas resolutions without retraining <ref:2603.22570#pg2>. The model achieves 45. p. 0 On
ADE20K segmentation, a frozen CanViT-B achieves 38. 5% mIoU in a single lowresolution glimpse, outperforming the best active model’s 27. Given
additional glimpses, CanViT-B reaches 45. 9% ADE20K mIoU. On both tasks, CanViT-B generalizes across policies, temporal horizons, and canvas resolutions,
even with frozen weights (Figure 4B,C). The inference-time-only C2F policy dramatically outperforms the R-IID and F-IID policies on ADE20K in both peak accuracy and efficiency, and provides a
modest efficiency boost early into IN1k rollouts.Where the paper says it
Where the paper says it
Where the paper says it
Efficiency and Analysis
The architecture achieves computational decoupling by having the canvas stream forgo self-attention and fully-connected layers, leading to fast sequential inference <ref:2603.22570#pg2>. The Canvas Attention mechanism is designed to be asymmetric, with a FLOP footprint of 7. p. 4 The glimpse token count Ng must be kept low due to the quadratic cost of
the ViT backbone’s self-attention, and it is desirable for the canvas to have high information capacity
both spatially, by tiling the scene in a fine-grained manner with a large number H × W of canvas
patch tokens (making up the bulk of the Ncan canvas tokens), and semantically at any given position
within the scene, with a large canvas embedding dimension Dcan. This makes the asymmetric design
of Canvas Attention highly advantageous, as the FLOP footprint of a single canvas-side projection
relative to the Scaled Dot Product Attention (SDPA) call that it accompanies would be:
projection FLOPs
SDPA FLOPs
=
2NcanD2
can
4NgNcanDcan
=
Dcan
2Ng
, (1)
which is a 7. This makes the asymmetric design
of Canvas Attention highly advantageous, as the FLOP footprint of a single canvas-side projection
relative to the Scaled Dot Product Attention (SDPA) call that it accompanies would be:
projection FLOPs
SDPA FLOPs
=
2NcanD2
can
4NgNcanDcan
=
Dcan
2Ng
, (1)
which is a 7. 2× ratio with Dcan = 1024 and Ng = 71 (64 glimpse patches, 5 registers, and CLS +
VPE tokens). , for small, distant or occluded objects) (Figure 5A)
or using multiple timesteps (Figure 5B). Together, the accuracy gains provided by finer canvases
and the low computational overhead of Canvas Attention allow CanViT-B to offer an attractive
accuracy-efficiency frontier on ADE20K segmentation at low FLOP budgets (Figure 5C).Where the paper says it
Where the paper says it
Where the paper says it
Inference Considerations
CanViT’s constant-memory recurrent design makes it an attractive candidate for real-time, high-FPS video processing and embodied active perception <ref:2603.22570#pg2>. The latency analysis shows that the minimum latency scales with the output resolution, with GPU inference being significantly faster than CPU inference across various scene grid sizes <ref:2603. p. 6 We use 1282 px glimpses consistently. We
provide additional details on pretraining (Appendix C) and evaluation (Appendix D), as well as a
detailed ablation study (Appendix E) and inference latency measurements (Appendix F). Similarly, RL-based policy learning would be a natural extension,
particularly in goal-directed settings such as visual search; we leave this to future work. We expect
CanViT-B to be directly applicable to monocular depth estimation without retraining, similarly
to its DINOv3 teacher and passive models distilled from DINOv2/v3 features 36.Where the paper says it
Where the paper says it
Limitations and Future Work
Like most active-vision models, CanViT was trained and evaluated on static scenes, but its constant-memory recurrent design makes it ideal for future adaptation to real-time video processing <ref:2603.22570#pg2>. RL-based policy learning is a natural extension for goal-directed settings such as visual search <ref:2603. p. 8 In those contexts, the addition of a lightweight gating mechanism may be desirable
in order to facilitate forgetting. Similarly, RL-based policy learning would be a natural extension,
particularly in goal-directed settings such as visual search; we leave this to future work. This may prove limiting at very high
scene resolutions due to the quadratic cost of teacher self-attention and the large storage footprint of
precomputed features, especially when working with video, which would involve distinct per-timestep
targets. We thus see active-to-active self-distillation as a promising avenue for future research. We thus see active-to-active self-distillation as a promising avenue for future research. Finally,
our results use a single model size pretrained using a modest compute budget and a dataset over 100×
smaller than DINOv3’s LVD-1689M; we expect further gains from more extensive pretraining and
larger models, though the latter may be undesirable for edge deployment.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Active computer vision foundation models can be built that are task-agnostic and policy-agnostic by using a passive-to-active dense latent distillation scheme to reconstruct scene-wide DINOv3 embeddings from sequences of low-resolution glimpses. This allows CanViT to generalize across tasks, temporal horizons, and policies without retraining.
-
The Canvas Vision Transformer architecture can be utilized for vision models capable of understanding the spatial and semantic structure of scenes across arbitrary sequences of glimpses by decoupling thinking (backbone-level) and memory (canvas-level). This is achieved through a recurrent architecture built around a latent scene-wide representation called the canvas.
-
A novel Canvas Attention mechanism, which is asymmetric, can be employed to support efficient interaction between the relatively small set of glimpse tokens and its much larger set of canvas tokens, allowing CanViT to support fine-grained scene/canvas grids without incurring the quadratic cost of self-attention.
-
Viewpoint Encoding (VPE) tokens can be used to facilitate future end-to-end policy learning by allowing the next viewpoint to be decoded from a rich transformation of the current viewpoint, which is parameterized as (x/s, y/s, log s).
-
The model can perform dense prediction tasks like semantic segmentation or depth estimation by decoding updated canvas tokens into DINOv3-space reconstructions via token-wise linear projections. This capability is supported by the fact that
dense predictions require explicit architectural handling.
-
Inference latency can be managed efficiently by leveraging the dual-stream design which decouples glimpse (instantaneous vision) resolution from canvas (memory/output) resolution, allowing for a
low FLOP budget
on ADE20K segmentation. -
The model can perform policy-agnostic active vision pretraining using dual rollouts, such as Random-then-IID (R-IID) and Full-then-IID (F-IID), which ensures robustness to arbitrary rollout starts and provides a low-resolution but high spatial coverage view of each scene at least once.
Abstract
Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce the Canvas Vision Transformer (CanViT), the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.
Sources
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Hierarchical Reasoning Model
- Less is More: Recursive Reasoning with Tiny Networks
- Adaptive Computation Time for Recurrent Neural Networks
- Layer Normalization
- Are we done with ImageNet?
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs