Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

arXiv:2610.12060 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Look Back, Think Ahead".

Tom: The gist The proposed ViMoD framework addresses computational expense in multimodal Large Language Models by maintaining a compact visual context while preserving access to original fine-grained evidence as reasoning needs…

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, what we're looking at here is how they’ve built this ViMoD system to save computation when you're doing complex reasoning tasks with images that have a lot of visual data.

Jane: It’s basically about keeping a small visual context in mind while still being able to pull out the specific image details whenever the reasoning needs them.

Tom: Right, and the big idea is they split the problem into two parts to manage this efficiency. First, they have DART for making those compact coarse representations linked back to the original fine tokens.

Lu: What’s cool about DART is that it learns how to group tokens based on their content affinity. So if you're looking at a blank area, it compresses that differently than if you're looking at text.

Meng: That sounds like smart compression for specific parts of the image, not just a blanket compression of everything.

Jane: Exactly. And then they have TRACE, which is Temporal Routing for Adaptive Contextual Evidence. This part uses the decoding history to guess what visual information you’ll need next and decides whether to keep, update, or replace those fine-grained groups of tokens.

Tom: It's really interesting that TRACE doesn't just use a fixed context; it changes its strategy based on what the model is actually trying to figure out at that moment in the reasoning process.

Jane: And they’ve used supervised training and reinforcement learning to tune both parts. DART gets trained by matching full-visual teacher predictions, and TRACE parameters are guided by a loss function targeting specific actions, like deciding when to update versus when to hold onto evidence.

Lu: The results they're showing are pretty strong. They’re getting state-of-the-art performance in visual reasoning even when they only use about twenty percent of the original visual token budget.

Tom: That’s what really catches my eye, Jane. Exceeding the strongest one-shot baseline by thirty-nine point zero percent at that twenty percent budget level is a big deal for how much information we can squeeze into these models.

Jane: It also shows they lead the general average across every budget they tested, which suggests it’s a reliable way to get good performance without needing the full visual input constantly.

Meng: I'm looking at that inference time reduction too, going from twenty percent down to ninety percent on MMMUPro while keeping the output length close to full. That means you get both better accuracy and much faster processing for long responses.

Tom: So, what this means for us is that we can start thinking about how multimodal models handle more complex visual tasks without having to process every single pixel or token all the time.

Jane: But they also pointed out some real hurdles. They mentioned that the level of detail in the coarse-to-fine mapping and those fixed routing intervals put constraints on how precisely you can access those fine tokens.

Lu: And they're upfront about that, which is good because it tells us where we need to focus future research if we want to push this further.

Tom: Yeah, they also said that building this coarse memory adds some initial work just to set it up, and keeping both the coarse context and the fine token banks means you’re storing more data overall.

Jane: And there’s a cost model they use, which suggests that for some longer responses, those small per-token savings don't actually translate into less total computation because you end up doing more work across the whole response.

Tom: So it’s not just about saving tokens; it’s about saving computation in a way that depends on the length of what you're asking the model to do.

Jane: It shows that efficient visual compression depends on how you make that evidence available across different stages of reasoning, not just which pieces you keep.

The paper's summary: Tom: So, we've seen how ViMoD handles efficient visual reasoning by using DART for compact memory and TRACE for adaptive access, but what are they suggesting next to make it even better?

Jane: The paper hints that the current setup has some constraints, especially around the granularity of that coarse-to-fine mapping and those fixed routing intervals.

Tom: Right, so what's the fix for those limitations? I’m looking at how they propose improving multi-step reasoning efficiency by implementing ViMoD itself.

Lu: They say that by making this framework work better, you can maintain that compact visual context while still having access to the original fine-grained evidence as your reasoning needs keep evolving.

Meng: So it’s about making the system more flexible so it doesn't get stuck with a fixed view of the image from the start.

Jane: And on top of that, they introduce temporal routing for TRACE to handle those shifting needs better, using a state-space model to integrate decoding history and decide what evidence to select or replace dynamically.

Tom: That sounds like a way to get more adaptive control over the visual context without increasing the computational load of keeping everything active at once.

Lu: And they also talk about optimizing these components using reinforcement learning from offline optimization, RLOO, guided by success-conditioned feedback on mean fine occupancy to improve how TRACE makes its choices.

Meng: That's a practical way to tune the system beyond just the initial supervised training and get those selection policies really sharp.

Jane: Plus, they look at cost-aware reasoning using the total cost model, showing that we need to consider inference savings over complete responses rather than just per-token savings alone.

Tom: That’s a key point. It suggests that for longer tasks, the total decoder computation might actually go up if you aren't careful with how you use this compression method.

Lu: So what this means for the future is that efficient visual compression depends not just on which evidence you keep, but also on how it’s made available across those different reasoning stages.

Jane: It shows value in adapting visual evidence access to changing needs while cutting down the overall computation needed for a complete response.

Tom: So we're moving beyond just making things smaller and now focusing on making the retrieval smarter based on the task at hand.

Meng: For practical AI deployment, this means we need to build systems that can dynamically adjust their visual attention budget during long reasoning chains, instead of using a fixed strategy.

Lu: It opens up possibilities for models to handle more intricate, multi-stage visual problems efficiently by managing their attention budget intelligently rather than just blindly compressing everything upfront.

The paper's improvements: Tom: So we've looked at how ViMoD manages visual memory on demand for efficient multimodal reasoning today.

Jane: It’s about balancing keeping a compact visual context while still letting you pull out specific image evidence as the reasoning needs change.

Lu: The paper shows that by using DART and TRACE, they can achieve state-of-the-art performance even when they only use twenty percent of the original visual token budget.

Meng: That’s a significant jump in accuracy for a smaller memory footprint, which is what we need for real-world deployment on resource-constrained hardware.

Lalam: From my side, it means that AI can get much better at handling long sequences of visual input without needing to keep every single detail loaded into the active context constantly.

Tom: And the cost model they use shows that this isn't just about saving tokens; it’s about reducing total decoder computation for longer responses.

Jane: It changes how we think about scaling these models, moving away from a fixed memory approach to one that adapts its visual access based on the specific reasoning step.

Lu: The constraints they mentioned are real though, like the fixed routing intervals and the initialization work needed to build that coarse memory structure.

Meng: I'm interested in how those constraints affect real-world tasks where we don't have perfect control over what visual information is needed at each step.

Lalam: If models can dynamically select evidence, it could make our vision-language tools much more flexible for complex, open-ended tasks.

Tom: Overall, the "Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning" paper gives us a really solid framework for smarter visual processing in AI.

Jane: It’s about making the visual memory accessible only when it's actually needed for that specific part of the reasoning process.

Tom: We’ll keep an eye on how they refine those routing intervals and what those next steps look like to make this system even more robust.

Conclusion: Tom: So today we've been digging into "Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning," which is about how to make multimodal models process images without loading everything at once.

Jane: Basically, it’s about balancing keeping a small visual context while still being able to pull out specific image evidence whenever the reasoning needs change.

Lu: The authors propose ViMoD, which is this lightweight framework designed to keep a compact visual memory while still letting you access the original fine-grained evidence whenever your reasoning demands it.

Meng: So, if I'm driving and I need to look at a road sign right now but maybe review a specific part of my map later, ViMoD suggests we don't throw away all that visual detail upfront.

Tom: Exactly. The core idea behind ViMoD is twofold: first you have this Deformable Aggregation of Region-wise Tokens, which creates those compact coarse representations linked back to the original fine tokens. And second you have Temporal Routing for Adaptive Contextual Evidence to manage when and how you pull that evidence out as reasoning progresses.

Jane: That sounds complicated on paper, but let's unpack it a little. The DART component learns content-adaptive groups and aggregation capacities. It figures out how to make those compact coarse representations based on the image content, using a learned affinity score to decide which tokens stick together.

Lu: Right. And this allows for stage-specific evidence access through something they call a Coarse-to-Fine map, so you don't have to attend to every single visual token continuously as you reason through the steps.

Tom: And then there's the TRACE component, which is Temporal Routing for Adaptive Contextual Evidence. This part integrates the decoding history to anticipate what evidence will be needed next and selects, retains, or replaces those fine-grained token groups dynamically.

Jane: It uses a state-space model where a learned combination and projection of hidden states across backbone layers forms this temporal state that guides the selection process. It's all about proposing evidence through a learned boundary function to get a set of proposals at each step, which they call Abt.

Wrap-up: Meng: So, when you're reasoning through something complex, TRACE is essentially making smart decisions about which visual pieces are relevant right now versus those you might need later.

Tom: And the training part is quite detailed too. DART is trained to distill compact visual memory by matching full-visual teacher predictions using token cross-entropy and KL distillation from those teacher predictions.

Jane: While DART learns to compress, TRACE parameters are supervised with a loss function targeting a specific set of actions, aiming for an UPDATE action when the required evidence set changes and HOLD otherwise.

Lu: They also use reinforcement learning from offline optimization, or RLOO, to sample actions based on correctness feedback about the mean fine occupancy. This helps refine how TRACE makes those decisions beyond just supervised training.

Tom: Now let's talk about what this actually means for performance and efficiency. The results show that ViMoD achieves state-of-the-art performance in visual reasoning even when you limit the budget to twenty percent of the original visual tokens.

Jane: That’s a big number there. They found that at a twenty percent target budget, the average normalized score exceeds the strongest one-shot baseline by thirty-nine point zero percent. It also leads the general-suite average across every tested budget level.

Meng: That's impressive compression efficiency if you look at how much visual information you can retain while still getting high accuracy on complex visual tasks.

Tom: On top of the performance gains, they looked at inference time, and ViMoD reduces total inference time relative to the full model across evaluated budgets from twenty percent down to ninety percent on MMMUPro.

Jane: But they also pointed out a key limitation for anyone looking at this kind of compression. They said that efficient visual compression depends not only on which evidence you keep, but also on how that evidence is made available across the different reasoning stages.

Wrap-up: Lu: And they acknowledged some constraints too, like the granularity of the Coarse-to-Fine mapping and fixed routing intervals, which limits how precisely you can access the fine tokens.

Tom: They also mentioned that building this Coarse memory adds initialization work, and keeping both the coarse context and the fine token banks increases storage requirements for you.

Jane: And there's a cost model they use, which suggests that while per-token savings are real, for some longer responses those savings don't translate to lower total decoder computation because you end up doing more overall work.

Meng: So if I'm just looking at a quick image classification task this might be overkill. But for those long reasoning tasks where the evidence needs shift constantly, this framework seems designed to handle that dynamic need.

Lu: So what this means for the future is that efficient visual compression depends not just on which evidence you keep but also on how it’s made available across those different reasoning stages.

Jane: It shows value in adapting visual evidence access to changing needs while cutting down the overall computation needed for a complete response.

Tom: So we're moving beyond just making things smaller and now focusing on making the retrieval smarter based on the task at hand.

Meng: For practical AI deployment, it seems like a trade-off between the initial setup cost of building the memory and the actual inference savings depends heavily on how long and complex your visual task is.

Lu: It opens up possibilities for models to handle more intricate, multi-stage visual problems efficiently by managing their attention budget intelligently rather than just blindly compressing everything upfront.

Tom: Overall "Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning" gives us a really solid framework for smarter visual processing in AI.

Jane: It’s about making the visual memory accessible only when it's actually needed for that specific part of the reasoning process.

Tom: We’ll keep an eye on how they refine those routing intervals and what those next steps look like to make this system even more robust.

Yicheng Xue, Han Wu, Jufeng Yang, Minjing Dong, Xinghao Chen, Hanting Chen

City University of Hong Kong · Zhejiang University · Peking University

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 27 pages, 8 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The proposed ViMoD framework addresses computational expense in multimodal Large Language Models by maintaining a compact visual context while preserving access to original fine-grained

Key concepts

Deformable Aggregation of Region-wise Tokens (DART)
DART learns how to group visual tokens in a way that changes based on the image content. It creates compact 'Coarse' representations by learning adaptive group sizes and membership scores. This allows the model to distill the visual information into a smaller, manageable summary without losing essential details.
Temporal Routing for Adaptive Contextual Evidence (TRACE)
TRACE uses the decoding history to predict future evidence requirements. It employs a state-space model to track hidden states across layers and proposes which fine-grained tokens should be retained, updated, or replaced. This ensures that only the necessary visual evidence is actively accessed at any given reasoning stage.
Coarse-to-Fine (C2F) Map
This map connects the compact Coarse context generated by DART back to the original Fine tokens. It acts as a bridge, allowing the model to retrieve specific, detailed visual evidence from its memory when needed during a particular step of reasoning. This enables stage-specific access without needing to process all visual tokens continuously.

Terminology

Summary

The gist The proposed ViMoD framework addresses computational expense in multimodal Large Language Models by maintaining a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve.

How it works

ViMoD introduces two key components: Deformable Aggregation of Region-wise Tokens (DART) and Temporal Routing for Adaptive Contextual Evidence (TRACE), which work together to manage visual token sequences during multi-step reasoning DART learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. This mechanism involves learning group sizes under a fixed total and adapting membership based on image content through a learned content affinity score. The resulting Coarse context is then linked to the original Fine tokens via a Coarse-to-Fine (C2F) map, allowing for stage-specific evidence access without continuously attending to all visual tokens.

How it works

TRACE integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. It uses a state-space model where a learned combination and projection of hidden states across backbone layers forms ut ∈ R ds. This temporal state is then used to propose evidence via a learned boundary function, yielding a proposal set Abt. The gate compares the retained and proposed evidence to decide between HOLD or UPDATE actions, allowing for Coarse-only reasoning between Fine accesses.

How it works

The system optimizes these components through supervised training and reinforcement learning DART is trained to distill compact visual memory by matching full-visual teacher predictions using token cross-entropy and KL distillation from full-visual teacher predictions. TRACE parameters are supervised by a loss that targets the target set A⋆t, aiming for an UPDATE action when the annotated set changes and HOLD otherwise. Furthermore, policy optimization uses RLOO to sample actions based on correctness feedback, where Cost advantages are not divided by their standard deviation.

Performance and Efficiency

ViMoD achieves state-of-the-art performance in visual reasoning under limited visual token budgets. At a 20% target budget, its average normalized score exceeds the strongest one-shot baseline by 39.0%. ViMoD also leads the general-suite average at every budget. The framework reduces total inference time relative to Full across evaluated budgets from 20% to 90% on MMMUPro while keeping output length close to Full, allowing decoding gains to translate into lower total time.

Key Findings and Limitations

The paper demonstrates that efficient visual compression depends not only on which evidence is retained, but also on how that evidence is made available across reasoning stages. The framework shows value in adapting visual evidence access to evolving reasoning needs while reducing computation over complete responses However, limitations exist, as C2F granularity, fixed routing intervals, and teacher supervision constrain evidence access. Additionally, Constructing Coarse memory adds initialization work and retaining both Coarse and Fine KV banks increases storage requirements. The cost model suggests that while per-token savings exist, longer responses outweigh per-token savings, increasing total decoder computation for some compression methods.

The paper is a scientific paper on arXiv:2610.12060v1 [cs.CV] 8 Oct 2026.

How it works

Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. This mechanism involves learning group sizes under a fixed total and adapting membership based on image content through a learned content affinity score.

How it works

Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. It uses a state-space model where a learned combination and projection of hidden states across backbone layers forms ut ∈ R ds.

How it works

The system optimizes these components through supervised training and reinforcement learning DART is trained to distill compact visual memory by matching full-visual teacher predictions using token cross-entropy and KL distillation from full-visual teacher predictions.

How it works

TRACE parameters are supervised by a loss that targets the target set A⋆t, aiming for an UPDATE action when the annotated set changes and HOLD otherwise.

Key Findings and Limitations

The paper demonstrates that efficient visual compression depends not only on which evidence is retained, but also on how that evidence is made available across reasoning stages. The framework shows value in adapting visual evidence access to evolving reasoning needs while reducing computation over complete responses. However, limitations exist, as C2F granularity, fixed routing intervals, and teacher supervision constrain evidence access. Additionally, Constructing Coarse memory adds initialization work and retaining both Coarse and Fine KV banks increases storage requirements. The cost model suggests that while per-token savings exist, longer responses outweigh per-token savings, increasing total decoder computation for some compression methods.

The paper is a scientific paper on arXiv:2610.12060v1 [cs.CV] 8 Oct 2026. DART learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens.

How it works

TRACE integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. It uses a state-space model where a learned combination and projection of hidden states across backbone layers forms ut ∈ R ds.

How it works

TRACE parameters are supervised by a loss that targets the target set A⋆t, aiming for an UPDATE action when the annotated set changes and HOLD otherwise <ref:19 A.3.

Improvements for AI systems

  1. Improve multi-step reasoning efficiency by implementing ViMoD, which maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. This allows the model to perform complex tasks under a 20% target visual token budget with significant performance gains, such as exceeding the strongest one-shot baseline by 39.0% on Qwen3-VL-4B.

  2. Develop adaptive evidence access via TRACE, which uses a state-space model to integrate decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. This capability enables the system to dynamically select visual evidence based on decoding history rather than relying on a fixed compressed context, addressing the issue that visual evidence needs can shift across successive stages of reasoning.

  3. Enhance compression granularity using DART, which learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. This allows the system to adapt compression based on image content, as seen in Figure 5(a), where Blank regions generally undergo stronger compression, while text-bearing regions retain finer granularity.

  4. Integrate cost-aware reasoning by leveraging the total cost model described in Section C.3.1 and C.3.2 to assess inference savings over complete responses, as opposed to per-token savings alone, noting that longer responses outweigh per-token savings, increasing total decoder computation.

  5. Implement policy optimization through RLOO (Reinforcement Learning from Offline Optimization) guided by Success-conditioned feedback on mean Fine occupancy to improve the selection policy, as this method shows gains when training TRACE beyond supervised initialization.

Abstract

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

Sources

Related papers