Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA

arXiv:2605.28641 · cs.IR · Submitted 2026-05-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Subtraction Gets You More".

Jane: Here's the gist of the paper. In multimodal multi-hop question answering, we point out that conventional iterative retrieval frameworks often suffer from Semantic Anchoring, where previously fetched evidence traps the retriever and yields entity-centric redundancy Under these settings, we propose a Task-Adaptive Hybrid Framework, achieving a 35.7% macro-averaged performance gain on the MultimodalQA evidence set completion task.

Tom: Problem Identification and Semantic Anchoring Conventional iterative retrieval frameworks often suffer from Semantic Anchoring, where previously fetched evidence traps the retriever and yields entity-centric redundancy.

Jane: This leads to entity-centric redundancy, blindly fetching redundant assets while pushing the true target evidence–the complementary fact regarding Bolton’s emblem–out of the top-K rank.

Tom: The authors systematically characterize Semantic Anchoring in MMQA, demonstrating how conventional iterative loops suffer from entity-centric redundancy (Figure 1).

Title and authors: Tom: So we're looking at this paper today, "Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA." It’s about how to make those retrieval systems better when you have multiple questions and different types of information like images and text.

Jane: Right. The main idea is that most of the current iterative methods fail because they get stuck on what they already found, which we call Semantic Anchoring, where old evidence traps the search.

Lu: Exactly. When you fetch evidence iteratively, it often just repeats stuff about what you already have instead of finding that one crucial piece of missing information you really need for the final answer.

Meng: From an engineering standpoint, that means we’re wasting compute just re-scanning things we already know, which is bad when the search space gets huge.

Lalam: This paper proposes something called GRAIL, or Gap-aware Retrieval via Adaptive Implicit Localization, which rewrites the query directly in the embedding space instead of relying on text.

Tom: Rewriting in the embedding space sounds interesting because it avoids that linguistic bottleneck we often see when we try to use large language models to guide those queries.

Jane: That's right. GRAIL works by taking your original question and implicitly subtracting all the information you’ve already seen from that query, which pushes the search toward what’s missing instead of what’s present.

Lu: The framework then uses this implicit steering to maintain a trajectory that keeps escaping the semantic gravity of the previous results, which is a key part of their sequential pool construction method.

Meng: So they're using subtraction to actively find novelty in each step, rather than just adding new, potentially redundant evidence on top of old evidence.

Tom: They then take those strengths—the additive updates and the subtractive steering—and put them into a Task-Adaptive Hybrid Framework to get that thirty-five point seven percent macro-averaged performance gain on the MultimodalQA evidence set completion task <ref:2605.28641#pg1,35.7% macro-averaged performance gain on the MultimodalQA evidence set completion>.

Jane: That performance lift over the standard additive methods is significant, and it shows how combining different retrieval strategies can actually help you perform better across various types of questions.

Lu: The way they structure this hybrid framework is smart; it uses a front-end selector to decide whether to use additive reinforcement or that subtractive steering for a specific query before the search even starts.

Tom: And when you look at the iterative retrieval part, they show that GRAIL is robust to noise and doesn't propagate errors like some other additive methods do; their Jump metric shows a much bigger effective search horizon.

Title and authors: Jane: That robustness is important because in real-world multimodal tasks, you’ve got so much noise, and you don’t want a small error in the first step to ruin the whole thing.

Lu: They show that GRAIL maintains robust positive margins across different partition schedules, which means it performs well even when you change how you chunk your search budget.

Meng: If this works reliably across those different budgets, it suggests it’s a solid system for deployment because we don't have to worry about the exact size of our retrieval pool at runtime.

Tom: Now, let's talk about how they measure performance against other methods; they compare their implicit update models against explicit baselines that use LLM-generated text queries.

Jane: That comparison highlights a structural advantage for Gap-aware, showing that relying on explicit LLM reasoning to generate text is less effective than this direct embedding manipulation.

Lu: They find that the dynamic gating coefficient in their model maintains an average of zero point two eight eight, and it shows higher activation in complex tasks like Compare and Intersect relative to simpler ones like Uni-modal queries.

Meng: So it's not just better on simple tasks; it’s actually more active when the reasoning required is genuinely multi-modal and complex.

Tom: And for efficiency, this is a big one; GRAIL completely bypasses the need to run an LLM inference loop during retrieval by doing that context-subtractive query steering via a lightweight gating mechanism.

Jane: That means near-instant latency, just three to six milliseconds per question, which gives you a throughput advantage of about three hundred fifty times over traditional methods.

Lu: By executing the subtraction directly in the embedding space without turning it into text first, they avoid that linguistic loss and enable a self-correcting retrieval trajectory.

Meng: That zero API cost for inference during retrieval is huge for large-scale systems; we can run this kind of deep multimodal search without incurring massive token costs every time.

Tom: To check its generalizability, they tested GRAIL on the WebQA dataset, and when retrieving within the same modality, context and target evidence are often in very close subspaces.

Jane: But they showed that when you give it enough training data to learn this non-cross-modal environment, GRAIL shows a massive reversal in performance.

Lu: The dynamic gating coefficient adapts the steering intensity based on the specific single-modality query category, which is what allows it to generalize across different types of retrieval patterns.

Meng: It sounds like they managed to make it adaptable enough that you don't need a totally new model setup every time you switch from one type of query to another.

Title and authors: Tom: So, when we look at the overall conclusion for this paper, they summarize that conventional additive methods suffer from entity-centric redundancy and that GRAIL provides an implicit context-subtractive query rewriting mechanism operating in the shared latent space.

Jane: And they wrap it up by proposing a Task-Adaptive Hybrid Framework to leverage both additive and subtractive strengths for better performance on tasks like evidence set completion.

Lu: They also mention their limitations, specifically that their iterative retrieval experiments used fixed step scheduling rather than adjusting the steps dynamically during runtime.

Meng: And they note that since this work focuses only on the retrieval and evidence set completion phases, it doesn't necessarily cover the whole multimodal pipeline for open-ended web environments.

Lalam: So, in short, Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA uses implicit context subtraction to overcome semantic anchoring and achieves better results with a task-adaptive hybrid approach.

Tom: It’s a solid result because the performance gain of thirty-five point seven percent on that evidence set completion task is backed by the analysis of how GRAIL manages those iterative pools without compounding errors <ref:2605.28641#pg1>.

Jane: It means for someone who only listens to this, it shows that instead of blindly adding everything you find, you should be actively subtracting what you already have to focus on the missing pieces.

Lu: The implication for research is that doing explicit query rewriting via LLMs can be a bottleneck because they execute implicit query steering directly in the embedding space instead.

Meng: For deployment, it means we can build systems that are fast and cost-effective because they don't have to call an expensive language model during the retrieval process itself.

Tom: So, to wrap up on this paper today, "Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA" addresses the trap of entity redundancy by using GRAIL’s implicit context subtraction within a task-adaptive framework.

Jane: And it suggests that by steering the query toward what is missing, we can achieve substantial performance gains on complex multimodal reasoning tasks.

Lu: The next step for this line of research would be to explore how they could integrate dynamic step scheduling into the iterative process for even finer control over the search trajectory.

Meng: And from an engineering perspective, it would be great to see if we can push this retrieval mechanism into a system that handles truly open-ended web queries without those bounded knowledge corpora.

Lalam: We'll keep an eye on how this gap-aware steering performs when we move toward more complex, unbounded search scenarios in future work.

The paper's summary: Tom: So, we’re looking at this paper today about Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA, and the main thing they did is address that problem where old search results trap you, which they call Semantic Anchoring.

Jane: That means when you’re doing multi-hop questions with images and text, your computer keeps finding stuff it already knows instead of finding that one crucial piece of missing evidence for the final answer.

Lu: Exactly. Conventional systems just keep adding to their pile, and that usually leads to a lot of repetition, which is inefficient when the search space is big.

Meng: So it sounds like they’re trying to stop the system from blindly repeating itself and actually push it toward what it hasn't found yet.

Tom: Right. They propose this thing called GRAIL, Gap-aware Retrieval via Adaptive Implicit Localization, which rewrites the query right in the embedding space before you even start searching.

Jane: Rewriting in the embedding space is smart because they skip that messy step of turning everything into text and using a language model to guide it; it’s direct manipulation of meaning.

Lu: GRAIL works by implicitly subtracting all the information already found from your original question, which pushes the search toward what’s missing instead of what you already have.

Meng: That sounds like a way to maintain structural robustness even if the first step is totally blind, because it’s constantly correcting itself based on what it needs next.

Tom: And they built this whole Task-Adaptive Hybrid Framework around it, which means the system knows whether to use additive updates or that subtractive steering for a specific question.

Jane: That hybrid part is key; instead of one single method, you get a front-end selector that decides the best tool—additive reinforcement or subtractive steering—for each specific task at hand.

Lu: The authors show this whole setup achieves a thirty-five point seven percent macro average performance gain on the MultimodalQA evidence set completion task, which is pretty solid compared to just using standard additive methods.

Meng: Thirty-five point seven percent is a big number when you’re trying to improve retrieval quality without slowing everything down too much.

Tom: And they also show that because GRAIL bypasses the LLM inference loop during retrieval, it runs with near-instant latency, about three to six milliseconds per question.

Jane: That speed is huge; it means you can do this kind of deep multimodal search in real time, which is something you just couldn't do before.

Lu: They tested this on the WebQA dataset too and showed that when they give the system enough training data, it generalizes well even when switching between different single-modality queries.

Meng: It’s good to see that adaptability; it means you don't need a completely new setup every time you switch from one type of question to another.

Tom: So, the main idea is using implicit context subtraction within a task-adaptive framework to overcome that entity redundancy problem we talked about earlier.

Jane: It’s about actively steering the search away from what’s already there and toward the actual missing facts needed for that complex multi-hop answer.

Lu: Their limitation they mention is that their iterative retrieval experiments used fixed step scheduling instead of adjusting the steps during runtime, so it's not fully dynamic yet.

Meng: That makes sense; we need to see how they handle those truly open-ended web environments where the knowledge isn't neatly bounded by a fixed set of documents.

Tom: So, if you’re listening and you’re building a multimodal system right now, this paper suggests that actively subtracting what you have is a much better strategy than just blindly adding more context.

Jane: It really shows how changing the query itself, not just the search step size, can give you that significant performance boost.

Lu: Next up, we’re going to look at some of those specific numbers they used to prove this works and see exactly what the math is showing us.

The paper's improvements: Tom: So, we’re moving on to what the authors suggest as future improvements for this Gap-aware Retrieval system, and basically they’re looking at making those iterative steps more dynamic rather than fixed.

Jane: It sounds like they are trying to move away from those fixed step sizes in their search process and instead make the system adjust how big each retrieval step is while it’s running.

Lu: That would mean the trajectory of the query, which we call q of t, has to continuously escape the semantic gravity of what was found in all previous steps, ensuring novel information gaps are always being looked for.

Meng: If you can dynamically change the search budget on the fly, that could make it much more efficient at scale because you only spend computation where it actually needs to go.

Tom: And they are also looking at how they handle those complex environments, like when retrieving within a single modality on the WebQA dataset, and they’re trying to learn how to adapt their steering intensity based on the query type.

Jane: That adaptation is important because it means the system can handle different kinds of questions with different search intensities, which we saw them doing with that dynamic gating coefficient.

Lu: They are exploring how this works when you have lots of training data so the model learns to navigate those very close, near-collinear subspaces where context and target evidence often live together.

Tom: This whole direction points toward a system that can be much more flexible, moving beyond the fixed step scheduling they used in their initial tests.

Jane: So what this means for you listening is that future versions of this AI will get better at figuring out when to be aggressive with the search and when to be cautious, based on what kind of question it’s trying to answer.

Lu: It opens up possibilities for making these retrieval systems much more versatile across different types of multimodal tasks and data distributions.

Meng: From a production standpoint, if they can get that dynamic scheduling working reliably, it means we might be able to deploy these kinds of retrieval systems in more unpredictable real-world scenarios without needing to pre-define every single search parameter.

Tom: It’s about moving from a set of good fixed numbers to a system that learns the optimal search path for every single query on the fly.

Jane: And that flexibility is what makes it so interesting, because it shows the AI isn't just following a script; it's actually learning how to navigate the missing information itself.

Lu: So, as they work on these improvements, they’re really focusing on making sure this gap-aware mechanism stays robust even when you introduce more complexity into the search process itself.

Conclusion: Tom: So we’re wrapping up on Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA, and basically they showed that by using this implicit context subtraction method, you can overcome the problem of entity redundancy in iterative retrieval.

Jane: It means instead of just adding more things to your search pile, you’re actively subtracting what you already found to focus on those crucial missing pieces for the final answer.

Lu: The main implication is that this approach offers a structural advantage over conventional additive methods by maintaining robustness when the initial search step is completely blind under noisy conditions.

Meng: For us engineers, it means we can build retrieval systems that don't get bogged down in repeating old data and actually find novel evidence faster.

Tom: And the speed is still there; they maintain near-instant latency of three to six milliseconds per question because they skip the heavy LLM inference loop during retrieval.

Jane: That is something we can really talk about—the throughput advantage of three hundred fifty times over traditional methods because it’s so lightweight.

Lu: They also showed that this works well when you test it on diverse datasets, meaning the model learns to adapt its search intensity based on the specific query category you give it.

Meng: It's good they tested it across different modalities and query types; we need to know that this isn't just a trick for one specific kind of data.

Tom: So, in summary, Subtraction Gets You More: Gap-Aware Retrieval for Multimodal Multi-Hop QA uses a context-subtractive rewrite within a task-adaptive framework to maximize performance gains.

Jane: It suggests that steering the query toward what is missing, rather than just adding everything you see, is the right way to handle complex reasoning tasks.

Lu: Their limitation they flag is that their iterative retrieval experiments relied on fixed step scheduling instead of dynamically adjusting those search steps during runtime.

Meng: That’s a fair caveat; we still need to see how that dynamic scheduling holds up when we push it into truly open-ended web environments where the knowledge isn't neatly contained.

Tom: Yeah, so while the results on MultimodalQA are strong, the next big test will be seeing how well this framework handles those more unpredictable search spaces.

Jane: I think that’s exactly what we need to watch for—does it stay smart when things get messy and unstructured?

Lu: This work sets a solid baseline for implicit steering in multimodal retrieval, and it shows the potential of operating directly in the shared latent space.

Seoul National University

cs.IR

Submitted: 2026-05-27

Updated: 2026-10-19

Comments: Accepted to EMNLP 2026 main conference

Journal ref: 2026.emnlp-main.2132

DOI: 10.18653/2026.emnlp-main.2132

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 100/100

The gist: In multimodal multi-hop question answering, we point out that conventional iterative retrieval frameworks often suffer from Semantic Anchoring…

Key concepts

Semantic Anchoring
This is when previous search results trap the retrieval system, causing it to repeatedly fetch the same types of information. Instead of finding new answers, the system gets stuck on entity-centric redundancy, missing crucial complementary facts.
GRAIL (Gap-aware Retrieval via Adaptive Implicit Localization)
GRAIL rewrites the search query directly in the embedding space by subtracting known context from it. This implicitly steers the query toward unanswered parts of a question, ensuring that each new step actively seeks novel information gaps instead of repeating what has already been found.
Task-Adaptive Hybrid Framework
This framework intelligently chooses between two retrieval strategies based on the question. It uses a selector to decide whether to use additive reinforcement (to consolidate context) or subtractive steering (to isolate missing facts), optimizing performance for different query types.
Implicit Query Steering
Instead of using complex, slow text-based rewriting, this mechanism modifies the query's vector representation in the embedding space. This allows for efficient, self-correcting retrieval trajectories that avoid linguistic bottlenecks and maintain high performance even under noisy conditions.

Terminology

Summary

In multimodal multi-hop question answering, we point out that conventional iterative retrieval frameworks often suffer from Semantic Anchoring, where previously fetched evidence traps the retriever and yields entity-centric redundancy Under these settings, we propose a Task-Adaptive Hybrid Framework, achieving a 35. p. 07% macro-averaged performance gain on the MultimodalQA evidence set completion task. p. 0 p. 0

Problem Identification and Semantic Anchoring

Conventional iterative retrieval frameworks often suffer from Semantic Anchoring, where previously fetched evidence traps the retriever and yields entity-centric redundancy. This leads to entity-centric redundancy, blindly fetching redundant assets while pushing the true target evidence–the complementary fact regarding Bolton’s emblem–out of the top-K rank. p. 0 The authors systematically characterize Semantic Anchoring in MMQA, demonstrating how conventional iterative loops suffer from entity-centric redundancy (Figure 1). p. 1

Implicit Latent Steering Mechanism

We propose GRAIL (Gap-aware Retrieval via Adaptive Implicit Localization), a paradigm that performs implicit query rewriting directly at the embedding level. Rather than blindly contextualizing existing context, the rewritten query consistently shifts toward the unanswered facets of the question, increasing its similarity to missing evidence by subtracting the already covered information from the query. p. 1 Operating entirely in the embedding space, this mechanism avoids any linguistic loss and enables an efficient, self-correcting retrieval trajectory for complex MMQA. p. 1

Task-Adaptive Framework and Hybridization

We introduce a Task-Adaptive Hybrid Framework, achieving a 35.7% macro-averaged performance gain on the MultimodalQA evidence set completion task. This framework is governed by a front-end query selector that pre-determines the most effective specialist framework before executing the document search. p. 4 For example, certain queries benefit from additive reinforcement (qadd) to consolidate ongoing context, while others strictly demand subtractive query steering (qgap) to isolate missing information. p. 4

Robust Iterative Retrieval and Efficiency

When applied iteratively on multimodal evidence, our mechanism constructs high-quality Top-K candidate pools; unlike conventional additive methods prone to compounding errors, GRAIL remains robust to noise and prevents error propagation. The behavior of the Jump metric in Table 3 demonstrates a substantial expansion of the effective search horizon. p. 7 GRAIL maintains robust, positive margins across granular partition schedules, whereas the Additive framework suffers severe performance degradation. p. 6

Inference Efficiency and Cost Analysis

GRAIL completely bypasses the LLM inference loop during retrieval. By executing context-subtractive query steering via a lightweight gating mechanism, our framework achieves a near-instantaneous latency of just 3–6 ms per question. p. 4 This translates to a ∼350× throughput advantage, enabling realtime, large-scale multimodal retrieval. p. 9

Effectiveness of Implicit Query Steering

Table 2 compares our implicit update models with explicit baselines relying on LLM-generated text queries. While methods like IRCoT attempt to trace human-like reasoning, they suffer from a linguistic bottleneck. p. 7 In contrast, Gap-aware demonstrates a decisive structural advantage, maintaining robust performance where conventional attention-based aggregation falters. p. 1 The dynamic gating coefficient maintains an overall average of 0. p. 6288, displaying higher activation in complex multi-modal reasoning tasks such as Compare (0. p. 6286) and Intersect (0. p. 6264) relative to simpler Uni-modal queries (0. p. 6233).

Generalization on WebQA

To verify the dataset-agnostic scalability and explore the boundary conditions of our framework, we evaluate GRAIL on the WebQA dataset. When retrieving within the exact same modality, captured context and target evidence often reside in densely clustered, near-collinear subspaces. p. 0 GRAIL shows a massive reversal when given enough training data to learn this non-cross-modal environment. p. 8 The dynamic gating adapts vector steering intensity across various single-modality query categories. p. 4

Conclusion

In this paper, we addressed Semantic Anchoring in MMQA retrieval, where conventional additive methods suffer from entity-centric redundancy. To overcome this, we introduced GRAIL, an implicit context-subtractive query rewriting mechanism operating directly in the shared latent space without error-prone textification. p. 8 Moreover, our Hybrid Framework dynamically routes queries to task-specific specialists to maximize baseline synergies. p. 8

Limitations

Fixed Step Scheduling In our iterative retrieval experiments (Section 5.4), we explored a variety of fixed partitioning strategies (e. p. 8g., 3+2+3+2, 5+5) rather than dynamically adjusting the step sizes during runtime. p. 8 Lack of Generative Answer Synthesis Our framework focuses strictly on the retrieval and evidence set completion phases of the multimodal multi-hop pipeline. p. 8 Scalability to Open-Ended Web Environments Although our empirical evaluations comprehensively cover both standard benchmarks and deep multi-hop scenarios (extending to 3+ hops), our projection-based subtraction has primarily been verified within bounded knowledge corpora. p. 8

Acknowledgments

This work was supported in part by the National Research Foundation of Korea (NRF) grant (RS2023-00280883, RS-2023-00222663); by the National Research Foundation, Korea, under project BK21 FOUR(Dept. of Data Science, SNU, No. p. 8 5199990914569); by the Korea Institute of Science and Technology Information (KISTI) in 2026 (No. p. 8 (KISTI)K26L3M1C1), aimed at developing KONI (KISTI Open Neural Intelligence), a large language model specialized in science and technology; and by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (RS-2025-02263754, Human-Centric Embodied AI Agents with Autonomous DecisionMaking); by grant (25202MFDS003) from Ministry of Food and Drug Safety in 2025; by AI-BIO Research Grant through Seoul National University; Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. p. 9 RS2025-25442149, LG AI STAR Talent Development Program for Leading Large-Scale Generative AI Models in the Physical AI Domain). p. 9

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Subtractive query steering can be used to implicitly subtract covered information from a query, which helps in navigating back to the target assets even when starting from a completely noisy context. This allows the system to maintain structural robustness even when the initial step is completely blind under noise-heavy environments.

  2. A task adaptive hybrid framework can be introduced, where a lightweight selector analyzes an incoming query to determine whether to use additive reinforcement or subtractive query steering, achieving a 'best-of-both-worlds' performance (+35.7% macro avg. gain over Additive).

  3. The system can perform sequential pool construction by iteratively updating the retrieval vector using the formula q(t) = f(q, E(t-1)p), ensuring that the trajectory of q(t) must continuously escape the semantic gravity of E(t-1)p to discover novel information gaps.

  4. The system can achieve extended effective top-K by employing granular efficiency under various step configurations, where iterative GRAIL demonstrates sustained gains over the baseline across all expanded budget scales (K ∈ 10, 20, 50).

  5. The system can be deployed with millisecond-level latency and zero API cost because it completely bypasses the LLM inference loop during retrieval by executing context-subtractive query steering via a lightweight gating mechanism.

Sources

Related papers