Out of Sight, Still in Mind: Token Compression for Omni-LLMs

summary

The gist

The goal of this paper is to reduce the input token cost of Omni-modal large language models (Omni-LLMs) at inference time because encoding all three modalities inflates the input sequence to tens of…

In short

ReMo is a training-free method to reduce input token cost for Omni-LLMs by compressing visual tokens. It achieves this by aligning audio and video embeddings in a common space, removing redundant visual information, and replacing complex visual tokens with compact text proxies. This exploits cross-modal redundancy to significantly cut the input sequence length while maintaining or improving model performance.

Key concepts

ReMo
A training-free framework designed to compress visual tokens in Omni-LLMs. It works by projecting audio and video into a shared space, removing redundant visual information explained by other modalities, and substituting object tokens with short text descriptions.
Cross-modal Projection
This involves mapping audio and visual tokens into a common embedding space. This is done using specific mathematical techniques like centering embeddings and whitening them to find canonical correlations between the different modalities, effectively revealing shared information.
Text Proxies
Instead of keeping every detailed visual token, ReMo replaces them with compact text descriptions of objects and their locations. This technique offloads high-level visual semantics onto the audio and text streams, allowing the model to retain necessary context using fewer tokens.

Terminology used across episodes

This episode discusses

The paper

Out of Sight, Still in Mind: Token Compression for Omni-LLMs · Read on arXiv

KAIST · University of Oxford

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Out of Sight, Still in Mind: Token Compression for Omni-LLMs".

Paper summary: Tom: So, we're looking at this paper called "Out of Sight, Still in Mind: Token Compression for Omni-LLMs". The main problem they're tackling is that these big models that handle audio, video, and text together have a huge input cost. It turns out most of those tokens come from the visual part, and a lot of them are just repeating information.

Jane: What's the core idea here? Why does this compression matter for someone listening to the show?

Tom: Well, their proposal is called ReMo, which is a training-free framework. It aims to cut down on input tokens by taking visual tokens and spreading that information across the audio and text streams. They claim you can keep a visual token only if its meaning isn't already captured somewhere else in the model.

Lu: What's interesting about their approach is how they do this redistribution, specifically by aligning audio and video in a common embedding space first. This helps them figure out which visual tokens are redundant because they can directly measure the agreement between those modalities, even though their original spaces didn't match very well.

Meng: So, if we think about what that means for us shipping these models, are we talking about a massive reduction in what we have to feed the model at inference time?

Lalam: They show that ReMo removes fifty-four percent of the input tokens on Qwen2 point 5-Omni at two model scales without losing any accuracy <ref:2607.21179#pg1,tokens on Qwen2.5-Omni at two model scales>. That's a big number for reducing the total input length, which is usually what slows things down during real use.

Jane: Fifty-four percent sounds substantial, but how do they actually achieve this reduction? Are they just deleting tokens randomly, or is there a smarter way to decide what to keep?

Tom: They use two specific ways. First, by aligning the audio and video embeddings so they overlap in a common space. If an audio token explains the same thing as a visual token, that visual one gets dropped because it's redundant; and second, they replace object-level visual tokens with short text descriptions.

Lu: That second part is clever because it uses a lightweight detector to find objects and insert their names as text tokens. This means a few words can stand in for many visual tokens, which is what they call replacing object-level visual tokens with compact text proxies.

Paper summary: Meng: I worry about that second part at scale. If we're inserting these text descriptions, how do we make sure those descriptions don't introduce new errors or just add noise to the input stream?

Tom: The authors place each description based on where the object is spatially located in the scene, so the text stays anchored to where its content appears. This keeps things grounded and helps ensure the text actually matches what's happening visually.

Lalam: They also have a mechanism for deciding which tokens to keep by fusing intra-visual saliency with audio saliency using a per-chunk weight. That lets them fine-tune the decision based on context, rather than just one fixed rule for every token.

Jane: So, it sounds like they are essentially offloading the high-level visual meaning onto the audio and text streams before the main language model even sees it. What does that imply about how these models reason?

Tom: It implies that the high-level semantics, like identifying an object or understanding its location, don't need to be encoded in a massive stream of visual data if they can be represented succinctly in language or sound. The visual tokens are only kept for the very residual information that no other modality can supply.

Lu: It really changes how we think about the modality gap. Instead of trying to force perfect alignment before feeding it into a monolithic decoder, they are using projection to bridge that gap, and then using text proxies to cover what's missing in the audio or video.

Meng: From an engineering standpoint, if this works as well as they claim on those two model scales, it suggests that the redundancy isn't just a problem for smaller models; it’s a property of how these large Omni-LLMs operate at scale.

Tom: Exactly. And they showed that when you project audio and video into that common space, the audio-to-video retrieval accuracy goes from two point seven to thirty point one three, which is a big jump for understanding how well the model connects what it hears to what it sees.

Jane: So, looking at the whole picture of this paper, "Out of Sight, Still in Mind: Token Compression for Omni-LLMs", what's the simple message for people who just want to know why this research matters?

Tom: The simple message is that we can significantly cut down on the input cost for these multimodal models without sacrificing performance. It shows that exploiting cross-modal redundancy through projection and text substitution is a practical way to make Omni-LLMs more efficient at inference time.

Paper summary: Lu: And they pointed out some limitations, which is important for us to keep in mind. They noted that their evaluation focused on relatively short clips, and because they use an off-the-shelf detector for the text proxies, those proxies can't explicitly identify things that aren't in its vocabulary.

Meng: That makes sense; if the detector misses something, we lose that information when we replace it with a word. And they also mentioned that finding the right compression ratio is hard because it changes depending on the task or what kind of input you give them.

Jane: So, even with these caveats, what's the big picture implication for how we build and use these large models in production environments?

Tom: It suggests that future work should focus on adaptive, input-conditioned compression. If we can figure out how to compress differently based on the specific task or the content of the input, we can move beyond just a fixed compression ratio.

Lu: Yeah, exploring that adaptive compression is definitely where the next research direction lies if we want to make this technique really robust for real-world deployment.

Meng: For production, robustness across different inputs is key. Right now, it's showing stability across hyperparameter choices on those tested models, but we need proof it holds up when we throw completely new data at it.

Lalam: I'm confident in the results they presented; the method works in practice for the scenarios they tested. It offloads high-level visual semantics onto audio and text streams, and reserves visual tokens for residual information.

Jane: So, to wrap up on "Out of Sight, Still in Mind: Token Compression for Omni-LLMs", we've seen how ReMo uses cross-modal projection and text proxies to trim down the massive token count coming from visual streams by over fifty percent with no accuracy loss on those models.

Tom: It's a solid demonstration of exploiting redundancy across modalities to reduce input cost at inference time.

Lu: It proves that the gap between audio and video embeddings isn't just noise; it can be leveraged for compression if you project them correctly.

Meng: The challenge now is moving from these controlled experiments on short clips to reliably handling long, streaming data where those limitations they mentioned become a real issue.

Jane: That adaptive compression idea they touched on seems like the most important thing to watch for in how we advance token-efficient multimodal inference.

Conclusion: Tom: So, we're wrapping up on "Out of Sight, Still in Mind: Token Compression for Omni-LLMs". Basically, this paper is about how we can cut down on the massive amount of data these big models need to process at inference time.

Jane: Right. It shows a training-free way to compress the visual tokens by spreading that information across the audio and text streams.

Tom: Yeah, they use cross-modal projection to figure out which parts of the visual data are just redundant and can be dropped or replaced with something shorter, like a text description.

Lu: The core idea is that you align audio and video in a common space first to remove those duplicate visual tokens before you start pruning anything.

Meng: From what I see on the engineering side, this means we could run these models with much lower latency, maybe zero point six five times the original speed, which is pretty good for production deployment.

Lalam: The authors show that when they project audio and video embeddings into that common space, their audio-to-video retrieval accuracy jumps from two point seven to thirty point one three.

Jane: thirty point one three is a significant jump in how well the model connects what it hears to what it sees, even after the compression.

Tom: It suggests that this method works at scale because they found that audio and video embeddings actually overlap quite a bit when projected correctly.

Meng: But I gotta ask, does this work for long videos or streaming data? The paper only tested relatively short clips, and that's where I worry about real-world deployment.

Lalam: They did flag that limitation. They also mentioned that because they use an existing detector to make those text proxies, the system can't explicitly identify things it hasn't seen before.

Jane: So, the big picture here is a training-free technique that cuts input cost substantially, but we still need to figure out how to make it work reliably for longer inputs and different types of content.

Tom: Exactly. That means the next big step in this research has to be exploring adaptive compression based on what the specific task or input data actually looks like.

More episodes

← Home