On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs
The study analyzed attention sinks in Omnimodal Large Language Models (Omni-LLMs) to understand how they shape reasoning during inference without extra training. Researchers found that high sink attention isn't just redundancy; sink values act as a global signal organizing the representation. They proposed OutRo, a method that aligns non-sink tokens with this sink bias and sharpens this signal, improving performance on video QA benchmarks with minimal overhead.