Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families

arXiv:2610.15095 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families".

Jane: LLM multi-agent systems split work across agents that communicate through text, which constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of multi-agent systems The gist.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on from the mechanics, let’s look at what they actually found in terms of performance and capabilities with this Swarm Telepathy method. The big takeaway here is that SwaT beats text-to-text communication in both accuracy and speed when we look across seventeen benchmarks for twelve LLMs and six families >

Jane: They also demonstrated that SwaT can transfer vision capabilities to a text-only receiver LLM, which is pretty significant because SwaT was trained mostly on text data >

Lu: And they also showed specialized knowledge transfer across languages, where models like Qwen three 4B can transfer knowledge to Llama three 8B >

Meng: That specific example about closing three quarters of the gap on subjects like CMMLU shows that this isn't just a general communication trick; it’s transferring actual specialized skills >

Tom: And they also showed increased context handling beyond the receiver's native context window by chunking and transferring KV-cache across layers >

Jane: And they also showed retention is high, even in the small-to-large pairing where the large model has to use a smaller model's cache compared to using its own native cache >

Lu: The fact that SwaT shows success across many model-pairs and families suggests that what they are transferring is fundamental knowledge that remains useful regardless of how it’s expressed >

Meng: It seems like the core idea here is making the transferred state more robust than just a simple text summary, which I think is a necessary step for real utility >

Tom: So we're looking at a method that boosts accuracy, speed, and capability by using learned mappings to move KV-caches directly between agents without any prefill cost for the receiver >

Jane: It fundamentally changes how agents can talk, moving from discrete tokens to transferring actual latent knowledge states without re-doing all the setup work >

Lu: And they also showed that combining this transfer with something called KV-cache compaction, SwaT plus AM, makes it significantly faster and more accurate than just sending the raw cache state >

Meng: So if a model has an 8k limit, this method lets it process information that exceeds that limit by moving the cache around its layers. That’s a big feature for long contexts >

Tom: It’s about being able to handle much longer documents than the native context window allows for, which is something we all struggle with when dealing with huge inputs >

Tom: So we're looking at a method that boosts accuracy, speed, and capability by using learned mappings instead of just sending raw text or relying on simple prefilling >

Jane: It fundamentally changes how agents can talk, moving from discrete tokens to transferring actual latent knowledge states without re-doing all the setup work.

The paper's summary: Tom: Now let’s talk about what they suggest as improvements to push this research further with Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families. They suggest that the main implication is that we can achieve more accurate and faster agent communication than text-to-text methods by transferring internal states directly instead of sending long text prompts back and forth >

Jane: They suggest that this method proves that cross-family KV-cache transfer is viable without receiver prefill, which opens up a multitude of optimizations in the field >

Lu: This method proves that cross-family KV-cache transfer is viable without receiver prefill, which opens up a multitude of optimizations in the field >

Meng: For deployment at scale, I just need to figure out how to manage those training costs for all those individual mappers effectively across many agents >

Lalam: From my perspective, this is a solid direction for making AI systems more integrated and capable of sharing knowledge internally >

Tom: They suggest having a shared latent space to train those mappers so you don't need one mapper per pair, which would total to only 2n mappers being needed >

Jane: That’s interesting because it addresses the scaling issue mentioned earlier, suggesting a way to make this much more scalable for massive agent swarms >

Lu: If they can solve the problem of training those mappers in a more shared latent space, then this whole cross-family transfer thing becomes much more scalable >

Meng: I’m still focused on the engineering challenge; if we can reduce the number of mappers needed, that helps immensely with managing compute for real-world deployment >

The paper's improvements: Tom: To wrap up this discussion on Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families, the main implication is that we can achieve more accurate and faster agent communication than text-to-text methods >

Jane: It’s a way to enable agents to have deeper, faster communication pathways by transferring internal states directly instead of relying on text exchange >

Tom: And they showed that combining this transfer with something called KV-cache compaction, SwaT plus AM, makes it significantly faster and more accurate than just sending the raw cache state >

Jane: The results on those benchmarks across seventeen models and twelve families show that the performance is actually quite strong compared to standard text-to-text methods >

Lu: It’s also notable how this transfer works even when you try to move capabilities across different model families, like moving vision knowledge from a Vision Language Model to a text-only one >

Meng: But I gotta point out the limitation they mentioned—each specific sender and receiver pair requires a trained mapper, which gets really hard for huge multi-agent systems with thousands of agents >

Lalam: That’s the bottleneck I see; scaling those individual mappings up to a massive swarm is definitely where we need to focus next >

Tom: So the main point is that SwaT achieves better accuracy and speed by using a learned mapping instead of just sending raw text or relying on simple prefilling >

Meng: I'm just thinking about how much compute that learning for each pair adds up when you try to manage a hundred agents all communicating >

Lalam: I’m confident the evidence shows this is a solid step forward for prefill-free communication, even with those scaling hurdles left to tackle.

Tom: Exactly. So we're looking at Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families as a way to build smarter, more interconnected agent systems where agents talk using their internal states instead of just exchanging text strings.

Jane: It shows that the way agents communicate can be fundamentally different than just exchanging text strings.

Conclusion: Tom: So we’ve just talked through Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families, and the main implication is that we can get faster, more accurate agent communication by moving internal states directly instead of sending long text prompts back and forth.

Jane: It’s a way to enable agents to have deeper, faster communication pathways by transferring those internal states directly instead of relying on those discrete tokens that text communication forces you into.

Meng: For deployment at scale, I just need to figure out how to manage those training costs for all those individual mappers effectively across many agents >

Lalam: From my perspective, this is a solid direction for making AI systems more integrated and capable of sharing knowledge internally >

Tom: They showed that combining SwaT with KV-cache compaction, SwaT plus AM, makes it significantly faster and more accurate than just sending the raw cache state >

Jane: The results on those benchmarks across seventeen models and twelve families show that the performance is actually quite strong compared to standard text-to-text methods >

Lalam: That’s the bottleneck I see; scaling those individual mappings up to a massive swarm is definitely where we need to focus next >

Tom: Exactly. So we're looking at Swarm Telepathy: Prefill-Free KV-Cache Transfer Across Families as a way to build smarter, more interconnected agent systems >

Jane: It shows that the way agents communicate can be fundamentally different than just exchanging text strings.

Abdulrahman Alabdulkareem, Irene Tenison Yang Zhang, Yada Zhu, Boris Katz, Dan Gutfreund

MIT CSAIL · MIT-IBM Computing Research Lab

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Code: https://github.com/Ar-Kareem/SwaT

License: http://creativecommons.org/licenses/by/4.0/

The gist: LLM multi-agent systems split work across agents that communicate through text, which constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of

Key concepts

KV-Cache Transfer: This is a technique where an agent sends its internal memory state, known as the Key-Value (KV) cache, to another agent instead of sending raw text. This allows for direct knowledge sharing between different language models without requiring the receiver to process input text first.
Mapper M_i→j: This is a learned function that translates the sender's KV-cache into a format that approximates the receiver's native prefill cache. The system learns this mapping during training so it can efficiently transfer knowledge across different model families and architectures.
KV-Cache Compaction (SwaT+AM): This involves combining SwaT with compaction techniques to reduce the size of the transferred cache while preserving its performance. This combination significantly boosts speed and accuracy, making the communication more practical for real-world multi-agent applications.

Summary

LLM multi-agent systems split work across agents that communicate through text, which constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of multi-agent systems The gist.

SwaT (Swarm Telepathy)

The paper introduces SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method that allows prefillfree communication in multi-family multi-agent systems. This method enables an agent to communicate by transferring its internal state, the KV-cache, directly to another agent without requiring the receiver to prefill any text In an agent-to-agent communication setting, combining SwaT with recent advances in KV-cache compaction [41] beats text-to-text communication in both accuracy and speed.

Mechanism of SwaT

SwaT seeks to communicate without agent A(j) receiving text by using agent A(i)’s cache C(i). Specifically, for any sender–receiver pair, it learns a mapper M i→j that transforms C(i)x into Cb(j)x which approximates the receiver’s own prefill cache while preserving (and sometimes exceeding) downstream performance. The core idea is to learn a mapping function that translates the sender’s latent state (KV-cache) to the receiver’s latent space SwaT combines this transfer with KV-cache compaction techniques such as AM (SwaT+AM) by translating the sender’s cache and compacting it.

The architecture involves training SwaT, a transformer mapper M i→j, while keeping both LLMs frozen The SwaT blocks update the hidden state h b through causal self-attention, causal cross-attention to m b, and a feed-forward network. The cross-attention uses m b as its key/value memory, and the output projection predicts keys and values for all layers in R b.

Performance and Capabilities

SwaT demonstrates high performance on 17 benchmarks across 12 LLMs and 6 families. When combined with KV-cache compaction, SwaT+AM is on average 40% more accurate than text summarization, at 4× the speed. Furthermore, given a sender Vision-Language Model (VLM), SwaT is able to transfer the vision capabilities to a text-only receiver LLM exceeding text-to-text communication despite SwaT being trained on text-only data.

Retention and Cross-Family Transfer

The experiments evaluate how much of the receiver’s own performance survives when it reads SwaT’s transferred cache instead of the text Across many model–pairs, families, and sizes, an evaluation across 17 benchmarks shows that SwaT is capable of transferring caches with high performance while exceeding the receiver model’s own capabilities in certain setups. In table 2, SwaT (ours) — indicates success across different organizations with different architectures and data still share what they represent, differing mainly in how they express it.

Limitations and Future Work

A limitation of SwaT is that each sender–receiver pair requires a trained mapper, which becomes infeasible for large multi-agent systems Future work might investigate having a shared latent space to train mappers in and out of it for each agent which would total to only 2n mappers being needed. Another limitation is the focus on individual message handoffs instead of repeated transfers or aggregating multiple messages together, both of which introduce challenges as KV-cache degrades and isn’t as composable as text.

The paper concludes that SwaT is a cross-family KV-cache transfer method between LLMs without receiver prefill. It achieves more accurate and faster communication compared to text-to-text communication. The effectiveness of SwaT+AM is shown across various datasets, budgets, and sender–receiver pairs Across many model–pairs and datasets, SwaT+AM exceeds T2T in all metrics (task score, sender GPU time, receiver GPU TTFT). The overall mean score is 42.9 across the tested pairs.

The gist.

SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method that allows prefillfree communication in multi-family multi-agent systems, achieves more accurate and faster communication compared to text-to-text communication.

How it works

SwaT introduces a mapper M i→j that transforms C(i)x into Cb(j)x which approximates the receiver’s own prefill cache while preserving (and sometimes exceeding) downstream performance.
The training objectives involve two goals: first, to reconstruct the receiver’s native cache using RMSnormalized mean squared error LKV Second, to optimize the transferred cache for downstream generation using the next-token prediction loss LNLL.
The architecture involves B blocks where block b cross-attends to S’s KV-cache for the layer group S b and outputs R’s KV-cache for the layer group R b.
Across many model–pairs, families, and sizes, an evaluation across 17 benchmarks shows that SwaT is capable of transferring caches with high performance while exceeding the receiver model’s own capabilities in certain setups.
SwaT beats T2T and even beats the sender VLM answering by itself on several VQA benchmarks
Across the 10 model–pairs evaluated on 17 benchmarks, the lowest mean retention is > 90% even in the ‘small to large’ pairing where the large model has to solve the task using the smaller model’s KV-cache compared to using its own (smaller model) KV-cache
Table 7 shows that SwaT exceeds others by ≥20 points in the cross-generation/family cases
Each sender–receiver pair requires a trained mapper, which quickly becomes infeasible for large multi-agent systems
Future work might investigate having a shared latent space to train mappers in and out of it for each agent which would total to only 2n mappers being needed
Each sender–receiver pair requires a trained mapper, which quickly becomes infeasible for large multi-agent systems

Improvements for AI systems

Improved agent communication via SwaT enables cross-family KV-cache transfer without receiver prefilling any text. This allows for a child agent to hand its parent a compacted cache instead of a summary which saves compute on both sides and carries more information.
Enhanced context switching capability is achieved by allowing the incoming model to inherit the outgoing model’s cache instead of re-prefilling the entire context, which skips the cost of switching models mid-context. This is critical for dynamic routing systems where a router can pass its cache to the chosen model rather than forcing a second read.
Cross-modal capability transfer allows a text-only receiver LLM to gain vision capabilities. Given a sender Vision-Language Model (VLM), SwaT enables this by transferring the vision capabilities to a text-only receiver LLM exceeding text-to-text communication despite SwaT being trained on text-only data.
Specialized knowledge transfer across languages allows models lacking specific training data to leverage the sender's strengths. For instance, SwaT can transfer specialized knowledge from Qwen 3 4B to Llama 3 8B, making the receiver close three quarters of the gap to Qwen on subjects like CMMLU.
Increased context handling is possible beyond the receiver's native context window by chunking and transferring KV-cache across layers. This allows a sender with a long context length (such as Qwen 3 which supports a 32k context length) to enable an 8k receiver to process information that exceed[s] its own 8k limit.

Abstract

LLM multi-agent systems split work across agents that communicate through text. An agent summarizes its findings, another agent receives the text. This constrains communication to discrete tokens and costs redundant compute, limiting bandwidth and capabilities of multi-agent systems. Recent works demonstrate KV-cache transfer between LLMs but they either require the LLMs to be within the same family or cannot function across context, thus requiring the receiver to prefill text. We present SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method that allows prefill-free communication in multi-family multi-agent systems. In an agent-to-agent communication setting, combining SwaT with recent advances in KV-cache compaction beats text-to-text communication in both accuracy and speed, with a mean of 40% higher accuracy and a mean of 4 times faster wall-clock time. Experiments show SwaT achieves high performance on 17 benchmarks across 12 LLMs and 6 families. Additionally, given a sender Vision-Language Model (VLM), SwaT is able to transfer the vision capabilities to a text-only receiver LLM exceeding text-to-text communication despite SwaT being trained on text-only data. SwaT keeps both sender and receiver models frozen without fine-tuning and only requires <50M tokens and <1 GPU-day to train.