QueenBee: A Recursively Self-Improving Harness for Communication Topology Design in LLM-Based Multi-Agent Systems

summary

The gist

Large language model (LLM) multi-agent systems increasingly depend not only on how individual agents reason…

In short

QueenBee Planner treats inter-agent communication topology as a learnable design skill rather than memorized answers. It uses an outer LLM planner to generate and refine temporal communication Directed Acyclic Graphs (DAGs) based on learned design rules, leading to more efficient, reusable, and task-conditioned agent architectures.

Key concepts

Temporal Communication DAG
This is the explicit plan generated by the LLM planner that dictates how agents communicate over multiple rounds. It specifies which agent sends a message to whom in which round and when messages are merged, creating a structured flow of information necessary for task completion.
Design Skill Bank
The planner's long-term memory is not just task knowledge but a set of reusable design rules called skills. Each skill describes how to preserve, modify, or avoid certain communication patterns based on observed performance, allowing the system to learn architectural knowledge.
Preserve/Modify/Avoid Actions
These are the three core actions distilled from execution traces that form the basis of a design rule. 'Preserve' keeps successful communication structures; 'Modify' suggests changes to existing ones; and 'Avoid' flags patterns that lead to failure, guiding future topology generation.
Design Prior RS (G, x)
This mathematical prior measures how well a candidate communication graph (G) fits the learned design skills (s). Skills like 'Preserve' boost scores for compatible structures, while 'Avoid' lowers scores for structures that resemble past failures, guiding the planner toward better designs.

Terminology used across episodes

This episode discusses

The paper

QueenBee: A Recursively Self-Improving Harness for Communication Topology Design in LLM-Based Multi-Agent Systems · Read on arXiv

Congjia Tian, Yuhang Yao, Jiaming Cui

Virginia Tech · Carnegie Mellon University

LLM-based multi-agent systems show great potential for complex tasks through role specialization and collaborative reasoning. However, existing automated workflow design methods primarily focus on one-off search and candidate optimization, lacking sustained accumulation and reuse of structural design experience to meet evolving coordination demands. We propose QueenBee, an experience-guided framework for multi-agent workflow evolution. QueenBee represents workflows as executable programs that generate temporal communication DAGs and maintains Structural Feedback and Transition Bank (SFT-Bank), a persistent structural memory of execution diagnoses, design hypotheses and validation outcomes, and historical programs. This accumulated experience guides subsequent workflow generation, validation, and optimization, enabling the co-evolution of collaboration programs and design experience. Experiments on Silo-Bench show that QueenBee outperforms fixed communication baselines across team sizes and achieves the highest mean task score among recent automated multi-agent workflow optimization methods. Ablation studies further show that SFT-Bank components provide complementary performance gains, while evolution dynamics reveal a consistently improving average trend in the quality of the best collaboration structures found across independent runs as structural experience accumulates.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "QueenBee: A Recursively Self-Improving Harness for Communication Topology Design in LLM-Based Multi-Agent Systems".

Jane: Here's the gist of the paper. Large language model (LLM) multi-agent systems increasingly depend not only on how individual agents reason, but also on how agents are connected <ref:2606.27492#pg2> This paper introduces QueenBee Planner, a framework that treats inter-agent communication topology as a retrievable and self-improving design skill <ref:2606.27492#pg2> The best generated graph reduces RMSE from 12.53 for the strongest fixed topology to 7.87 while also reducing messages, model calls, and token cost <ref:2606.27492#pg2> These results suggest that multi-agent systems can learn reusable architectural design knowledge rather than merely memorizing task answers <ref:2606.27492#pg2>.

Tom: Pruning redundant reasoning tokens The core innovation of QueenBee Planner is treating inter-agent communication topology as a retrievable and self-improving design skill <ref:2606.27492#pg2> A pool of worker agents, the task adapter, and the scoring function are frozen; only an outer LLM planner learns to generate temporal communication DAGs specifying who sends information to whom, in which round, who merges messages, and who emits the final answer <ref:2606.27492#pg2> Execution traces are distilled into evidence-backed design rules with three actions: Preserve, Modify, and Avoid <ref:2606.27492#pg2> The hard part is not generating one good topology; the hard part is preventing a self-improving design loop from turning noise into policy <ref:2606.27492#pg2>.

Jane: How the training signal is built The planner’s long-term memory is a set of retrievable design skills, where each skill can be understood as an evidence-backed design rule <ref:2606.27492#pg2> A skill is written as s = (z, r, a, q, E), describing the applicability condition z, the design rule r for graph generation and action a in Preserve/Modify/Avoid mode, the statistical summary q including average loss and uncertainty q is a statistical summary including average loss and uncertainty <ref:2606.27492#pg2> The effect of the skill bank on a candidate graph G can be written as a design prior RS (G, x) = X s∈ρ(x) ws compat(G; s), where Preserve and Modify skills increase the score of candidate graphs compatible with their structures, while Avoid skills lower the score of candidates that reproduce failed structures <ref:2606.27492#pg2>.

Tom: Where the gains come from The system employs several mechanisms to ensure disciplined self-improvement, including a held-out generation gate, variance-aware credit, motif-level attribution, and insight falsification <ref:2606.27492#pg2> The acceptance gate decides whether a batch of new rules ∆S can be written into the long-term skill bank by comparing held-out performance of the generative planner <ref:2606.27492#pg2> Variance-aware credit gives candidate rules conservative credit during the acceptance phase, ensuring that a generated structure with low mean but high variance and low sample count will not dominate the planner because of a single lucky run <ref:2606.27492#pg2>.

Title and authors: Tom: So, Jane, we’re looking at this paper called QueenBee: A Recursively Self-Improving Harness for Communication Topology Design in LLM-Based Multi-Agent Systems. It sounds like they are tackling how to make the way these agents talk to each other actually learn and improve over time.

Jane: It does sound like it, Tom. The title itself suggests that these systems aren't just static structures for a single task; they're supposed to be learning how to design better communication patterns across different tasks. What do you think is the core idea behind this?

Tom: Well, the main thing they’re proposing is treating inter-agent communication topology as a retrievable and self-improving design skill. Instead of just having one fixed way agents talk, they want the outer LLM planner to learn how to generate those temporal communication Directed Acyclic Graphs, or DAGs.

Lu: That's the key shift from what most of this work has been doing, which is usually finding one good topology for a single task and then stopping. QueenBee wants that optimizer to accumulate reusable design knowledge about communication structure across different problems instead of just memorizing the answers for each specific task <ref:2606.27492#pg2>.

Meng: From my perspective, what I’m interested in is how they handle the practical side of this. If you're shipping models, learning a new topology sounds great, but how do you prevent that self-improvement loop from just turning noise into something useless or even harmful?

Jane: That brings us to the summary of their method. Basically, they freeze the worker agents and the task adapter and the scoring function. Only an outer LLM planner learns to generate those temporal communication DAGs specifying who sends information to whom, in which round, who merges messages, and who ultimately emits the final answer <ref:2606.27492#pg1>.

Tom: And they distill execution traces from these runs into evidence-backed design rules with three actions: Preserve, Modify, and Avoid. That sounds like a very structured way to capture what works and what doesn't in terms of communication flow <ref:2606.27492#pg1>.

Lu: They aren't just generating random graphs; they are distilling the execution traces into these rules using those three actions, which is how they build their long-term memory of design skills <ref:2606.27492#pg3>.

Meng: I like the idea of a skill bank where each skill is written as s = (z, r, a, q, E), describing the applicability condition z and the design rule r for graph generation and action a in Preserve/Modify/Avoid mode <ref:2606.27492#pg3>. But how do you actually make sure those skills are useful when you're dealing with thousands of potential communication structures?

Jane: They address that by writing a design prior, RS (G, x) = X s∈ρ(x) ws compat(G; s), where Preserve and Modify skills increase the score of candidate graphs compatible with their structures while Avoid skills lower the score of candidates that reproduce failed structures <ref:2606.27492#pg3>. That’s how they guide the planner's generation process.

Title and authors: Tom: So, it’s not just learning patterns; it’s using those learned skills as a prior to filter out bad designs before they even get fully generated <ref:2606.27492#pg3>. That seems like a solid mechanical improvement for generating better structures.

Lu: And the way they handle that self-improvement is interesting, too. They employ several mechanisms to ensure disciplined self-improvement, including a held-out generation gate and variance-aware credit <ref:2606.27492#pg3>. That acceptance gate decides whether a batch of new rules, Delta S, can be written into the long-term skill bank by comparing the held-out performance of the generative planner <ref:2606.27492#pg3>.

Meng: Variance-aware credit sounds like a good guard against those lucky runs dominating the planner because of a single successful but unstable configuration <ref:2606.27492#pg3>. I worry that if you keep filtering based on variance, you might end up only learning very conservative structures, which could limit the system's potential for novel solutions.

Jane: That’s a fair concern, Meng. But the goal seems to be disciplined improvement rather than just maximizing raw performance on any single task. They are trying to avoid turning noise into policy <ref:2606.27492#pg1>.

Tom: And I like how they use motif-level attribution for that skill distillation process <ref:2606.27492#pg3>. It means they are looking at the structure of the successful runs themselves, not just the final output quality. That’s a nice way to connect execution traces back to actual design choices.

Lu: The results show that the best generated graph reduces RMSE from twelve point five three for a strongest fixed topology down to seven point eight seven while also reducing messages and model calls <ref:2606.27492#pg1>. It’s a measurable efficiency gain, not just a qualitative improvement on accuracy <ref:2606.27492#pg1>.

Meng: That reduction in tokens and model calls is huge for deployment cost, which is exactly what I care about when we look at running these systems at scale <ref:2606.27492#pg1>. So, the improvement isn't just better accuracy on a test set; it’s a more efficient system overall.

Jane: And when we look at the empirical validation, they tested this on tasks like Count-Frequency aggregation and Silo-Bench style distributed coordination tasks <ref:2606.27492#pg1>. The successful graphs they generated aren't always dense; they tend to be structures like hybrid reduce-audit, binary-tree reduce, or dual-root convergence <ref:2606.27492#pg1>.

Tom: So the system is learning these specific structural motifs that are useful across different task types rather than just one perfect answer for one type of problem <ref:2606.27492#pg1>. That’s what I was hoping to see—transferable architectural knowledge.

Lu: And the representative skill cards they show, like skill id: silo cf accuracy peer star, illustrate that these accepted skill cards encode reusable communication design knowledge rather than task-specific answers <ref:2606.27492#pg1>. That’s evidence that the system is accumulating structure, not just task-solving behavior <ref:2606.27492#pg3>.

Meng: I'm still concerned about the scope of this learning. They show improvement over cold and fixed-topology baselines in Silo results, but what about generalization outside of those specific task sets? How robust is this skill bank when we throw it at something completely new?

Title and authors: Jane: The paper itself points out that they are not trying to discover one universally optimal topology; instead, they accumulate transferable priors about when to preserve, modify, or avoid communication patterns <ref:2606.27492#pg1>. This suggests the planner’s generation prior becomes less dependent on unconstrained cold sampling and more likely to propose deployable DAGs that control merge difficulty and cost <ref:2606.27492#pg1>.

Tom: So they aren't trying to find one magic topology; they are learning a set of guidelines for when to use certain structures, which makes sense from an engineering standpoint.

Lu: The implication is that the architecture of a multi-agent system is not just an implementation detail anymore; it’s becoming an explicit object of learning <ref:2606.27492#pg1>. That really changes how we think about designing these complex systems <ref:2606.27492#pg3>.

Meng: For deployment, that means we might start designing the communication structure first, and then the LLM planner refines it based on what it has learned from past successful runs <ref:2606.27492#pg1>. That sounds like a more proactive design approach.

Jane: To wrap things up, QueenBee Planner isn't about finding one universally optimal topology; it’s about accumulating transferable priors about when to preserve, modify, or avoid communication patterns <ref:2606.27492#pg1>. The findings suggest that the planner’s generation prior becomes less dependent on unconstrained cold sampling and more likely to propose deployable DAGs that preserve global information while controlling merge difficulty and cost <ref:2606.27492#pg1>.

Tom: So, the big picture here is that we are moving toward systems where the architecture itself is learned, not just implemented. It’s a significant step for building more efficient multi-agent setups <ref:2606.27492#pg3>.

Lu: I think future work needs to evaluate this mechanism on larger task suites and longer evolution horizons to see how those skills actually evolve under more pressure <ref:2606.27492#pg1>. That’s where the real testing is needed <ref:2606.27492#pg3>.

Meng: From a production standpoint, we need to know how long it takes for this skill bank to stabilize and become reliable enough for deployment, especially when dealing with longer task chains <ref:2606.27492#pg1>.

Jane: QueenBee: A Recursively Self-Improving Harness for Communication Topology Design in LLM-Based Multi-Agent Systems is a paper that shows how multi-agent systems can learn reusable architectural design knowledge rather than just memorizing task answers <ref:2606.27492#pg1>. It’s about learning the communication skill itself.

Tom: Yeah, it moves us from brittle, task-specific setups to something more adaptive and self-improving in terms of how agents interact <ref:2606.27492#pg3>.

Lu: We're looking at a framework that treats inter-agent communication topology as a retrievable and self-improving design skill <ref:2606.27492#pg1>. It’s about accumulating experience through skill libraries and cross-task lessons <ref:2606.27492#pg3>.

Meng: The practical gain is seeing that the best generated graph reduces RMSE from twelve point five three to seven point eight seven, plus a reduction in messages and model calls <ref:2606.27492#pg1>. That’s a concrete win for cost efficiency <ref:2606.27492#pg3>.

Jane: It really suggests that the architecture of a multi-agent system is not merely an implementation detail, but an explicit object of learning <ref:2606.27492#pg1>. That's a big idea for how we approach these kinds of complex AI systems.

The paper's summary: Tom: So, QueenBee is essentially about teaching multi-agent systems how to design their own communication structure over time. It’s not just picking one fixed way for them to talk, but having the system learn how to build better ways to communicate as it goes.

Jane: Right, so instead of giving the agents a set blueprint and letting them run with it, you let the agents figure out a better blueprint by learning from their past runs. That sounds like a really smart way for an AI system to improve its own architecture.

Lu: Exactly. The paper treats the communication topology—that whole network of who talks to whom and when—as a skill that gets retrievable and self-improving <ref:2606.27492#pg1>. It’s treating the design itself as something you can learn and save.

Meng: From a deployment standpoint, that’s interesting because it suggests the system isn't stuck with a rigid structure that might be inefficient later on. If it can adapt its communication flow based on what worked before, that could mean lower token costs down the line <ref:2606.27492#pg1>.

Tom: And they show some concrete numbers there. They took a strong fixed topology and got its error rate from twelve point fifty-three all the way down to seven point eighty-seven <ref:2606.27492#pg1>. That’s a significant drop in performance just by having this learned structure instead of just picking the best fixed one.

Jane: That reduction in error is interesting, but what about the process? How does it actually learn to make those structural changes? What are the rules it learns?

Lu: They distill their execution traces into evidence-backed design rules using three actions: Preserve, Modify, and Avoid <ref:2606.27492#pg1>. So when something works well, it gets preserved; when something fails or causes trouble, it gets avoided or modified.

Meng: Three actions sounds manageable for a system to follow. But how do you stop the system from just getting stuck in a loop where it keeps tweaking the structure without actually improving its task performance?

Tom: That’s the tricky part they call preventing a self-improving design loop from turning noise into policy <ref:2606.27492#pg1>. They use things like a held-out generation gate and variance-aware credit to make sure the new rules are actually good, not just lucky runs <ref:2606.27492#pg3>.

Jane: So they're building a system that learns when to change its communication pattern, but they have mechanisms in place to keep that learning disciplined and focused on real improvements rather than just random tinkering.

Lu: They define these learned skills as s = (z, r, a, q, E), which is the applicability condition z and the design rule r for graph generation and action a in Preserve/Modify/Avoid mode <ref:2606.27492#pg3>. This skill bank is what they call the planner’s long-term memory.

Tom: And how does that skill bank actually guide the system when it’s tackling a brand new task? It uses a design prior RS, which basically scores candidate graphs based on whether they match those learned skills <ref:2606.27492#pg3>.

Jane: So the system doesn't just guess; it looks at all the ways it’s seen successful communication designs in the past and uses that knowledge to suggest a graph structure that’s likely to work better for this new job.

Meng: That means instead of cold generation, which is just throwing out random connections, you get proposals that are already filtered by prior experience. That sounds like a way to make the process much more targeted <ref:2606.27492#pg1>.

Tom: And the results show that these successful structures aren't always dense; they’re often hybrid things like "hybrid reduce-audit" or "binary-tree reduce" <ref:2606.27492#pg1>. It seems to be learning specific, reusable motifs rather than just one single perfect configuration.

Jane: That means the system is accumulating transferable knowledge about communication design—like, "for this kind of task, I should probably use a peer-to-peer structure with a star sink"—instead of just memorizing the answer for this one specific problem.

Lu: That’s the core idea: the architecture becomes an explicit object of learning <ref:2606.27492#pg1>. It moves beyond just solving a task to optimizing how the agents cooperate for any task that looks similar.

The paper's improvements: Tom: So, we’re talking about how QueenBee actually improves things beyond just finding one good setup for a task <ref:2606.27492#pg1>. It’s about the learning mechanism itself getting smarter and more robust.

Jane: Right, so the paper isn't just saying "this setup is good," it’s showing how the whole system can learn *how* to design better setups for future problems. That shifts things from static engineering to something that evolves <ref:2606.27492#pg1>.

Lu: The key improvement here is treating the communication topology as a skill, which means it’s not just a one-off graph but a piece of reusable architectural knowledge <ref:2606.27492#pg3>. This allows the system to adapt its interaction strategy based on what it has already learned.

Meng: If the AI can learn these structural patterns, that should translate directly into better efficiency at scale. We’re talking about structures that are inherently more efficient for token usage and fewer model calls <ref:2606.27492#pg1>.

Tom: They show a significant win on efficiency, where the best generated graph reduced the error rate from twelve point fifty-three to seven point eighty-seven while cutting down messages by seventy-four point two percent <ref:2606.27492#pg1>. That’s a measurable improvement in how much work the system has to do.

Jane: That’s a lot of data for one metric, but what about the robustness? How does this learning process handle tasks that are completely different from the ones it trained on?

Lu: They introduce mechanisms like insight falsification and variance-aware credit to prevent the system from just memorizing noise or getting stuck in bad patterns <ref:2606.27492#pg3>. This acceptance gate checks if a new rule actually performs well on data it hasn't seen before.

Meng: I’m worried about that gate, though. If you filter out anything that has low variance or seems slightly unpredictable, you might end up with a system that is too conservative and can’t explore genuinely novel solutions <ref:2606.27492#pg3>.

Tom: That’s a fair point, Meng. But the authors argue this discipline is necessary to stop the self-improvement from turning into something useless or buggy for production deployment <ref:2606.27492#pg1>. They are trying to ensure it proposes deployable DAGs that control merge difficulty and cost rather than just random connections.

Jane: It sounds like the goal is creating a system that learns the *principles* of good communication design, not just a specific answer for this one problem <ref:2606.27492#pg1>.

Lu: Precisely. They’re building a library of reusable design skills, like "preserve this kind of peer propagation" or "avoid this type of mesh structure," that can be applied across many different scenarios <ref:2606.27492#pg3>.

Meng: So, for us in production, it means we could potentially give the system a library of proven communication patterns instead of having to design the whole flow from scratch every single time <ref:2606.27492#pg1>. That reduces our engineering overhead and risk significantly.

Tom: It changes the workflow from "design this task" to "retrieve and apply this skill," which is a huge operational shift for large-scale AI systems.

Jane: This really suggests that we’re moving toward systems where the communication architecture itself is something we explicitly train and manage, rather than just treating it as an afterthought.

Conclusion: Tom: So, to wrap up, QueenBee Planner isn't just about finding one perfect communication structure for a task <ref:2606.27492#pg1>. It’s about building a system that learns transferable rules on when to preserve, modify, or avoid certain communication patterns <ref:2606.27492#pg1>.

Jane: Exactly. This means the system is learning the architectural knowledge of how agents should talk, not just memorizing answers for specific problems <ref:2606.27492#pg1>. That’s a big shift in how we think about building these AI setups.

Lu: The implication is that the planner’s generation prior becomes less dependent on random cold sampling and more likely to propose deployable Directed Acyclic Graphs <ref:2606.27492#pg1>. It’s proposing structures that control merge difficulty and cost based on what it has learned <ref:2606.27492#pg1>.

Meng: From a production standpoint, that means we could design the communication structure first, and then the AI refines it based on its experience with past runs <ref:2606.27492#pg1>. It sounds like a way to make the whole system more predictable at scale.

Tom: I think so. The paper is pushing us toward an architecture that learns from its own interaction structure, making the entire system self-improving in a very structured way <ref:2606.27492#pg1>.

Jane: It’s fascinating because it moves the architecture of a multi-agent system from being just an implementation detail to being something that is actively learning and evolving <ref:2606.27492#pg1>.

Lu: I think future work needs to look at how these skills evolve over much longer time horizons and with even larger task suites to see how this prior holds up under more pressure <ref:2606.27492#pg1>.

Meng: For me, the question is just how fast that skill bank stabilizes into something we can actually trust for a live deployment, you know?

Lalam: I am confident in this direction. My literature review confirms that learning to design communication topology is a very high-leverage area for agent systems <ref:2606.27492#pg1>. This paper’s approach is highly structured and evidence-backed.

Tom: So, we’ve seen how QueenBee Planner uses skill evolution to build reusable architectural design knowledge <ref:2606.27492#pg1>. It’s a solid mechanical improvement for agent coordination.

Jane: We’re leaving with the idea that the architecture of an AI system is not just something you implement, but something you explicitly learn and optimize <ref:2606.27492#pg1>. That opens up a lot of new ways to approach complex AI design.

Lu: It really re-frames the entire problem from solving a task to optimizing the way the agents collaborate across many tasks <ref:2606.27492#pg1>.

More episodes

← Home