CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies
cs.AI
Submitted: 2026-09-26
Updated: 2026-09-29
Code: https://github.com/crewAIInc/crewAI
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated.
Terminology
Abstract
Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information must remain isolated. We call this task-conditioned scope of information a collaborative memory boundary. Workflow topology determines which intermediate artifacts are applicable to which downstream workers and when they cease to be valid, thereby providing a structural stress dimension for sharing and isolation. Existing memory benchmarks primarily evaluate retention and retrieval, whereas multi-agent benchmarks emphasize coordination and end-to-end completion, leaving topology-conditioned memory boundaries largely unmeasured. We introduce CoMemBench, an execution-grounded benchmark for collaborative memory sharing and isolation across multi-agent workflow topologies. It constructs 800 composite workflows across four domains from source-grounded dependency graphs, with node-local specifications, verifiable artifact handoffs, native evaluators, and matched isolation challenges. CoMemBench measures workflow completion, verified node progress, required-handoff reliability, isolation robustness, and token cost. Experiments reveal a sharing-isolation trade-off: broader context improves information availability but can weaken isolation, while system rankings shift across topologies and artifact violations.
Sources
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- miniCTX: Neural Theorem Proving with (Long-)Contexts
- REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
- GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
- StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
- Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
- GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems