CANDO: Cooperative Agentic Network for Layout Design Optimization

arXiv:2610.10044 · cs.MA · Submitted 2026-10-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "CANDO: Cooperative Agentic Network for Layout Design Optimization".

Jane: Here's the gist of the paper. Layout generation for real-world facilities is a complex problem involving reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility.

Tom: Existing benchmarks often focus on simpler placements over rectangular domains and utilize distributional metrics like FID and IoU, which reward conformity to dataset priors rather than design innovation.

Jane: To address these limitations, the authors introduce ALPS-Bench, a benchmark featuring 1,000 professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual.

Tom: As a strong baseline for ALPS-Bench, they propose CANDO, a training-free multi-agent framework.

Paper summary: Tom: So we’re looking at this paper called CANDO: Cooperative Agentic Network for Layout Design Optimization. It tackles the really hard problem of designing layouts for real-world facilities, which involves dealing with irregular site boundaries and different object orientations, plus making sure there's a way to get in and out of the place and move around.

Jane: Right, so most existing layout benchmarks focus on simpler rectangular areas and use metrics like FID or IoU that just reward fitting things into what the dataset already looks like, instead of actually trying new design ideas.

Lu: That’s exactly the gap they’re targeting. They introduce ALPS-Bench, which is a benchmark with one thousand professionally annotated real-world facility layouts and a set of rules based on a structured design manual to score them.

Meng: So, CANDO is their answer to that, right? It’s presented as this training-free multi-agent framework where different specialized agents work together to refine the layout configuration.

Lalam: Exactly. It sets up four specific agents: an Initializer, a Verifier, a Historian, and a Planner. The goal is to find the best configuration by maximizing some total weighted score based on both hard constraints and softer properties like how well it fits the design manual.

Tom: The way they define that aggregate score is interesting; it splits the total into two parts: one part checks if you met all those hard constraints, and another part looks at things like compositionality and orientation.

Jane: And they use a specific protocol for scoring compliance, where if you meet the hard constraints, there’s this weighted measure of how well those physical requirements are satisfied. It sounds like they are making sure the layout is actually physically possible before worrying about aesthetics.

Lu: The refinement loop is split into two phases: first, the Initializer uses a large language model to get a starting configuration based on similar layouts from development data, and then the four agents iteratively refine that seed.

Meng: I'm curious about how they handle the iteration part. Does it just keep tweaking things randomly, or does it have a systematic way of improving the design over time?

Lu: They are inspired by some blockwise sampling schemes from another system, and they use a deterministic tool for verification during the refinement process to select the best candidate at each step.

Jane: That’s important because it means they aren't just guessing; they are using these tools to check if a proposed placement actually makes sense according to the design manual.

Tom: Then you have this Historian agent keeping track of how the design has evolved, which feeds into the Planner, giving it context on what has been tried before.

Meng: And then there's the Planner drafting remediation plans based on all that history and feedback, using specialized sub-planners for things like hard constraints or flow. That sounds like a lot of coordination to manage.

Paper summary: Lalam: The final part is where the Generator takes those plans and translates them into actual revised candidate configurations by applying specific actions, like moving or rotating elements.

Tom: And they have this Supervisor Planner that looks at everything holistically and prioritizes actions based on category weights, making sure hard constraints always come first, even if the worker planners suggest something else.

Jane: So it’s a very structured way of letting the agents collaborate to solve this complex spatial optimization problem. It sounds like they are trying to build a system that doesn't just generate an image, but generates a feasible plan.

Lu: They deliberately offload the low-level geometric calculations to deterministic tools, which keeps the core AI reasoning focused on the strategic decisions about placement and orientation rather than doing all that hard math themselves.

Meng: That’s smart for deployment because running complex spatial reasoning models on real traffic at scale would be tough; separating the computation helps. But I wonder how much work goes into training those specialized worker planners?

Lalam: They use a Qwen model as the backbone, and the results show that even this simplified version achieves state-of-the-art mIoU and alignment on other benchmarks like PubLayNet and RICO.

Tom: That’s a strong result, especially because they achieved those high scores with what is essentially a training-free approach for the agentic framework itself. It suggests that the cooperative agentic design method is effective for this kind of constraint-aware synthesis.

Jane: What does this actually mean for someone who just listens to the show? It means that instead of a computer just guessing where things go, we're seeing a system that follows a set of explicit rules—like "this door must be reachable from the entrance"—and it systematically fixes those rules.

Lu: The implication is that this approach of structured cooperation among agents, combined with tool-grounded verification, is a solid recipe for layout synthesis problems where feasibility and adherence to complex manuals are key.

Meng: But I still have to ask about the scale; how reliable is this when you introduce novel constraints that weren't in the original design manual? Does it just break down?

Lalam: The authors state their limitations, saying that while they show strong performance on PubLayNet and RICO, they haven't fully tested its robustness against entirely new or unseen complex constraints outside of the structured manual.

Tom: So it’s a powerful tool for the problems it's designed for—real-world facilities with known constraints—but we need to be careful when applying it to completely unpredictable situations.

Jane: It’s a really cool way to think about AI design, moving away from just asking the model to "draw me a layout" toward building a system that can actually reason through the physics and access rules of the space.

Conclusion: Tom: So we're wrapping up CANDO, which stands for Cooperative Agentic Network for Layout Design Optimization.

Jane: It’s basically a system where different AI agents work together to figure out how to put things in a real space correctly, using a lot of rules from a design manual.

Tom: Yeah, the authors are showing that this framework can find layouts that actually fit those hard physical constraints, which is harder than just drawing something pretty.

Jane: What this means for us is that instead of just asking an AI to make a picture look good, we can build systems that respect real-world physics and access requirements.

Tom: Exactly. It's about moving from generating a design idea to generating a feasible plan that actually works in the real world.

Jane: And they are using this specific multi-agent setup with verification tools to make sure the plans they generate don't just look right, but are actually buildable or usable.

Tom: The results show it beats other methods on benchmarks like ALPS-Bench and even PubLayNet and RICO, which is a strong signal that the cooperation between agents is effective.

Jane: It’s not about one single smart model doing everything; it’s about specialized agents handling different parts of the problem, like one agent checking physical overlap while another looks at orientation.

Tom: The core idea they are pushing here is that breaking down a massive spatial problem into smaller, constrained tasks and letting agents collaborate on those tasks works really well.

Jane: But even with these good results, they admit that the system's performance depends heavily on how detailed the design manual is to begin with.

Tom: That’s a fair caveat. It suggests that while it’s great for structured environments like real facilities, if you throw totally novel shapes at it without clear rules, the whole system might struggle to find a solution.

Jane: So CANDO is really pointing toward a future where AI design isn't just about creative output, but about rigorous spatial reasoning and constraint satisfaction.

Tom: And that leaves us wondering how this kind of structured agent cooperation could translate to designing complex physical systems in manufacturing or logistics down the road.

Shell Information Technology International · Delft University of Technology

cs.MA

Submitted: 2026-10-07

Updated: 2026-11-16

Comments: Published at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Code: https://github.com/PKU-ICST-MIPL/PosterLayout-CVPR2023

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Layout generation for real-world facilities is a complex problem involving reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements…

Key concepts

ALPS-Bench
A new benchmark featuring 1,000 professionally annotated real-world facility layouts. It is designed to test layout generation on complex sites with irregular boundaries, heterogeneous orientations, and access requirements. It uses a structured design manual for scoring.
CANDO Agents
Four specialized agents work together: the Initializer creates a starting layout, the Verifier checks it against rules, the Historian tracks design changes over time, and the Planner drafts fixes. They cooperate to find an optimal configuration that balances various design goals.
Hard Constraints (HC)
These are non-negotiable rules for a layout, such as ensuring facilities do not overlap or stay within site boundaries. CANDO prioritizes meeting these constraints absolutely before optimizing other softer properties like proximity or orientation.
Two-Phase Refinement Loop
The process starts by generating a seed layout (Phase 1) using an LLM and geometric checks. Then, the four agents enter a loop where the Verifier selects the best candidate, the Historian provides context, and the Planner generates remediation plans to improve that candidate iteratively.

Terminology

Summary

Layout generation for real-world facilities is a complex problem involving reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility. p. 0

Where the paper says it

CANDO: Cooperative Agentic Network for Layout Design Optimization Athanasios Masouris1 Jing Zheng2 Benjamin Chandler1 Hadi Jamali-Rad1,3 1 Shell Information Technology International, 2 Shell China Limited 3 Delft University of Technology (TU Delft) Abstract Layout generation for real-world facilities is a challenging problem, requiring reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility. Yet, most existing layout benchmarks in the generative AI space target simpler placements over rectangular domains and rely on distributional metrics such as FID and IoU that reward conformity to dataset priors, thus discounting design innovation.

Existing benchmarks often focus on simpler placements over rectangular domains and utilize distributional metrics like FID and IoU, which reward conformity to dataset priors rather than design innovation. p. 0
Where the paper says it

CANDO: Cooperative Agentic Network for Layout Design Optimization Athanasios Masouris1 Jing Zheng2 Benjamin Chandler1 Hadi Jamali-Rad1,3 1 Shell Information Technology International, 2 Shell China Limited 3 Delft University of Technology (TU Delft) Abstract Layout generation for real-world facilities is a challenging problem, requiring reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility. Yet, most existing layout benchmarks in the generative AI space target simpler placements over rectangular domains and rely on distributional metrics such as FID and IoU that reward conformity to dataset priors, thus discounting design innovation.

To address these limitations, the authors introduce ALPS-Bench, a benchmark featuring 1,000 professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual. p. 0
Where the paper says it

Yet, most existing layout benchmarks in the generative AI space target simpler placements over rectangular domains and rely on distributional metrics such as FID and IoU that reward conformity to dataset priors, thus discounting design innovation. Motivated by these gaps, we introduce ALPS-Bench, a benchmark of 1, 000 professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual.

As a strong baseline for ALPS-Bench, they propose CANDO, a training-free multi-agent framework. p. 0

Where the paper says it

Motivated by these gaps, we introduce ALPS-Bench, a benchmark of 1, 000 professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual. As a strong baseline for ALPS-Bench, we propose CANDO, a training-free multi-agent framework in which specialized agents iteratively refine layouts through a verification-grounded loop, concentrating reasoning on strategic spatial decisions.

CANDO operates as a cooperative agentic network consisting of four specialized agents: the ICL Initializer, the Verifier, the Historian, and the Planner. p. 5
Where the paper says it

First, an in-context learning initializer (ICL Initializer) produces a seed configuration q (0) by conditioning an LLM on structurally similar layouts retrieved from the development data. Second, four specialized agents (Verifier, Historian, Planner, and Generator) cooperate to refine q (0) toward q ⋆ through a parallel-branching iterative loop adapted from CoDe’s blockwise sampling scheme for diffusion models [46].

The goal is to find an optimal configuration q that maximizes a total weighted score J(q), subject to hard constraints (HC) like containment and non-overlap, and soft constraints (SC) quantifying properties such as Compositionality, Orientation, and Proximity. p. 2
Where the paper says it

Encodes stacking relations, clearance zones, and inter-element distance bounds. The goal is to find a valid configuration that maximizes total weighted score: q ⋆ = arg max q ∈ Qvalid J (q) ≜ X S s=1 ws fs(q), (2) with S = |SC|, and scoring function fs ∈ [0, 1] indicating compliance against soft constraint s ∈ SC when fs(q) = 1.

The Aggregate Score is defined as AGG = 1 over 2(CS + RE), where CS depends on whether HC are met (alpha times HCM times SCM if HCM=1, or 0 otherwise), and RE = 1 over 3(PL + MN + PC). p. 4
Where the paper says it

Aggregate Score (AGG). With all sub-metrics bounded in [0, 1], we define the final scores: AGG = 1 2 (CS + RE), CS = SCM if HCM = 1 α × HCM × SCM if HCM < 1 , RE = 1 3 (PL + MN + PC).

CANDO employs a two-phase refinement loop. p. 5

Where the paper says it

Crucially, since language models struggle with spatial understanding and coordinate arithmetic [56], CANDO deliberately offloads low-level geometric computations to deterministic tools. The two-phase design reflects the asymmetric interplay between feasibility and design quality, as hard-constraint violations would render soft-constraint quality irrelevant.

First, the ICL Initializer produces a seed configuration q(0) by conditioning an LLM on structurally similar layouts retrieved from development data, ranked using a geometric hard-constraint satisfaction proxy sigma(q). p. 5
Where the paper says it

The proposed solution operates in two phases. First, an in-context learning initializer (ICL Initializer) produces a seed configuration q (0) by conditioning an LLM on structurally similar layouts retrieved from the development data.

Second, the four agents iteratively refine this configuration through parallel branching inspired by CoDe’s blockwise sampling scheme. p. 5
Where the paper says it

First, an in-context learning initializer (ICL Initializer) produces a seed configuration q (0) by conditioning an LLM on structurally similar layouts retrieved from the development data. Second, four specialized agents (Verifier, Historian, Planner, and Generator) cooperate to refine q (0) toward q ⋆ through a parallel-branching iterative loop adapted from CoDe’s blockwise sampling scheme for diffusion models [46].

The Verifier evaluates candidates against the design manual M using deterministic tools to select the best candidate q bar(t). p. 5
Where the paper says it

It lies at the core of our proposed approach, providing reward signals that guide the refinement of intermediate layouts. Given K candidates in Q(t) , the Verifier evaluates them against the design manual M using dedicated deterministic tools and selects the best-scoring candidate ¯q (t) .

The Historian agent maintains a running record of design evolution, passing a sliding window H(t) to the Planner. p. 5
Where the paper says it

History Management. The Historian agent maintains a running record of how the layout design evolves.

The Planner drafts remediation plans pi(t) based on diagnostic feedback, history H(t), and the current state, utilizing specialized worker planners for each constraint category (Hard constraints, Assembly, Proximity, Orientation, Flow). p. 5
Where the paper says it

The Planner agent drafts its remediation plan along the same five evaluation categories used by the Verifier. Category-level plans are produced by the same agent, if the agent is capable of handling the full context (with a large backbone), or decomposed and processed in parallel by specialized planning sub-agents/workers, with task-specific tools, and then reconciled by a supervisor 6

Finally, the Generator translates these plans into a set of revised candidate configurations Q(t) by applying modifications to q bar(t-1). p. 6
Where the paper says it

This agent translates the remediation plan Π(t) into a set of K revised candidate configurations Q(t) = {q (t) k } K k=1 by applying a sequence of modifications to q¯ (t−1) = {(pi , θi)} E i=1 (Eq. The Generator adopts a strategy-diversified ensemble generation, with candidates either applying the plan conservatively through local repairs, prioritizing a single category or allowing larger layout-wide modifications.

The Supervisor Planner uses a holistic view to prioritize actions based on category weights (prioritizing Hard constraints and Flow first), overriding worker suggestions with visual spatial reasoning when contradictions arise. p. 6
Where the paper says it

into a remediation plan Π(t) at each time step t (as shown in Fig. To resolve cross-category conflicts and prioritize repairs, the supervisor grounds its decisions in a multi-modal input context defined by four complementary pieces of information: (i) the targeted diagnostic feedback from the specialist workers, (ii) the history H(t) , (iii) the geometric coordinate representation of the layout, and (iv) its visual rendering created by a simple plotting tool.

The system deliberately offloads low-level geometric computations to deterministic tools. p. 5

Where the paper says it

Second, four specialized agents (Verifier, Historian, Planner, and Generator) cooperate to refine q (0) toward q ⋆ through a parallel-branching iterative loop adapted from CoDe’s blockwise sampling scheme for diffusion models [46]. Crucially, since language models struggle with spatial understanding and coordinate arithmetic [56], CANDO deliberately offloads low-level geometric computations to deterministic tools.

The refinement process involves strategy diversification in the Generator with eight configurations to explore different repair behaviors, and the core principle emphasizes that hard constraint violations take absolute priority over all other categories. The final output is a JSON object detailing corrective actions per category for the Generator, utilizing specific action types such as move, rotate, stack, cluster, align group, and move group.

The results demonstrate that CANDO variants substantially outperform all considered baselines on ALPS-Bench and other benchmarks like PubLayNet and RICO. p. 7

Where the paper says it

Figure 3: Mean CS Score vs. T for Table 1 The summary statistics demonstrate that CANDO variants substantially outperform all considered baselines, including the two models trained specifically on ALPS-Bench, while requiring an order of magnitude smaller total number of refinement steps (T) than the SingleAgent baseline, with a time-evolution graph shown in Fig.

CANDO achieves state-of-the-art mIoU and alignment scores on PubLayNet and RICO with a comparatively lightweight backbone (Qwen), suggesting that cooperative agentic design with tool-grounded verification is a broadly effective recipe for constraint-aware layout synthesis. p. 1
Where the paper says it

CANDO offers several innovative design choices, including structured cooperation among agents, tool-grounded verification, and motion planning. We demonstrate that even a simplified version of CANDO achieves state-of-the-art mIoU and alignment on the widely adopted PubLayNet and RICO benchmarks, and leading overlap and strict-underlay performance on PKU-PosterLayout, corroborating that cooperative agentic frameworks are a broadly effective recipe for layout generation.

The framework generates two-dimensional element placements for operational sites, and the routing feasibility checks are conservative by design. p. 3
Where the paper says it

Intrigued by these gaps, we introduce the Asset Layout Planning and Synthesis Benchmark (ALPS-Bench), a collection of 1,000 annotated layouts from real operational sites. Polygon-level annotations were produced by professional annotators working directly from original CAD design drawings of each facility, yielding geometrically accurate element boundaries that reflect the as-built state of the sites.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Cooperative Agentic Network for Design Optimization (CANDO) can be used to refine layouts through a verification-grounded loop, concentrating reasoning on strategic spatial decisions rather than relying solely on distributional metrics like FID or IoU.

  2. CANDO can be applied to complex, real-world asset layout design problems by reasoning over irregular site boundaries, enforcing orientation consistency across heterogeneous functional groups, and certifying motion-planning feasibility for multiple vehicle scenarios.

  3. The system can achieve state-of-the-art performance on constraint satisfaction metrics such as mIoU and alignment on benchmarks like PubLayNet and RICO, demonstrating that cooperative agentic design is a broadly effective recipe for constraint-aware layout synthesis.

  4. The framework utilizes an in-context learning initializer to produce a seed configuration by retrieving structurally similar layouts from development data, significantly improving the initialization phase compared to purely random generation.

  5. CANDO can be adapted to resource-constrained settings by leveraging specialized agents and offloading low-level geometric computations to deterministic tools, allowing it to achieve competitive overall complexity despite increased operational complexity per step.

  6. The system can perform iterative refinement using a parallel-branching loop, where the blockwise BoN approach allows the system to simultaneously explore structurally different refinements without committing prematurely to any single branch.

  7. The Planner agent can use a holistic view of the entire layout and verification feedback to determine an optimal set of corrective actions per category, prioritizing hard constraints and flow violations first as dictated by the scoring function.

  8. The Generator's strategy library allows for diverse refinement behaviors, enabling it to select specialized strategies like hard-constraints focused or proximity focused based on the specific violations detected in the current iteration.

  9. The system can utilize visual analysis of the layout image to override worker suggestions when they are spatially suboptimal or visually congested, ensuring that decisions are grounded in a global spatial picture rather than purely numeric data.

Abstract

Layout generation for real-world facilities is a challenging problem, requiring reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility. Yet, most existing layout benchmarks in the generative AI space target simpler placements over rectangular domains and rely on distributional metrics such as FID and IoU that reward conformity to dataset priors, thus discounting design innovation. Motivated by these gaps, we introduce ALPS-Bench, a benchmark of 1,000 professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual. As a strong baseline for ALPS-Bench, we propose CANDO, a training-free multi-agent framework in which specialized agents iteratively refine layouts through a verification-grounded loop, concentrating reasoning on strategic spatial decisions. We demonstrate that CANDO surpasses state-of-the-art trained and LLM-based baselines on the widely adopted PubLayNet, RICO, and PKU-PosterLayout benchmarks, establishing cooperative agentic design as a broadly effective recipe for constraint-aware layout synthesis. Code and benchmark are available through our project page: https://shell-oss.github.io/cando/

Sources

Related papers