Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks
cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: 25 pages, 10 figures, 6 tables
Code: https://github.com/Arcadia-1/analog-design-bench
Project page: https://analog-design-bench.tokenzhang.com
License: http://creativecommons.org/licenses/by/4.0/
The gist: Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured.
Terminology
Abstract
Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured. We introduce Analog Design Bench, a long-horizon agentic benchmark of 50 transistor-level design tasks contributed by 17 chip designers. Agents work with an open-source simulator, while an isolated verifier evaluates the submitted circuit using specification-based electrical tests. We evaluate 15 agent configurations across 2,250 two-hour attempts and observe full-specification pass rates from 8.0% to 78.0%. Coding-benchmark performance correlates with analog results but leaves much of the performance spread unexplained. Our failure analysis shows that most unsuccessful submissions have no recorded legality rejection but fail electrical acceptance, identifying electrical closure as the dominant endpoint challenge. We test time, reasoning effort, agent harness, and supplied design knowledge as interventions. Longer budgets and higher reasoning effort improve performance, while general skill documents provide little benefit and sometimes reduce performance. Supplying a task-matched reference topology, an idealized form of circuit-IP retrieval, raises DeepSeek V4 Pro by 18.7 percentage points and mainly accelerates GPT-5.6 Sol.
Sources
- CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process
- AnalogAgent: Self-Improving Analog Circuit Design Automation with LLM Agents
- White-Box Reasoning: Synergizing LLM Strategy and gm/Id Data for Automated Analog Circuit Design
- DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey
- AnalogGym: An Open and Practical Testing Suite for Analog Circuit Synthesis
- AmpAgent: An LLM-based Multi-Agent System for Multi-stage Amplifier Schematic Design from Literature for Process and Performance Porting
- VerilogEval: Evaluating Large Language Models for Verilog Code Generation
- RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model
- Large Language Model Agent: A Survey on Methodology, Applications and Challenges
- NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
- AICircuit: A Multi-Level Dataset and Benchmark for AI-Driven Analog Integrated Circuit Design
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- A Survey of Research in Large Language Models for Electronic Design Automation
- Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification
- HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimization
- AutoCkt: Deep Reinforcement Learning of Analog Circuit Designs
- AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuits
- AMSNet: Netlist Dataset for AMS Circuits
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems