CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generation

CHORUS is a post-training framework for high-coverage testbench stimulus generation in hardware verification. The paper's key findings and contributions are:

Core Problem: Large language models have advanced code generation, but hardware verification—particularly generating testbench stimuli that maximize coverage in a hardware simulator—remains difficult even for frontier models. Scale alone is insufficient: Scale alone does not solve testbench generation (Figure 1), as even a 671B-parameter frontier model (DeepSeek-R1) remains well below achievable performance.

Key Observations: The paper builds on two observations. First, staged SFT produces behaviorally diverse checkpoints, and dense-reward RL turns them into strong experts with comparable aggregate performance but distinct task-level strengths. Second, these complementary strengths can be exploited through either training-free model merging or further post-training to outperform the best individual expert.

Method: CHORUS takes the staged-SFT checkpoints from LLM4Cov (Stage-0, Stage-1, Stage-2) and applies identical execution-guided RL (DAPO) to each independently. This produces three RL experts (π0RL, π1RL, π2RL) that converge to similar aggregate performance (85% Pass@1) but retain complementary task-level strengths. The paper then consolidates these experts through two approaches:

1. Training-free weight merging (Model Soup, DARE-TIES, DELLA)

2. Adaptive multi-teacher on-policy distillation (OPD), which routes each task to its best teacher based on execution reward and skips tasks where no teacher beats the student

Key Results:

• The final CHORUS model (4B parameters) achieves 88.0% Pass@1 on CVDP-ECov under agentic refinement, outperforming DeepSeek-R1 (671B) by 13.5 percentage points

• It also outperforms all general-purpose, coding-specialized, and hardware-specific baselines evaluated

• The three RL experts individually score ~85% Pass@1, but their oracle union reaches 90.8%, showing ~5 points of exploitable headroom

• Training-free merging (Model Soup) recovers about 1 point above the best individual expert

• Adaptive multi-teacher OPD recovers over half the oracle-union headroom, reaching 88.0% Pass@1

Ablations and Analysis:

• Staged SFT barely changes eventual RL performance: the later, more elaborate SFT stages buy almost nothing once execution-guided RL is applied

• The experts are strongly complementary: the leader rotates: an expert that nearly saturates one design can fall to little more than half coverage on the next, where a sibling saturates instead

• The reward gate in adaptive OPD is load-bearing: Always distilling from the current best teacher... recovers only about a third of what the full method obtains

• The skip rule (not training on tasks where no teacher beats the student) is also important

• Worst-state refinement is more stable than best-state refinement: Best-state selection rises faster early but destabilizes mid-run and ends lower; worst-state selection (ours) climbs steadily and finishes higher

Limitations: The study is confined to hardware testbench generation, and the diversity exploited originates in a specific SFT curriculum; other curricula may induce more or less of it.

Conclusion: "We introduced CHORUS, a post-training framework that turns related SFT checkpoints into complementary RL experts and consolidates their strengths into one model... The resulting 4B model reaches 88.0% Pass@1 on CVDP-ECov, showing that consolidating complementary experts can push performance beyond single-model RL saturation."

Improvements to AI systems:

1. Complementary Expert Consolidation via Adaptive Multi-Teacher Distillation

• What it adds: Instead of training one model to saturation, the system trains multiple experts from diverse initialization checkpoints with identical RL, then distills them into a single student using a reward-gated routing mechanism. The student only learns from a teacher when that teacher’s execution reward exceeds the student’s current performance, and skips tasks where no teacher is superior.

• Improved capability: The final model outperforms any single expert and even a 671B-parameter frontier model on the target task, achieving 88.0% Pass@1 vs. 74.5% for DeepSeek-R1. This method is task-agnostic and can be applied to any domain with a dense reward signal (e.g., code generation with unit tests, robotic control with simulator rewards).

2. Staged SFT + Dense-RL as a Diversity Engine

• What it adds: Rather than using a single SFT checkpoint, the system deliberately uses multiple SFT checkpoints from a staged curriculum (Stage-0, 1, 2) as separate starting points for RL. This induces behavioral diversity that persists after RL, even though all experts converge to similar aggregate performance (85% Pass@1).

• Improved capability: The system can automatically discover complementary solution strategies for different subproblems (e.g., different hardware designs) without manual prompt engineering or task-specific tuning. The oracle union of experts reaches 90.8%, revealing exploitable headroom that a single model cannot reach.

3. Training-Free Weight Merging as a Cheap Performance Boost

• What it adds: Before resorting to distillation, the system applies simple weight-averaging methods (Model Soup, DARE-TIES, DELLA) to the RL experts. This recovers 1 point above the best individual expert with zero additional training.

• Improved capability: Any AI system with multiple fine-tuned models can immediately gain performance by merging their weights, even when the models are trained on the same task but from different initializations. This is especially useful in resource-constrained settings where distillation is too expensive.

4. Reward-Gated Distillation with Skip Rule

• What it adds: The distillation process uses a binary reward gate (teacher must beat student on that task) and a skip rule (if no teacher beats the student, the task is excluded from training). This prevents the student from learning from weaker teachers and avoids overfitting to already-mastered tasks.

• Improved capability: The system avoids catastrophic forgetting and maintains stable improvement. Ablations show that always distilling from the best teacher recovers only a third of the full method’s gain, proving the gate is load-bearing. This principle can be applied to any multi-teacher knowledge distillation setup.

5. Worst-State Refinement for RL Stability

• What it adds: During RL, the system refines the model on its worst-performing states (lowest reward) rather than best-performing states. This leads to steadier climbing and higher final performance, whereas best-state selection rises fast but destabilizes mid-run.

• Improved capability: RL agents trained with worst-state refinement are more robust and reach higher final performance in sparse-reward or high-variance environments. This is directly transferable to other RL domains like dialogue generation or game playing.

6. Post-Training Consolidation Beyond Single-Model Saturation

• What it adds: The framework demonstrates that single-model RL saturation (85% Pass@1) is not a ceiling. By consolidating complementary experts, the system pushes to 88.0% Pass@1, recovering over half of the oracle-union headroom.

• Improved capability: AI systems can exceed the performance limits of any single training run by exploiting diversity from multiple checkpoints, without needing larger models or more compute per run. This is a general recipe for squeezing extra performance from existing training pipelines.

What the improved AI system can do specifically:

• Generate hardware verification testbenches that achieve 88% coverage Pass@1 on complex designs, outperforming models 160x larger.

• Automatically route each verification task to the best-suited internal expert during training, then collapse into a single deployable model with no routing overhead at inference.

• Adapt to new hardware designs without retraining from scratch—just apply the same staged-SFT + RL + consolidation pipeline to new design families.

• Provide a drop-in performance boost for any existing multi-checkpoint training pipeline via weight merging, even without distillation.

• Maintain stable RL training in environments with noisy or sparse rewards by focusing on worst-case states.

Large language models (LLMs) have advanced code generation, where executable feedback provides a more reliable learning signal than textual imitation alone. Hardware verification is an important application of code generation and accounts for a substantial fraction of modern chip design effort, with high-coverage testbench stimulus generation as a key task. We present CHORUS, a post-training framework that pushes performance beyond what a conventional supervised fine-tuning (SFT)-to-reinforcement learning (RL) pipeline achieves. CHORUS builds on two observations. First, staged SFT produces behaviorally diverse checkpoints, and dense-reward RL turns them into strong experts with comparable aggregate performance but distinct task-level strengths. Second, these complementary strengths can be exploited through either training-free model merging or further post-training to outperform the best individual expert. By consolidating the resulting specialists into a single 4B model, CHORUS achieves 88.0% Pass@1 on CVDP-ECov, outperforming DeepSeek-R1 (671B) by 13.5 percentage points.