Failure-Driven Workflow Refinement
cs.AI
Submitted: 2025-10-11
Updated: 2026-11-03
Comments: Accepted at ICML 2026 as a Spotlight. Code:https://github.com/jushengzhang/CE-Graph
License: http://creativecommons.org/licenses/by/4.0/
The gist: Optimizing LLM-based workflows is typically formulated as a global search, where candidate workflows are evaluated based on a scalar metric.
Terminology
Abstract
Optimizing LLM-based workflows is typically formulated as a global search, where candidate workflows are evaluated based on a scalar metric. This paradigm, however, suffers from a critical flaw: information collapse. By reducing rich, multi-step execution traces to simple success/failure signals, existing methods are rendered blind to the underlying structure of failures, fundamentally preventing them from modeling the workflow's failure distribution. We reconceptualize this challenge as a distributional problem. We propose a new paradigm where the optimization goal is not to maximize a scalar score, but to directly minimize a workflow's Expected Failure Mass, i.e., the integral of its failure probability density function defined over a high-dimensional Failure Signature Space (FSS). This distributional lens allows us to move from inefficient, zero-order optimization to a principled, gradient-like descent on the failure landscape itself. We introduce CE-Graph, a framework that operationalizes this paradigm through a novel, failure-driven refinement process. CE-Graph approximates the failure distribution from a pool of counterexamples, identifies its densest regions as recurring failure modes, and applies targeted, operator-constrained graph edits via a Propose-and-Verify mechanism to greedily reduce the failure mass. On math, code, and QA benchmarks, our CE-Graph achieves higher robustness at a significantly lower cost than strong baselines. This suggests that a system's reliability emerges not from avoiding failures, but from systematically learning and reshaping the geometric structure of its failure distributions.
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- CigaR: Cost-efficient Program Repair with LLMs
- Exploring the True Potential: Evaluating the Black-box Optimization Capability of Large Language Models
- AutoFlow: Automated Workflow Generation for Large Language Model Agents
- GAIA: a benchmark for General AI Assistants
- Large Language Models: A Survey
- GPT-4 Technical Report
- AgentSquare: Automatic LLM Agent Search in Modular Design Space
- Practical Bayesian Optimization of Machine Learning Algorithms
- To Err is Machine: Vulnerability Detection Challenges LLM Reasoning
- Attention Is All You Need
- CF-VLM:CounterFactual Vision-Language Fine-tuning
- OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaboration
- GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning
- A Survey of Large Language Models
- AdaptFlow: Adaptive Workflow Optimization via Meta-Learning
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents