FlowScout: From Execution Feedback to Reliable Tool-Using Agent Workflows

F LOW S COUT is an execution-guided framework for automatically generating tool-integrated agentic workflows from historical task-solving records. It addresses the limitation of prior automatic workflow generation approaches (e.g., AFlow, A2 Flow) that produce LLM-centric workflows where real tool executions are abstracted and simulated by LLM nodes, limiting usability and stability. F LOW S COUT represents an agentic workflow as a directed graph composed of LLM nodes, tool-calling nodes, and dependency edges that encode data flow and control flow. The generation is formulated as a graph search problem over valid workflow graphs.

The framework consists of three modules: Workflow Miner, Workflow Executor, and Workflow Optimizer. The Workflow Miner first splits historical task-solving records into optimization and validation sets, mines a common tool coordination skeleton from validated orchestration logs using PM4Py process mining, and constructs an initial agentic workflow by augmenting the skeleton with necessary LLM nodes for planning and response synthesis. The Workflow Executor compiles candidate workflows into executable Python programs with observation hooks, and executes them on historical queries to collect optimization feedback (tool orchestration traces, execution results, and node-level observations) and validation feedback. The Workflow Optimizer uses Monte Carlo tree search (MCTS) guided by execution feedback to refine the workflow topology through four stages: selection, expansion, simulation, and backpropagation. The expansion is guided by an LLM-based optimizer that analyzes execution feedback and proposes refinement plans containing node-level edits, edge-level edits, control-flow edits, and prompt-level edits. The search terminates when the validation score does not improve for 5 iterations or the maximum of 30 iterations is reached.

The paper evaluates F LOW S COUT on four task domains from ToolBench (finance, sports, travel, weather) and compares it with three baselines: PM4Py, ReAct, and AFlow. The experimental results show that F LOW S COUT improves tool invocation correctness (TC) by at least 92.69% and execution score (ES) by at least 17.66% over the baselines. Specifically, F LOW S COUT achieves an average TC of 0.7515 and average ES of 0.7381 across all domains, compared to ReAct's 0.3900 TC and 0.6273 ES, and AFlow's 0.6172 ES. PM4Py performs markedly worse with 0.1047 TC and 0.1636 ES. F LOW S COUT also reduces the coefficient of variation of ES across repeated runs by 62.09% and 27.90% compared with ReAct and AFlow, respectively, demonstrating better execution stability.

For efficiency, F LOW S COUT finds its best workflow within 20 iterations across all domains, with validation scores improving by 92.20% to 109.47% over initial workflows. At runtime, F LOW S COUT requires 22.80 seconds per query on average, compared to 11.44 seconds for ReAct and 18.37 seconds for AFlow, representing at least 24.12% higher runtime cost for improved effectiveness.

The ablation study on the finance domain shows that removing MCTS (F LOW S COUT-N O MCTS) reduces TC by 30.15% and ES by 71.18%, while removing the workflow miner (F LOW S COUT-N O M INER) reduces TC by 24.27% and ES by 23.51%. Replacing MCTS with beam search (F LOW S COUT-B EAM) yields 2.81% higher TC but 27.51% lower ES, while taking 60.42% more time (38.88 hours vs. 15.39 hours).

The generalization evaluation shows that F LOW S COUT retains 85.98% of its original TC and 91.86% of its original ES when deployed with unseen tools, and retains 84.28% and 89.39% of its original TC with DeepSeek-V3.2 and Qwen3-32B respectively, while retaining 98.26% and 98.50% of its original ES. Overall, the workflows generated by F LOW S COUT retain 91.37% of their original TC and ES on average when deployed with different tools and LLM backbones.

Improvements to AI systems:

1. Execution-grounded workflow generation: Replace LLM-simulated tool calls with actual tool executions during workflow construction, ensuring generated workflows are verified against real API responses rather than hallucinated outcomes.

2. Graph-based workflow representation with explicit control flow: Model agentic behavior as directed graphs containing both LLM nodes and tool-calling nodes with dependency edges, enabling precise data routing and conditional branching that current linear or tree-based planners lack.

3. MCTS-driven iterative refinement from execution feedback: Use Monte Carlo tree search to explore workflow topology variants, where each expansion is informed by node-level observations (e.g., tool output mismatches, execution errors) rather than purely semantic reasoning.

4. Process-mining-based skeleton extraction: Automatically discover reusable tool coordination patterns from historical successful task logs using process mining (PM4Py), reducing the search space for new workflows and improving generalization to unseen queries.

5. Stability-aware optimization: Optimize not just average performance but also variance across repeated runs, producing workflows with lower coefficient of variation in execution scores, which is critical for production reliability.

6. Cross-domain and cross-backbone transferability: Design workflows that retain over 85% of tool-calling accuracy and 91% of execution scores when deployed with unseen tools or different LLMs (e.g., DeepSeek, Qwen), enabling portable agentic solutions.

What the improved AI system can do:

• Automatically generate and refine tool-integrated agentic workflows (e.g., for finance, travel, weather queries) that execute real API calls with 92.69% higher tool-calling correctness and 17.66% higher execution scores than ReAct or AFlow baselines.

• Adapt to new tools and LLM backbones with minimal performance loss (e.g., retaining 91.37% of original effectiveness on average), making it deployable across changing environments.

• Produce stable, reproducible agent behavior—reducing execution score variance by up to 62% compared to prior methods—while converging to optimal workflows within 20 iterations.

• Handle complex multi-step tasks requiring conditional logic, parallel tool calls, and iterative error correction, with runtime overhead of ~22.8 seconds per query, acceptable for interactive use.