AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning

arXiv:2604.07900 · cs.CV, cs.AI · Submitted 2026-04-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning".

Jane: The gist Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled "AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning," and the core idea is this agent uses self-reflection and knowledge retrieval to make its anomaly generation process better over time.

Jane: They equip it with five specific tools to work with: Prompt Generation, Image Generation, Quality Evaluation, Knowledge Retrieval, and Mask Generation.

Lu: The crucial part here is that the agent doesn't just follow a fixed path; it decides which tool to call next based on the feedback it gets while generating an image I prime and its corresponding mask M.

Meng: So instead of one big generation step, it’s a loop where the agent checks its work, gets feedback, and then uses that feedback to decide what its next action should be? That sounds like something that would actually be useful for industrial applications.

Lalam: Right, and they build this whole process by constructing structured trajectories from real anomaly images first. This is how the agent learns the correct sequence of tool calls without needing someone to manually annotate every single step.

The paper's summary: Tom: The authors show that this approach lets AnomalyAgent generate more challenging and informative signals for tasks like anomaly detection because it’s so much better than previous zero-shot methods on the MVTec-AD benchmark.

Jane: They claim performance exceeding previous zero-shot methods on that same benchmark.

Lu: Specifically, they report achieving an IS/IC-L of two point one zero/zero point three three for anomaly generation and a fifty-seven percent classification accuracy using ResNet34 on that dataset, which beats all other zero-shot methods they tested.

Meng: From a practical standpoint, they also show efficiency gains that are important for deployment; they report a time per good sample of one hundred seventeen point two seconds and the cost per good sample comes out to zero point zero five six seven.

Lalam: And their ablation studies show that adding the supervised fine-tuning stage stabilizes tool-use behavior and gives about fifty-two point four percent improvement, but the reinforcement learning part provides the biggest boost to accuracy, getting it up to fifty-seven point zero percent.

The paper's improvements: Tom: To sum this up, AnomalyAgent proposes this new agentic framework that treats anomaly synthesis as a multi-turn decision-making process with structured tool use and iterative reasoning. It’s about making the generation controllable and semantically consistent beyond just a single step.

Jane: The training strategy, moving from supervised fine-tuning on constructed trajectories to reinforcement learning with verifiable rewards for long-range decision making, is what really makes this approach different from older methods.

Lu: This setup allows the agent to adapt its generation strategy based on environmental feedback during the reinforcement learning phase, which significantly improves the realism and diversity of the samples it creates.

Meng: It’s a solid method because it shows that this kind of iterative optimization through reinforcement learning can produce results that are actually useful for downstream anomaly detection tasks.

Lalam: The paper's main limitation they point out is that they focus heavily on this specific industrial anomaly synthesis task, so we don't necessarily know how well it generalizes to completely different types of data without more training.

Conclusion: Tom: So we’ve looked at AnomalyAgent, which is Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning, and basically this is an agent that learns how to create industrial anomalies by using a set of tools and then getting better through some smart reinforcement learning.

Jane: Right, and the core idea is that it’s not just one generation step anymore; it’s a whole loop where the agent checks its work, gets feedback on quality, and then uses that feedback to refine what it does next.

Lu: They built this sequential decision-making process using five tools: Prompt Generation, Image Generation, Quality Evaluation, Knowledge Retrieval and Mask Generation. It lets the agent dynamically decide which tool to call based on how well the image I prime and its mask M look along the way.

Meng: So instead of just guessing what an anomaly looks like from scratch, this AI is reasoning about the problem and fixing its own mistakes through that loop. That sounds genuinely practical for real-world industrial problems.

Lalam: And to get that iterative refinement going, they construct these structured trajectories from real anomaly images first. That’s how the agent learns the right sequence of tool calls without needing someone to manually annotate every single step.

Tom: They show that this approach lets AnomalyAgent generate more challenging and informative signals for tasks like anomaly detection because it’s so much better than previous zero-shot methods on the MVTec-AD benchmark.

Jane: They claim performance exceeding previous zero-shot methods, achieving an IS/IC-L of two point one zero/zero point three three for anomaly generation and fifty-seven percent classification accuracy using ResNet34 on that dataset.

Lu: They also reported efficiency gains that are important for deployment because the time per good sample is one hundred seventeen point two seconds and the cost per good sample comes out to zero point zero five six seven.

Meng: That efficiency data is key; it shows this isn't just a clever experiment, it’s something that can actually run on a practical timeline without breaking the bank.

Tom: So to wrap this up AnomalyAgent proposes this new agentic framework that treats anomaly synthesis as a multi-turn decision-making process with structured tool use and iterative reasoning. It’s about making the generation controllable and semantically consistent beyond just a single step.

Lu: It allows the agent to adapt its generation strategy based on environmental feedback during the reinforcement learning phase, which significantly improves realism and diversity of the samples it creates.

Tom: Exactly, and that’s where the next piece comes in. We’ve seen how AnomalyAgent uses these tools to create high-quality samples; now we need to see if this agent framework can handle more complex industrial scenarios down the line.

Shanghai Jiao Tong University · Tongji University · Fudan University

cs.CV, cs.AI

Submitted: 2026-04-09

Updated: 2026-10-09

Comments: 23 pages, 10 figures. Accepted at the ACM International Conference on Multimedia (ACM MM) 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks.

Key concepts

AnomalyAgent: A novel agent framework that synthesizes industrial anomalies. It functions as an intelligent system capable of sequential decision-making, using specialized tools and self-reflection to iteratively refine image generation prompts and masks for highly realistic defect synthesis.
Tool-Augmented Synthesis: The process where the agent uses five distinct tools—Prompt Generation (PG), Image Generation (IG), Quality Evaluation (QE), Knowledge Retrieval (KR), and Mask Generation (MG)—to solve the anomaly synthesis problem. This allows for a closed-loop optimization cycle where the agent can dynamically choose which tool to use next based on its current progress.
Two-Stage Training Framework: A training strategy consisting of supervised fine-tuning (SFT) and reinforcement learning (RL). SFT teaches the model the basic steps of using the tools by learning from structured trajectories. RL then optimizes the agent's decision-making strategy through long-horizon rewards, ensuring it learns to make better, more complex choices during synthesis.
Three-Part Reward Mechanism: A unified reward function combining three components: task rewards (for final image quality), reflection rewards (to encourage prompt improvement across turns), and behavioral rewards (to ensure the agent follows the intended decision path). This system guides the agent toward generating not just a good anomaly, but one that is iteratively refined and adheres to logical synthesis steps.

Terminology

Reinforcement Learning — 2× in this paper · explained in 4 episodes
Accuracy — 1× in this paper · explained in 1 episode
Closed-Loop Optimization — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Iterative Refinement — 1× in this paper · explained in 3 episodes
Mask Generation — 1× in this paper · explained in 2 episodes
Two-stage training — 1× in this paper · explained in 2 episodes

Summary

The gist Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks.

AnomalyAgent Overview

AnomalyAgent is an anomaly synthesis agent with self-reflection, knowledge retrieval, and iterative refinement capabilities, aiming to generate realistic and diverse anomalies. Specifically, it is equipped with five tools: Prompt Generation (PG), Image Generation (IG), Quality Evaluation (QE), Knowledge Retrieval (KR), and Mask Generation (MG), enabling closed-loop optimization. To improve decision-making and self-reflection, it constructs structured trajectories from real anomaly images and designs a two-stage training framework: supervised fine-tuning followed by reinforcement learning. This process is driven by a three-part reward mechanism: (1) task rewards to supervise the quality and location rationality of generated anomalies; (2) reflection rewards to train the model’s ability to improve anomaly synthesis prompt; (3) behavioral rewards to ensure adherence to the trajectory.

Agentic Framework and Trajectory Construction

AnomalyAgent reformulates anomaly synthesis as a sequential decision-making task, equipping the MLLM with five tools. The process follows an iterative loop where the agent dynamically coordinates multiple tools to produce a high-quality anomaly image I' along with a corresponding anomaly mask M. The sequence involves:

Prompt Generation (PG) to generate an initial textual prompt.
Image Generation (IG) conditioned on the original image and the generated prompt to synthesize an anomaly image I'.
Quality Evaluation (QE) to analyze both the original and generated images, outputting a quality score s together with textual feedback f.
Knowledge Retrieval (KR) to retrieve textual descriptions of typical defects given the category and anomaly type, providing semantic guidance for prompt refinement.
Mask Generation (MG) to generate a pixel-level anomaly mask M by comparing the original image I with the synthesized anomaly I'.

Training Strategy

The training proceeds in two complementary stages. The first stage involves cold-start supervised fine-tuning (SFT) on the constructed trajectories, enabling the model to master the format and basic strategies of tool invocations. Trajectories are constructed using a reverse synthesis strategy, starting from each anomaly image Iano to reconstruct its defect-free counterpart Inormal, followed by an N-step paradigm where intermediate images I(t)ano are generated to simulate imperfect attempts. The second stage introduces Reinforcement Learning with Verifiable Rewards (RLVR), which optimizes the agent’s decision-making strategy and self-reflection mechanism through trajectory-level reinforcement learning.

Reward Mechanism

The unified reward function R = αRtask + βRref + γRbeh models three critical capabilities.

Task Reward (Rt) evaluates realism and industrial plausibility using LLM-as-a-Judge, where Rt = Sfinal, a scalar score measuring both defect localization rationality and visual fidelity of the final output image.
Reflection Reward (Rref) encourages iterative refinement by rewarding improvements across consecutive generation turns: Rref = ∑M m=1 max(0, Sm − Sm−1), where Sm denotes the quality score of the m-th synthesized anomaly.
Behavior Reward (Rbeh) unifies tool correctness, format validity, and efficiency into a single objective. This includes a penalty term Φ(at−1, at) that enforces valid tool transition rules, and a knowledge-driven term weighted by λKR that encourages invoking KR when the current synthesis quality S t falls below a threshold δ.

Performance and Results

AnomalyAgent achieves performance exceeding previous zeroshot methods on the MVTec-AD benchmark, demonstrating that Agentic RL-driven anomaly synthesis can generate more challenging and informative supervisory signals for downstream tasks. On the MVTec-AD dataset, AnomalyAgent achieves an IS/IC-L of 2.10/0.33 for anomaly generation, 57.0% classification accuracy using ResNet34, and 99.3%/74.2% AP at the image/pixel level using a simple UNet. This surpasses all zero-shot SOTA methods. The method achieves the best IS and IC-L scores on the MVTec dataset, surpassing previous zero-shot SOTA methods.

Ablation and Efficiency

Ablation studies show that starting from a minimal setup, progressively enabling key modules consistently improves performance. Specifically, adding SFT stabilizes tool-use behavior and yields consistent gains (52.4%), while RL provides the largest improvement, boosting accuracy to 57.0% by enabling iterative optimization and long-horizon decision-making. The three rewards are complementary, jointly promoting high-quality generation, progressive improvement, and disciplined decision-making. AnomalyAgent achieves the lowest time per good sample of 117.2s and the lowest cost per good sample at 0.0567, indicating that our approach dominates all baselines in terms of practical efficiency.

Conclusion

In this paper, we propose AnomalyAgent, a novel agent-based framework that formulates industrial anomaly synthesis as a multi-turn decision-making process. By integrating structured tool use with iterative reasoning, the proposed method enables controllable and semantically consistent anomaly generation beyond conventional single-step pipelines. We further design a two-stage training strategy: SFT based on the constructed synthetic trajectory, followed by optimization of long-range decision-making capabilities through RL. Extensive experiments demonstrate that AnomalyAgent outperforms existing zero-shot methods in both anomaly generation quality and downstream tasks. Future work will explore scaling to more complex industrial scenarios and extending the agent framework to broader multimodal generation tasks.

--- Page 1 ---

The gist Industrial anomaly generation is a crucial method for alleviating the data scarcity problem in anomaly detection tasks.

How it works

Training Strategy

The training proceeds in two complementary stages.

Improvements for AI systems

Agentic framework for industrial anomaly synthesis: AnomalyAgent can generate realistic and diverse anomalies by employing a closed-loop optimization mechanism through its structured trajectories and iterative refinement capabilities.
Tool-augmented reasoning: The system moves beyond single-step generation to perform self-reflection, knowledge retrieval, and iterative refinement, enabling it to construct a perception-reflection-action closed loop.
Two-stage training pipeline: The combination of cold-start supervised fine-tuning (SFT) on constructed trajectories followed by Reinforcement Learning with Verifiable Rewards (RLVR) optimizes the agent’s decision-making strategy and self-reflection mechanism for superior performance.
Multi-component reward design: The unified reward function, combining task, reflection, and behavioral rewards (R = αRtask + betaRref +gammaRbeh), explicitly models quality assessment, iterative improvement, and disciplined decision-making.
Scalable trajectory construction: The method introduces a trajectory construction strategy based on real anomaly images, which allows for scalable training without requiring additional manual annotation.