Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?

arXiv:2610.12310 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?".

Jane: The gist The single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU).

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today, "Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?". It basically asks if just training a single multimodal large language model is enough to handle the whole job of finding, describing, and reasoning about defects in industrial settings.

Jane: The main point they’re making is that in-domain training doesn't close that gap between what models can do and what humans can do across all those different tasks.

Tom: That's right, so they look at a benchmark called MMAD, which covers seven different tasks for industrial anomaly understanding. And their finding is pretty sobering—trained specialists only reach at most seventy-five point five percent accuracy in defect localization on that benchmark, compared to ninety-two point three percent for actual human experts > <ref:2610.12310#pg1,reach at most 75.5% accuracy in defect localization>

Jane: And they also found that these trained specialists can sometimes detect anomalies less accurately than their untrained base models, which suggests the training isn't helping in the right way for everything >

Tom: Exactly. They point out that different multimodal models have complementary strengths, but they all share this weakness in fine-grained perception, so just combining them doesn't fix it >

Jane: So what they propose instead is something called SiGMA, which is a spatially grounded multi-agent framework that splits the work between different kinds of multimodal models and a dedicated visual defect expert >

Meng: That sounds like a lot of moving parts. How does this SiGMA actually work to solve the problem they identified in "Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?" >

Tom: Well, the SiGMA framework follows a search–reason–ground–aggregate workflow. First, there's a multimodal searcher that brings in industrial knowledge and normal references as context for all the agents >

Lu: That searcher is supposed to ground everything semantically while anchoring the normal appearance for both the agents and this defect expert >

Jane: Then you have these heterogeneous multimodal models, where each one independently produces a candidate prediction based on that context >

Meng: And then there’s this crucial part where a fine-grained defect expert steps in to localize query–reference discrepancies without relying on the MLLM's perception itself >

Tom: It goes from there to a calibration step, where that map is calibrated against normal residual statistics to suppress any unreliable responses >

Jane: And finally, there's this reliability controller that weighs each source based on its competence and the quality of the evidence it provides at query time >

Tom: So they’re using this combination of agents and expert to get a final answer by combining task-wise estimates with query-dependent evidence weighting >

Meng: It sounds like a very complex system, but what are the actual results they got when they tested this SiGMA framework on the MMAD benchmark? >

Jane: They found that SiGMA reaches an average accuracy of eighty-five point two percent on MMAD, which is four point zero percent above both JUDO and Gemini-two point five-Pro >

Tom: And they noted that its largest gains over the trained methods were in anomaly detection, where it was up by thirteen point eight percent, and defect localization was up by four point one percent >

Jane: That's a pretty big jump for those specific tasks, especially since they said composition lifts semantic tasks but doesn't improve perception >

Tom: They also found that adding the defect expert alone raised the average accuracy by two point zero percent, entirely through anomaly detection and defect localization, which is interesting because it bypasses the MLLM perception limitation > <ref:2610.12310#pg1>

Jane: And that reliability controller added another one point nine percent to the accuracy, with its biggest gains being in defect localization at six point zero percent and anomaly detection at two point one percent >

Tom: So they avoided updating any of the model parameters during this process, which is important for deployment because it means we don't have to retrain anything >

Lu: From a research perspective, what this implies is that you don't need one massive model to do everything; instead, dividing specialized tasks among different models and adding a dedicated expert can help when those models have different errors >

Meng: Practically speaking, if I were building something like this on my end, the inference cost scales with how many participating models there are >

Jane: Right, so while it adds complexity to the system architecture, it doesn't necessarily mean we need a single gigantic model to get good results for complex industrial tasks >

Tom: So what does this paper mean for us who just listen and aren't building the AI systems? It suggests that instead of relying on massive models getting better at everything through more training, a smarter architecture that divides labor might be the way forward >

Jane: It means we need to look at how we structure these multimodal AIs to intentionally separate perception from reasoning or description >

Tom: Exactly. The SiGMA framework shows that separating the task of identifying a physical defect from the general language understanding of an MLLM can actually improve performance in those specific areas >

Jane: It’s about moving away from hoping one giant model learns all these fine-grained details simultaneously, and instead building a system where different components handle what they do best >

Conclusion: Tom: So we're wrapping up our look at this paper, "Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?"

Jane: It really boils down to whether just teaching a single model more about industrial parts is enough to solve the whole problem of finding small defects and understanding them deeply.

Tom: Yeah, the authors are showing that in-domain training often leaves models uneven across all the different tasks they need to perform, like detection and localization.

Lu: It’s not just a matter of how much data you feed it; composition doesn't magically fix the perception weakness that most MLLMs share.

Meng: So if we look at the results, SiGMA is performing better than some trained methods, but they still trail human experts by a bit in those fine-grained visual tasks.

Lalam: The system they propose uses a lot of different models working together, and a separate expert that looks at the actual pixels instead of relying on just the language understanding.

Jane: It means we can't rely on one giant model getting better at everything; we have to build systems where different parts handle what they do best.

Tom: Exactly. The final accuracy they hit is around eighty-five point two percent, which is better than some of the specialized training methods they tested.

Lu: What’s really cool about their conclusion is that this whole SiGMA framework doesn't require any new model training to work; it just uses the models as frozen components.

Meng: That's a practical point for us because it means we don't have to spend months retraining massive models every time we want to adapt them for a new factory floor.

Jane: It suggests that instead of chasing bigger and bigger foundational models, focusing on how you structure the interaction between different specialized AI components is where the real gain is coming from.

Tom: It shows that dividing labor—giving one model a job for reasoning and another expert just to look at image patches—can actually boost performance in those tricky visual areas.

Lu: And they even show how the reliability controller estimates how trustworthy each model’s output is based on the evidence it sees at that exact moment.

Jane: So, the big picture here is moving away from hoping one monolithic AI learns all these tiny details at once, toward a more modular approach for industrial understanding.

Tom: And this leaves us wondering if this separation of concerns could be useful in other complex visual domains outside of just manufacturing defects.

Xingwu Zhang, Duanyang Du, Huiling Zhu, Jiayue Dai, Yixiao Liu, Guozhi Liu, Zhihan Zhang, Zijun Long

Hunan University · University of Aberdeen South China Normal University South China University of Technology

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 21 pages, 5 figures, 8 tables

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 90/100

The gist: The gist The single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding

Key concepts

MM-IAU: Multimodal Industrial Anomaly Understanding is the complex task of analyzing images from industrial settings to detect defects, pinpoint where they are, describe them, and reason about their potential impact. It involves multiple distinct capabilities that a single model often struggles to master simultaneously.
In-Domain Training Limitations: Specializing an MLLM using in-domain data helps it perform better on specific tasks. However, the paper finds this specialization is insufficient; models trained this way still lag behind human experts in critical perceptual skills like anomaly detection and defect localization, suggesting a fundamental limitation.
SiGMA Framework: SiGMA is a multi-agent system that divides labor. It uses several frozen MLLMs for semantic tasks while employing a specialized 'defect expert' that identifies physical defects by analyzing image differences directly, bypassing the MLLM's perception limitations.
Reliability Controller: This component dynamically weights the predictions from different models based on their competence and how much evidence they provide for a specific query. It ensures that the final answer is more trustworthy by prioritizing reliable sources of information.

Terminology

Accuracy — 38× in this paper · explained in 1 episode
Calibration — 29× in this paper · explained in 7 episodes
Aggregation — 17× in this paper
Benchmark — 10× in this paper · explained in 2 episodes
Adam — 6× in this paper · explained in 2 episodes
Ablation — 4× in this paper · explained in 1 episode
Conference on Computer Vision and Pattern Recognition — 3× in this paper
Grounding — 3× in this paper · explained in 3 episodes
Reinforcement Learning — 3× in this paper · explained in 4 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
GRPO — 2× in this paper · explained in 6 episodes
Multi-Agent Framework — 2× in this paper · explained in 3 episodes
Multimodal Reasoning — 2× in this paper · explained in 5 episodes
Abstention — 1× in this paper · explained in 1 episode
Active Learning — 1× in this paper · explained in 6 episodes
Adaptation — 1× in this paper · explained in 1 episode
Agentic Approaches — 1× in this paper · explained in 1 episode
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Frozen model — 1× in this paper · explained in 3 episodes
Generalization — 1× in this paper · explained in 13 episodes
Latent Reasoning — 1× in this paper · explained in 2 episodes
Memory Augmentation — 1× in this paper · explained in 3 episodes

Summary

The gist The single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU).

Problem Motivation

Multimodal industrial anomaly understanding (MM-IAU) requires detecting whether a product is defective, localizing the defect, describing its appearance, and reasoning about its potential impact (Jiang et al., 2025). Defects such as small scratches or missing components often occupy only a small fraction of the image (Bergmann et al., 2019; Zou et al., 2022). The prevailing remedy is to specialize a single MLLM through in-domain supervised fine-tuning or reinforcement learning (Chao et al., 2025; Zhao et al., 2025; Guan et al., 2025; Kang et al., 2016). Our analysis of 39,670 questions from MMAD (Jiang et al., 2016), a widely adopted MM-IAU benchmark covering seven tasks, suggests that it is not enough for in-domain training to close the gap (Table 1). Specialization improves particular capabilities but leaves models uneven across tasks (Jiang et al., 2025).

Limitations of In-Domain Training

Trained specialists trail human experts most in anomaly detection and defect localization, a perceptual weakness that different MLLMs share and that composing them does not remove (Table 1). The strongest trained method for defect localization reaches only 75.5% accuracy, compared to 92.3% for human experts (Jiang et al., 2025). Both OmniAD and JUDO detect anomalies less accurately than their untrained base model (68.8% and 65.0% vs. 70.5%) (Jiang et al., 2016). This suggests that in-domain training mainly improves domain-specific language, while perceptual limitations persist (Section 4.2). Furthermore, scale does not close the gap either, as the best defect-localization accuracy among large-scale MLLMs is 69.1% (GPT-5-mini), which is 10.5% below SiGMA (Table 1).

SiGMA Framework

SiGMA proposes a spatially grounded multi-agent framework that divides labor between heterogeneous MLLM agents and a dedicated visual defect expert (Section 1). The workflow follows a search–reason–ground–aggregate workflow (Figure 2) (Section 3.2).

A multimodal searcher retrieves industrial knowledge and normal references as shared context, grounding the heterogeneous agents semantically while anchoring normal appearance for both the agents and the defect expert (Section 3.4).
Heterogeneous MLLM agents independently produce candidate predictions, where agent m produces oim = fm(Ii, qi, A, di, ri) for m = 1..., N (Section 3.3).
A fine-grained defect expert localizes query–reference discrepancies without relying on MLLM perception (Section 3.4). This involves extracting patch-level features from the query image and the retrieved normal reference to construct a dense anomaly map Hi = hi(p)p∈omegai (Section 3.4).
The map is calibrated against normal residual statistics to suppress unreliable responses, yielding the calibrated anomaly map He i = n ehi(p) o p∈omegai (Equation 2) (Section 3.4).

Reliability Controller and Aggregation

A label-free reliability controller weighs each source by task-wise competence and query-level evidence quality (Section 1). This controller operates on the predictions of frozen models, estimating statistics from unlabeled predictions transductively (Appendix B) (Section 3.5).

Query-level spatial evidence determines expert reader admission: readers are queried only when bi > 0 and bi ≥ θi,area for anomaly detection (Section 3.5).
The controller estimates source competence by maximizing a regularized aggregation objective JD(C) (Equation 4) (Section 3.5).
Task-wise competence and query-level evidence weighting adapt the aggregation to complementary semantic and spatial predictions, resulting in the final answer yˆi = arg maxl∈Ai π final i(l) (Equation 9) (Section 3.5).

Key Findings

SiGMA reaches an average accuracy of 85.2% on MMAD, which is 4.0% above both JUDO and Gemini-2.5-Pro, with its largest gains over trained methods in anomaly detection (13.8% ↑) and defect localization (4.1% ↑) (Section 1). Composition lifts semantic tasks but not perception, while adding the defect expert raises average accuracy by 2.0%↑ entirely through anomaly detection and defect localization (Table 5). The reliability controller adds a further 1.9%↑, with its largest gains in defect localization (6.0%↑) and anomaly detection (2.1%↑) (Section 4.3). SiGMA avoids in-domain parameter updates, while its inference cost scales with the number of participating models (Table 7).

Conclusion

SiGMA pairs heterogeneous frozen MLLM agents, which improve the semantic tasks, with a defect expert whose calibrated evidence improves detection and localization (Section 5). It reaches 85.2% average accuracy, 4.0% above the strongest trained specialist and Gemini-2.5-Pro, and 84.4% with three agents of at most 9B parameters (Section 1). SiGMA requires no model training, and its source-reliability parameters can be estimated automatically from unlabeled predictions on new data (Section 5). The controller estimates source reliability from the unlabeled evaluation batch, making the full result transductive (Section 3.5). The final accuracy is achieved by combining task-wise estimation and query-dependent evidence weighting (Equation 9) (Section 3.5).

How it works

(See SiGMA Framework section above for detailed steps)

Key Components and Roles

Multimodal Searcher: Encodes the query image with a frozen CLIP model to retrieve industrial knowledge di and select a normal reference ri from a category-specific gallery (Section 3.2).
Heterogeneous MLLM Agents: These agents are retained as frozen agents that generate complementary candidate predictions, where agent m independently produces oim = fm(Ii, qi, A, di, ri) for m = 1..., N (Section 3.3).
Fine-grained Defect Expert: This expert localizes query–reference discrepancies without relying on MLLM perception by extracting patch-level features from the query image and the retrieved normal reference to construct a dense anomaly map Hi = hi(p)p∈omegai (Section 3.4).
Noise-robust Calibration: The map is normalized against normal residual statistics to establish an absolute anomaly scale, yielding the calibrated anomaly map He i = n ehi(p) o p∈omegai (Equation 2) (Section 3.4).
Expert Readers: These frozen agents interpret the spatially grounded evidence by relating localized deviations to semantic requirements, producing oie = ge(Ii, He i, qi, A) ∈ A (Section 3.4).
Reliability Controller: This controller accounts for task-wise competence and query-level evidence quality by estimating global and task-wise source competence and modulating reader influence using spatial evidence (Section 3.5).

Experimental Setup

(See Experimental Setup section above for details)

Evaluation Results Summary

Table 1 compares SiGMA with trained specialists, showing SiGMA improves most on the two perception-centric tasks, raising anomaly detection by 13.8%↑ and defect localization by 4.1%↑ over the best training-based methods (Section 4.2). The full ten-agent pool reaches 84.9%, which is 3.7%↑ above JUDO and Gemini-2.5-Pro (Table 1). The three-agent team reaches 85.2% accuracy at a latency of 1.5 s per question (Table 8).

Limitations

SiGMA still trails human experts by 12.6% in anomaly detection and 12.7% in defect localization, though by less than any trained method (Section 5). Aggregation cannot correct errors that every source shares, which SiGMA mitigates with a defect expert independent of MLLM perception (Section 5). SiGMA still trails human experts by 12.6% in anomaly detection and 12.7% in defect localization, though by less than any trained method (Table 2). The largest remaining gaps are anomaly detection (12.6%↓) and defect localization (12.7%↓), the capabilities that Section 1 identified as the shared weakness of MLLMs (Section 4.2).

Conclusion

Improvements for AI systems

Bold header: Division of labor between heterogeneous MLLM agents and a dedicated visual defect expert. This system can achieve an 85.2% average accuracy on MMAD, specifically by dividing labor such that the agents contribute complementary judgments across inspection tasks, while the defect expert supplies localized defect evidence through explicit comparison with normal references.
Bold header: Spatially grounded anomaly localization via calibrated residual maps. The system can generate a map where Local discrepancies between their patch features are used to construct a dense anomaly map Hi = ⌊hi(p)p∈omegai⌋, where omegai denotes the spatial domain of the map and hi(p) is the anomaly response at location p, anchoring localization to normal residual statistics rather than raw MLLM perception.
Bold header: Noise-robust calibration for reliable expert evidence injection. The system can suppress spurious responses on normal images by normalizing maps using category-matched normal reference and defining bounds such that Responses below θi,low are suppressed, whereas stronger deviations are retained with comparable intensity across queries.
Bold header: Task-wise competence and query-level evidence quality weighting. The reliability controller can adaptively determine influence by weighting sources based on their performance, as described by the formula that sets ρi = 1 + βti, where ti = 1/Vτ(i) Σj∈Vτ(i) 1[uj ≤ ui], Ei ≠ ∅,0, Ei = ∅.
Bold header: Adaptive expert reader admission based on spatial coherence. The system can selectively query expert readers only when "bi > 0 and bi ≥ θi,area," controlling the injection of localized evidence to maximize accuracy while avoiding reliance on potentially misleading MLLM outputs that respond spuriously on normal images.

Abstract

A single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU). We show that in-domain training does not close this gap. On MMAD, a widely adopted MM-IAU benchmark, trained specialists reach at most 75.5% accuracy in defect localization, against 92.3% for human experts, and even detect anomalies less accurately than their untrained base model. Meanwhile, different MLLMs offer complementary strengths but share this weakness in fine-grained perception, so combining them alone cannot remove it. We therefore propose SiGMA, a spatially grounded multi-agent framework that divides labor between heterogeneous MLLM agents and a dedicated visual defect expert. A multimodal searcher supplies industrial knowledge and normal references, the defect expert turns query-reference comparison into calibrated anomaly evidence, and a label-free reliability controller weighs each source by task-wise competence and query-level evidence quality. SiGMA reaches 85.2% average accuracy on MMAD, 4.0% above the strongest trained specialist and Gemini-2.5-Pro and within 1.5% of human experts. Even with three agents of at most 9B parameters, it reaches 84.4%, and new MLLMs join without retraining.