Co-Evolutionary Prompt Optimization with Cross-Category Transfer for Zero-Shot Anomaly Detection

arXiv:2608.29467 · cs.CV · Submitted 2026-08-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Co-Evolutionary Prompt Optimization with Cross-Category Transfer for Zero-Shot Anomaly Detection".

Jane: Here's the gist of the paper. Zero-shot anomaly detection (ZSAD) has gained significant attention for its practical value in industrial inspection The proposed framework achieves state-of-the-art performance across multiple anomaly detection datasets Framework Introduction CoEvoAD is a co-evolutionary framework for discrete prompt selection that performs prompt search in the discrete natural-language space using an evolutionary algorithm It preserves the interpretability and composability of natural language by iteratively generating, evaluating, and selecting candidate prompts throughout population evolution Furthermore, it introduces a Cross-Category Transfer Objective (CCTO) which treats held-out source categories as proxies for unseen categories to score prompt rules based on their estimated cross-category transferability Discrete Prompt Optimization CoEvoAD formulates prompt optimization as a discrete search over natural-language prompt rules rather than relying on opaque blackbox vector representations It maintains separate populations, Pn and Pa, for normal and abnormal roles, updating them through role-preserving mutation while keeping the role type fixed to prevent mixing descriptions during co-evolution The fitness function combines the evaluation score on source data with the semantic contrast between normal and abnormal descriptions to enable a transparent optimization process without gradients Cross-Category Transfer Objective (CCTO) The key idea behind CCTO is to construct systematic held-out splits within the source domain, where a subset of source categories is treated as proxy unseen categories This design enables the evaluation of candidate prompt rules under a simulated zero-shot transfer scenario, allowing assessment of their consistency and robustness when applied across categories The roleadjusted selection scores are defined as ψn(p, c) = (1 − λn)ηn + λnχn, ψa(p, c) = ηa + λa (ηa − χa), where the normal branch interpolates between in-category performance and cross-category consistency Experimental Validation Extensive experiments validate the effectiveness of CoEvoAD, showing it achieves state-of-the-art performance across multiple anomaly detection datasets The framework shows that interpretable prompt search is feasible for ZSAD via role-separated co-evolution guided by held-out cross-category signals Table 14 demonstrates that CoEvoAD consistently improves both transfer directions, suggesting the gain is not explained by candidate budget alone The framework's inference latency is competitive, running only under 0.7 ms difference compared to the class-name control CoEvoAD attains the highest mean image AUROC/AP (95.0/96.2) and pixel AUROC/PRO (96.9/92.5) among compared CLIP-based ZSAD baselines The benefits of CoEvoAD concentrate on pixellevel localization rather than image-level AUROC, and the gains are not uniform across target categories The selected rules remain auditable as natural-language strings, complementing continuous prompt embeddings References Institute of Information Science, Beijing Jiaotong University, Beijing, China Extensive experiments are conducted to validate the effectiveness of CoEvoAD, and the results show that it achieves state-of-the-art performance across multiple anomaly detection datasets.

Tom: The code is available at https://github.com/rstao-bjtu/CoEvoAD.

Jane: To this end, we propose CoEvoAD, a co-evolutionary framework for discrete prompt selection CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm.

Tom: Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language.

Paper summary: Jane: So, wrapping up the discussion on this paper, what’s the real takeaway for someone just listening who isn't deep in the math? What does "Co-Evolutionary Prompt Optimization with Cross-Category Transfer for Zero-Shot Anomaly Detection" actually mean in simple terms?

Tom: Well, it means they found a way to make prompt selection interpretable and scalable by searching through natural language rules instead of using complex vector embeddings. The key is using that cross-category transfer objective to ensure the prompts they select aren't just good for the categories they trained on, but actually work when faced with new ones.

Lu: It’s about building a system where you can audit exactly which natural language rules are being used, and that system is robust because it has been tested on simulated zero-shot transfers using those held-out source categories.

Meng: From my side, the fact that they keep the prompt search discrete means it’s more manageable for deployment than if we were trying to optimize a massive continuous embedding space. It gives us something auditable at scale.

Jane: And the authors claim they achieved state-of-the-art performance across multiple anomaly detection datasets, specifically mentioning a mean image AUROC of nine point five and pixel AUROC of nine point six point two. So, they’re showing solid results for this zero-shot capability.

Tom: But I do have to mention the caveat: the benefits are concentrated on pixellevel localization rather than just overall image-level AUROC, and those gains aren't uniform across target categories; for example, the MVTecAD to VisA direction showed a worst category pixel AP regression on pcb4.

Lu: That’s a fair point about uneven gains. It suggests that while the framework is powerful for finding good rules, the quality of those rules still depends on which specific target category you are looking at.

Jane: So, in short, they offer a way to get interpretable prompts that generalize better across different categories than existing methods do, even if the performance bump isn't perfectly even everywhere. That’s what this paper is about.

Conclusion: Tom: So, we're wrapping up this look at CoEvoAD, which is Co-Evolutionary Prompt Optimization with Cross-Category Transfer for Zero-Shot Anomaly Detection.

Jane: To start off, what’s the actual big idea of that title? It sounds pretty dense.

Lu: It’s about using an evolutionary algorithm to pick the best natural language prompts for anomaly detection, and they add this cross-category transfer objective to make sure those prompts work even when seeing something totally new.

Meng: So, we’re talking about searching through language rules instead of just tweaking the numbers in a vector space, right?

Tom: Exactly. They’re keeping the search discrete so it stays interpretable and you can actually read what the AI is doing, which is huge for us when we need to debug something.

Jane: But what does that cross-category transfer thing actually *do* for the detector?

Lu: It treats held-out categories as if they were unseen ones, so it scores the prompt rules based on how well they might generalize across different kinds of anomalies.

Tom: The results show state-of-the-art performance across several datasets, hitting image AUROC numbers around nine point five and pixel AUROC around nine point six.

Jane: Those are high numbers for zero-shot detection, but what’s the catch?

Lu: The authors admit that while the gains are good for pixellevel localization—finding where exactly something is—the improvements aren't uniform across every single target category they tested.

Meng: That means if you're focusing on one specific type of anomaly, you might see a bigger jump than someone else.

Tom: Right. And the rules themselves are still just natural language strings, which means we can actually look at them and understand why the AI picked that particular phrase.

Jane: So it gives us a way to build models that are both powerful for finding anomalies and transparent about how they arrived at those findings.

Lu: It’s a solid framework for showing how you can optimize discrete choices while ensuring those choices aren't just memorizing the training data.

Institute of Information Science, Beijing Jiaotong University · Institute of Automation, Chinese Academy of Sciences

cs.CV

Submitted: 2026-08-29

Updated: 2026-11-11

Comments: 25 pages, 25 figures. Accepted to EMNLP 2026 Main Conference. v2: removes an inaccurate description in Sec. 3.3 and corrects the VisA image AP in Table 1 (from 89.7 to 89.8); all other results are unchanged

Code: https://github.com/rstao-bjtu/CoEvoAD

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: Zero-shot anomaly detection (ZSAD) has gained significant attention for its practical value in industrial inspection The proposed framework achieves state-of-the-art performance across multiple…

Key concepts

CoEvoAD
A co-evolutionary framework that uses an evolutionary algorithm to search for the best discrete natural-language prompts. It iteratively generates, evaluates, and selects candidate prompts while keeping the roles (normal/abnormal) separate to maintain interpretability.
Discrete Prompt Optimization
This method treats prompt optimization as a discrete search over natural-language rules instead of using complex vector representations. It maintains distinct populations for normal and abnormal roles, updating them through role-preserving mutation without mixing descriptions.
Cross-Category Transfer Objective (CCTO)
A criterion designed to improve cross-category generalization by treating held-out source categories as proxies for unseen categories. It scores prompt rules based on their estimated transferability between these related categories, simulating zero-shot transfer.
Role Adjustment Scores
Mathematical functions used to define the selection score for prompts. They blend in-category performance (normal branch) with cross-category consistency (abnormal branch), allowing the framework to intelligently select rules that are both accurate and transferable.

Terminology

Summary

Zero-shot anomaly detection (ZSAD) has gained significant attention for its practical value in industrial inspection The proposed framework achieves state-of-the-art performance across multiple anomaly detection datasets

Framework Introduction

CoEvoAD is a co-evolutionary framework for discrete prompt selection that performs prompt search in the discrete natural-language space using an evolutionary algorithm It preserves the interpretability and composability of natural language by iteratively generating, evaluating, and selecting candidate prompts throughout population evolution Furthermore, it introduces a Cross-Category Transfer Objective (CCTO) which treats held-out source categories as proxies for unseen categories to score prompt rules based on their estimated cross-category transferability

Discrete Prompt Optimization

CoEvoAD formulates prompt optimization as a discrete search over natural-language prompt rules rather than relying on opaque blackbox vector representations It maintains separate populations, Pn and Pa, for normal and abnormal roles, updating them through role-preserving mutation while keeping the role type fixed to prevent mixing descriptions during co-evolution The fitness function combines the evaluation score on source data with the semantic contrast between normal and abnormal descriptions to enable a transparent optimization process without gradients

Cross-Category Transfer Objective (CCTO)

The key idea behind CCTO is to construct systematic held-out splits within the source domain, where a subset of source categories is treated as proxy unseen categories This design enables the evaluation of candidate prompt rules under a simulated zero-shot transfer scenario, allowing assessment of their consistency and robustness when applied across categories The roleadjusted selection scores are defined as ψn(p, c) = (1 − λn)ηn + λnχn, ψa(p, c) = ηa + λa (ηa − χa), where the normal branch interpolates between in-category performance and cross-category consistency

Experimental Validation

Extensive experiments validate the effectiveness of CoEvoAD, showing it achieves state-of-the-art performance across multiple anomaly detection datasets The framework shows that interpretable prompt search is feasible for ZSAD via role-separated co-evolution guided by held-out cross-category signals Table 14 demonstrates that CoEvoAD consistently improves both transfer directions, suggesting the gain is not explained by candidate budget alone The framework's inference latency is competitive, running only under 0.7 ms difference compared to the class-name control

Conclusion and Limitations

CoEvoAD attains the highest mean image AUROC/AP (95.0/96.2) and pixel AUROC/PRO (96. p. 5

Where the paper says it

9 – – 95. 2 Table 2: Pixel-level benchmark comparison (%; AUROC and Per-Region Overlap (PRO)).

9/92.5) among compared CLIP-based ZSAD baselines The benefits of CoEvoAD concentrate on pixellevel localization rather than image-level AUROC, and the gains are not uniform across target categories The selected rules remain auditable as natural-language strings, complementing continuous prompt embeddings

References

Institute of Information Science, Beijing Jiaotong University, Beijing, China

Extensive experiments are conducted to validate the effectiveness of CoEvoAD, and the results show that it achieves state-of-the-art performance across multiple anomaly detection datasets. The code is available at https://github. p. 0

Where the paper says it

Extensive experiments are conducted to validate the effectiveness of CoEvoAD, and the results show that it achieves state-of-the-art performance across multiple anomaly detection datasets. The code is available at https:// github.

com/rstao-bjtu/CoEvoAD. p. 0
Where the paper says it

The code is available at https:// github. com/rstao-bjtu/CoEvoAD.

To this end, we propose CoEvoAD, a co-evolutionary framework for discrete prompt selection CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm. p. 0

Where the paper says it

To this end, we propose CoEvoAD, a co-evolutionary framework for discrete prompt selection. CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm.

Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language. p. 0
Where the paper says it

CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm. Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language.

Furthermore, we introduce a Cross-Category Transfer Objective (CCTO), which treats held-out source categories as proxies for unseen categories and scores prompt rules based on their estimated cross-category transferability, effectively improving cross-category generalization. p. 0
Where the paper says it

Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language. Furthermore, we introduce a Cross-Category Transfer Objective (CCTO), which treats held-out source categories as proxies for unseen categories and scores prompt rules based on their estimated cross-category transferability, effectively improving cross-category generalization.

CoEvoAD treats cross-category ZSAD as discrete prompt-rule selection, as shown in Figure 2. p. 2

Where the paper says it

3 Method 3. 1 Overview CoEvoAD treats cross-category ZSAD as discrete prompt-rule selection, as shown in Figure 2.

The prompt-conditioned scorer is kept fixed during rule search, so the optimization only changes the natural-language rule strings used to render prompts. p. 2
Where the paper says it

1 Overview CoEvoAD treats cross-category ZSAD as discrete prompt-rule selection, as shown in Figure 2. The prompt-conditioned scorer is kept fixed during rule search, so the optimization only changes the natural-language rule strings used to render prompts.

We maintain two separate populations, Pn = (p n1,. p. 3

Where the paper says it

Candidates are generated through role-preserving mutation and selected by the paired fitness function and CCTO, while keeping the role type fixed prevents normal and abnormal descriptions from being mixed during co-evolution. We maintain two separate populations, P n = {p n 1 , .

.., p nN), Pa = (p a1,..., p aN), where N is the population size and each p r i is a roletyped rule. p. 3
Where the paper says it

, pn N }, P a = {p a 1 , . , pa N }, (2) where N is the population size and each p r i is a roletyped rule.

Initial populations are constructed from role-specific templates and descriptor pools; since each candidate is an explicit natural-language rule, the same rule can be instantiated for another category by substituting the class-name placeholder. p. 3
Where the paper says it

, pa N }, (2) where N is the population size and each p r i is a roletyped rule. Initial populations are constructed from role-specific templates and descriptor pools; since each candidate is an explicit natural-language rule, the same rule can be instantiated for another category by substituting the class-name placeholder.

To address the cross-category generalization issue, we introduce the Cross-Category Transfer Objective (CCTO), a prompt-rule selection criterion specifically designed for cross-category transfer. p. 1

Where the paper says it

Overall, the framework transforms prompt-rule search from a single black-box vector into a structured, multirole, multi-candidate co-evolutionary system, effectively improving interpretability and scalability. To address the cross-category generalization issue, we introduce the Cross-Category Transfer Objective (CCTO), a prompt-rule selection criterion specifically designed for cross-category transfer.

The key idea behind CCTO is to construct systematic held-out splits within the source domain, where a subset of source categories is treated as proxy unseen categories. p. 1
Where the paper says it

To address the cross-category generalization issue, we introduce the Cross-Category Transfer Objective (CCTO), a prompt-rule selection criterion specifically designed for cross-category transfer. The key idea behind CCTO is to construct systematic held-out splits within the source domain, where a subset of source categories is treated as proxy unseen categories.

This design enables the evaluation of candidate prompt rules under a simulated zero-shot transfer scenario, allowing us to assess their consistency and robustness when applied across categories. p. 1
Where the paper says it

The key idea behind CCTO is to construct systematic held-out splits within the source domain, where a subset of source categories is treated as proxy unseen categories. This design enables the evaluation of candidate prompt rules under a simulated zero-shot transfer scenario, allowing us to assess their consistency and robustness when applied across categories.

CoEvoAD runs the same frozen scorer call as the class-name control, differing only in the prompt string: the two differ by under 0. p. 8

Where the paper says it

Table 8 compares per-image inference latency under one implementation-level protocol: single RTX 4090, batch size 1, 30 warmup followed by 300 timed images per direction, with the timer wrapping the entire per-image call. CoEvoAD runs the same frozen scorer call as the class-name control, differing only in the prompt string: the two differ by under 0.

7 ms with opposite signs across the two directions, so no systematic overhead is attributable to the evolved rules; the routing lookup, measured separately, adds 0. p. 8
Where the paper says it

CoEvoAD runs the same frozen scorer call as the class-name control, differing only in the prompt string: the two differ by under 0. 7 ms with opposite signs across the two directions, so no systematic overhead is attributable to the evolved rules; the routing lookup, measured separately, adds 0.

19/0.27 ms per image (under 0. p. 5
Where the paper says it

Evaluation metrics. Following common ZSAD evaluation, we report image-level AUROC and AP and pixel-level AUROC and Per-Region Overlap (PRO) for benchmark comparison.

5%). CoEvoAD’s inference is 1. p. 2
Where the paper says it

31 cls_N × Semantic / Template route 𝑒𝑡 Global & Patch Features Evaluate rule pairs Role-preserving mutation Selected rule pair for a source category 𝒑𝒏,⋆, 𝒑 Select Elite 𝒂,⋆ Trained Scorer ❄ Figure 2: Overview of CoEvoAD. A frozen source-trained scorer evaluates role-separated normal and abnormal rule populations; CCTO uses held-out source categories to select rules for unseen-category inference.

4–4.0× faster than AnomalyCLIP, MRAD, AdaCLIP, and Bayes-PFL retimed under the same protocol. p. 8
Where the paper says it

CoEvoAD’s inference is 1. 0× faster than AnomalyCLIP, MRAD, AdaCLIP, and Bayes-PFL retimed under the same protocol.

The benefits of CoEvoAD concentrate on pixellevel localization rather than image-level AUROC: matched-control gains in image AUROC are modest and not uniformly positive, and CCTO gains vary across target datasets. p. 8

Where the paper says it

The saved rules remain auditable as natural-language strings, complementing continuous prompt embeddings. Limitations The benefits of CoEvoAD concentrate on pixellevel localization rather than image-level AUROC: matched-control gains in image AUROC are modest and not uniformly positive, and CCTO gains vary across target datasets.

The gains are not uniform across target categories: the MVTecAD→VisA direction shows a worst-category pixelAP regression on pcb4 (Table 25). p. 8
Where the paper says it

Limitations The benefits of CoEvoAD concentrate on pixellevel localization rather than image-level AUROC: matched-control gains in image AUROC are modest and not uniformly positive, and CCTO gains vary across target datasets. The gains are not uniform across target categories: the MVTecAD→VisA direction shows a worst-category pixelAP regression on pcb4 (Table 25).

Table 10 lists the full implementation and hyperparameters. p. 8

Where the paper says it

Inference latency. Table 8 compares per-image inference latency under one implementation-level protocol: single RTX 4090, batch size 1, 30 warmup followed by 300 timed images per direction, with the timer wrapping the entire per-image call.

Component Setting Backbone CLIP ViT-L/14@336px Input resolution 518 × 518 Stage 1 epochs / optimiser 30 / AdamW, cosine annealing Stage 1 learning rates prompt 1×10−3, other 1×10−4 Prompt groups M / context Lc / state Ls 3 / 5 / 5 Search population N = 16 Search generations G = 5 Search elite count Ke = 4 Search pairing count K = 3 Rule-search evaluation split source test partition Rule-search score weights (img AUROC / px AP / px F1)

Table 29: Evolved prompt rules per VisA source category, for the VisA→MVTec-AD direction.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. CoEvoAD improves prompt search by performing prompt search in the discrete natural-language space using an evolutionary algorithm instead of relying on opaque blackbox vector representations, which enhances interpretability and composability of natural language rules.

  2. The Cross-Category Transfer Objective (CCTO) allows the system to score prompt rules based on their estimated cross-category transferability, effectively improving cross-category generalization by treating held-out source categories as proxies for unseen categories.

  3. The system can generate interpretable, human-readable rule strings, such as a photo of a damaged pcb4 or faulty flawed deteriorated pcb4, which are stored for auditability and can be instantiated with any category name via the class-name slot.

  4. The framework enables role separation by maintaining independent populations for the two roles (normal and abnormal), ensuring that normal descriptions remain focused on acceptable appearance while abnormal descriptions focus on defect evidence.

  5. The system can perform inference for an unseen target category by using a prespecified rule-transfer policy, selecting a stored pair based only on category names, and rendering the rules to produce anomaly scores.

Abstract

Zero-shot anomaly detection (ZSAD) has gained significant attention for its practical value in industrial inspection. Recently, CLIP-based approaches have been widely adopted in ZSAD due to their strong vision-language generalization capabilities. However, existing methods commonly employ continuous prompt embeddings for prompt optimization and encode semantics in latent vectors, which lack interpretability and scalability. To this end, we propose CoEvoAD, a co-evolutionary framework for discrete prompt selection. CoEvoAD performs prompt search in the discrete natural-language space using an evolutionary algorithm. Candidate prompts are iteratively generated, evaluated, and selected throughout population evolution, thus preserving the interpretability and composability of natural language. Furthermore, we introduce a Cross-Category Transfer Objective (CCTO), which treats held-out source categories as proxies for unseen categories and scores prompt rules based on their estimated cross-category transferability, effectively improving cross-category generalization. Extensive experiments are conducted to validate the effectiveness of CoEvoAD, and the results show that it achieves state-of-the-art performance across multiple anomaly detection datasets. The code is available at https://github.com/rstao-bjtu/CoEvoAD.

Related papers