Testing concept-level audits of melanoma classifiers with diffusion counterfactuals and matched positive controls
cs.CV, cs.AI, cs.LG
Submitted: 2026-11-01
Updated: 2026-11-04
Comments: 25 pages, 7 figures, 9 tables. Under review at Scientific Reports. Code and data at https://github.com/srahman7900/melanoma-concept-audit
Code: https://github.com/srahman7900/melanoma-concept-audit
License: http://creativecommons.org/licenses/by/4.0/
The gist: Concept-based explanations of melanoma classifiers are rarely tested against controls with known targets.
Terminology
Abstract
Concept-based explanations of melanoma classifiers are rarely tested against controls with known targets. We built a counterfactual audit from a diffusion model conditioned on seven 7-point checklist concepts and tested it with such controls. Colour-calibrated start noise reduced colour outliers from 78.5-84.8% to 6.6-13.3%. Four of seven knobs were target-dominant but also changed other concepts. Quartile-based positive controls were confounded, and the audit failed its pre-specified rule. It recovered the targets of all 16 controls matched on the generator's pseudo-labels and of 12 of 12 controls labelled by a second evaluator, for three concepts; concept activation vectors recovered fewer (11 of 16). In 18 permuted-label classifiers, significant sweep responses and target rankings also arose, whereas standardised effect margins separated the controls from these null classifiers. The pigmentation sweep raised the melanoma logit in 15 of 15 ISIC 2020 classifiers (+3.12 on average) and 6 of 6 HAM10000 classifiers (+4.01), about four times the null response in scale-free units; the regression effect did not transfer. An exploratory decomposition, evaluated against the controls, suggested that knob cross-talk partly hides vascular responses. Matched controls and null classifiers are practical checks of concept audits.
Sources
- On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors
- Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation
- Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence
- Explaining Classifiers with Causal Concept Effect (CaCE)
- CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
- Classifier-Free Diffusion Guidance
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- Captum: A unified and generic model interpretability library for PyTorch
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs