A Comparative Explainability Framework for DeBERTa-v3 in Zero-Shot Medical Abstract Classification
cs.AI, cs.CL
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: 18 pages, 6 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: A comparative explainability framework is presented to audit DeBERTa-v3 under zero-shot classification of medical abstracts.
Terminology
Abstract
A comparative explainability framework is presented to audit DeBERTa-v3 under zero-shot classification of medical abstracts. The work addresses the disagreement problem in Explainable Artificial Intelligence, where different attribution methods produce divergent explanations for the same input and prediction. A natural language inference engine is implemented over the Medical Abstracts corpus with five enriched hypotheses per diagnostic category and a balanced sample of one thousand texts per class. Five explanation methods are compared: SHAP and LIME as model-agnostic approaches, occlusion and Input x Gradient as deep-learning-specific approaches, and Attention x Gradient as a transformer-specific approach. Explanations are standardized through top-token attribution, and pairwise agreement is quantified using the Jaccard index. High predictive accuracy is achieved across well-defined clinical domains, whereas performance degrades under high semantic ambiguity. Explanatory stability directly mirrors predictive certainty, exhibiting strong convergence in univalent categories and a marked drop under diagnostic uncertainty. Furthermore, qualitative error auditing uncovers three systemic failure mechanisms: lexical hypersensitivity, semantic overlap, and loss of attribution coherence. The results support the combined use of several explanation methods and quantitative agreement metrics when auditing transformer-based models in medical text classification, and suggest prioritizing specific clinical ontologies over broad diagnostic labels.
Sources
- Quantifying Attention Flow in Transformers
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
- On the Robustness of Interpretability Methods
- A Close Look at Decomposition-based XAI-Methods for Transformer Language Models
- Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
- Captum: A unified and generic model interpretability library for PyTorch
- The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
- Building Efficient Universal Classifiers with Natural Language Inference
- A Unified Approach to Interpreting Model Predictions
- Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
- Attention Is All You Need
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems