Challenges in Evaluating Explanation Methods for Static and Evolving Data

This paper, authored by Jerzy Stefanowski of Poznan University of Technology, addresses fundamental limitations in the evaluation of Explainable Artificial Intelligence (XAI). The author argues that while the need for transparency and explainability in AI systems has grown—particularly for high-stakes domains such as medicine, aerospace, and mission-critical autonomous systems—the field of XAI suffers from insufficient and inadequate evaluation practices. As the paper states, "in XAI there is now a trend to constantly introduce more and more new methods, while the assessment of their usefulness is too limited. Moreover, the number of comparative studies of XAI methods is too limited and the spectrum of studied measures is too narrow. The author draws a parallel to a historical problem in supervised classification research, which, to paraphrase David Hand's famous text, is more of an 'illusion of progress'."

The paper covers several interconnected topics: the general difficulties of evaluating XAI methods, two detailed case studies (one automated/debiasing-oriented and one human-grounded), the ambiguity of counterfactual evaluation, and the relatively unexplored problem of explaining evolving data streams with concept drift.

The paper identifies several core challenges in XAI evaluation. First, the field has produced a wide range of mainly post-hoc techniques that differ substantially in data types, explanation representations, target audiences, and methodological paradigms. According to the cited work of Moshkovitz et al., XAI approaches are not easy to apply, are highly sensitive to arbitrary design choices, and are often misused in practice.

Second, the very concept of XAI is rather ill-defined, and explanations are intended for people, making usefulness highly dependent on the recipient's background, domain knowledge, preferences, and the specific task. The paper notes that currently proposed XAI methods take too much into account the perspective of the AI system developers, not other users.

Third, human evaluation is severely underutilized. The paper cites research showing that 97% of the reviewed works pointed out that such explanations serve users, but most of them did not evaluate them through any user study. Another review of 381 XAI papers found that only 5% explicitly focused on a deeper evaluation of XAI methods. Additional issues include a lack of scientific consensus on evaluations, a lack of multi-disciplinarity, a lack of standardized procedures, and a lack of good tools for visualization and human interaction.

The paper then presents the standard taxonomy of XAI evaluation, influenced by Doshi-Velez and Kim:

• Application-grounded evaluation—real tasks with human experts in real settings.

• Human-grounded evaluation—simpler tasks with lay users, often via surveys.

• Functionally-grounded evaluation—no human experiments; proxy measures evaluate explanation properties automatically (currently the dominant approach).

For automatic evaluation, the paper lists key desiderata: fidelity, identity, stability, feature importance, compactness/simplicity, and representativeness. However, the author warns that fulfilling them does not always translate into the usefulness or correctness of the explanations themselves. Citing Guidotti's experimental work, the paper notes that focusing on high fidelity when evaluating proxy explanations may be incorrect with respect to the background knowledge—for example, rule-based explainers failed to retrieve all features in rule conditions as the number of features increased, and feature importance explainers failed to consider all relevant features or assign correct weights. The paper also highlights the insufficient number of well-prepared benchmark datasets with precise annotations of the expected explanations and recommends using multiple evaluation metrics.

The first case study demonstrates using XAI for automated bias detection and removal in image recognition. The work considers convolutional neural networks (CNNs) applied to human face datasets, where the issue of protected attributes (like gender) and proxy features arises. In the CelebA dataset, the authors discovered that wearing some items, such as a necktie, is highly correlated with the person gender, meaning the necktie feature could be used as a proxy to infer gender, creating potential unintended pathways for discrimination. Furthermore, the models consistently misclassifies individuals wearing neckties as not smiling, revealing unintended bias and decreasing values of selected fairness measures.

To detect this bias, the DetoXAI system used saliency maps, ultimately employing Concept-wise Relevance Propagation (CRP), where relevance scores are propagated backward through the network layers, while also allowing us to identify which neurons (based on their scores) are most closely related to the undesired concept (bias). Quantitative evaluation was performed automatically using novel metrics for saliency maps introduced in the authors' related work.

To eliminate the bias, rather than retraining the network from scratch (which is costly), the authors used concept unlearning methods, testing several recent post-hoc approaches. The results "clearly showed that, although the methods differed, most of them led to the elimination of the network's undesirable focus on biases—which can be observed qualitatively by a visual inspection of saliency maps and quantitatively in improvement of selected group-level fairness metrics." This case illustrates an application-grounded/functionally-grounded evaluation approach with quantitative metrics.

The second case study concerns a human-grounded evaluation conducted as a master's thesis by Zofia Frackowiak. Three different XAI methods generating salience maps were compared: ACE (Automatic Concept-based Explanations with Concept Activation Vectors), ProtoPNet (Prototypical Network), and RISE (Randomized Input Sampling for Explanations).

The paper describes several significant challenges encountered in designing this study. The first was selecting a task and dataset understandable to a broad audience; after a pilot study, the researchers chose ten easily recognizable animals from ImageNet (e.g., elephant, crocodile, hippopotamus, panda, tiger, zebra, lion, bison, frog). The second challenge was standardizing the visualizations: although all of them returned salience maps, their graphical formats differed radically—which, during the initial pilot tests, led to confusion among respondents and made it more difficult to compare results. The researchers converted all outputs into a uniform graphical interface.

In the final survey, respondents were shown up to six explanations per animal photo (without knowing which method produced them) and asked to select 0–2 of the best. Demographic information was collected, along with questionnaires about which anatomical animal features respondents considered most important for recognition. A total of 148 completed surveys were analyzed.

The main results showed a clear preference for ProtoPNet (1,045 ranking points), followed by ACE (763) and RISE (643). ProtoPNet won for most animals (strongest for zebra, panda, and tiger), while ACE won for bison and nearly tied for elephant. Interestingly, respondents with experience using XAI methods showed a stronger preference for the ProtoPNet method, while those without such experience preferred ACE. Strong correlations were found between expected animal features and chosen explanations—for example, the ACE score had the strongest correlation with the expected presence of an elephant's trunk, while ProtoPNet was associated with the mouth in photos of hippos and the presence of stripes on tigers or zebras.

The author concludes that human-centered evaluation should not be conducted as relatively simple user satisfaction questionnaires...but rather requires careful planning and preparation, including drawing from psychology survey research, conducting pilot studies, designing appropriate user interfaces, and including additional verification questions.

The paper then discusses counterfactual explanations, which specify the minimal changes to feature values required to obtain a different, typically more favorable, prediction from the model. They directly address the question: what would need to change for the decision to be different?

The paper notes that many algorithms for generating counterfactuals exist, based on different principles, with common expected properties including: validity, proximity to the input, sparsity, actionability (not modifying immutable features), plausibility (locating counterfactuals in high-density data regions), and discriminative power. However, in the case of counterfactuals, there are no clear guidelines for using single evaluation measures...beyond checking validity. Furthermore, many of them are not aligned and even contradictory, and even very similar methods generate counterfactuals that differ significantly. The paper concludes that there is no single, definitive best method.

Rather than seeking one best method, the authors propose in their prior work (Stepka et al.) an ensemble approach: instead of looking for single method for generating counterfactuals, we proposed to use several good methods for providing a diversified, richer set of explanations. A multiple criteria approach (dominance relation and Pareto front construction) reduces the set, and the Ideal Point Method selects a final counterfactual without requiring additional user preference elicitation. The experiments demonstrated the utility of it over several datasets, with selected counterfactuals quite competitive with the solutions offered by the best of the single methods. The author emphasizes this aligns with a human perspective, as humans could make trade-offs between criteria when selecting the best explanations.

The paper highlights that most XAI methods assume static settings, but many AI systems operate in non-stationary environments with concept drift—changes in the underlying data-generating process over time and its impact on the model performance. While substantial research addresses drift detection and model adaptation, significantly less attention has been paid to explaining how and why these changes occur. Static XAI applied to evolving data could provide outdated information or lead to incorrect interpretations.

Existing attempts include snapshot-based analyses that repeatedly apply static methods at different time points, but these remain limited. The paper presents two of the author's original contributions:

1. Prototype-based explanation of drift (Karolczak and Stefanowski, 2025): The work considers stream adaptation of prototype explanations for tree ensembles (e.g., bagging, random forests). The authors introduced a set of measures for comparing two sets of prototypes, possibly before and after a change in the data stream, including mean minimal distance between prototypes, mean prototype centroid displacement, and prototype reassignment impact. Experiments on synthetic and real-world streams showed these measures could support detecting concept drift, and combining them with feature parallel plots can support the analysis of the causes of drift concerning the most influential features.

2. Group counterfactual explanations for drift (Stępka and Stefanowski, 2026): This work generalizes the GLANCE algorithm to generate group counterfactual explanations (GCEs), which extend per-sample counterfactuals to groups of similar instances sharing a common explanation vector. By studying the evolution of cluster centroids and associated counterfactual vectors before and after drift, the authors showed they could serve as an interpretable proxy for assessing locality of drifts and examine shifts in cluster positions to check if the decision boundary has moved. Differences between vectors were quantified using cosine similarity, and feature-wise vector directions identified which features were most affected. The authors emphasized that examining data-level changes, model performance decreases, and explanation vector changes jointly could provide a more comprehensive understanding the drift that analyzing each part independently.

The paper concludes that a well defined, unified, 'machine-learning-style' evaluation framework for XAI is still missing. Key open issues listed include:

• Evaluation measures depend heavily on explanation type and task, and do not generalize.

• Most measures are functional proxies for complex, hidden model behaviors; more comparative studies are needed.

• A wider, diversified set of measures should be applied, with inspiration from multi-criteria decision analysis.

• Research focuses mainly on local post-hoc explanations, while global explanations are understudied.

• There is a lack of benchmark datasets with annotated ground-truth explanations; more work, including synthetic generators, is needed.

The paper stresses that explanations must serve humans, and although application- and human-grounded evaluations are most important, they are too neglected. More interdisciplinary research, drawing from experimental psychology and survey research frameworks, is needed. Remaining human-study questions include how to organize interaction sessions, how to prepare users, how to validate answer correctness, whether to move toward NLP sequential dialog, how to incorporate causality and background knowledge, and Do the explanations really increase human's trust in predictions and using AI systems?

Finally, the paper emphasizes that evaluating explanations in dynamic environments remains one of the most challenging aspects for XAI, since existing measures focus on static settings and single points in time, whereas we need to take into account issues of longer temporal dynamics (and also possible interactions between data, model and explanation changes). This area, the author states, essentially requires more intensive fundamental research and interesting application problems to verify new proposed ideas, and he expresses hope that aerospace, earth observation, and medical applications may provide inspiration for these challenges.

The work was funded by the National Science Centre, Poland, under OPUS programme grant 2023/51/B/ST6/00545, and the paper was accepted for publication in the proceedings of the EASi 2026 Workshop at IJCAI-ECAI 2026 in Bremen (Springer CCIS vol. 3107).

Based on the paper's findings, here are specific improvements for AI systems and what they can then do:

1. Multi-criteria XAI evaluation with Pareto-optimal selection

Instead of relying on a single fidelity metric, the AI system evaluates explanations against a diversified set of non-aligned criteria (validity, proximity, sparsity, actionability, plausibility, discriminative power) and uses dominance relations and an Ideal Point Method to select the best explanation.

Capability: Generate counterfactual explanations that balance competing properties, avoiding the pitfalls of single-metric optimisation, and present users with a Pareto-front of trade-off options.

2. Ensemble-based counterfactual generation

The AI system runs multiple counterfactual algorithms (with different underlying principles) and combines their outputs into a richer, diversified set, rather than searching for one best method.

Capability: Provide more robust and comprehensive answers to what would need to change, reducing dependence on arbitrary design choices and improving user trust.

3. Prototype-based concept-drift explanation

The AI system maintains prototypes from tree ensembles (bagging, random forests) and computes drift measures — mean minimal distance between prototypes, centroid displacement, prototype reassignment impact — combined with feature parallel plots.

Capability: Detect and explain why concept drift occurred, identifying the most influential features and their changes over time, instead of merely flagging a performance drop.

4. Group counterfactual explanations for evolving data

The AI system extends per-sample counterfactuals to groups of similar instances (GLANCE-style), tracking cluster centroids and counterfactual vectors before/after drift, and quantifies shifts with cosine similarity and feature-wise vector analysis.

Capability: Explain how and where decision boundaries moved during drift, providing an interpretable proxy for drift locality and showing which features caused the boundary shift.

5. Automated bias detection via concept-based relevance propagation

The AI system uses Concept-wise Relevance Propagation (CRP) to identify neurons most related to protected attributes or proxy features (e.g., necktie → gender), then applies post-hoc concept unlearning to remove those pathways.

Capability: Detect and eliminate unintended bias in image recognition models without costly retraining, with measurable improvements in group-level fairness and visual saliency maps.

6. Human-grounded evaluation with standardised, pilot-tested interfaces

The AI system follows a rigorous human-study protocol: user-friendly datasets (e.g., recognisable animals), uniform visualisation formats across different explanation methods, pilot studies, demographic data collection, and verification questions.

Capability: Produce explanations that are genuinely preferred by end-users (e.g., ProtoPNet-style prototypes for laypeople), with results that distinguish between expert and non-expert preferences and align with user expectations about relevant features.

7. Adaptive XAI for non-stationary environments

The AI system refrains from applying static explanation methods directly to evolving data; instead, it couples explanation generation with drift detection, updating prototypes or counterfactual vectors as the data stream changes.

Capability: Avoid outdated or misleading interpretations in dynamic settings such as aerospace, earth observation, and medical monitoring, delivering explanations that reflect the current data-generating process.

8. Benchmark-aware explanation generation with synthetic ground-truth generators

The AI system can generate synthetic data with annotated expected explanations, enabling functional-grounded evaluation of fidelity and correctness against known ground truth.

Capability: Automatically validate whether an explanation method retrieves all relevant features and assigns correct weights, preventing the illusion of progress caused by high-fidelity but semantically wrong explanations.

This paper addresses the limitations of Explainable Artificial Intelligence (XAI) with respect to insufficient evaluation. They are illustrated through the DetoxAI image recognition system for bias detection and concept unlearning. Then, an example of a human-grounded evaluation of methods for explaining image classification is presented. The paper further explores methods for adapting explanations to evolving data streams with concept drift. Experiences with adapting counterfactuals for this problem are discussed. Finally it is related to the challenges of tracking the co-evolution of data, models, and explanations.