X Hacking: The Threat of Misguided AutoML

arXiv:2401.08513 · cs.LG, cs.CR · Submitted 2024-01-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "X Hacking: The Threat of Misguided AutoML".

Jane: The gist The paper introduces X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're diving into this paper called "X Hacking: The Threat of Misguided AutoML" today. It's about how people can deliberately mess with AI tools to get explanations that look good but aren't actually true, especially when it comes to things like SHAP values for XAI metrics >

Jane: Right. Basically, the authors introduce this idea of X-hacking, which is like p-hacking applied specifically to those explanation scores we use in machine learning models >

Lu: It’s about finding models that give you a nice explanation while still keeping the accuracy at an acceptable level, which they call "model multiplicity" >

Meng: So what does that actually mean for us, Tom? How easy is it for someone to do this manipulation?

Tom: Well, the paper shows how easily an automated pipeline can be adapted to exploit model multiplicity at scale by searching for a Rashomon set of defensible models >

Jane: It suggests that analysts can find a set of models with similar predictive performance but different explanations just by making smart choices in their analysis pipeline >

Lu: They frame this as a multi-objective optimization problem where you try to balance accuracy and the explanation quality, which is what they call the trade-off >

Tom: So it’s not just about finding *a* good model, but optimizing for both things simultaneously, which is a bigger picture than just picking one model >

Jane: Exactly. And they show that this can be done using tools that are already out there, even with attacks on different types of SHAP summaries in a post-hoc manner >

The paper's summary: Tom: Moving on to the summary of "X Hacking: The Threat of Misguided AutoML," the paper lays out three specific metrics an actor might use when trying to hack those XAI scores >

Jane: That’s interesting because it makes it concrete; they aren't just talking abstractly about manipulation, they’re giving us specific targets >

Lu: They mention explanations SHAP values, where someone could try to find a data point xi that makes a SHAP value go in the desired direction, like making it zero >

Meng: So if I’m trying to hide something, I could be looking for a specific input that makes the explanation look like it doesn't matter >

Tom: Then there’s another metric they discuss involving model- or large grouplevel explanations where you look at rankD(mean(SHAP(x))) to see if your feature of interest is not in the top k most important features >

Jane: And finally, they talk about hacking partial dependence plots by looking at the correlation between SHAP and a specific feature, searching for positive or negative correlations >

Lu: These are all ways an actor can probe the system to find weaknesses in how the XAI is generated across different pipeline stages >

Tom: It really shows that there’s a whole pipeline of decisions where someone could be trying to manipulate things innocuously rather than overtly >

Jane: It paints a picture of how subtle these kinds of manipulations can be when you look at the whole analysis process, not just the final output >

The paper's improvements: Tom: Now for the parts where they suggest ways to improve this situation, specifically their proposed strategies for X-hacking >

Jane: They present two main tactics: cherry picking and a directed search approach, which are different ways of trying to find those defensible models >

Lu: Cherry picking is using off-the-shelf software with attacks on SHAP summaries in a post-hoc manner to retrieve a set of ‘good’ models that exceed some baseline >

Meng: That sounds like something an analyst could do quickly, just grabbing the best ones they can find right now >

Tom: But the paper pushes for something more robust, which is the directed search strategy using Bayesian multi-objective optimization to find defensible models instead of relying on post-hoc cherry picking >

Jane: That means instead of just picking a handful and optimizing one thing, you use a vector of objectives where you try to balance accuracy and explanation metrics simultaneously >

Lu: They detail this directed search by defining a vector QD, where q1 is minimizing performance and q2 is maximizing the explanation metric for your target feature >

Tom: And they show that when you use this Bayesian optimization approach compared to just random sampling, it’s on average three times faster in finding those defensible models >

Jane: It’s a lot faster than what was previously possible with simpler methods, which is important for real-world application >

Conclusion: Tom: So wrapping up the paper "X Hacking: The Threat of Misguided AutoML," the main implication is that we need to be aware of this misuse of XAI tools because they can undermine trust in our AI systems >

Jane: It also highlights a potential avenue for adversarial misuse, which means biased or misleading explanations could have real ethical and legal ramifications >

Lu: The authors suggest that pre-registered analysis plans could be valuable if those plans are comprehensive enough to cover the whole search space of models and hyperparameters in an AutoML search >

Meng: From an engineering standpoint, they show that redundancy among features determines how easy it is to manipulate SHAP values; if all features are independent, manipulation without changing accuracy isn't possible >

Tom: So they end by saying AI methods make these dishonest practices semi-automated and scalable for researchers at lower effort and cost >

Jane: The authors really advocate for us to include detection strategies as a routine part of XAI evaluations to mitigate these risks >

Deutsches Forschungszentrum fur¨ Kunstliche Intelligenz GmbH

cs.LG, cs.CR

Submitted: 2024-01-16

Updated: 2025-07-15

Comments: Accepted to ICML 2025

Journal ref: Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:54326-54363, 2025

Code: https://github.com/datasciapps/x-hacking

Project page: https://eytan.github.io/papers/ae_workshop.pdf

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: The gist The paper introduces X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values, which refers to the practice of deliberately searching for and selecting models that produce a

Key concepts

X-hacking
"X-hacking" is p-hacking applied to XAI metrics such as SHAP values, involving the deliberate search for models that yield a specific explanation while keeping accuracy at an acceptable level.
Model Multiplicity
This occurs when different machine learning models can achieve nearly identical predictive performance on the same data. X-hacking exploits this by searching through a 'Rashomon set' of these similar models to find one with the desired explanation.
Cherry Picking
This strategy involves retrieving a pre-selected set of 'good' models that meet an initial baseline and then optimizing another metric, such as the desired explanation quality, from within that chosen group.

Terminology

Summary

The gist The paper introduces X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values, which refers to the practice of deliberately searching for and selecting models that produce a desired explanation while maintaining ‘acceptable’ accuracy.

X-Hacking Concept

X-hacking is defined as a form of p-hacking (Head et al., 2015) applied to XAI metrics such as SHAP values. It involves the practice of deliberately searching for and selecting models that produce a desired explanation while maintaining ‘acceptable’ accuracy. This strategy explores plausible combinations of analysis decisions to build a pipeline that might otherwise have been found innocuously. X-hacking exploits model multiplicity (Black et al., 2022), where different models offer equivalent predictive performance. The authors extend existing work on model multiplicity to incorporate the full analysis pipeline, including data preprocessing, feature extraction, choice of model class and hyperparameter tuning.

X-Hacking Framework

The paper outlines a framework for X-hacking using three quantifiable metrics an actor might consider when trying to hack XAI metrics. These metrics include Explanations SHAP values, which can be manipulated by seeking a specific data point xi such that a SHAP value in the desired direction, e.g. I(m, xi) → 0. Another metric involves model- or large grouplevel explanations, where an adversary might set "I(m, x) = rankD(mean(SHAP(x))) > k to seek models where the feature of interest is not in the ‘top k’ most important features. Finally, partial dependence plots can be hacked" by examining the relationship between SHAP and a specific feature approximately with the slope of a simple linear model, or even I = Corr(x, SHAP(x)) and searching for models with positive, negative or no correlation.

X-Hacking Strategies

The paper details two primary strategies for X-hacking:

  1. Cherry Picking: This involves retrieving a (Rashomon) set of ‘good’ models exceeding a pre-specified baseline, e.g. Ms = to then optimize the other metric, e.g., arg maxm∈Ms ID(m, x). The authors show that this approach can be performed using off-the-shelf software with attacks on different types of SHAP summaries in a post-hoc manner".

  2. Directed Search: This is a more efficient, but higher-effort, ad-hoc approach involving custom multi-objective optimization where the vector QD = (q1,..., qn) includes objectives like q1 = − perf(m, D) and q2 = Ik(m, x) for m ∈ M. The authors demonstrate that this directed search can be accelerated through Bayesian multi-objective optimisation compared to random sampling, showing that it is on average three times faster in finding defensible models.

Vulnerability and Detection

The vulnerability of a dataset to X-hacking is equivalent to measuring predictive multiplicity, with metrics such as Rashomon Capacity, based on the KLdivergence for a set of probabilistic classifiers. The authors show that the greater the number of independent child nodes associated with a specific feature, the more susceptible that feature becomes to a phenomenon referred to as “flipping”. Furthermore, they demonstrate in simulation that "as the magnitude of independent information in the dependent variables is augmented, the corresponding reduction in predictive accuracy resulting from changes in SHAP (SHapley Additive exPlanations) values also experiences a corresponding increase. Detection methods include Explanation histograms, where a reviewer can visualize the marginal distribution of XAI metrics over likely pipelines to locate reported metrics within extreme tails and Pipeline analysis," which converts pipeline similarity into a graph isomorphism problem using graph edit distance.

Comparison of Strategies

The comparison between post-hoc and ad-hoc X-hacking shows that directed search is superior in speed for vulnerable features, demonstrating that for 14 such datasets, a defensible model could be found in less than 6 minutes of directed X-hacking. The authors conclude that on average Bayesian optimisation is 3 times faster than random sampling for a majority of features to defensibly hack explanations. Redundancy among features determines the ease of manipulation, as if all input features are completely independent of one other, manipulation of SHAP values—without change of model predictive accuracy—is not possible. The paper suggests that pre-registered analysis plans could be valuable if they are comprehensive enough to cover the range of models and hyperparameters in an AutoML search space. The study concludes by emphasizing that AI methods enable such dishonest practices to be (semi-)automated, scalable and accessible to researchers and practitioners at lower effort and cost. The societal implications are broad, as biased or misleading explanations could have significant ethical and legal ramifications.

Impact Statement

This paper introduces the concept of ‘X-hacking’, a misuse of XAI and AutoML tools to produce defensible but misleading model explanations. The study highlights a potential avenue for adversarial misuse that could undermine trust in machine learning systems and exacerbate the reproducibility crisis in scientific research. The authors advocate for the inclusion of detection strategies as part of routine XAI evaluations. This work aims to mitigate these risks through awareness and contribute to the development of more robust and trustworthy AI systems.

Acknowledgements

The project on which this report was based was funded by the Federal Ministry of Education and Research under the funding code 03ZU1202JA. The responsibility for the content of this publication lies with the author. The paper includes references to various tools and frameworks, such as auto-sklearn, optuna, and Ray Tune.

References

The paper cites numerous works across machine learning, fairness, and XAI fields including foundational texts on model multiplicity like Breiman, L. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author), work on adversarial attacks like "Baniecki & Biecek. Adversarial attacks and defenses in explainable artificial intelligence: A survey, and research on specific tools like Optuna: A Next-generation Hyperparameter Optimization Framework".

--- Page 15 ---

X-Hacking: The Threat of Misguided AutoML

The paper is 2401.08513. EVERY sentence must end with a reference token.

X-Hacking Strategies

The paper details two primary strategies for X-hacking:

  1. Cherry Picking: This involves retrieving a "(Rashomon) set of ‘good’ models exceeding a pre-specified baseline, e.g.

Improvements for AI systems

  1. Model selection can be fortified by implementing a directed search strategy using Bayesian multi-objective optimization to find defensible models rather than relying on post-hoc cherry picking. This approach allows an analyst to select from the Pareto frontier of predictive performance and explanations, optimizing for a vector of objectives where QD = (q1,..., qn) is a vector, thereby finding a model that balances accuracy and desired explanation metrics.

  2. A custom AutoML solution can be developed by incorporating optuna for MOO and Ray Tune for distributed computing to perform multi-objective optimization at scale, which is necessary because an off-the-shelf solution that incorporates all the required capabilities in a single package does not exist. This enables the system to deliberately seek models where, as described in Section 4.2, m (Pareto)-dominates another model m′ in Λ for a feature x if and only if there is no criterion qi in which m′ ≻ m (m′ is superior to m), and at least one criterion qj in which m ≻ m′.

  3. Dataset vulnerability can be quantified by measuring predictive multiplicity using metrics like Rashomon Capacity, allowing for proactive mitigation of X-hacking risks. Specifically, the system can determine if a feature is robust to manipulation by examining the redundancy among the features, as the paper demonstrates that the greater the number of independent child nodes associated with a specific feature, the more susceptible that feature becomes to a phenomenon referred to as “flipping”.

Sources

Related papers