Trustworthy AI Must Account for Interactions

The gist The overall goal of Trustworthy AI research is to enable not one or two aspects of trust, but many simultaneously.

Trustworthy AI Aspects

The paper reviews five core concepts that are central to Trustworthy AI: Fairness, Privacy, Robustness, Explainability, and Uncertainty Quantification (Section 2). These aspects are considered for a classification model Fθ: X → Y parameterized by θ (Page 2). The review focuses on these five TAI aspects whose interactions will be discussed in Section 3.

1. Fairness is defined by the dichotomy between procedural fairness, which emphasizes treating all individuals equally, and substantive fairness, which aligns with equity through fairness through awareness (Page 2). Disparate impact is measured by accuracy disparity: ∆acc = max a,b∈A [acc(Fθ, Da) − acc(Fθ, Db)] (Page 2).

2. Privacy is quantified using Differential Privacy (DP), which ensures that the function M cannot strongly depend on any one datapoint (Page 2). DP is typically achieved in ML through DPSGD, a stochastic gradient descent method that adds noise to gradients to satisfy Eq. (3) (Page 2).

3. Robustness refers to the ability of a model to maintain its performance under various conditions, often tested adversarially by maximizing the model’s loss on the correct answer rather than minimizing it (Page 2). Adversarial training is a defense against this by exposing the model to adversarial inputs during training (Page 2).

4. Explainability enables understanding and auditing of decisions, focusing on local explanations like LIME, which provide feature importance Eθ(x) (Page 2).

5. Uncertainty Quantification addresses the lack of confidence in point predictions by using methods like conformal prediction (CP), which outputs sets of predictions where larger sets indicate greater model uncertainty (Page 2).

Negative Interactions Between TAI Aspects

The paper exhaustively considers every pair of the five aspects to detail negative interactions, using the notation A ⇀ B to indicate that applying a concept from A has a negative impact on B (Page 3).

** Fairness and Privacy:**

Fairness interventions often necessitate collecting group identifiers, which are sensitive personal information, exposing individuals to greater risks of conventional data leaks (Page 3). Furthermore, fairness-aware ML algorithms tend to memorize from underrepresented groups, weakening privacy (Page 3). Conversely, DPSGD can cause disparate impact by introducing bias into the gradients through uneven clipping of large gradients (Page 4).

** Fairness and Robustness:**

Fairness interventions can increase the relative influence of underrepresented groups, making them more susceptible to adversarial attacks (Page 4). Adversarial training, while improving robustness, can lead to larger disparities in class-wise performance due to properties of the data distribution (Page 4).

** Fairness and Explainability:**

Fairness interventions can alter feature importance by modifying training data through re-balancing or constraints, obscuring true relationships (Page 3). Conversely, explanations may fail to represent that a model is relying on sensitive attributes, covering up discrimination (Page 3).

** Fairness and Uncertainty Quantification:**

Equalized coverage for fairness in CP can increase the average size of prediction sets, which expresses a greater level of uncertainty than marginal CP would use (Page 4). Furthermore, equalizing coverage can make human accuracy worse for groups that are under-covered using marginal CP (Page 4).

** Privacy and Robustness:**

Models trained with DPSGD tend to be less adversarially robust than those without DP guarantees because the clipping and noise addition steps slow convergence, leading to less smooth decision boundaries (Page 5). Adversarial training can increase the influence of individual datapoints, making models more susceptible to MIAs (Page 5).

** Privacy and Explainability:**

DPSGD's noise addition can degrade the fidelity of post hoc explanations by clouding true relationships between input and output variables (Page 6). Local explanations like LIME can be exploited for MIAs because their behavior varies depending on whether an input was in Dtrain (Page 6).

** Privacy and Uncertainty Quantification:**

Applying CP to a differentially private model requires a calibration set Dcal, which does not inherit DP guarantees, and the noise added for DP degrades empirical coverage of prediction sets, requiring larger sets to retain coverage (Page 7).

Position

The paper takes the position that current research practices of improving one or two aspects in isolation are insufficient because technical solutions designed to improve one aspect frequently harm other aspects unintentionally (Page 8). The authors advocate for TAI research to account for interactions between aspects in a holistic framework that accounts for interactions (Page 8). They provide guidance on how practitioners can work towards integrated trust by enumerating all relevant TAI aspects, establishing clear metrics, deliberately analyzing interactions, and performing ablations to measure impact on all other aspects (Page 8). The financial industry vignette illustrates how siloed approaches fail when negative interactions are not managed across jurisdictions (Page 9).

Alternative Views

An alternative view suggests that TAI improvements may detract from the utility of AI systems because many technical solutions have a detrimental effect on model accuracy (Page 10). It could also be argued that the trade-offs between some TAI aspects are fundamental, such as the intrinsic tension between fairness and privacy (Page 10). However, without holistic TAI, even accurate models may never reach real-world application due to laws and regulations (Page 10).

Conclusion

True alignment means achieving all of these goals simultaneously through a holistic framework that accounts for interactions (Page 8). The authors call for TAI researchers to broaden the scope of their work beyond one or two aspects in isolation, moving towards a more holistic and resilient approach (Page 8). This approach relies on interdisciplinary expertise to bring together knowledge from disparate fields and recognizes real-world constraints (Page 8).

--- Page 1 ---

Trustworthy AI Must Account for Interactions

Abstract

Trustworthy AI encompasses many aspirational aspects for aligning AI systems with human values, including fairness, privacy, robustness, explainability, and uncertainty quantification. However, efforts to enhance one aspect often introduce unintended trade-offs that negatively impact others. In this position paper, we review notable approaches to these five aspects and systematically consider every pair of our five aspects: Fairness (F), Privacy (P), Robustness (R), Explainability (E), and Uncertainty Quantification (UQ). Drawing on these findings, we take the position that current research practices of improving one or two aspects in isolation are insufficient. Instead, research on Trustworthy AI must account for interactions between aspects and adopt a holistic view across all relevant axes at once.

--- Page 2 ---

Trustworthy AI Must Account for Interactions

1. Introduction

Artificial intelligence (AI) systems have become widespread for automated decision making and as productivity aids. Industries such as banking increasingly rely on AI models that directly impact customers, while the healthcare sector explores AI-driven advancements in patient care. Increased scrutiny by regulators and concerns around the trustworthiness of these systems call for a more measured approach to AI development with considerations beyond raw performance. In response, the field of Trustworthy AI (TAI) has blossomed, with the general goal of aligning AI to human values. Chief among the tenets of TAI are fairness, privacy, robustness, explainability, and uncertainty quantification – each of which is a noble pursuit, but all of which must be harmonized to promote deep trust. The ultimate goal of TAI as a field is to achieve these aspects simultaneously.

2. Trustworthy AI Aspects

Throughout the discussions below we will typically consider a classification model Fθ: X → Y parameterized by θ, which maps a feature space X to a discrete set of labels Y. We use capital letters (e.g. X and Y) to denote random variables, while lower case (e.g. x ∈ X and y ∈ Y) indicates data instances. Fθ is trained to minimize a loss function L over its training set Dtrain. Softmax outputs are denoted as fθ: X → [0, 1]m, such that Fθ(x) = argmaxi∈Y fθ(x)i.

2.1. Fairness

Fairness is a foundational pillar in the development of TAI, ensuring systems treat diverse populations equally or equitably. Since fairness is a nuanced and highly contextual topic, it cannot be boiled down into a single set of guidelines to follow in all cases. Instead, TAI researchers and practitioners must consider the appropriate fairness definitions and methodologies to use in each circumstance.

2.2. Privacy

Privacy has become a crucial area of research in TAI as systems increasingly rely on sensitive data for training (Page 2). The use of personal information, such as financial transactions and social media activity, has led to growing concerns about privacy breaches, unauthorized data access, and the risk of re-identification (Page 2). In the context of machine learning (ML), privacy concerns are usually demonstrated adversarially, where an attack is employed to extract as much private information as possible. The standard example is a membership inference attack (MIA) in which the adversary tries to determine if a test datapoint xtest was included in Dtrain (Page 2). Differential privacy (DP) is the primary framework for quantifying how much private information could be exposed by an ML model. Formally, let M be a probabilistic function acting on datasets D. We say that M is (ϵ, δ)-differentially private if for all subsets of possible outputs S ⊆ Range(M), and for all pairs of datasets D and D′ that differ by the addition or removal of one element, Pr[M(D) ∈ S] ≤ exp(ϵ) Pr[M(D′) ∈ S] + δ (Page 2).

2.3.

1. The system can achieve fairness while protecting privacy by employing fairness-aware ML algorithms that reduce privacy risks, although this may lead to a trade-off where "fairness-aware ML algorithms tend to memorize from underrepresented groups, improving model accuracy, but weakening privacy (Chang & Shokri, 2021)."

2. The system can achieve robustness while maintaining fairness by incorporating adversarial training and ensuring that robustness bias has been attributed to properties of the data distribution like feature distributions across groups (Benz et al., 2021).

3. The system can provide reliable uncertainty quantification while managing fairness by using group-wise conditional coverage in conformal prediction, which is preferred over marginal coverage because it addresses the issue that equalized coverage negatively impacts the usefulness of prediction sets for uncertainty quantification by increasing their average size.

4. The system can ensure that explanations are fair and trustworthy by implementing methods that mitigate bias in feature importance, such as utilizing techniques like LIME while being aware of potential issues where "explanations may fail to represent that a model is relying on sensitive attributes, covering up discrimination (Lakkaraju & Bastani, 2020)."

5. The system can be made more resilient against adversarial attacks by using adversarial training, but this necessitates managing the trade-off where adversarial training increases the overall uncertainty of models, leading to larger set sizes even for clean datapoints xtest.