Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
cs.LG, cs.CL, cs.CY
Submitted: 2026-01-06
Updated: 2026-08-31
Comments: Accepted and published at ACL Findings 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Models (LLMs) exhibit systematic biases across demographic groups.
Terminology
Abstract
Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g., Δ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40 times fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at epsilon=0.02 for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.
Sources
- Limitations of Pinned AUC for Measuring Unintended Bias
- Active Fourier Auditor for Estimating Distributional Properties of ML Models
- Cooper: A Library for Constrained Optimization in Deep Learning
- Black-Box Audits for Group Distribution Shifts
- Holistic Evaluation of Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- Red Teaming Language Models with Language Models
- A Brief Tutorial on Sample Size Calculations for Fairness Audits
- Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
- Statistical Hypothesis Testing for Auditing Robustness in Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks