Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models
cs.AI, cs.CL
Submitted: 2026-10-31
Updated: 2026-10-31
Code: https://github.com/bespokelabsai/nimble
Terminology
Sources
- Labels Override Definitions in Jev-Style Typed Decision Models
- Evaluating and Benchmarking the System One Model Jev
- More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
- On Calibration of Modern Neural Networks
- Are AI Decision Models Biased? A Case Study of Jev
- SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models
- Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
- Lost in the Middle: How Language Models Use Long Contexts
- Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation
- Jev in Medicine: A Benchmark Evaluation
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- BBQ: A Hand-Built Bias Benchmark for Question Answering
- Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
- Jev thinks "I don't know'', but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration
- Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model
- Benchmarking System One decision models against trained classifiers and language models for automated decision gates
- Born With a Silver Spoon? Investigating Socioeconomic Bias in Large Language Models
- Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It
- Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
- The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents