BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
cs.AI, cs.CL
Submitted: 2026-08-31
Updated: 2026-10-15
Comments: 10 pages, 5 figures, 3 tables. Code: https://github.com/AdrSkapars/bloom-wilt
Code: https://github.com/AdrSkapars/bloom-wilt
License: http://creativecommons.org/licenses/by/4.0/
The gist: Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation
Terminology
Abstract
Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.
Sources
- Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
- Estimating Tail Risks in Language Model Output Distributions
- Data Swarms: Optimizable Generation of Synthetic Evaluation Data
- The Llama 3 Herd of Models
- Refusal in Language Models Is Mediated by a Single Direction
- Classifier-Free Diffusion Guidance
- Eliciting Behaviors in Multi-Turn Conversations
- Best-of-N Jailbreaking
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Forecasting Rare Language Model Behaviors
- Tuning Language Models by Proxy
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
- Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
- Frontier Models are Capable of In-context Scheming
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- Large Language Models Often Know When They Are Being Evaluated
- Estimating the Probabilities of Rare Outputs in Language Models
- Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
- Universal and Transferable Adversarial Attacks on Aligned Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems