Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation
Kevin Miller, Arjun Chandra, Venkatesh Saligrama
cs.CL
Submitted: 2026-08-12
Updated: 2026-08-14
License: http://creativecommons.org/licenses/by/4.0/
The gist: Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence.
Terminology
Abstract
Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paralinguistic evidence. We introduce counterfactual audits for paralinguistic response evaluation. Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift, forcing a valid judge to track the audio cue rather than lexical content or response style. We evaluate ALM judges using a native one-context judgment protocol and a contrastive recoverability control, then further decompose each item into its constituent perception and response-mapping skills. This yields useful diagnostic states that identify different sources of judge failures. Across Gemini, GPT, and open audio models, we find that contrastive success often overstates native judge reliability, and that similar aggregate accuracies can hide different failure modes. These results suggest that ALM judges should not be evaluated by accuracy alone, instead requiring thorough behavioral audits before deployment.
Sources
- Audio-Aware Large Language Models as Judges for Speaking Styles
- MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Analyzing Musical Characteristics of National Anthems in Relation to Global Indices
- DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
- WavReward: Spoken Dialogue Models With Generalist Reward Evaluators
- S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
- Voxtral
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
- AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors
- Qwen2.5-Omni Technical Report
- ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction
- "What do others think?": Task-Oriented Conversational Modeling with Subjective Knowledge
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering