Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Beyond Text Following".
Jane: Here's the gist of the paper. Audio-language models (ALMs) often follow text that conflicts with audio, even when the audio evidence is clear The same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern suggests that the relevant audio evidence is encoded but loses in arbitration GACL improves nAUC by 17.8 points over the best contrastive baseline and transfers without retuning to vision–text arbitration (up to +40.5,pp) The core finding of the study is that a repairable arbitration reversal exists where models can support the audio answer, but the joint input reverses the decision This reversal is localized to an answer-position residual stream and its direction tracks output score differences at the rank level with Spearman ρ=0.93 The proposed GACL decoding rule interpolates between joint and same-audio scores, improving conflict audio-following accuracy by 17.8 nAUC points under a 5 pp faithfulness budget This method generalizes to vision–text arbitration without retuning, achieving up to +40.5 pp gain GACL's success is attributed to its diagnostic-derived gating and bounded interpolation mechanism Diagnostic Behavior of Arbitration Failure The study examines a same-audio counterfactual that keeps the audio fixed and removes only the conflicting text to measure preference shift Across five ALMs and four conflict tasks, 64.1% of conflict samples show a sign flip where the same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern is referred to as repairable arbitration reversal Activation patching localizes this reversal to the answer-position residual stream, and patching effects closely track output candidate-score differences (Spearman ρ=0.93) The diagnostic region O is defined by MA > 0 and MJ < 0, indicating the audio answer is available in the same-audio reference branch but text-favored in the joint branch This behavioral evidence suggests that many conflict errors are not reducible to perceptual failure, as removing only conflicting text often recovers the audio-supported answer Mechanism of Repair and GACL Implementation The authors propose Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule that interpolates between joint and same-audio reference scores gated by branch disagreement and reference reliability The GACL formula is defined as Nout(x) = 1
ˆyA ≠ ∅: · 1
ˆyA ≠ ˆyJ: , ∆A(x) = sA(ˆyA) − max y∈H(x) sA(c), RA(x) = clip(∆A(x)/τA, 0, 1), and α(x) = clipλ RA(x) Nout(x), 0, 1 The rank alignment between the hidden repair direction and the observable difference sA − sJ is Spearman ρ = 0.93 across twelve model–task configurations GACL's performance is robust, achieving a macro average of 0.81 rescue under a 5 pp faithfulness drop budget Performance and Generalization Under a strict 5 pp faithfulness-drop budget, GACL improves conflict audio-following accuracy by 17.8 nAUC points over the best contrastive decoding baseline Applied without retuning to vision–text arbitration on MC2, the same rule raises adversarial accuracy by up to +40.5 pp GACL dominates in the low-drop regime, showing gains of 51.0 nAUC for Qwen2-Audio-Inst.
Tom: on AQA at a 5 pp budget The method is sensitive to reference reliability, as GACL stays conservative when the audio reference is weak, defining an operating range where it converts repairable opportunities Ablation and Stress Testing Component-level ablation shows that the answer-side residual patching reaches a macro average of 0.
Paper summary: Jane: What they claim is that this preference shift happens in about sixty-four point one percent of their test cases, where you keep the audio fixed and remove only the conflicting text, and you see a sign flip.
Lu: That pattern suggests that the relevant audio evidence is encoded but it loses its influence once that conflicting text is added to the joint input.
Tom: They call this pattern repairable arbitration reversal, and they've localized this reversal to a specific part of the model called an answer-position residual stream.
Meng: So when you look at that stream, you can see exactly where the preference is shifting compared to just looking at the score differences between the audio branch and the joint branch.
Jane: And they found that this alignment tracks those score differences really well, with a Spearman correlation of zero point nine three across twelve different model and task setups.
Conclusion: Tom: So, to wrap up, this paper introduces Gated Audio Counterfactual Logit Correction or GACL, which is a training-free decoding rule.
Jane: It works by interpolating between the joint and same-audio reference scores, using a gating mechanism based on how much disagreement there is between the branches and how reliable that audio reference actually is.
Lu: And they've shown that under a five point five faithfulness drop budget, this GACL method improves conflict audio-following accuracy by seventeen point eight nAUC points over the best contrastive baseline.
Meng: The real kicker is that this rule transfers without any further training to vision and text arbitration, actually boosting adversarial accuracy by up to forty point five points when applied on a different task.
Tom: It sounds like they found a way to fix this specific kind of conflict without having to retrain the whole model, which is pretty important for deployment.
Jane: It’s about finding that specific repairable reversal and then building a simple decoding rule that leverages it.
Yichen Gao, Yiqun Zhang, Zijing Wang, Yujia Li, Heng Guo, Xi Wu, Xiaocui Yang, Shi Feng, Yifei Zhang
Northeastern University, China · Shanghai Artificial Intelligence Laboratory, China
cs.SD, cs.CL
Submitted: 2026-06-03
Updated: 2026-06-03
Journal ref: Proceedings of EMNLP 2026
DOI: 10.18653/2026.emnlp-main.1088
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: Audio-language models (ALMs) often follow text that conflicts with audio, even when the audio evidence is clear The same-audio branch prefers the audio-supported answer…
Key concepts
- Repairable Arbitration Reversal
- This is a specific failure pattern where an AI model shows conflicting preferences: one branch (same-audio) favors the audio answer, while another (joint) favors the text answer. This reversal can be fixed by modifying how the model makes its final choice.
- Gated Audio Counterfactual Logit Correction (GACL)
- GACL is a training-free decoding rule designed to fix these arbitration errors. It uses branch disagreement and reference reliability to gate an interpolation between the joint and same-audio scores, allowing the model to correctly choose the audio answer when it's available.
- Diagnostic Behavior
- Researchers tested what happens when they deliberately remove conflicting text from an audio prompt while keeping the audio fixed. This diagnostic test revealed that many errors are not due to perceptual failure but rather a preference shift caused by the joint input structure.
Terminology
Summary
Audio-language models (ALMs) often follow text that conflicts with audio, even when the audio evidence is clear The same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern suggests that the relevant audio evidence is encoded but loses in arbitration GACL improves nAUC by 17. p. 08 points over the best contrastive baseline and transfers without retuning to vision–text arbitration (up to +40. p. 05,pp)
**The core finding of the study is that a repairable arbitration reversal exists where models can support the audio answer, but the joint input reverses the decision This reversal is localized to an answer-position residual stream and its direction tracks output score differences at the rank level with Spearman ρ=0.93 The proposed GACL decoding rule interpolates between joint and same-audio scores, improving conflict audio-following accuracy by 17. p. 18 nAUC points under a 5 pp faithfulness budget This method generalizes to vision–text arbitration without retuning, achieving up to +40. p. 05 pp gain GACL's success is attributed to its diagnostic-derived gating and bounded interpolation mechanism **
Diagnostic Behavior of Arbitration Failure
The study examines a same-audio counterfactual that keeps the audio fixed and removes only the conflicting text to measure preference shift Across five ALMs and four conflict tasks, 64.1% of conflict samples show a sign flip where the same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern is referred to as repairable arbitration reversal Activation patching localizes this reversal to the answer-position residual stream, and patching effects closely track output candidate-score differences (Spearman ρ=0. p. 093) The diagnostic region O is defined by MA > 0 and MJ < 0, indicating the audio answer is available in the same-audio reference branch but text-favored in the joint branch This behavioral evidence suggests that many conflict errors are not reducible to perceptual failure, as removing only conflicting text often recovers the audio-supported answer
Mechanism of Repair and GACL Implementation
The authors propose Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule that interpolates between joint and same-audio reference scores gated by branch disagreement and reference reliability The GACL formula is defined as Nout(x) = 1[ˆyA ≠ ∅] · 1[ˆyA ≠ ˆyJ], ∆A(x) = sA(ˆyA) − max y∈H(x) sA(c), RA(x) = clip(∆A(x)/τA, 0, 1), and α(x) = clipλ RA(x) Nout(x), 0, 1 The rank alignment between the hidden repair direction and the observable difference sA − sJ is Spearman ρ = 0.93 across twelve model–task configurations GACL's performance is robust, achieving a macro average of 0.81 rescue under a 5 pp faithfulness drop budget
Performance and Generalization
Under a strict 5 pp faithfulness-drop budget, GACL improves conflict audio-following accuracy by 17.8 nAUC points over the best contrastive decoding baseline Applied without retuning to vision–text arbitration on MC2, the same rule raises adversarial accuracy by up to +40. p. 15 pp GACL dominates in the low-drop regime, showing gains of 51. p. 60 nAUC for Qwen2-Audio-Inst. p. 3 on AQA at a 5 pp budget The method is sensitive to reference reliability, as GACL stays conservative when the audio reference is weak, defining an operating range where it converts repairable opportunities
Ablation and Stress Testing
Component-level ablation shows that the answer-side residual patching reaches a macro average of 0.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
GACL implements a training-free decoding rule that interpolates between joint and same-audio reference scores, improving conflict audio-following accuracy by 17.8 nAUC points under a strict 5 pp faithfulness budget. This allows the AI system to rescue audio answers when they are encoded but overridden by conflicting text, even when the model's internal computation fails to prioritize the audio evidence at the answer position.
-
The system can be equipped with gates (Nout and RA) that condition on branch disagreement and reference reliability to dynamically apply interpolation, ensuring correction is only attempted when there is a repairable opportunity O = where
the same candidate pair is audio-favored in A but text-favored in J.
This prevents the model from applying corrections where they are not needed or would degrade output validity. -
The system can be configured with a bounded interpolation parameter alpha, which controls the correction strength, ensuring that the generated answer remains close to the original joint prediction while being moved toward the audio-supported side. This boundedness prevents parse failures when using weak references during free-form generation, as shown by Table E.9 confirming that removing this cap leads to
26–30% parse failures.
-
The system can leverage activation patching to localize the reversal to the answer-position residual stream and measure its effect on candidate scores with high correlation (Spearman rho=0.93). This diagnostic provides a training-free proxy for the hidden repair direction, allowing for output-space correction without requiring parameter updates or internal model access.
-
The system can be tuned using parameters lambda and tau A selected on a development set under a faithful drop budget to maximize conflict accuracy while staying within constraints. These settings are then frozen for testing, ensuring that the decoding rule is robust across different tasks and models when applied without retraining.
Sources
- Understanding intermediate layers using linear classifier probes
- When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration
- Qwen2-Audio Technical Report
- Kimi-Audio Technical Report
- How Contrastive Decoding Enhances Large Audio Language Models
- Voxtral
- Simple synthetic data reduces sycophancy in large language models
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
- Evaluating and Steering Modality Preferences in Multimodal Large Language Model
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems