SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
cs.CL, cs.AI, cs.SD
Submitted: 2026-06-24
Updated: 2026-11-02
Project page: https://binomial14.github.io/speecheq-demo
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI…
Terminology
Abstract
As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce SpeechEQ, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, SpeechEQ reveals that current multimodal models remain bottlenecked by a text-reliant ``modality shortcut,'' an alignment-induced ``safety trap,'' and ``contextual amnesia,'' highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and demo page at https://speecheq.github.io
Sources
- Towards a Human-like Open-Domain Chatbot
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
- Qwen2-Audio Technical Report
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Moshi: a speech-text foundation model for real-time dialogue
- MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models
- Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence
- Kimi-Audio Technical Report
- Qwen3-TTS Technical Report
- Qwen2.5-Coder Technical Report
- GPT-4o System Card
- DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset
- LLMs Get Lost In Multi-Turn Conversation
- EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
- AgentBench: Evaluating LLMs as Agents
- EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models
- ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models
- AudioPaLM: A Large Language Model That Can Speak and Listen
- Towards Understanding Sycophancy in Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data