EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

Abstract

Despite significant advances in instruction-following and auditory comprehension, the evaluation of Emotional Intelligence (EI) in Spoken Language Models (SLMs) remains confined to rudimentary paralinguistic perception, lacking a systematic, theory-driven cognitive framework. We introduce EmoSBench, the first comprehensive EI evaluation benchmark for SLMs constructed upon the four-branch theoretical model, covering Perceiving, Understanding, Using, and Managing Emotion across ten sub-tasks. Preliminary assessments on EmoSBench reveal a substantial gap: even leading proprietary models like GPT-4o-Audio achieve only 52.6%, significantly trailing human baselines. To bridge this gap, we develop EmoS, a specialized evaluator model optimized via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To facilitate its effective training, we curate EmoDialogue, a bilingual dataset providing necessary fine-grained supervision through response pairs with rigorously defined EI gradations. Concurrently, we introduce a reward mechanism integrating a Steep Exponential Accuracy Reward (SEAR) and a Rationale Fidelity Reward (RFR) to enforce precise ordinal scoring and valid reasoning. Experiments demonstrate that EmoS reaches 83.8% accuracy, approaching human-level performance. Furthermore, evaluations on authentic, unconstrained spoken interactions validate its robust real-world generalization, establishing a foundational framework for advancing emotionally intelligent dialogue systems.

1. Introduction

Current Spoken Language Models (SLMs) have evolved into the core engines of modern conversational systems, achieving sophisticated auditory comprehension, context awareness, and instruction-following. Early evaluation frameworks primarily assessed the IQ of SLMs by focusing on acoustic perception and textual reasoning. Although subsequent dialogue benchmarks successfully introduced tasks requiring contextual interaction, their core focus remains confined to semantic understanding and logical deduction, typically treating the speech modality merely as an auxiliary input for text processing.

However, human spoken dialogue is intrinsically rich in paralinguistic cues, such as emotion, prosody, speed, volume, and rhythm, which significantly modulate semantic interpretation. For instance, the phrase I am aware of this schedule conveys distinctly different meanings when articulated with a tone of disappointment versus relief. Recognizing this, recent works like WavReward and ParaS2S have begun to address this by incorporating the perception of paralinguistic features into preliminary assessments of model Emotional Intelligence (EI).

Nevertheless, genuine EI, as rigorously defined in psychological theory, transcends mere acoustic perception. Fundamentally, EI represents a complex cognitive ability to reason about emotions and utilize emotional information to enhance thought and problem-solving. This conceptualization is formalized in the influential four-branch model proposed by Mayer and Salovey, which characterizes EI as a hierarchical progression—starting from the basic sensory recognition of affective signals and ascending to the higher-order regulation of emotions to achieve social or personal goals. Drawing upon this theoretical foundation, our work posits that EI in SLMs comprises four hierarchical skills: Perceiving (decoding emotional and non-verbal cues), Understanding (comprehending emotional transitions across turns), Using (harnessing emotions for decision-making), and Managing (regulating emotions to achieve goals). Based on this, we define ten sub-task scenarios to systematically simulate the multifaceted challenges of high-EI dialogue in the real world.

Preliminary evaluations reveal a significant performance disparity: even the most advanced contemporary models, including leading open-source SLMs such as Qwen3-Omni and Kimi-Audio, and closed-source counterparts such as GPT-4o-Audio and Gemini 2.5 Pro, perform considerably below the human baseline on this comprehensive EI task. This finding highlights a critical methodological limitation: the prevailing practice of relying on proprietary models (e.g., GPT-4o-Audio) as evaluators for generating high-EI dialogue inadvertently sets an artificially low performance ceiling. This approach masks the intrinsic deficiencies of current systems and underscores the urgent need for specialized EI evaluation models. However, the development of such specialized evaluators is currently obstructed by a fundamental resource gap, as existing datasets primarily focus on coarse-grained emotion classification and paralinguistic perception, thereby lacking the fine-grained, theoretically aligned annotations necessary to train models with valid EI reasoning capabilities.

To bridge this data scarcity and enable the construction of reliable evaluators, we introduce EmoDialogue, a large-scale dataset comprising both single-turn and multi-turn samples. Each instance consists of user speech paired with four candidate responses, rigorously graded from 1 to 4 based on their EI quality. Leveraging this, we develop EmoS, a specialized reward model for spoken EI evaluation, employing a sequential two-stage training paradigm. Specifically, EmoS is first initialized through Supervised Fine-Tuning (SFT) to establish a robust foundational distribution of EI. Subsequently, by integrating Group Relative Policy Optimization (GRPO) with multi-reward reinforcement learning (RL) techniques, we effectively align EmoS's predictive scoring and analytical rationales with our theoretical guidelines.

The contributions are threefold:

• We propose EmoSBench, a novel, theoretically grounded benchmark that evaluates EI across four key dimensions (Perceiving, Understanding, Using, Managing) through ten sub-tasks, offering a comprehensive and realistic reflection of high-EI interactions.

• We introduce EmoDialogue, the first high-quality dataset featuring responses with clear, theoretically defined scoring gradients, emphasizing the quality of emotional interaction over simple emotion classification.

• We develop EmoS, the first specialized reward model for spoken dialogue EI evaluation, alleviating the performance ceilings and scoring biases inherent in relying on proprietary models like GPT-4o-Audio as judges.

2. Related Work

2.1 Spoken Language Models

The evolution of large language models (LLMs) and large-scale audio-text paired data has transitioned spoken language models (SLMs) from traditional modular pipelines (ASR-LLM-TTS) toward unified architectures. These advanced SLMs have acquired strong capabilities in auditory understanding, open-domain dialogue, and multi-turn instruction-following. Early advances in this field primarily focused on enhancing cross-modal interaction. For instance, Qwen2-Audio improved fluent speech interaction and robust instruction-following by scaling up pretraining data and applying Direct Preference Optimization (DPO) to the audio-text alignment process. Building on this, AudioReasoner introduced Chain-of-Thought (CoT) reasoning techniques into the audio modality, significantly augmenting the model's capacity for complex logical deduction based on spoken input. However, concerning speech synthesis, these models did not operate in a strictly end-to-end manner; the decoupling of semantic generation from acoustic synthesis often created an information bottleneck, leading to significant loss of paralinguistic nuances during response generation.

Recent research has pivoted toward end-to-end SLMs to achieve unified modeling. GLM-4-Voice utilizes a low-frame-rate tokenizer to discretize continuous speech signals, enabling the model to learn bidirectional dependencies between speech and text more efficiently. Similarly, Step-Audio 2 integrates the generation of discrete audio tokens directly into the language modeling process, thereby enhancing its responsiveness to paralinguistic cues and improving the naturalness of interactions. Furthermore, to address the latency challenges inherent in real-time dialogue, Qwen2.5-Omni introduces a Thinker-Talker architecture, which decouples reasoning from streaming generation, allowing for the simultaneous output of text and speech with minimal latency. Kimi-Audio adopts a dual-head generation mechanism combined with a chunk-wise flow-matching detokenizer to balance low latency and high-fidelity expressive speech synthesis. Most recently, Qwen3-Omni significantly scales the architecture up to 30 billion parameters, leveraging optimized training objectives to further push the performance ceiling of multi-modal alignment and complex reasoning. Despite substantial architectural progress in achieving fluency, low latency, and expressiveness, no existing model has systematically addressed the complex challenges of EI in spoken dialogue. The assessment and optimization of EI for speech remains a largely unexplored frontier. To the best of our knowledge, EmoS represents the first work dedicated to evaluate the EI capabilities of SLMs within a theoretically grounded framework.

2.2 Benchmark for Spoken Language Models

Evaluation paradigms for SLMs have shifted from early tasks focused on discrete recognition towards a more holistic assessment of speech-to-speech interaction. Early benchmarks like AIR-Bench, SD-Eval, and VoxDialogue primarily utilize text-based metrics or LLMs to gauge semantic alignment and paralinguistic cues. Other frameworks target specific capabilities, such as VoxEval for knowledge comprehension and VoiceBench for instruction-following. However, a common limitation among these frameworks is their heavy reliance on transcription or a narrow focus on basic acoustic information perception.

To address the information loss inherent in text-based metrics, recent work has pivoted toward direct, audio-centric evaluation. URO-Bench serves as a speech-to-speech benchmark covering multi-turn dialogue and paralinguistic assessment, and WavReward further incorporates implicit conversational scenarios to directly evaluate the authenticity of acoustic interaction. Despite these advances, existing benchmarks tend to conflate EI with rudimentary acoustic perception. While a few methods have begun to explore the Understanding dimension, the Using and Managing dimensions remain notably understudied. In contrast, EmoSBench constitutes the first benchmark that systematically addresses all four hierarchical dimensions of EI, extending the evaluation scope from mere perception to the complex cognitive processes involved in using and managing emotions.

3. Methodology

3.1 EmoSBench

To address the limitations of prevailing approaches that equate Emotional Intelligence (EI) with simple paralinguistic perception, we establish a systematic evaluation framework anchored in the influential four-branch model. This framework is designed to comprehensively assess the advanced emotional capabilities required by Spoken Language Models (SLMs) for complex interpersonal interactions. We categorize EI into four hierarchical branches encompassing ten specific sub-task scenarios.

Perceiving Emotion: This dimension evaluates the foundational ability to decode emotional information and non-verbal cues from single-turn speech. It comprises two sub-tasks: (1) Basic Acoustic Information Perception, which tests the accuracy in identifying explicit emotional and paralinguistic features; and (2) Implicit Attitude Analysis, which assesses the model's capacity to capture subtle cues from faint acoustic hints or textual implications to generate empathetic responses.

Understanding Emotion: This dimension assesses the model's capacity to comprehend the causes and dynamic evolution of emotions in multi-turn contexts. This is examined through: (3) Emotional State Tracking, which requires the model to identify emotional transitions across dialogue turns; and (4) Emotional Causation Analysis, which evaluates the ability to connect the current complex emotional state to previously mentioned historical triggers.

Using Emotion: This dimension examines the proactive harnessing of emotions to facilitate cognitive processing and decision-making. We propose: (5) Emotion-Cognition Matching, investigating the understanding of how emotional states facilitate cognitive tasks; and (6) Emotion-Driven Plan Adjustment, requiring the model to detect acoustic-semantic conflicts (e.g., hesitant tone vs. committed text) and guide users to reconsider or optimize decisions rather than offer blind agreement.

Managing Emotion: As the pinnacle of spoken EI, this dimension evaluates the strategic capability to regulate emotions to achieve specific communicative goals. It encompasses four sub-tasks: (7) Social Strategy Execution, coordinating text and tone to achieve challenging social objectives like constructive criticism; (8) Proactive Mitigation and Emotional Buffering, employing textual and acoustic means to soften the impact of negative information; (9) Conflict Resolution and De-escalation, utilizing an empathy-first strategy in high-pressure scenarios; and (10) Value Alignment and Safety Response, maintaining dialogue while upholding principles against harmful or biased inputs.

3.2 EmoDialogue

To facilitate the training and rigorous evaluation of EI in spoken dialogue systems, we construct EmoDialogue, a comprehensive dataset comprising over 70,000 pairs of user inputs and model responses in both Chinese and English. Given the absence of speech dialogue data with fine-grained annotations for multidimensional EI, we predominantly employ a synthesis strategy based on advanced LLMs to generate training data.

Data Synthesis and Annotation. We leverage the reasoning capabilities of the DeepSeek-R1 LLM to simulate realistic user-model interactions. For every user input, the system generates four distinct candidate responses, hierarchically graded on a four-point ordinal scale to reflect varying levels of EI. Generally, Score 1 denotes responses characterized by logical incoherence or mechanical repetition, rendering them unusable; Score 2 represents factually correct but emotionally detached or blunt responses, indicating a deficiency in empathy; Score 3 corresponds to responses that are fundamentally empathetic, socially appropriate, and helpful, meeting the standard for general conversation; and Score 4 is reserved specifically for excellent responses that demonstrate high-EI capabilities, such as optimal emotion utilization or proactive emotional guidance. Concurrently, the LLM produces detailed metadata, including the intended emotional tone, paralinguistic descriptions, and a rationale for each assigned score based on sub-task criteria.

Dialogue Audio Synthesis. To obtain high-fidelity and expressive speech synthesis, we utilize the Doubao TTS API with advanced emotional control, ensuring acoustic diversity by explicitly controlling parameters such as emotion category, speech rate, and volume. Furthermore, to enhance realism, non-verbal paralinguistic cues like sighs, laughter, or hesitation pauses are strategically injected into the audio stream.

Quality Control. A rigorous multi-stage quality control pipeline is implemented to ensure the acoustic quality and consistency of the dataset. First, all synthesized audio is automatically transcribed using the Whisper-Large-V3 model, and samples with a Word Error Rate (WER) or Character Error Rate (CER) exceeding 5% are filtered out. Subsequently, a manual review is conducted to perform secondary verification on the remaining samples, checking for audio naturalness, accuracy of emotional expression, and consistency with annotated descriptions.

Benchmark Test Set Construction. To ensure reliable and unbiased evaluation, we curate EmoSBench, a core evaluation benchmark consists of approximately 4,000 audio dialogue samples. Five expert annotators select samples from each sub-task based on the principles of score discriminability, topic diversity, and high inter-annotator agreement.

Real-World Evaluation Set. While synthetic datasets provide large-scale supervision signals, authentic spoken interac…

Improvements to AI Systems Based on This Paper:

1. Emotionally Intelligent Speech Evaluation Module: Integrate EmoS as a specialized reward model into SLM training pipelines to replace generic LLM judges (e.g., GPT-4o-Audio) that cap performance. This enables systems to score responses on a 1–4 ordinal EI scale with 83.8% accuracy, approaching human baselines, and provides valid Chain-of-Thought rationales for each score.

2. Four-Branch EI Reasoning Capability: Equip SLMs with hierarchical emotional cognition—Perceiving (acoustic/paralinguistic decoding), Understanding (emotional state tracking and causation across turns), Using (emotion-cognition matching and plan adjustment), and Managing (social strategy, mitigation, conflict resolution, value alignment). This allows systems to handle ten distinct sub-tasks, from detecting implicit attitudes to de-escalating high-pressure conflicts.

3. Steep Exponential Accuracy Reward (SEAR) for Precise Scoring: Implement a reward function where utility decays exponentially with prediction error (Δ), penalizing near-miss scores (e.g., predicting 3 instead of 4) far more heavily than binary or linear rewards. This forces the model to avoid vague guesses and achieve exact ordinal alignment, improving scoring precision by 10% over standard RL.

4. Rationale Fidelity Reward (RFR) with Adaptive Weighting: Use an automated critic (e.g., Qwen3-8B) to verify that correct scores are derived from psychologically valid reasoning, not spurious correlations. The dynamic coefficient ω(Δ) gives full weight (1.0) to rationale fidelity when predictions are correct and residual weight (0.1) when incorrect, preventing reasoning collapse during training and ensuring explainable, trustworthy evaluations.

5. Two-Stage Training Paradigm (SFT + GRPO): Combine Supervised Fine-Tuning on a large bilingual dataset (EmoDialogue, 70k pairs) with Group Relative Policy Optimization using multi-reward RL. This yields a synergistic boost from 57.0% (SFT only) to 83.8% (final), enabling the model to explore optimal reasoning paths and align with complex decision boundaries beyond what supervised learning alone achieves.

6. Fine-Grained EI-Graded Dialogue Dataset (EmoDialogue): Train systems on responses with four ordinal EI levels (1=incoherent, 2=blunt, 3=empathetic, 4=high-EI) with detailed rationales, emotional tones, and paralinguistic annotations. This provides supervision for nuanced emotional interaction quality, not just emotion classification, enabling models to distinguish between factually correct but emotionally detached and proactively emotionally intelligent responses.

7. Real-World Generalization via Acoustic Robustness: Apply the training pipeline to handle unconstrained speech with background noise, disfluencies, and spontaneous paralinguistic cues (validated on YouTube-derived data). The system maintains 62.9% accuracy in real-world conditions, outperforming GPT-4o-Audio (36.0%) and Gemini 2.5 Pro (43.3%) by 19.6%+, enabling deployment in noisy, authentic conversational settings.

8. Bilingual Emotional Intelligence Support: Leverage the bilingual (Chinese and English) training data to enable EI evaluation and reasoning across languages, allowing systems to assess and generate emotionally intelligent responses for diverse global user bases without language-specific retraining.

What the Improved AI System Can Do:

• Evaluate spoken dialogue responses on a 4-point EI scale with near-human accuracy, providing transparent reasoning for each score.

• Generate high-EI responses that perceive subtle acoustic cues (e.g., sighs, hesitation), understand emotional transitions, use emotions to guide decisions, and manage conflicts strategically.

• Detect acoustic-semantic conflicts (e.g., hesitant tone vs. committed text) and proactively guide users to reconsider decisions.

• De-escalate high-pressure scenarios using empathy-first strategies and uphold safety values against harmful inputs.

• Operate reliably in real-world, noisy environments with spontaneous speech, outperforming commercial APIs by significant margins.

• Serve as a reward model for aligning future SLMs, enabling the development of emotionally intelligent agents that reason about and regulate emotions in dialogue.

Despite significant advances in instruction-following and auditory comprehension, the evaluation of Emotional Intelligence (EI) in Spoken Language Models (SLMs) remains confined to rudimentary paralinguistic perception, lacking a systematic, theory-driven cognitive framework. We introduce EmoSBench, the first comprehensive EI evaluation benchmark for SLMs constructed upon the four-branch theoretical model, covering Perceiving, Understanding, Using, and Managing Emotion across ten sub-tasks. Preliminary assessments on EmoSBench reveal a substantial gap: even leading proprietary models like GPT-4o-Audio achieve only 52.6%, significantly trailing human baselines. To bridge this gap, we develop EmoS, a specialized evaluator model optimized via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To facilitate its effective training, we curate EmoDialogue, a bilingual dataset providing necessary fine-grained supervision through response pairs with rigorously defined EI gradations. Concurrently, we introduce a reward mechanism integrating a Steep Exponential Accuracy Reward (SEAR) and a Rationale Fidelity Reward (RFR) to enforce precise ordinal scoring and valid reasoning. Experiments demonstrate that EmoS reaches 83.8% accuracy, approaching human-level performance. Furthermore, evaluations on authentic, unconstrained spoken interactions validate its robust real-world generalization, establishing a foundational framework for advancing emotionally intelligent dialogue systems.