EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving
cs.CL, cs.AI, cs.CV
Submitted: 2026-11-09
Updated: 2026-11-09
Comments: 18 pages, 11 figures, 6 tables. Project page: https://emo-harness.github.io. Code: https://github.com/Emo-Harness/EmoHarness
Project page: https://emo-harness.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Small multimodal models can misjudge emotion when words, voice, and visual cues conflict.
Terminology
Abstract
Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by-0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.
Sources
- Qwen3-VL Technical Report
- What is the Visual Cognition Gap between Humans and Multimodal LLMs?
- Gemma 4 Technical Report
- Trimming the Long-Tail of Visual World Modeling Evaluation
- Toward Cognitive Supersensing in Multimodal Large Language Model
- CogniRoute: Learning to Route Social Evidence in Omni-Modal Models
- STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification
- Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
- R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
- Where LLM Agents Fail and How They can Learn From Failures
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data