Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
HEAR2ACT tests if an assistant can use prosodic cues to improve decisions in dialogue tasks. The study found that prosody is valuable when a user need is missing from spoken words, but audio-capable models don't reliably use it directly. Success significantly improved when models inferred the concern status from audio and used that text representation for action selection, showing a perception-action gap.