How Much Prosody Does Your ASR Encoder Already Know
eess.AS
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by/4.0/
The gist: End-of-turn (EoT) detection decides in real-time whether a speaker has finished.
Terminology
Abstract
End-of-turn (EoT) detection decides in real-time whether a speaker has finished. Traditional detectors rely on hand-built prosodic features (pitch, speaking rate, pauses), while modern detectors run on a neural ASR encoder that has processed the same audio. Because the encoder reads the signal those features are computed from, it may already hold much of the information they were meant to supply. We test this three ways, reading a streaming Zipformer encoder at four depths from raw input to the final layer. First, we measure how much prosody helps over the encoder alone. Second, we test whether prosody can be recovered from the encoder. Third, we ask whether any single feature still helps given the encoder. The three tests agree; prosody adds a large gain on raw audio, but that gain collapses once the signal passes through even the first encoder layer. The same pattern holds for two open-source encoders, wav2vec2 and Whisper, both of which are offline.
Sources
- How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models
- Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Zipformer: A faster and better encoder for automatic speech recognition
- Robust Speech Recognition via Large-Scale Weak Supervision
- Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs
- Voice Activity Projection: Self-supervised Learning of Turn-taking Events
- Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion
- Moshi: a speech-text foundation model for real-time dialogue
- Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
- Layer-wise Analysis of a Self-supervised Speech Representation Model
- A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models
- Probing Classifiers: Promises, Shortcomings, and Advances
- Designing and Interpreting Probes with Control Tasks
- Information-Theoretic Probing with Minimum Description Length
- The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations
- Shortcut Learning in Deep Neural Networks
- On Calibration of Modern Neural Networks
- Turn-Taking Prediction for Natural Conversational Speech
- What Do Self-Supervised Speech Models Know About Words?
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models