NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding".
Dev: Here's the gist of the paper. Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information <ref:2609.09940#pg2> NVV-Locator achieves the best aggregate performance across all three metrics <ref:2609.09940#pg2>.
Rosa: Problem Statement Early NVV studies mainly formulate the problem as clip- or utterance-level acoustic event classification, providing limited supervision for continuous-speech localization <ref:2609.09940#pg2> More recent work has collected, synthesized, or annotated NVV corpora through automatic detectthen align pipelines, including NonverbalTTS
3: and NonVerbalSpeech-38K
23: ; synthesis- or editing-based approaches, such as SynParaSpeech
1: and CapSpeech
16: ; and manually curated or hybrid resources, including MNV-17
7: , SMIIP-NV
17: , and NVSpeech-170k
6: <ref:2609.09940#pg2> Many existing approaches still represent NVVs primarily as discrete semantic tags, for example by inserting
laughter: into a transcript <ref:2609.09940#pg2> Such representations capture event identity and textual context, but do not explicitly preserve acoustic onset, offset, or duration on the waveform timeline <ref:2609.09940#pg2> Timestamped NVV data therefore remain scarce, while manual boundary annotation is expensive and difficult to scale <ref:2609.09940#pg2>.
Dev: Data Construction Pipeline To construct large-scale training data for transcript-conditioned NVV temporal grounding under the unified taxonomy, we develop a four-stage automatic data construction pipeline <ref:2609.09940#pg2> 1.
Rosa: Phase 1: Candidate Data Acquisition and Dual-LLM Verification involves acquiring candidate training instances from three complementary sources and subjecting them to a sequential dual-LLM verification procedure <ref:2609.09940#pg2>.
Paper summary: Rosa: So, we're looking at this paper called "NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding." Essentially, the authors are tackling the problem of figuring out exactly when a nonverbal vocalization happens on an audio file.
Dev: Right. They start by saying that most early studies just look at classifying an event in a clip or an utterance, which is not really helpful for figuring out where it sits in continuous speech.
Rosa: Exactly. The big gap they're addressing is that existing methods usually just use transcript-level tags, like putting a "laughter" tag into the text. This works for knowing *what* happened and *when* in the words, but it doesn't tell you when the actual sound starts or stops on the waveform timeline.
Dev: So, what they claim is they are proposing NVV-Locator to do fine-grained temporal grounding for these nonverbal vocalizations. They build a system that links lexical units with the actual acoustic boundaries of those events.
Rosa: And how they claim to do it is by building a huge dataset first, which involves unifying twenty-six different NVV categories from four separate public resources <ref:2609.09940#pg1>. Then they use dual large language models for verification and some transcript-guided alignment techniques to get those initial timestamps.
Taro: I wonder about that data construction part. If you're unifying twenty-six categories from different sources, how do you make sure the taxonomy is actually consistent enough for the model to learn from it?
Dev: That’s a good point, Taro. The paper mentions they use acoustic normalization for physically defined vocal events and functional abstraction for lexicalized paralinguistic expressions to handle that inconsistency between datasets.
Rosa: And once they have this unified dataset, they introduce NVV-TimeBench, which is their expert refined benchmark with six hundred sixty-seven utterances and one thousand ninety-four events <ref:2609.09940#pg1,benchmark with 667 utterances and 1,094 events>. This benchmark seems important because it gives a concrete way to measure performance beyond just looking at simple clip classification.
Dev: But the core of their proposed model is NVV-Locator itself, which they call a lightweight non-autoregressive slot-filling model. They use this architecture to do two things simultaneously: estimate the temporal boundaries of the lexical units, and identify locally associated NVV events along with their categories and onset and offset times.
Taro: A non-autoregressive approach sounds interesting for real-time performance, but what’s the actual mechanism for how it avoids those temporal inconsistencies they mentioned?
Dev: They state that by avoiding autoregressive timestamp decoding, they mitigate temporal inconsistency, which leads to strong temporal grounding performance. The model inserts a five-slot template after every transcript unit to encode all that information about the lexical start and end times, the NVV type, and those onset and offset times.
Paper summary: Rosa: So the training optimization involves three different losses working together: one for lexical timestamps, one for NVV type, and a third one specifically for event timestamps. They define this event timestamp loss by looking at how much the predicted boundary aligns with the actual acoustic energy profile within a five millisecond analysis window.
Taro: That sounds like they are grounding the prediction in physical audio characteristics rather than just relying on language model predictions for time. What about when things go wrong? If you're running this in real-time, what's the safety case there?
Dev: The loss function is set up to prevent the model from learning arbitrary timestamps for events that aren't actually there, ensuring that event boundary gradients only come from acoustically grounded NVV instances. That’s a crucial part of their safety mechanism in terms of avoiding false positives.
Rosa: And they claim they achieved very good results on NVV-TimeBench, reaching a Micro F1 of seventy-one point zero percent and a Macro F1 of seventy point two percent. They also report a Macro mIoU of eighty point four percent and a Macro mean absolute error for the boundary at fifty-nine point six milliseconds.
Taro: Fifty-nine point six milliseconds for boundary error is quite precise, especially when you compare it to Gemini-two point five-Pro, which they state is an improvement of thirty three point zero milliseconds in boundary error. That’s a measurable gain in accuracy for the actual sound timing.
Dev: They also looked at how different types of events perform, and things like Whistle or Cough achieved very high Event-Level F1 scores with relatively tight boundaries, while sustained or repetitive events tended to have larger boundary errors. Lip Smack, being an ultra-short transient under fifty milliseconds, was noted as being easily masked by background noise.
Rosa: And for generalization, they tested NVV-Locator on WESR-Bench without any specific fine-tuning on the dataset. They found it still performed well with a Micro F1 of seventy point two percent and a Macro F1 of forty-five point seven percent. This suggests the model has learned something about NVVs that isn't just specific to the distribution of NVV-TimeBench data, which is pretty promising for real-world use cases.
Taro: So if you take this as a whole, it seems they’ve taken an old problem—just tagging events—and built a system that provides actual waveform-time localization, and they claim it's robust across different datasets.
Dev: The authors conclude by summarizing the entire process: unifying the taxonomy, building that four-stage data pipeline with dual LLM verification, alignment, energy refinement, and augmentation. They introduce NVV-TimeBench and propose NVV-Locator as this non-autoregressive slot-filling model for joint recognition and localization.
Rosa: The main implication for listeners is that this isn't just about recognizing laughter; it’s about understanding the actual acoustic event structure in speech, which could mean better interaction modeling or more robust speech understanding systems down the road.
Dev: It shifts the focus from discrete tags to continuous temporal boundaries, which is what you need if you want a system that can react to subtle changes in vocal behavior over time rather than just recognizing isolated sounds.
Conclusion: Rosa: So, we've been talking about NVV-Locator, which is this new way to pinpoint exactly when nonverbal vocalizations happen on an audio file instead of just tagging them in text.
Dev: Right. It’s this system that takes those tags and tries to find the real acoustic boundaries on the waveform itself, like the exact start and stop times for a sigh or a laugh.
Rosa: The paper is titled "NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding." It’s about taking that gap we talked about—the difference between text tags and actual sound timing—and building a method to bridge it.
Dev: The authors are focused on creating this whole pipeline, from gathering data across different sources to building the model that actually does the grounding.
Rosa: What this means for us is moving beyond just identifying *what* sound happened to understanding the precise temporal structure of speech and non-speech vocalizations. It’s about getting a much finer view of human communication.
Dev: And it shows how important it is to have these detailed boundary estimates because that timing information matters for real-time systems, like voice assistants or interaction modeling.
Rosa: The results on the benchmark show this model actually performs better than some of the existing large models we looked at, especially in terms of temporal overlap and error reduction.
Dev: It’s not just about getting a higher F1 score; it's about reducing that boundary error, which is measured in milliseconds. That reduction matters for how accurate a system needs to be when reacting to vocal cues.
Rosa: So, the big picture here is that we're developing tools that can map human behavior from sound directly onto time on an audio file, which could lead to much smarter ways of understanding and interacting with spoken language.
Dev: And it opens the door for more robust systems where we don't just hear a word, but understand the entire acoustic event structure around that word.
Yuang Cao, Bingshen Mu, Zhennan Lin, Guojian Li, Haoyue Zhan, Jie Liu, Chuan Xie, Qiang Zhang, LLiumeng Xue☁⋆] Lei Xie☁⋆
ASLP@Northwestern Polytechnical University, China · Nanjing University, China · Shanghai Lingguang Zhaxian Technology, China
eess.AS, cs.SD
Submitted: 2026-09-09
Updated: 2026-11-03
Project page: https://nvv-locator.github.io/Demo-Page
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information <ref:2609.09940#pg2> NVV-Locator achieves the…
Key concepts
- Nonverbal Vocalizations (NVVs)
- These are sounds like laughter, sighs, breaths, or coughs that convey feelings or interaction but aren't spoken words. The paper focuses on accurately identifying these specific sounds in audio recordings to understand human communication better.
- Transcript-Conditioned Model
- This model uses the written text of a speech as an input guide while analyzing the raw audio. Instead of just tagging a sound like '[laughter]', it tries to find where that laughter actually starts and stops on the sound wave, making it much more detailed.
- Four-Stage Data Construction Pipeline
- This is a systematic process used to build massive training data for the model. It involves acquiring audio examples, verifying them using two different large language models (LLMs), aligning them with text, refining the exact boundaries using sound energy, and finally balancing the data for better learning.
Terminology
Summary
Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information <ref:2609. p. 009940#pg2> NVV-Locator achieves the best aggregate performance across all three metrics <ref:2609. p. 209940#pg2>.
Problem Statement
Early NVV studies mainly formulate the problem as clip- or utterance-level acoustic event classification, providing limited supervision for continuous-speech localization <ref:2609.09940#pg2> More recent work has collected, synthesized, or annotated NVV corpora through automatic detectthen align pipelines, including NonverbalTTS [3] and NonVerbalSpeech-38K [23]; synthesis- or editing-based approaches, such as SynParaSpeech [1] and CapSpeech [16]; and manually curated or hybrid resources, including MNV-17 [7], SMIIP-NV [17], and NVSpeech-170k [6] <ref:2609. p. 109940#pg2> Many existing approaches still represent NVVs primarily as discrete semantic tags, for example by inserting [laughter] into a transcript <ref:2609. p. 109940#pg2> Such representations capture event identity and textual context, but do not explicitly preserve acoustic onset, offset, or duration on the waveform timeline <ref:2609. p. 109940#pg2> Timestamped NVV data therefore remain scarce, while manual boundary annotation is expensive and difficult to scale <ref:2609. p. 109940#pg2>.
Data Construction Pipeline
To construct large-scale training data for transcript-conditioned NVV temporal grounding under the unified taxonomy, we develop a four-stage automatic data construction pipeline <ref:2609.09940#pg2>
-
Phase 1: Candidate Data Acquisition and Dual-LLM Verification involves acquiring candidate training instances from three complementary sources and subjecting them to a sequential dual-LLM verification procedure <ref:2609. p. 309940#pg2>. Specifically, Qwen3-Omni [18] first analyzes each audio–label pair and produces an initial diagnostic report concerning NVV presence, category consistency, and potential ambiguity <ref:2609. p. 309940#pg2> Gemini-2. p. 95 subsequently reviews the original audio and candidate annotation with the first-stage report provided as auxiliary evidence, and makes the final binary keep/discard decision <ref:2609. p. 309940#pg2>.
-
Phase 2: Coarse Alignment via LLM-ForcedAligner involves removing NVV tags from the corresponding transcription to obtain a lexical transcript and then applying LLM-ForcedAligner [9] to align the lexical transcript with the audio and estimate character-level timestamps for Chinese and word-level timestamps for English <ref:2609. p. 409940#pg2>.
-
Phase 3: Energy-Based Boundary Refinement involves refining the boundary estimates using the local energy profile within W, where we compute an RMS-based short-time energy contour using a 5 ms analysis window and smooth it with a 10 ms moving window <ref:2609. p. 409940#pg2>.
-
Phase 4: Class Balancing and Decoupled Data Augmentation performs class balancing and decoupled data augmentation to mitigate class imbalance and improve discrimination between lexical speech and NVVs <ref:2609. p. 509940#pg2>.
Model Architecture
NVV-Locator is a transcript-conditioned model for fine-grained temporal grounding of nonverbal vocalizations (NVVs) and a dedicated baseline for NVV-TimeBench <ref:2609.09940#pg2> Given a speech recording and its lexical transcript, with all NVV annotations removed, NVV-Locator jointly estimates the temporal boundaries of lexical units and identifies locally associated NVV events, including their categories and acoustic onset and offset times <ref:2609. p. 609940#pg2>. The model uses a non-autoregressive slot-filling architecture <ref:2609. p. 009940#pg2> where a five-slot template is inserted after each transcript unit to encode lexical start and end times, NVV type, NVV onset and offset times <ref:2609. p. 609940#pg2>. The decoder produces representations for all inserted slots in one forward pass <ref:2609. p. 709940#pg2>.
Training Optimization
We optimize the Timestamp & NVV Prediction Layer using three complementary objectives: lexical timestamp loss Ltext time, NVV type loss Lnv type, and NVV timestamp loss Levent time <ref:2609.09940#pg2> The total loss is defined as Ltotal = Ltext time + Lnv type + Levent time <ref:2609. p. 809940#pg2>. The event timestamp loss is defined as Levent time = 1/2omegaNV X i∈omegaNV h CE yˆ(i) nvs, y(i) nvs + CE yˆ(i) nve, y(i) nve i <ref:2609. p. 809940#pg2>. This conditional masking prevents the model from learning arbitrary timestamps for non-existent events and ensures that event-boundary gradients originate only from acoustically grounded NVV instances <ref:2609. p. 809940#pg2>.
Evaluation Results
NVV-Locator achieves the best aggregate performance across all three metrics on NVV-TimeBench <ref:2609.09940#pg2> It attains a Micro F1 of 71. p. 00% and a Macro F1 of 70. p. 02%, improving over the strongest baseline by 25. p. 62 and 20.4 absolute percentage points, respectively <ref:2609. p. 009940#pg2>. It further achieves a Macro mIoU of 80. p. 04% and a Macro mMAE of 59. p. 06 ms, corresponding to a 7. p. 06-point gain in temporal overlap and a 33. p. 00 ms reduction in boundary error relative to Gemini-2. p. 95-Pro <ref:2609. p. 009940#pg2>.
Category Analysis
Performance is shaped by acoustic saliency, event duration, and boundary ambiguity <ref:2609.09940#pg2> Whistle (96. p. 39% F1, 44.2 ms mMAE), Cough (78. p. 01%, 50.0 ms), Sneeze (89. p. 33%, 47.0 ms), and Shush (91. p. 32%, 28.7 ms) achieve high Event-Level F1 with relatively precise boundaries <ref:2609. p. 009940#pg2>. In contrast, sustained or repetitive events can achieve high Event-Level F1 while exhibiting larger boundary errors <ref:2609. p. 709940#pg2>. Lip Smack (21. p. 31% F1) is typically an ultra-short transient, often shorter than 50 ms, and can be easily masked by reverberation or background noise <ref:2609.09940#pg2>.
Generalization
We further evaluate NVV-Locator on WESR-Bench to assess its cross-corpus transferability <ref:2609.09940#pg2> After mapping its labels to our unified taxonomy, NVV-Locator is evaluated zero-shot without dataset-specific fine-tuning, achieving a Micro F1 of 70. p. 92% and a Macro F1 of 45. p. 07% <ref:2609. p. 009940#pg2>. The zero-shot performance suggests that NVV-Locator retains effective NVV event recognition ability beyond the NVV-TimeBench distribution <ref:2609. p. 909940#pg2>.
Conclusion
In this paper, we present a scalable approach to fine-grained nonverbal vocalization (NVV) temporal grounding, covering taxonomy construction, automatic timestamp annotation, benchmark curation, and model formulation <ref:2609.09940#pg2> We unify 26 NVV categories and develop a four-stage data construction pipeline that combines dual-LLM verification, transcript-guided alignment, energy-based boundary refinement, and decoupled augmentation <ref:2609. p. 009940#pg2> We further introduce NVV-TimeBench and propose NVV-Locator, a non-autoregressive slot-filling model for joint NVV recognition and waveform-time localization <ref:2609. p. 209940#pg2>. Experimental results show that the proposed approach substantially outperforms the evaluated large audio model counterparts, while ablation studies validate the importance of the data construction pipeline for both event recognition and boundary localization <ref:2609.09940#pg2>. Evaluation on an external corpus further supports its cross-corpus generalization <ref:2609. p. 009940#pg2>.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Unified 26-category taxonomy allows for cross-corpus integration by harmonizing existing resources, resolving annotation fragmentation across datasets to ensure consistency in physical vocal event labeling and functional abstraction for lexicalized expressions.
-
A four-stage automated data construction pipeline—including DualLLM Verification, Transcript-Guided Coarse Alignment via LLM-ForcedAligner, Energy-Based Boundary Refinement, and Decoupled Data Augmentation—creates a large scale training dataset with timestamp supervision to resolve the scarcity of continuous speech localization data.
-
NVV-Locator, a nonautoregressive slot filling model, jointly predicts lexical timestamps, NVV categories, and event boundaries in parallel rather than autoregressively, mitigating temporal inconsistency and achieving fine-grained waveform time grounding.
-
The system can perform zero shot evaluation on external corpora like WESR-Bench by mapping labels to the unified taxonomy without target corpus training, demonstrating cross corpus generalization capability beyond the initial benchmark distribution.
-
The model can identify specific high saliency events with precise boundaries, such as Whistle (96.9% F1) and Cough (78.1% F1), while maintaining accurate temporal localization even for sustained events by using energy-based refinement to reduce mismatch between transcript-derived lexical gaps and acoustic event extent.
-
The system can distinguish between different event types based on acoustic cues; for example, it shows high Event-Level F1 scores for abrupt transitions like Sniff (85.0% F1) compared to sustained events like Laughter (83.7% F1), allowing the model to adapt its temporal prediction strategy based on event characteristics.
-
The system can leverage category-aware Hungarian matching during checkpoint selection, focusing on event-level temporal grounding quality measured by t-IoU rather than just token accuracy, ensuring that selected checkpoints prioritize accurate NVV event recognition and boundary localization.
Sources
- NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- NVSpeech: An Integrated and Scalable Pipeline for Human-Like Speech Modeling with Paralinguistic Vocalizations
- LLM-ForcedAligner: A Non-Autoregressive and Accurate LLM-Based Forced Aligner for Multilingual and Long-Form Speech
- NV-Bench: Benchmark of Nonverbal Vocalization Synthesis for Expressive Text-to-Speech Generation
- Qwen3-ASR Technical Report
- Step-Audio-R1 Technical Report
- Qwen3-Omni Technical Report
- NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
- MOSS-Audio Technical Report
- Beyond Words: Towards Effective Modeling of Non-Verbal Vocalizations in ASR
- A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models