RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery".
Jane: Here's the gist of the paper. Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.
Tom: If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult.
Jane: RECOVER, an agentic correction framework that serves as a tool-using agent, leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints.
Tom: It achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points across five diverse datasets.
Paper summary: Tom: So looking at RECOVER, the core thesis is that using multiple ASR hypotheses and then using a constrained Large Language Model to propose and verify corrections makes a huge difference in finding rare entities Jane It moves us beyond just picking the single best output from the ASR system >
Lu: The title itself says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery Tom That points to the fact that this isn't just about a clever prompt; it’s about building a whole system where different sources of evidence are used together to recover accuracy >
Meng: For deployment, the real implication is that you get much better recall on those tricky terms, which is what matters when you're shipping products and need high precision for specific data types Jane But they also show that while methods like ROVER Ensemble might give a higher recovery score, they can actually increase the overall word error rate on some noisy domains >
Lalam: The paper confirms that substitution errors are the biggest thing we can fix with this constrained LLM editing Tom It's a solid result showing that if you have good evidence and strict rules for what an AI is allowed to change, you get measurable gains in accuracy >
Conclusion: Tom: So, RECOVER is this agentic framework for fixing errors in Automatic Speech Recognition when those specific terms are rare or weird, right?
Jane: Exactly. The title says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery. It sounds a bit technical, but the main idea is that it uses multiple ASR guesses to find the best answer and then applies a Large Language Model correction under strict rules.
Lu: What I'm digging into is how it does that evidence gathering. They use Multi-Hypothesis Generation to create several transcript guesses, and then they score those hypotheses using exact token matches, fuzzy similarity, and phonetic matching against a list of potential entity candidates.
Meng: From a deployment standpoint, I’m curious about the actual tools. The paper mentions three tools: Fuse Hypotheses, Propose Corrections using an LLM prompt, and Verify and Apply with deterministic guardrails. How complex is that orchestration in practice?
Tom: It sounds modular. They break it down into these distinct steps so you can test each part separately and make sure the AI isn't just making things up when it tries to correct the text. That structure is pretty clean, if you ask me.
Jane: And the results are pretty strong for recall. They see gains of up to twenty-two percentage points in entity recall across different datasets, like on Common Voice where speech is very varied. So, the naive question is, what does this actually mean for someone who just listens to a podcast?
Lu: It means that even if the initial ASR makes a mistake—like mishearing a rare medical term—this system has multiple chances to find it. The gains come mostly from finding entities that were totally missed in one guess but caught in another.
Meng: But what about the cost of running this? They tested it across five different domains, including financial and medical stuff. It’s not just about high recall; we have to worry about the overall word error rate on noisy data, where things like ATCO2 comes into play.
Tom: Yeah, and they show that substitution errors—swapping one word for another—are the main thing getting fixed by this LLM correction, accounting for a huge chunk of the total error reduction. That's concrete evidence.
Jane: So, to wrap up on this paper: RECOVER shows that combining multiple ASR hypotheses with a constrained Large Language Model gives you reliable entity recovery without completely wrecking the overall accuracy of the transcription. Next time we talk, we're looking at how they actually tested this LLM correction against other models and what those specific numbers tell us about its reliability.
Abhishek Kumar, Aashraya Sachdeva
Observe.AI
cs.CL, eess.AS
Submitted: 2026-03-17
Updated: 2026-11-13
Comments: Accepted to EMNLP 2026 Industry Track
Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2939-2951, Budapest, Hungary
DOI: 10.18653/2026.emnlp-industry.173
Code: https://github.com/SYSTRAN/faster-whisper
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.
Key concepts
- Multi-Hypothesis Generation
- This process generates several different possible transcripts from the same audio using various decoding settings. The key is that these different versions of the transcript should have complementary errors, meaning entities missed in one version might be correctly transcribed in another, providing richer evidence for correction.
- Dynamic Entity Candidate Retrieval
- This tool dynamically selects the most likely entity candidates from a large pool based on every phrase in the audio. It uses a scoring function that checks for exact matches, fuzzy similarities, and phonetic prefix matches to find the top-K relevant terms for correction.
- Agentic Orchestration
- The framework is structured as an agent that manages three sequential tools: Fuse Hypotheses, Propose Corrections (using an LLM), and Verify & Apply. This modular design allows the system to analyze evidence, suggest fixes based on LLM knowledge under strict rules, and then verify those changes deterministically.
Terminology
Summary
Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. p. 0 sachdeva}@observe. ai
Abstract
Entity recognition in Automatic Speech Recognition (ASR) is
challenging for rare and domain-specific terms. In domains
such as finance, medicine, and air traffic control, these errors
are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. In domains
such as finance, medicine, and air traffic control, these errors
are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this,
we introduce RECOVER, an agentic correction framework that
serves as a tool-using agent. It leverages multiple hypotheses
as evidence from ASR, retrieves relevant entities, and applies
Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely,
1-Best, Entity-Aware Select, Recognizer Output Voting Error
Reduction (ROVER) Ensemble, and LLM-Select. Evaluated
across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases
recall by up to 22 percentage points. Evaluated
across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases
recall by up to 22 percentage points. The LLM-Select achieves
the best overall performance in entity correction while maintaining overall WER.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The framework is decomposed into three agent tools (Fuse Hypotheses, Propose Corrections, Verify & Apply) which are orchestrated by an agent. These tools work in sequence to produce the corrected transcript. p. 1 Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained
LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript. Given a list of known entity phrases and one or more ASR
hypotheses, our goal is to correct entity-level errors in the transcript while leaving the rest of the text intact. To address this,
we propose a framework as in Figure 1: Multi-Hypothesis Generation to produce transcript hypotheses, Dynamic Entity Candidate Retrieval to select the top-K entity candidates (K=200),
and three agent tools: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections), and Tool 3 (Verify & Apply).Where the paper says it
Where the paper says it
How the training signal is built
The framework utilizes Multi-Hypothesis Generation to produce N ASR hypotheses for each audio segment, which can be obtained through various methods such as temperature sampling from a single ASR model. The key requirement for these hypotheses is that when N > 1, the hypotheses contain complementary errors so that entity tokens missed or corrupted in one hypothesis may be correctly transcribed in another. p. 1 The hypotheses can be obtained using any method, such
as temperature sampling from a single ASR model, beam search
with diverse decoding, or an ensemble of different ASR systems. The key requirement is that, when N > 1, the hypotheses contain complementary errors so that entity tokens missed
or corrupted in one hypothesis may be correctly transcribed in
another. Sending all phrases in the LLM prompt is wasteful
and may degrade quality. We therefore dynamically retrieve
the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list:
score(b) = nexact · we
| {z }
exact token hits
+ fbest · wf
| {z }
fuzzy similarity
+ phit · wp
| {z }
phonetic prefix
(1)
where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any
token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a
5-character prefix with any token’s phonetic key from any hypothesis. Sending all phrases in the LLM prompt is wasteful
and may degrade quality. We therefore dynamically retrieve
the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list:
score(b) = nexact · we
| {z }
exact token hits
+ fbest · wf
| {z }
fuzzy similarity
+ phit · wp
| {z }
phonetic prefix
(1)
where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any
token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a
5-character prefix with any token’s phonetic key from any hypothesis.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The dominant gain across all datasets comes from entity recall, with gains ranging from +0.9 pp on ContextASR-Bench to +21. p. 2 4 6,198 42,449 Medical terms
Common Voice 16,401 27. 1 3,098 7,889 Derived Named Entities
ContextASR-Bench 5,273 221. EkaMedical1
[32] is a medical ASR evaluation dataset with 3,619
utterances and 6,198 medical entity phrases covering diseases,
symptoms, medications, and procedures. Common Voice [33]
Corpus-22 English test set has 16,401 utterances. Table 3 presents the entity-level SCLITE alignment counts
for the baseline and LLM-Select method across all five datasets. Substitution reduction is the dominant correction mechanism,
accounting for 63–93% of the total error reduction. This shows
that the constrained LLM correction primarily resolves entity near-misses (e. , saitiva→cytiva, left hansa→lufthansa,
chronic holes→chronicles). Table 2 shows that selection-based
strategies, 1-Best and LLM-Select, preserve or improve overall WER (Medical: 17. 93%, ContextASR-Bench:
9.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Evaluation across diverse domains
Comprehensive evaluations were conducted across five diverse datasets spanning financial, ATC, medical, general speech, and dialogue domains. The framework demonstrates 8–46% relative entity-WER reduction. p. 3 Conclusions
We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves
8–46% relative entity-WER reduction, with entity recall as the
primary driver (gains of up to 22 pp), confirming that deletion
and substitution of entity phrases are the dominant ASR failure
modes. Among multi-hypothesis methods, LLM-Select is the most consistent strategy, achieving the
best or near-best E-WER on four of five datasets. This strategy
also preserves or improves the overall WER. Conclusions
We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves
8–46% relative entity-WER reduction, with entity recall as the
primary driver (gains of up to 22 pp), confirming that deletion
and substitution of entity phrases are the dominant ASR failure
modes. Even a single greedy hypothesis benefits substantially
from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is
the most consistent strategy overall, Entity-Aware Select excels
on well-defined financial entities, and ROVER Ensemble offers
strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates
analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected
before application.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Future work
Future work will explore adaptive per-segment strategy selection based on inter-hypothesis agreement. This includes integration with word-level ASR confidence scores, comparison with more LLMs, and extension to multi-agent architectures. p. 3 The three-tool agentic architecture cleanly separates
analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected
before application. Future work will explore adaptive per-segment strategy selection based on inter-hypothesis agreement, integration with
word-level ASR confidence scores, comparison with more
LLMs, and extension to multi-agent architectures. Even a single greedy hypothesis benefits substantially
from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is
the most consistent strategy overall, Entity-Aware Select excels
on well-defined financial entities, and ROVER Ensemble offers
strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates
analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected
before application. We also conduct an ablation study with GPT-4o-mini using the
best performing LLM-Select method in Table 2. With RWERR
lying in the range of 2. We
demonstrate 8-46% relative entity-phrase WER (E-WER) reduction and consistent gains across domains. arXiv:2603. arXiv:2603. 16411v1 [cs. 16411v1 [cs. CL] 17 Mar 2026Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes. p. 3 Conclusions
We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves
8–46% relative entity-WER reduction, with entity recall as the
primary driver (gains of up to 22 pp), confirming that deletion
and substitution of entity phrases are the dominant ASR failure
modes. Evaluated across five diverse domains, the framework achieves
8–46% relative entity-WER reduction, with entity recall as the
primary driver (gains of up to 22 pp), confirming that deletion
and substitution of entity phrases are the dominant ASR failure
modes. Even a single greedy hypothesis benefits substantially
from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is
the most consistent strategy overall, Entity-Aware Select excels
on well-defined financial entities, and ROVER Ensemble offers
strong deletion recovery at the cost of some overall WER degradation. Even a single greedy hypothesis benefits substantially
from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is
the most consistent strategy overall, Entity-Aware Select excels
on well-defined financial entities, and ROVER Ensemble offers
strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates
analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected
before application.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 4 ---
Table 2 reports overall WER, entity-phrase WER (E-WER) (with RWERR beneath), and entity F1 (with Precision / Recall beneath) for all five datasets. With constrained LLM editing effective across all domains, even the 1-Best strategy achieves 33. p. 2 Results and Discussion
Table 2 reports overall WER, entity-phrase WER (E-WER)
(with RWERR beneath), and entity F1 (with Precision / Recall
beneath) for all five datasets. With constrained LLM editing
effective across all domains, even the 1-Best strategy achieves
33. With constrained LLM editing
effective across all domains, even the 1-Best strategy achieves
33. 2% RWERR on Earnings-21, 41. EkaMedical1
[32] is a medical ASR evaluation dataset with 3,619
utterances and 6,198 medical entity phrases covering diseases,
symptoms, medications, and procedures. Common Voice [33]
Corpus-22 English test set has 16,401 utterances. Every proposed replacement is verified by multiple validations. We conduct comprehensive evaluations across five diverse
datasets spanning financial, ATC, medical, general speech,
and dialogue domains using different correction methods. 4 6,198 42,449 Medical terms
Common Voice 16,401 27. 1 3,098 7,889 Derived Named Entities
ContextASR-Bench 5,273 221. 7% on ContextASR-Bench. This
confirms that an LLM (GPT-4o) with entity-list guardrails is
a strong post-ASR corrector. This strategy
also preserves or improves the overall WER. On Earnings-21,
it reaches 33. Sending all phrases in the LLM prompt is wasteful
and may degrade quality. We therefore dynamically retrieve
the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list:
score(b) = nexact · we
| {z }
exact token hits
+ fbest · wf
| {z }
fuzzy similarity
+ phit · wp
| {z }
phonetic prefix
(1)
where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any
token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a
5-character prefix with any token’s phonetic key from any hypothesis. It leverages multiple hypotheses
as evidence from ASR, retrieves relevant entities, and applies
Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely,
1-Best, Entity-Aware Select, Recognizer Output Voting Error
Reduction (ROVER) Ensemble, and LLM-Select. 1 pp of Entity-Aware Select (34. On ContextASR-Bench, ROVER Ensemble achieves the highest RWERR (20. On ContextASR-Bench, ROVER Ensemble achieves the highest RWERR (20. 8%) but at the cost of increased overall WER.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
Table 3 presents the entity-level alignment counts for baseline vs. LLMSelect: Correct (C), Substitution (S), Deletion (D), Insertion (I). p. 3 86)
Table 3: Entity-level alignment counts for baseline vs. LLMSelect: Correct (C), Substitution (S), Deletion (D), Insertion (I). Table 3 presents the entity-level SCLITE alignment counts
for the baseline and LLM-Select method across all five datasets. Substitution reduction is the dominant correction mechanism,
accounting for 63–93% of the total error reduction. Substitution reduction is the dominant correction mechanism,
accounting for 63–93% of the total error reduction. This shows
that the constrained LLM correction primarily resolves entity near-misses (e. This shows
that the constrained LLM correction primarily resolves entity near-misses (e. , saitiva→cytiva, left hansa→lufthansa,
chronic holes→chronicles). , saitiva→cytiva, left hansa→lufthansa,
chronic holes→chronicles). On Eka-Medical, the substitution
drop (∆=–1,697) is the largest absolute change.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
Audio
Whisper-small
T ∈ [0.0, 0.8] N=5 transcript hypotheses. p. 1 Audio
Whisper-small
T ∈ {0. 8}
N=5 transcript hypotheses
Entity phrases
Candidate Retrieval
exact + fuzzy + phonetic
scoring → Top-K entities
Tool 1: Fuse
Hypotheses
Entity scoring +
1-Best / Entity-Aware
Select / ROVER
Ensemble / LLM-Select
Tool 2: Propose
Corrections
LLM + Top-K entity
candidates →
find/replace
Tool 3: Verify & Apply
Guardrail validation &
Non-overlapping
application
Corrected
Transcript
+
Correction log
fused text proposals
Agentic Orchestration
Multi-Hypothesis Generation Dynamic Entity Candidate Retrieval
N variants
Top-K
Figure 1: End-to-end system overview. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. p. 1 8}
N=5 transcript hypotheses
Entity phrases
Candidate Retrieval
exact + fuzzy + phonetic
scoring → Top-K entities
Tool 1: Fuse
Hypotheses
Entity scoring +
1-Best / Entity-Aware
Select / ROVER
Ensemble / LLM-Select
Tool 2: Propose
Corrections
LLM + Top-K entity
candidates →
find/replace
Tool 3: Verify & Apply
Guardrail validation &
Non-overlapping
application
Corrected
Transcript
+
Correction log
fused text proposals
Agentic Orchestration
Multi-Hypothesis Generation Dynamic Entity Candidate Retrieval
N variants
Top-K
Figure 1: End-to-end system overview. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained
LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained
LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained
LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are
selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained
LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery. Abstract Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. p. 0 sachdeva}@observe. ai
Abstract
Entity recognition in Automatic Speech Recognition (ASR) is
challenging for rare and domain-specific terms. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this,
we introduce RECOVER, an agentic correction framework that
serves as a tool-using agent. To address this,
we introduce RECOVER, an agentic correction framework that
serves as a tool-using agent. It leverages multiple hypotheses
as evidence from ASR, retrieves relevant entities, and applies
Large Language Model (LLM) correction under constraints.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
Figure 1: End-to-end system overview. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).
--- Page 5 ---
Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).
--- Page 5 ---
The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails). The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multi-hypothesis diversity allows for complementary error exploitation to spot corrections that a single best hypothesis might miss, as stated in
the hypotheses contain complementary errors so that entity tokens missed or corrupted in one hypothesis may be correctly transcribed in another.
-
Agentic architecture decomposes the correction pipeline into three specialized tools: Fuse Hypotheses, Propose Corrections, and Verify & Apply to orchestrate the process robustly.
-
Dynamic Entity Candidate Retrieval uses a threesignal scoring function score(b) = nexact · we exact token hits + fbest · wf fuzzy similarity + phit · wp phonetic prefix to select the top-K entity candidates robustly.
-
Constrained LLM editing restricts the LLM's proposed edits to
exactly one of the entity-list phrases, no generic word rewrites (grammar, punctuation, filler words, casing),
ensuring high precision in replacements. -
The system achieves 8–46% relative entity-WER reduction and up to 22 percentage points in recall across diverse domains by leveraging this agentic framework with a constrained LLM editor.
Abstract
Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air-traffic control, these errors are costly. If an entity is entirely absent from the ASR output, text-only post-correction has little signal to recover it. To address this, we introduce RECOVER, an agentic post-ASR entity-correction framework. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, proposes Large Language Model (LLM) correction under constraints, filters them through deterministic guardrails and uses an LLM critic that judges each surviving edit and can trigger a targeted feedback round to recover an entity the first pass missed. The hypotheses are used by one of several strategies, namely 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, LLM-Select and Adaptive Routing. Evaluated across five diverse datasets over a Whisper-large-v3 baseline, it achieves up to 49% relative reduction in entity-phrase word error rate (E-WER) and increases entity recall by up to 14 percentage points while preserving the overall word error rate (WER), with LLM-Select the strongest strategy.
Sources
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Two-Pass End-to-End Speech Recognition
- Earnings-21: A Practical Benchmark for ASR in the Wild
- Contextual RNN-T For Open Domain ASR
- Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network
- Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model
- Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion
- Error Correction in ASR using Sequence-to-Sequence Models
- Improving Domain-Specific ASR with LLM-Generated Contextual Descriptions
- N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space
- Large Language Models based ASR Error Correction for Child Conversations
- Multi-stage Large Language Model Correction for Speech Recognition
- Agentic AI Frameworks: Architectures, Protocols, and Design Challenges
- VoiceBench: Benchmarking LLM-Based Voice Assistants
- VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
- ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications
- ASR Under the Stethoscope: Evaluating Biases in Clinical Speech Recognition across Indian Languages
- ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark
- GPT-4o System Card
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data