RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

arXiv:2603.16411 · cs.CL, eess.AS · Submitted 2026-03-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery".

Jane: Here's the gist of the paper. Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

Tom: If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult.

Jane: RECOVER, an agentic correction framework that serves as a tool-using agent, leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints.

Tom: It achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points across five diverse datasets.

Paper summary: Tom: So looking at RECOVER, the core thesis is that using multiple ASR hypotheses and then using a constrained Large Language Model to propose and verify corrections makes a huge difference in finding rare entities Jane It moves us beyond just picking the single best output from the ASR system >

Lu: The title itself says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery Tom That points to the fact that this isn't just about a clever prompt; it’s about building a whole system where different sources of evidence are used together to recover accuracy >

Meng: For deployment, the real implication is that you get much better recall on those tricky terms, which is what matters when you're shipping products and need high precision for specific data types Jane But they also show that while methods like ROVER Ensemble might give a higher recovery score, they can actually increase the overall word error rate on some noisy domains >

Lalam: The paper confirms that substitution errors are the biggest thing we can fix with this constrained LLM editing Tom It's a solid result showing that if you have good evidence and strict rules for what an AI is allowed to change, you get measurable gains in accuracy >

Conclusion: Tom: So, RECOVER is this agentic framework for fixing errors in Automatic Speech Recognition when those specific terms are rare or weird, right?

Jane: Exactly. The title says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery. It sounds a bit technical, but the main idea is that it uses multiple ASR guesses to find the best answer and then applies a Large Language Model correction under strict rules.

Lu: What I'm digging into is how it does that evidence gathering. They use Multi-Hypothesis Generation to create several transcript guesses, and then they score those hypotheses using exact token matches, fuzzy similarity, and phonetic matching against a list of potential entity candidates.

Meng: From a deployment standpoint, I’m curious about the actual tools. The paper mentions three tools: Fuse Hypotheses, Propose Corrections using an LLM prompt, and Verify and Apply with deterministic guardrails. How complex is that orchestration in practice?

Tom: It sounds modular. They break it down into these distinct steps so you can test each part separately and make sure the AI isn't just making things up when it tries to correct the text. That structure is pretty clean, if you ask me.

Jane: And the results are pretty strong for recall. They see gains of up to twenty-two percentage points in entity recall across different datasets, like on Common Voice where speech is very varied. So, the naive question is, what does this actually mean for someone who just listens to a podcast?

Lu: It means that even if the initial ASR makes a mistake—like mishearing a rare medical term—this system has multiple chances to find it. The gains come mostly from finding entities that were totally missed in one guess but caught in another.

Meng: But what about the cost of running this? They tested it across five different domains, including financial and medical stuff. It’s not just about high recall; we have to worry about the overall word error rate on noisy data, where things like ATCO2 comes into play.

Tom: Yeah, and they show that substitution errors—swapping one word for another—are the main thing getting fixed by this LLM correction, accounting for a huge chunk of the total error reduction. That's concrete evidence.

Jane: So, to wrap up on this paper: RECOVER shows that combining multiple ASR hypotheses with a constrained Large Language Model gives you reliable entity recovery without completely wrecking the overall accuracy of the transcription. Next time we talk, we're looking at how they actually tested this LLM correction against other models and what those specific numbers tell us about its reliability.

Abhishek Kumar, Aashraya Sachdeva

Observe.AI

cs.CL, eess.AS

Submitted: 2026-03-17

Updated: 2026-11-13

Comments: Accepted to EMNLP 2026 Industry Track

Journal ref: Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2939-2951, Budapest, Hungary

DOI: 10.18653/2026.emnlp-industry.173

Code: https://github.com/SYSTRAN/faster-whisper

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

Key concepts

Multi-Hypothesis Generation
This process generates several different possible transcripts from the same audio using various decoding settings. The key is that these different versions of the transcript should have complementary errors, meaning entities missed in one version might be correctly transcribed in another, providing richer evidence for correction.
Dynamic Entity Candidate Retrieval
This tool dynamically selects the most likely entity candidates from a large pool based on every phrase in the audio. It uses a scoring function that checks for exact matches, fuzzy similarities, and phonetic prefix matches to find the top-K relevant terms for correction.
Agentic Orchestration
The framework is structured as an agent that manages three sequential tools: Fuse Hypotheses, Propose Corrections (using an LLM), and Verify & Apply. This modular design allows the system to analyze evidence, suggest fixes based on LLM knowledge under strict rules, and then verify those changes deterministically.

Terminology

Summary

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. p. 0

Where the paper says it

sachdeva}@observe. ai Abstract Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. p. 0
Where the paper says it

In domains such as finance, medicine, and air traffic control, these errors are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult.

p. 0
Where the paper says it

In domains such as finance, medicine, and air traffic control, these errors are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult.

RECOVER, an agentic correction framework that serves as a tool-using agent, leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. p. 0
Where the paper says it

To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints.

It achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points across five diverse datasets. p. 0
Where the paper says it

The hypotheses are used using different strategies, namely, 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, and LLM-Select. Evaluated across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points.

The LLM-Select achieves the best overall performance in entity correction while maintaining overall WER. p. 0
Where the paper says it

Evaluated across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points. The LLM-Select achieves the best overall performance in entity correction while maintaining overall WER.

Pruning redundant reasoning tokens

The framework is decomposed into three agent tools (Fuse Hypotheses, Propose Corrections, Verify & Apply) which are orchestrated by an agent. These tools work in sequence to produce the corrected transcript. p. 1

Where the paper says it

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.

The process involves Multi-Hypothesis Generation to produce transcript hypotheses, Dynamic Entity Candidate Retrieval to select the top-K entity candidates (K=200), and three agent tools: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections), and Tool 3 (Verify & Apply). p. 1
Where the paper says it

Given a list of known entity phrases and one or more ASR hypotheses, our goal is to correct entity-level errors in the transcript while leaving the rest of the text intact. To address this, we propose a framework as in Figure 1: Multi-Hypothesis Generation to produce transcript hypotheses, Dynamic Entity Candidate Retrieval to select the top-K entity candidates (K=200), and three agent tools: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections), and Tool 3 (Verify & Apply).

How the training signal is built

The framework utilizes Multi-Hypothesis Generation to produce N ASR hypotheses for each audio segment, which can be obtained through various methods such as temperature sampling from a single ASR model. The key requirement for these hypotheses is that when N > 1, the hypotheses contain complementary errors so that entity tokens missed or corrupted in one hypothesis may be correctly transcribed in another. p. 1

Where the paper says it

The hypotheses can be obtained using any method, such as temperature sampling from a single ASR model, beam search with diverse decoding, or an ensemble of different ASR systems. The key requirement is that, when N > 1, the hypotheses contain complementary errors so that entity tokens missed or corrupted in one hypothesis may be correctly transcribed in another.

Dynamic Entity Candidate Retrieval dynamically retrieves the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list. p. 1
Where the paper says it

Sending all phrases in the LLM prompt is wasteful and may degrade quality. We therefore dynamically retrieve the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list: score(b) = nexact · we | {z } exact token hits + fbest · wf | {z } fuzzy similarity + phit · wp | {z } phonetic prefix (1) where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a 5-character prefix with any token’s phonetic key from any hypothesis.

This scoring function includes exact token hits, fuzzy similarity, and phonetic prefix matching. p. 1
Where the paper says it

Sending all phrases in the LLM prompt is wasteful and may degrade quality. We therefore dynamically retrieve the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list: score(b) = nexact · we | {z } exact token hits + fbest · wf | {z } fuzzy similarity + phit · wp | {z } phonetic prefix (1) where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a 5-character prefix with any token’s phonetic key from any hypothesis.

Where the gains come from

The dominant gain across all datasets comes from entity recall, with gains ranging from +0.9 pp on ContextASR-Bench to +21. p. 2

Where the paper says it

4 6,198 42,449 Medical terms Common Voice 16,401 27. 1 3,098 7,889 Derived Named Entities ContextASR-Bench 5,273 221.

5 pp on Common Voice. p. 2
Where the paper says it

EkaMedical1 [32] is a medical ASR evaluation dataset with 3,619 utterances and 6,198 medical entity phrases covering diseases, symptoms, medications, and procedures. Common Voice [33] Corpus-22 English test set has 16,401 utterances.

The results show that substitution reduction is the dominant correction mechanism, accounting for 63–93% of the total error reduction. p. 3
Where the paper says it

Table 3 presents the entity-level SCLITE alignment counts for the baseline and LLM-Select method across all five datasets. Substitution reduction is the dominant correction mechanism, accounting for 63–93% of the total error reduction.

The constrained LLM correction primarily resolves entity near-misses, such as saitiva→cytiva. p. 3
Where the paper says it

This shows that the constrained LLM correction primarily resolves entity near-misses (e. , saitiva→cytiva, left hansa→lufthansa, chronic holes→chronicles).

Selection-based strategies like 1-Best and LLM-Select preserve or improve overall WER on Medical and ContextASR-Bench, while merge-based fusion can increase overall WER on noisy domains like ATCO2. p. 3
Where the paper says it

Table 2 shows that selection-based strategies, 1-Best and LLM-Select, preserve or improve overall WER (Medical: 17. 93%, ContextASR-Bench: 9.

Evaluation across diverse domains

Comprehensive evaluations were conducted across five diverse datasets spanning financial, ATC, medical, general speech, and dialogue domains. The framework demonstrates 8–46% relative entity-WER reduction. p. 3

Where the paper says it

Conclusions We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes.

LLM-Select is the most consistent strategy overall, achieving the best or near-best E-WER on four of five datasets. p. 2
Where the paper says it

Among multi-hypothesis methods, LLM-Select is the most consistent strategy, achieving the best or near-best E-WER on four of five datasets. This strategy also preserves or improves the overall WER.

The results confirm that deletion and substitution of entity phrases are the dominant ASR failure modes. p. 3
Where the paper says it

Conclusions We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes.

The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification. p. 3
Where the paper says it

Even a single greedy hypothesis benefits substantially from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is the most consistent strategy overall, Entity-Aware Select excels on well-defined financial entities, and ROVER Ensemble offers strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected before application.

Future work

Future work will explore adaptive per-segment strategy selection based on inter-hypothesis agreement. This includes integration with word-level ASR confidence scores, comparison with more LLMs, and extension to multi-agent architectures. p. 3

Where the paper says it

The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected before application. Future work will explore adaptive per-segment strategy selection based on inter-hypothesis agreement, integration with word-level ASR confidence scores, comparison with more LLMs, and extension to multi-agent architectures.

The paper concludes that the three-tool agentic architecture enables modular ablation and ensures that LLM hallucinations are rejected before application. p. 3
Where the paper says it

Even a single greedy hypothesis benefits substantially from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is the most consistent strategy overall, Entity-Aware Select excels on well-defined financial entities, and ROVER Ensemble offers strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected before application.

The authors also report an ablation with GPT-4o-mini on the best performing LLM-Select strategy, showing entity gains in the range of 2. p. 2
Where the paper says it

We also conduct an ablation study with GPT-4o-mini using the best performing LLM-Select method in Table 2. With RWERR lying in the range of 2.

2–15.7%. The paper is available as arXiv:2603. p. 0
Where the paper says it

We demonstrate 8-46% relative entity-phrase WER (E-WER) reduction and consistent gains across domains. arXiv:2603.

16411v1 [cs. p. 0
Where the paper says it

arXiv:2603. 16411v1 [cs.

CL] 17 Mar 2026. p. 0
Where the paper says it

16411v1 [cs. CL] 17 Mar 2026

--- Page 5 ---

RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes. p. 3

Where the paper says it

Conclusions We have presented RECOVER, an agentic post-ASR entity correction framework that combines multi-hypothesis ASR decoding with constrained LLM editing and deterministic guardrails. Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes.

Even a single greedy hypothesis benefits substantially from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is the most consistent strategy overall, Entity-Aware Select excels on well-defined financial entities, and ROVER Ensemble offers strong deletion recovery at the cost of some overall WER degradation. p. 3
Where the paper says it

Evaluated across five diverse domains, the framework achieves 8–46% relative entity-WER reduction, with entity recall as the primary driver (gains of up to 22 pp), confirming that deletion and substitution of entity phrases are the dominant ASR failure modes. Even a single greedy hypothesis benefits substantially from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is the most consistent strategy overall, Entity-Aware Select excels on well-defined financial entities, and ROVER Ensemble offers strong deletion recovery at the cost of some overall WER degradation.

The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected before application. p. 3
Where the paper says it

Even a single greedy hypothesis benefits substantially from constrained LLM editing, while multi-hypothesis diversity provides further domain-dependent gains: LLM-Select is the most consistent strategy overall, Entity-Aware Select excels on well-defined financial entities, and ROVER Ensemble offers strong deletion recovery at the cost of some overall WER degradation. The three-tool agentic architecture cleanly separates analysis & fusion, correction, and verification, enabling modular ablation and ensuring that LLM hallucinations are rejected before application.

--- Page 4 ---

Table 2 reports overall WER, entity-phrase WER (E-WER) (with RWERR beneath), and entity F1 (with Precision / Recall beneath) for all five datasets. With constrained LLM editing effective across all domains, even the 1-Best strategy achieves 33. p. 2

Where the paper says it

Results and Discussion Table 2 reports overall WER, entity-phrase WER (E-WER) (with RWERR beneath), and entity F1 (with Precision / Recall beneath) for all five datasets. With constrained LLM editing effective across all domains, even the 1-Best strategy achieves 33.

2% RWERR on Earnings-21, 41. p. 2
Where the paper says it

With constrained LLM editing effective across all domains, even the 1-Best strategy achieves 33. 2% RWERR on Earnings-21, 41.

2% on Common Voice, 20. p. 2
Where the paper says it

EkaMedical1 [32] is a medical ASR evaluation dataset with 3,619 utterances and 6,198 medical entity phrases covering diseases, symptoms, medications, and procedures. Common Voice [33] Corpus-22 English test set has 16,401 utterances.

4% on Medical, and 12. p. 0
Where the paper says it

Every proposed replacement is verified by multiple validations. We conduct comprehensive evaluations across five diverse datasets spanning financial, ATC, medical, general speech, and dialogue domains using different correction methods.

7% on ContextASR-Bench. p. 2
Where the paper says it

4 6,198 42,449 Medical terms Common Voice 16,401 27. 1 3,098 7,889 Derived Named Entities ContextASR-Bench 5,273 221.

This confirms that an LLM (GPT-4o) with entity-list guardrails is a strong post-ASR corrector. p. 2
Where the paper says it

7% on ContextASR-Bench. This confirms that an LLM (GPT-4o) with entity-list guardrails is a strong post-ASR corrector.

On Earnings-21, it reaches 33. p. 2
Where the paper says it

This strategy also preserves or improves the overall WER. On Earnings-21, it reaches 33.

4%, within 1. p. 1
Where the paper says it

Sending all phrases in the LLM prompt is wasteful and may degrade quality. We therefore dynamically retrieve the top-K most relevant candidates per segment using a threesignal scoring function applied to every phrase b in the list: score(b) = nexact · we | {z } exact token hits + fbest · wf | {z } fuzzy similarity + phit · wp | {z } phonetic prefix (1) where nexact counts how many words of b appear as exact tokens across all transcript hypotheses (the variants are concatenated to build a unified token set), fbest ∈ [0, 1] is the best normalised Levenshtein similarity between any word of b and any token from any hypothesis (restricted to tokens within ±3 characters), and phit ∈ {0, 1} is 1 if the phonetic key of b shares a 5-character prefix with any token’s phonetic key from any hypothesis.

1 pp of Entity-Aware Select (34. p. 0
Where the paper says it

It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely, 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, and LLM-Select.

5%). On ContextASR-Bench, ROVER Ensemble achieves the highest RWERR (20. p. 2
Where the paper says it

1 pp of Entity-Aware Select (34. On ContextASR-Bench, ROVER Ensemble achieves the highest RWERR (20.

8%) but at the cost of increased overall WER. p. 2
Where the paper says it

On ContextASR-Bench, ROVER Ensemble achieves the highest RWERR (20. 8%) but at the cost of increased overall WER.

--- Page 3 ---

Table 3 presents the entity-level alignment counts for baseline vs. LLMSelect: Correct (C), Substitution (S), Deletion (D), Insertion (I). p. 3

Where the paper says it

86) Table 3: Entity-level alignment counts for baseline vs. LLMSelect: Correct (C), Substitution (S), Deletion (D), Insertion (I).

Substitution reduction is the dominant correction mechanism, accounting for 63–93% of the total error reduction. p. 3
Where the paper says it

Table 3 presents the entity-level SCLITE alignment counts for the baseline and LLM-Select method across all five datasets. Substitution reduction is the dominant correction mechanism, accounting for 63–93% of the total error reduction.

This shows that the constrained LLM correction primarily resolves entity near-misses (e. p. 3
Where the paper says it

Substitution reduction is the dominant correction mechanism, accounting for 63–93% of the total error reduction. This shows that the constrained LLM correction primarily resolves entity near-misses (e.

g., saitiva→cytiva, left hansa→lufthansa, chronic holes→chronicles). p. 3
Where the paper says it

This shows that the constrained LLM correction primarily resolves entity near-misses (e. , saitiva→cytiva, left hansa→lufthansa, chronic holes→chronicles).

On Eka-Medical, the substitution drop (∆=–1,697) is the largest absolute change. p. 3
Where the paper says it

, saitiva→cytiva, left hansa→lufthansa, chronic holes→chronicles). On Eka-Medical, the substitution drop (∆=–1,697) is the largest absolute change.

--- Page 2 ---

Audio

Whisper-small

T ∈ [0.0, 0.8] N=5 transcript hypotheses. p. 1

Where the paper says it

Audio Whisper-small T ∈ {0. 8} N=5 transcript hypotheses Entity phrases Candidate Retrieval exact + fuzzy + phonetic scoring → Top-K entities Tool 1: Fuse Hypotheses Entity scoring + 1-Best / Entity-Aware Select / ROVER Ensemble / LLM-Select Tool 2: Propose Corrections LLM + Top-K entity candidates → find/replace Tool 3: Verify & Apply Guardrail validation & Non-overlapping application Corrected Transcript + Correction log fused text proposals Agentic Orchestration Multi-Hypothesis Generation Dynamic Entity Candidate Retrieval N variants Top-K Figure 1: End-to-end system overview.

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. p. 1
Where the paper says it

Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. p. 1

Where the paper says it

8} N=5 transcript hypotheses Entity phrases Candidate Retrieval exact + fuzzy + phonetic scoring → Top-K entities Tool 1: Fuse Hypotheses Entity scoring + 1-Best / Entity-Aware Select / ROVER Ensemble / LLM-Select Tool 2: Propose Corrections LLM + Top-K entity candidates → find/replace Tool 3: Verify & Apply Guardrail validation & Non-overlapping application Corrected Transcript + Correction log fused text proposals Agentic Orchestration Multi-Hypothesis Generation Dynamic Entity Candidate Retrieval N variants Top-K Figure 1: End-to-end system overview. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses.

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected.

p. 1
Where the paper says it

Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected.

The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails). p. 1
Where the paper says it

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.

p. 1
Where the paper says it

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.

p. 1
Where the paper says it

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.

p. 1
Where the paper says it

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails), producing the corrected transcript.

--- Page 1 ---

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery. Abstract Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. p. 0

Where the paper says it

sachdeva}@observe. ai Abstract Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. p. 0
Where the paper says it

If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent.

It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. p. 0
Where the paper says it

To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints.

--- Page 2 ---

Figure 1: End-to-end system overview. Audio is decoded at five temperatures using Whisper-small, producing five transcript hypotheses. Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).

--- Page 5 ---

Entity phrases are scored against the hypotheses using exact, fuzzy, and phonetic matching and the top-K candidates (K=200) are selected. The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).

--- Page 5 ---

The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails). The agentic orchestrates three tools in sequence: Tool 1 (Fuse Hypotheses), Tool 2 (Propose Corrections via a constrained LLM prompt), and Tool 3 (Verify & Apply with deterministic guardrails).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Multi-hypothesis diversity allows for complementary error exploitation to spot corrections that a single best hypothesis might miss, as stated in the hypotheses contain complementary errors so that entity tokens missed or corrupted in one hypothesis may be correctly transcribed in another.

  2. Agentic architecture decomposes the correction pipeline into three specialized tools: Fuse Hypotheses, Propose Corrections, and Verify & Apply to orchestrate the process robustly.

  3. Dynamic Entity Candidate Retrieval uses a threesignal scoring function score(b) = nexact · we exact token hits + fbest · wf fuzzy similarity + phit · wp phonetic prefix to select the top-K entity candidates robustly.

  4. Constrained LLM editing restricts the LLM's proposed edits to exactly one of the entity-list phrases, no generic word rewrites (grammar, punctuation, filler words, casing), ensuring high precision in replacements.

  5. The system achieves 8–46% relative entity-WER reduction and up to 22 percentage points in recall across diverse domains by leveraging this agentic framework with a constrained LLM editor.

Abstract

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air-traffic control, these errors are costly. If an entity is entirely absent from the ASR output, text-only post-correction has little signal to recover it. To address this, we introduce RECOVER, an agentic post-ASR entity-correction framework. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, proposes Large Language Model (LLM) correction under constraints, filters them through deterministic guardrails and uses an LLM critic that judges each surviving edit and can trigger a targeted feedback round to recover an entity the first pass missed. The hypotheses are used by one of several strategies, namely 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, LLM-Select and Adaptive Routing. Evaluated across five diverse datasets over a Whisper-large-v3 baseline, it achieves up to 49% relative reduction in entity-phrase word error rate (E-WER) and increases entity recall by up to 14 percentage points while preserving the overall word error rate (WER), with LLM-Select the strongest strategy.

Sources

Related papers