RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

summary

The gist

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

In short

RECOVER is an agentic framework that corrects entity recognition errors in Automatic Speech Recognition (ASR). It uses multiple ASR hypotheses to find evidence, retrieves relevant entities, and applies constrained Large Language Model (LLM) correction. This method significantly reduces entity-phrase word error rate while maintaining overall speech accuracy across diverse domains.

Key concepts

Multi-Hypothesis Generation
This process generates several different possible transcripts from the same audio using various decoding settings. The key is that these different versions of the transcript should have complementary errors, meaning entities missed in one version might be correctly transcribed in another, providing richer evidence for correction.
Dynamic Entity Candidate Retrieval
This tool dynamically selects the most likely entity candidates from a large pool based on every phrase in the audio. It uses a scoring function that checks for exact matches, fuzzy similarities, and phonetic prefix matches to find the top-K relevant terms for correction.
Agentic Orchestration
The framework is structured as an agent that manages three sequential tools: Fuse Hypotheses, Propose Corrections (using an LLM), and Verify & Apply. This modular design allows the system to analyze evidence, suggest fixes based on LLM knowledge under strict rules, and then verify those changes deterministically.

Terminology used across episodes

This episode discusses

The paper

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery · Read on arXiv

Abhishek Kumar, Aashraya Sachdeva

Observe.AI

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air-traffic control, these errors are costly. If an entity is entirely absent from the ASR output, text-only post-correction has little signal to recover it. To address this, we introduce RECOVER, an agentic post-ASR entity-correction framework. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, proposes Large Language Model (LLM) correction under constraints, filters them through deterministic guardrails and uses an LLM critic that judges each surviving edit and can trigger a targeted feedback round to recover an entity the first pass missed. The hypotheses are used by one of several strategies, namely 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, LLM-Select and Adaptive Routing. Evaluated across five diverse datasets over a Whisper-large-v3 baseline, it achieves up to 49% relative reduction in entity-phrase word error rate (E-WER) and increases entity recall by up to 14 percentage points while preserving the overall word error rate (WER), with LLM-Select the strongest strategy.

DOI: 10.18653/2026.emnlp-industry.173

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery".

Jane: Here's the gist of the paper. Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms.

Tom: If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult.

Jane: RECOVER, an agentic correction framework that serves as a tool-using agent, leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints.

Tom: It achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points across five diverse datasets.

Paper summary: Tom: So looking at RECOVER, the core thesis is that using multiple ASR hypotheses and then using a constrained Large Language Model to propose and verify corrections makes a huge difference in finding rare entities Jane It moves us beyond just picking the single best output from the ASR system >

Lu: The title itself says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery Tom That points to the fact that this isn't just about a clever prompt; it’s about building a whole system where different sources of evidence are used together to recover accuracy >

Meng: For deployment, the real implication is that you get much better recall on those tricky terms, which is what matters when you're shipping products and need high precision for specific data types Jane But they also show that while methods like ROVER Ensemble might give a higher recovery score, they can actually increase the overall word error rate on some noisy domains >

Lalam: The paper confirms that substitution errors are the biggest thing we can fix with this constrained LLM editing Tom It's a solid result showing that if you have good evidence and strict rules for what an AI is allowed to change, you get measurable gains in accuracy >

Conclusion: Tom: So, RECOVER is this agentic framework for fixing errors in Automatic Speech Recognition when those specific terms are rare or weird, right?

Jane: Exactly. The title says Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery. It sounds a bit technical, but the main idea is that it uses multiple ASR guesses to find the best answer and then applies a Large Language Model correction under strict rules.

Lu: What I'm digging into is how it does that evidence gathering. They use Multi-Hypothesis Generation to create several transcript guesses, and then they score those hypotheses using exact token matches, fuzzy similarity, and phonetic matching against a list of potential entity candidates.

Meng: From a deployment standpoint, I’m curious about the actual tools. The paper mentions three tools: Fuse Hypotheses, Propose Corrections using an LLM prompt, and Verify and Apply with deterministic guardrails. How complex is that orchestration in practice?

Tom: It sounds modular. They break it down into these distinct steps so you can test each part separately and make sure the AI isn't just making things up when it tries to correct the text. That structure is pretty clean, if you ask me.

Jane: And the results are pretty strong for recall. They see gains of up to twenty-two percentage points in entity recall across different datasets, like on Common Voice where speech is very varied. So, the naive question is, what does this actually mean for someone who just listens to a podcast?

Lu: It means that even if the initial ASR makes a mistake—like mishearing a rare medical term—this system has multiple chances to find it. The gains come mostly from finding entities that were totally missed in one guess but caught in another.

Meng: But what about the cost of running this? They tested it across five different domains, including financial and medical stuff. It’s not just about high recall; we have to worry about the overall word error rate on noisy data, where things like ATCO2 comes into play.

Tom: Yeah, and they show that substitution errors—swapping one word for another—are the main thing getting fixed by this LLM correction, accounting for a huge chunk of the total error reduction. That's concrete evidence.

Jane: So, to wrap up on this paper: RECOVER shows that combining multiple ASR hypotheses with a constrained Large Language Model gives you reliable entity recovery without completely wrecking the overall accuracy of the transcription. Next time we talk, we're looking at how they actually tested this LLM correction against other models and what those specific numbers tell us about its reliability.

More episodes

← Home