Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing

summary

The gist

Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls.

In short

CORTIS adapts speech-language models for task-oriented voice agents by fine-tuning them only on text instructions and structured outputs, avoiding the need for expensive paired speech data. This method uses text supervision to transfer adaptation to spoken queries at inference time, improving performance in semantic parsing and robustness under noise.

Key concepts

Text-Only Adaptation
This technique involves fine-tuning a large language model using only text instructions paired with structured outputs. The model leverages its existing speech-text alignment from pretraining to apply this text adaptation directly to spoken inputs during the final task.
Text-Form Task Supervision
Instead of needing specific speech-target pairs, the training signal is built entirely from text examples formatted as instructions and their corresponding structured outputs. This allows for adaptation without collecting costly paired speech data.
Inference Time Transfer
The core idea is that the adaptation learned during text fine-tuning can be transferred to spoken inputs at inference time. The model receives a spoken query in the same format as the training text, allowing it to generate structured outputs directly from speech.

Terminology used across episodes

This episode discusses

The paper

Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing · Read on arXiv

Youngwon Choi, Hyeonyu Kim, Taeyoun Kwon, Donghyuk Jung, Myeongkyun Cho

MAUM AI Inc. · Seoul National University · Korea Culture Technology Institute · KAIST

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing".

Title and authors: Tom: So let's talk about what this paper actually proposes with this "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" framework. The core idea is to fine-tune the spoken language models using only text-form task supervision.

Jane: That means they take a user instruction and a structured output, like telling the model, "How is the weather in Pangyo?" and then it learns from that text pair how to generate that structured response when it hears actual speech later.

Lu: The key mechanism here is pruning redundant reasoning tokens. They adapt the SLMs on this text-form task data and rely on their existing, pre-trained alignment between speech and text to transfer that learning over to real spoken inputs at inference time.

Meng: I'm curious about the training signal itself. If you aren't using speech data during fine-tuning, how does the model actually learn to connect the spoken words it hears in production back to those structured outputs?

Lalam: The training objective they use is defined as Ltext theta equals negative sum i from one to N log p theta of y sub i given T of x sub i where Dtext is text and N is the number of examples. This means they are fine-tuning based on predicting the correct structured output text given a task-specific prompt template for the user query.

The paper's summary: Tom: What’s interesting here is that they aren't just doing some general text fine-tuning; they are specifically using instruction following format to teach the model how to parse speech into structured outputs. They mention evaluating this on two different Qwen2 point 5 Omni backbones and three different task-oriented speech datasets <ref:2606.21453#pg1,Qwen2.5 Omni backbones and three>.

Jane: The summary really hammers home the gap they’re filling: while prior work explored text-only adaptation in general, this paper focuses specifically on task-oriented voice agents where it’s unclear how text-only models stack up against traditional ASR with LLM cascades trained on the same text supervision.

Lu: They are testing whether these text-only fine-tuned SLMs can reliably generate structured outputs directly from speech, and they compare them against those more conventional ASR with LLM cascades that were also trained using the exact same text-form supervision.

Meng: So, in plain terms, the paper is asking: can we get better structured output directly from speech by only training on text examples, instead of needing paired audio and answer examples?

Lalam: That’s right. CORTIS fine-tunes the model on text instructions paired with structured outputs while keeping the speech modules frozen. Then, at inference time, the adapted model receives a spoken user query in that same query slot it saw during training and generates the output directly.

The paper's improvements: Tom: One of the main improvements they highlight is how CORTIS performs competitively with those ASR with LLM cascades that were trained using matched text-form supervision, especially under acoustic degradation.

Jane: They show clear advantages when the audio quality gets worse. For example, in their in-house function calling results, CORTIS surpasses the cascade when there’s noise at five decibels or less and reaches an absolute error margin gain of four point five seven at zero decibels.

Lu: Qualitatively, they suggest that while CORTIS is good at recovering the high-level task semantics directly from speech, it might still struggle with getting the exact lexical grounding for specific slot values. That’s a known limitation they point out themselves.

Meng: So, if I'm deploying this in a real application where users might be talking in a busy environment, I need to know that this performance boost holds up against actual noisy traffic on production systems.

Lalam: The key is that CORTIS shows the clearest gains specifically in intent accuracy on SLURP, which indicates that direct speech-based inference is particularly effective when predicting those high-level semantic predictions.

Conclusion: Tom: So to wrap up, the paper with the title "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" suggests that using text-only adaptation can be a viable way to get speech models to generate structured outputs without needing paired speech and target data during the adaptation step.

Jane: The main implication is that for task-oriented voice agents, this method offers a competitive path against traditional ASR cascades when dealing with noise, even if it still has some issues with exact details like slot values.

Lu: The paper confirms that by focusing on text-form supervision, we can transfer that adaptation to speech inputs at inference time using the model’s pre-trained crossmodal alignment.

Meng: From a deployment standpoint, this means less data collection upfront, which is good for startups and smaller teams building these voice agents. But we still have to watch how it handles those edge cases where the acoustic conditions are really tough.

Lalam: I think CORTIS provides a practical framework that reduces the dependency on expensive speech-target annotations while still letting us leverage the power of spoken language models for direct structured output generation at inference time.

More episodes

← Home