Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing
summary
The gist
Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls.
In short
CORTIS adapts speech-language models for task-oriented voice agents by fine-tuning them only on text instructions and structured outputs, avoiding the need for expensive paired speech data. This method uses text supervision to transfer adaptation to spoken queries at inference time, improving performance in semantic parsing and robustness under noise.
Key concepts
- Text-Only Adaptation
- This technique involves fine-tuning a large language model using only text instructions paired with structured outputs. The model leverages its existing speech-text alignment from pretraining to apply this text adaptation directly to spoken inputs during the final task.
- Text-Form Task Supervision
- Instead of needing specific speech-target pairs, the training signal is built entirely from text examples formatted as instructions and their corresponding structured outputs. This allows for adaptation without collecting costly paired speech data.
- Inference Time Transfer
- The core idea is that the adaptation learned during text fine-tuning can be transferred to spoken inputs at inference time. The model receives a spoken query in the same format as the training text, allowing it to generate structured outputs directly from speech.
Terminology used across episodes
This episode discusses
- Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing · Paper Radio
- Multimodal Audio-textual Architecture for Robust Spoken Language Understanding
- Text-only adaptation in LLM-based ASR through text denoising · Paper Radio
- Octopus v2: On-device language model for super agent · Paper Radio
- Qwen2-Audio Technical Report
- Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades
- Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
- MUSAN: A Music, Speech, and Noise Corpus
- BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
- Qwen2.5-Omni Technical Report
- Qwen2 Technical Report
- Qwen2.5 Technical Report
- Tuning Large language model for End-to-end Speech Translation
The paper
Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing · Read on arXiv
Youngwon Choi, Hyeonyu Kim, Taeyoun Kwon, Donghyuk Jung, Myeongkyun Cho
MAUM AI Inc. · Seoul National University · Korea Culture Technology Institute · KAIST
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing".
Title and authors: Tom: So let's talk about what this paper actually proposes with this "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" framework. The core idea is to fine-tune the spoken language models using only text-form task supervision.
Jane: That means they take a user instruction and a structured output, like telling the model, "How is the weather in Pangyo?" and then it learns from that text pair how to generate that structured response when it hears actual speech later.
Lu: The key mechanism here is pruning redundant reasoning tokens. They adapt the SLMs on this text-form task data and rely on their existing, pre-trained alignment between speech and text to transfer that learning over to real spoken inputs at inference time.
Meng: I'm curious about the training signal itself. If you aren't using speech data during fine-tuning, how does the model actually learn to connect the spoken words it hears in production back to those structured outputs?
Lalam: The training objective they use is defined as Ltext theta equals negative sum i from one to N log p theta of y sub i given T of x sub i where Dtext is text and N is the number of examples. This means they are fine-tuning based on predicting the correct structured output text given a task-specific prompt template for the user query.
The paper's summary: Tom: What’s interesting here is that they aren't just doing some general text fine-tuning; they are specifically using instruction following format to teach the model how to parse speech into structured outputs. They mention evaluating this on two different Qwen2 point 5 Omni backbones and three different task-oriented speech datasets <ref:2606.21453#pg1,Qwen2.5 Omni backbones and three>.
Jane: The summary really hammers home the gap they’re filling: while prior work explored text-only adaptation in general, this paper focuses specifically on task-oriented voice agents where it’s unclear how text-only models stack up against traditional ASR with LLM cascades trained on the same text supervision.
Lu: They are testing whether these text-only fine-tuned SLMs can reliably generate structured outputs directly from speech, and they compare them against those more conventional ASR with LLM cascades that were also trained using the exact same text-form supervision.
Meng: So, in plain terms, the paper is asking: can we get better structured output directly from speech by only training on text examples, instead of needing paired audio and answer examples?
Lalam: That’s right. CORTIS fine-tunes the model on text instructions paired with structured outputs while keeping the speech modules frozen. Then, at inference time, the adapted model receives a spoken user query in that same query slot it saw during training and generates the output directly.
The paper's improvements: Tom: One of the main improvements they highlight is how CORTIS performs competitively with those ASR with LLM cascades that were trained using matched text-form supervision, especially under acoustic degradation.
Jane: They show clear advantages when the audio quality gets worse. For example, in their in-house function calling results, CORTIS surpasses the cascade when there’s noise at five decibels or less and reaches an absolute error margin gain of four point five seven at zero decibels.
Lu: Qualitatively, they suggest that while CORTIS is good at recovering the high-level task semantics directly from speech, it might still struggle with getting the exact lexical grounding for specific slot values. That’s a known limitation they point out themselves.
Meng: So, if I'm deploying this in a real application where users might be talking in a busy environment, I need to know that this performance boost holds up against actual noisy traffic on production systems.
Lalam: The key is that CORTIS shows the clearest gains specifically in intent accuracy on SLURP, which indicates that direct speech-based inference is particularly effective when predicting those high-level semantic predictions.
Conclusion: Tom: So to wrap up, the paper with the title "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" suggests that using text-only adaptation can be a viable way to get speech models to generate structured outputs without needing paired speech and target data during the adaptation step.
Jane: The main implication is that for task-oriented voice agents, this method offers a competitive path against traditional ASR cascades when dealing with noise, even if it still has some issues with exact details like slot values.
Lu: The paper confirms that by focusing on text-form supervision, we can transfer that adaptation to speech inputs at inference time using the model’s pre-trained crossmodal alignment.
Meng: From a deployment standpoint, this means less data collection upfront, which is good for startups and smaller teams building these voice agents. But we still have to watch how it handles those edge cases where the acoustic conditions are really tough.
Lalam: I think CORTIS provides a practical framework that reduces the dependency on expensive speech-target annotations while still letting us leverage the power of spoken language models for direct structured output generation at inference time.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game