Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing".
Title and authors: Tom: So let's talk about what this paper actually proposes with this "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" framework. The core idea is to fine-tune the spoken language models using only text-form task supervision.
Jane: That means they take a user instruction and a structured output, like telling the model, "How is the weather in Pangyo?" and then it learns from that text pair how to generate that structured response when it hears actual speech later.
Lu: The key mechanism here is pruning redundant reasoning tokens. They adapt the SLMs on this text-form task data and rely on their existing, pre-trained alignment between speech and text to transfer that learning over to real spoken inputs at inference time.
Meng: I'm curious about the training signal itself. If you aren't using speech data during fine-tuning, how does the model actually learn to connect the spoken words it hears in production back to those structured outputs?
Lalam: The training objective they use is defined as Ltext theta equals negative sum i from one to N log p theta of y sub i given T of x sub i where Dtext is text and N is the number of examples. This means they are fine-tuning based on predicting the correct structured output text given a task-specific prompt template for the user query.
The paper's summary: Tom: What’s interesting here is that they aren't just doing some general text fine-tuning; they are specifically using instruction following format to teach the model how to parse speech into structured outputs. They mention evaluating this on two different Qwen2 point 5 Omni backbones and three different task-oriented speech datasets <ref:2606.21453#pg1,Qwen2.5 Omni backbones and three>.
Jane: The summary really hammers home the gap they’re filling: while prior work explored text-only adaptation in general, this paper focuses specifically on task-oriented voice agents where it’s unclear how text-only models stack up against traditional ASR with LLM cascades trained on the same text supervision.
Lu: They are testing whether these text-only fine-tuned SLMs can reliably generate structured outputs directly from speech, and they compare them against those more conventional ASR with LLM cascades that were also trained using the exact same text-form supervision.
Meng: So, in plain terms, the paper is asking: can we get better structured output directly from speech by only training on text examples, instead of needing paired audio and answer examples?
Lalam: That’s right. CORTIS fine-tunes the model on text instructions paired with structured outputs while keeping the speech modules frozen. Then, at inference time, the adapted model receives a spoken user query in that same query slot it saw during training and generates the output directly.
The paper's improvements: Tom: One of the main improvements they highlight is how CORTIS performs competitively with those ASR with LLM cascades that were trained using matched text-form supervision, especially under acoustic degradation.
Jane: They show clear advantages when the audio quality gets worse. For example, in their in-house function calling results, CORTIS surpasses the cascade when there’s noise at five decibels or less and reaches an absolute error margin gain of four point five seven at zero decibels.
Lu: Qualitatively, they suggest that while CORTIS is good at recovering the high-level task semantics directly from speech, it might still struggle with getting the exact lexical grounding for specific slot values. That’s a known limitation they point out themselves.
Meng: So, if I'm deploying this in a real application where users might be talking in a busy environment, I need to know that this performance boost holds up against actual noisy traffic on production systems.
Lalam: The key is that CORTIS shows the clearest gains specifically in intent accuracy on SLURP, which indicates that direct speech-based inference is particularly effective when predicting those high-level semantic predictions.
Conclusion: Tom: So to wrap up, the paper with the title "Robustness of Text-Only Adapted Speech LLMs for Spoken Semantic Parsing" suggests that using text-only adaptation can be a viable way to get speech models to generate structured outputs without needing paired speech and target data during the adaptation step.
Jane: The main implication is that for task-oriented voice agents, this method offers a competitive path against traditional ASR cascades when dealing with noise, even if it still has some issues with exact details like slot values.
Lu: The paper confirms that by focusing on text-form supervision, we can transfer that adaptation to speech inputs at inference time using the model’s pre-trained crossmodal alignment.
Meng: From a deployment standpoint, this means less data collection upfront, which is good for startups and smaller teams building these voice agents. But we still have to watch how it handles those edge cases where the acoustic conditions are really tough.
Lalam: I think CORTIS provides a practical framework that reduces the dependency on expensive speech-target annotations while still letting us leverage the power of spoken language models for direct structured output generation at inference time.
Youngwon Choi, Hyeonyu Kim, Taeyoun Kwon, Donghyuk Jung, Myeongkyun Cho
MAUM AI Inc. · Seoul National University · Korea Culture Technology Institute · KAIST
cs.HC, cs.AI, cs.SD, eess.AS
Submitted: 2026-06-19
Updated: 2026-11-18
Comments: review in progress
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls.
Key concepts
- Text-Only Adaptation
- This technique involves fine-tuning a large language model using only text instructions paired with structured outputs. The model leverages its existing speech-text alignment from pretraining to apply this text adaptation directly to spoken inputs during the final task.
- Text-Form Task Supervision
- Instead of needing specific speech-target pairs, the training signal is built entirely from text examples formatted as instructions and their corresponding structured outputs. This allows for adaptation without collecting costly paired speech data.
- Inference Time Transfer
- The core idea is that the adaptation learned during text fine-tuning can be transferred to spoken inputs at inference time. The model receives a spoken query in the same format as the training text, allowing it to generate structured outputs directly from speech.
Terminology
Summary
Task-oriented voice agents need to map spoken user requests to structured outputs such as semantic frames, executable actions, and function calls. p. 0 CORTIS presents a text-only adaptation framework for task-oriented voice agents that fine-tunes spoken language models using text-form task supervision, enabling speech-based structured output generation at inference time without taskspecific speech–target annotations during adaptation. p. 0 This method is motivated by the cost of collecting paired speech–target data. p. 1
Pruning redundant reasoning tokens
The core idea behind CORTIS is to adapt SLMs on text-form task data, relying on their pretrained speech–text alignment to transfer this adaptation to speech inputs at inference time. The fine-tuning objective is defined as Ltext(θ) = −∑i=1 log pθ (yi T(xi)) where Dtext = 文本 and N i=1. p. 2 This process involves converting each example into an instruction-following format using a task-specific prompt template T(·). p. 2
How the training signal is built
The text-only adaptation relies solely on text instructions paired with structured outputs, denoted as Dtext = 文本 and N i=1. Unlike speech-supervised adaptation, CORTIS does not use task-specific speech–target pairs (ai, yi) during fine-tuning. p. 2 The SLM is fine-tuned with the standard next-token prediction objective Ltext(θ) = −∑i=1 log pθ (yi T(xi)). p. 2 At inference time, the fine-tuned SLM receives the spoken user query ai in the same query slot where the text query xi appears during training. p. 2
Where the gains come from
The evaluation compares CORTIS-tuned SLMs against ASR–LLM cascades trained with matched text-form supervision. Results show that CORTIS performs competitively with matched cascades and offers clearer advantages under acoustic degradation, particularly in preserving high-level task semantics. p. 0 Specifically, CORTIS shows the clearest gains in intent accuracy on SLURP, indicating that direct speech-based inference is particularly effective for high-level semantic prediction. p. 5
Robustness under acoustic degradation
The advantage of CORTIS becomes more pronounced under noisy conditions when compared to the ASR–LLM cascade. For instance, in the in-house function calling results, CORTIS surpasses the cascade at 5 dB and below, reaching an absolute EM gain of 4. p. 557 at 0 dB. The qualitative analysis suggests that while CORTIS recovers high-level task semantics directly from speech, it may struggle with exact lexical grounding for slot values. p. 5
Limitations and future directions
The work acknowledges several limitations, including the fact that FSC and SLURP are public benchmarks whose data might overlap with pretraining corpora. Furthermore, preliminary experiments with Qwen2-Audio show weaker results than expected, suggesting further investigation is needed to understand generalization across different SLM families. p. 6 The study also focuses on simple text-only fine-tuning and does not explore complementary adaptation strategies like text denoising or incorporating paired speech data. p. 6
Ethical considerations
The in-house data used for evaluation were collected internally for product development, and the speech content, transcripts, and training examples do not contain personally identifiable information. The authors explicitly state that they do not release the in-house dataset. p. 6
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI system can perform speech-based structured output generation at inference time without taskspecific speech–target annotations during adaptation by fine-tuning SLMs using text-form task supervision.
-
The improved system can be applied to task-oriented voice agents, enabling them to map spoken user requests directly to structured outputs like semantic frames, executable actions, and function calls while retaining the ability to process speech input at inference time.
-
CORTIS fine-tunes SLMs on text input–output pairs while keeping the speech-related modules frozen, relying on pretrained crossmodal alignment to transfer adaptation to speech inputs at inference time.
Sources
- Multimodal Audio-textual Architecture for Robust Spoken Language Understanding
- Text-only adaptation in LLM-based ASR through text denoising
- Octopus v2: On-device language model for super agent
- Qwen2-Audio Technical Report
- Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades
- Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
- MUSAN: A Music, Speech, and Noise Corpus
- BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
- Qwen2.5-Omni Technical Report
- Qwen2 Technical Report
- Qwen2.5 Technical Report
- Tuning Large language model for End-to-end Speech Translation
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems