A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "A Factorial Ablation of a Speech-to-SFT Pipeline".
Jane: Here's the gist of the paper. Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage’s marginal value unknown <ref:2608.20394#pg2>.
Tom: Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains <ref:2608.20394#pg2>.
Jane: Factorial Design and Evaluation Scope The study designs a productionready speech-to-SFT pipeline featuring a 2 × 2 factorial design by independently toggling transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) <ref:2608.20394#pg2>.
Tom: This setup was applied to Korean medical and finance conference recordings, involving training nine LLMs across five families and varying sizes <ref:2608.20394#pg2>.
Paper summary: Jane: Okay, so looking at this paper "A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer," what's the big picture here for us?
Tom: Basically, the authors set up this test to see how two stages—transcript refinement and data quality refinement—affect downstream question answering when they run a fixed setup.
Lu: And they found that even though both phases improved the QA quality numbers, you couldn't see a significant lift across all models for the final MCQA scores.
Jane: So what does that imply about building these speech fine-tuning pipelines?
Tom: It implies we can’t just optimize every step in isolation; you have to look at how those steps interact with the specific format of your ultimate task.
Lu: The paper is telling us that phase zero refinement, which deals with transcript accuracy, seems to be a stronger contributor to initial quality improvement than the question refinement stage does.
Jane: It suggests that if you're working on a speech-to-SFT pipeline, focusing on cleaning up the transcription might give you more immediate data quality gains.
Tom: And we have to be careful because those gains might not translate directly into better performance on downstream tasks unless the SFT data is also shaped correctly for what those tasks demand.
Lu: It’s a reminder that family or domain alignment is what actually drives positive transfer, not just any uniform improvement across the board.
Jane: So, in simple terms, this paper says that getting your input data right is only half the battle; you also need to make sure the data structure matches what your final benchmark needs.
Conclusion: Tom: So we've seen how these researchers looked at a speech fine-tuning pipeline and ran a factorial test, right? They were looking at two main things: changing the transcript refinement stage and changing the quality refinement stage for their training data.
Jane: Exactly. The main finding is that improving the quality of your input data doesn't automatically mean you get better results on those final downstream question answering tasks.
Lu: That’s what they found, specifically that improvements in QA data quality don't transfer uniformly into gains on the final MCQA scores across different models.
Meng: So, if I’m trying to ship a model fast, and I spend time making my initial training data super clean, does that actually help my deployment performance? What is the actual measurable difference here?
Tom: It seems like the structure of the data matters more than just having cleaner data points. They found that one stage—the transcript refinement—was a stronger driver for initial quality improvement than the second stage.
Jane: That suggests you might want to focus your energy on making sure your speech-to-text transcription is as accurate as possible before you start fine-tuning the language models.
Lu: And they pointed out that the final question type composition shifts toward more explanatory questions when you use that second refinement stage, but those changes don't translate into better performance on factoid recall tasks.
Tom: It’s like they’re moving the goalposts. The data gets better in one direction, but it doesn't help with what the final evaluation is actually testing for.
Jane: So what does this mean for someone building these complex speech systems? It means you need to look at how the data format matches what your final task requires.
Meng: From a deployment standpoint, if the transfer isn't uniform, then spending time on every single pipeline tweak might just be wasted effort unless you have a very specific downstream goal in mind.
Tom: Yeah, so while these two stages both added some quality to the data, they didn't provide that big lift across all models.
Jane: It highlights that family alignment or domain matching is still the most important factor for getting those real gains on performance.
Lu: The study also showed that while one stage contributed more to initial quality, the overall cross-cell effect wasn't statistically significant, emphasizing the importance of that specific alignment between training groups and their target domains.
Tom: So what we’re seeing is a trade-off where you can improve individual parts, but you don't guarantee a better final score without hitting those specific alignment targets.
Jane: It gives us a clearer picture of where to spend our time next when building these speech systems.
Wonsup Shin, Jingu Kim
cs.SD, cs.CL
Submitted: 2026-07-01
Updated: 2026-11-10
Comments: 20 pages, 2 figures; v2: added an evaluation prompt-format analysis and removed an unsupported claim
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 88/100
The gist: Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge…
Key concepts
- Factorial Design
- This experimental setup used a 2x2 design by independently toggling two pipeline stages: transcript refinement and SFT data quality refinement. This allowed researchers to see how changing one stage affects the final outcome, helping to isolate the specific impact of each step.
- QA Quality Improvement Dynamics
- The study measured how much the quality of the QA data improved after applying different pipeline stages. It found that both LLM judges and human experts confirmed improvements in QA quality, but these gains were not evenly distributed across all downstream tasks.
- Downstream Transfer Mismatch
- Improvements in QA data did not lead to consistent gains in downstream multiple-choice question answering (MCQA) benchmarks. This suggests a mismatch: Phase 2 changes the data composition (making questions more explanatory), but the MCQA task primarily tests factoid recall, leading to poor transfer.
- Family-Domain Alignment
- Positive performance transfer was only observed when models were aligned with their specific training family and domain. This indicates that uniform improvement across all models is not possible; instead, performance gains are highly dependent on matching the model's background to the specific task's context.
Terminology
Summary
Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage’s marginal value unknown <ref:2608. p. 0 com
Jingu Kim
Flitto
jin@flitto. com
Abstract
Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage
refinement are increasingly adopted but, to
our knowledge, have not been publicly ablated
stage-by-stage, leaving each stage’s marginal
value unknown. 4B–
70B); we evaluate with four cross-provider
LLM judges, a blind six-expert human evaluation, and 3 downstream MCQA benchmarks. Our central finding: under a fixed, standard
SFT recipe, improvements in QA data quality do not transfer uniformly into downstream
MCQA gains.Where the paper says it
Where the paper says it
Factorial Design and Evaluation Scope
The study designs a productionready speech-to-SFT pipeline featuring a 2 × 2 factorial design by independently toggling transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) <ref:2608.20394#pg2>. This setup was applied to Korean medical and finance conference recordings, involving training nine LLMs across five families and varying sizes <ref:2608. p. 0 We design a productionready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data
quality refinement (Phase 2) are independently toggleable, yielding a 2 × 2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and
fine-tune 9 models (5 LLM families, 2. Evaluation spans five axes: downstream
MCQA (KMMLU, KMMLU-Pro, MMLU) after
SFT, cell-blind QA-quality scoring by 4 LLM
judges (Zheng et al. , 2023) on a 200-QA stratified
public sample, a blind 6-expert human evaluation
of QA quality on the same sample, STT-engine
swap via Whisper-medium (Radford et al.Where the paper says it
Where the paper says it
QA Quality Improvement Dynamics
Improvements in QA data quality do not transfer uniformly into downstream MCQA gains <ref:2608.20394#pg2>. Specifically, the 4-judge LLM and 6-expert human evaluations both confirm robust QA quality improvement (∆2−0 = +0. p. 0 Main contributions. (i) Under a fixed, standard SFT recipe (§4), improvements in QA data
quality do not transfer uniformly into downstream
MCQA gains: 4-judge LLM and 6-expert human evaluations both confirm robust quality improvement (+0. 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. 23 (95% CI [+0. Phase 0 is the stronger single-stage contributor
(P
s
0 = +0. 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. Phase-2 mechanism: composition shift and
within-type quality. Phase 2 shifts questiontype composition toward more explanatory questions (Appendix G, Table 11): Exp 2 is
explanatory-dominant (+10 pp vs. 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Downstream Transfer and Format Mismatch
The differential pattern observed in QA quality does not translate into significant cross-model mean MCQA gains, suggesting a format mismatch where Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall <ref:2608.20394#pg2>. Positive transfer concentrates on family↔domain aligned pairs <ref:2608. p. 0 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. Typestratified analysis (Appendix G) shows withintype gains positive on all 4 types; re-weighting
Exp 2 to Exp 0’s composition yields typecontrolled ∆2−0 ≈ +0. 16, so within-type improvement accounts for ∼86% of ∆2−0 and composition shift ∼ 14% (per-type n = 5–20, Table 12).Where the paper says it
Where the paper says it
Pipeline Robustness and Limitations
The study confirms pipeline robustness through an STT-engine swap to Whisper-medium, which showed minimal aggregate downstream effect (mean absolute KMMLU difference ≤ 1.32 pp) <ref:2608. p. 0 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. (iv)
An LLM-difficulty audit shows that even the two
frontier LLMs (Opus 4. 4) cannot confidently answer 7. 4) cannot confidently answer 7. 8% of the public-sample QA
on average. 8 pp on
MMLU; seed variance (n = 2 per cell, worst-case
σ= 2. 87 pp) dominates per-sample bootstrap variance. Under our standard SFT recipe, 4-
judge LLM and 6-expert human evaluations both
confirm robust QA quality improvement (∆2−0 =
+0. 22), yet the cross-cell mean MCQA
∆2−0 is not significant; positive transfer concentrates on family↔domain aligned cells.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ethics and Release of Artifacts
The authors released the 200-QA sample plus a Phase-2-filtered variant, all SFT checkpoints, judge prompts, and analysis code for reproducibility <ref:2608.20394#pg2>. Data collection involved 40 conference sessions from Korean medical and finance domains with speaker consent <ref:2608. p. 7 Ethics Statement
Data collection and consent. The 40 conference sessions used in this work are a subset of
Korean-language session data collected through
an AI-mediated interpretation service, spanning
academic conferences, professional seminars, and
industry forums in the medical and finance sectors (2024–2026); speakers consented to research
and non-commercial use. The six human
evaluators used in our blind expert evaluation,
three medical and three finance, were recruited externally through referral from in-house contacts. No evaluator had an employment relationship with
the authors or their institution, nor participated
in pipeline design or model selection. The full pipeline implementation is
available upon reasonable request. Downstream
use of the released artifacts should respect the consent scope (research and non-commercial use).Where the paper says it
Where the paper says it
Where the paper says it
References
Amazon AGI. 2025. SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning. p. 7 References
Amazon AGI. SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning. SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning. In
Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). In
Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo,
Thomas Wolf, and Leandro von Werra. , 2023), Evol-Instruct (Xu et al. , 2024), and
Cosmopedia (Ben Allal et al. 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. co /
datasets/HuggingFaceTB/cosmopedia. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley,
Jonathan Frankle, and 1 others. , 2022), for parameterefficient SFT; Biderman et al. (2024) recently
showed that LoRA “learns less and forgets less”
than full fine-tuning, motivating our Full FT sanity
check (§5. Transactions on Machine
Learning Research (TMLR). Joya Chen and 1 others. Joya Chen and 1 others. LiveCC: Learning
video LLM with streaming speech transcription at
scale. In Conference on Computer Vision and Pattern Recognition (CVPR). Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and
Luke Zettlemoyer. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and
Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS). Dan Hendrycks, Collin Burns, Steven Basart, Andy
Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Dan Hendrycks, Collin Burns, Steven Basart, Andy
Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference
on Learning Representations. Hu, Yelong Shen, Phillip Wallis, Zeyuan
Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. Hu, Yelong Shen, Phillip Wallis, Zeyuan
Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. LoRA: Low-rank adaptation of
large language models. In International Conference
on Learning Representations (ICLR). Shuyue Jia and 1 others. Shuyue Jia and 1 others. PodGPT: An audioaugmented large language model for research and
education. Preprint,
arXiv:2402. Jing Pan and 1 others. Jing Pan and 1 others. COSMIC: Data efficient
instruction-tuning for speech in-context learning. In
Interspeech. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. Preprint,
arXiv:2402. Guijin Son and 1 others. Guijin Son and 1 others. From KMMLU-redux
to pro: A professional Korean benchmark suite for
LLM evaluation. Preprint, arXiv:2507. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng,
Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin
Jiang. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng,
Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin
Jiang. WizardLM: Empowering large pretrained language models to follow complex instructions. In International Conference on Learning Representations (ICLR). Zitong Yang, Neil Band, Shuangping Li, Emmanuel
Candès, and Tatsunori Hashimoto. Zitong Yang, Neil Band, Shuangping Li, Emmanuel
Candès, and Tatsunori Hashimoto. Synthetic
continued pretraining. In International Conference
on Learning Representations (ICLR). Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan
Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin,
Zhuohan Li, Dacheng Li, Eric P. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan
Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin,
Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang,
Joseph E. Xing, Hao Zhang,
Joseph E. Gonzalez, and Ion Stoica. Gonzalez, and Ion Stoica. Judging
LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS). A Per-phase token costs
Per-QA token-cost estimates (USD; OpenRouter
unified pricing, snapshot 2026-05), measured on a
representative log sub-sample:
• Phase 0: 0. 050 (analyze + augment +
strategize + generate)
• Phase 2: 0. 013 (judge + refine ≤ 2 +
dedup)
B Seed inventory
Table 7 lists the seed configuration used in every
experiment reported in this paper. 013 (judge + refine ≤ 2 +
dedup)
B Seed inventory
Table 7 lists the seed configuration used in every
experiment reported in this paper. C KMMLU and KMMLU-Pro domain
mapping
KMMLU 45 subjects → domain proxies used in
§5. C KMMLU and KMMLU-Pro domain
mapping
KMMLU 45 subjects → domain proxies used in
§5. 2:
• KMMLU-med (medical-relevant, 4 subjects): health, biology, chemistry,
psychology. • KMMLU-fin (finance-relevant, 6 subjects):
economics, accounting, taxation,
management, marketing, real_estate. MMLU 57 subjects → domain proxies used in
the Full FT sanity (§5. MMLU 57 subjects → domain proxies used in
the Full FT sanity (§5. 5, Appendix F):
• MMLU-med (8 subjects): clinical_knowledge,
medical_genetics, anatomy,
college_medicine,
professional_medicine,
college_biology, nutrition, virology. • MMLU-fin (7 subjects): econometrics,
high_school_macroeconomics,
high_school_microeconomics,
professional_accounting,
business_ethics, management,
marketing. KMMLU-Pro license categories → domain
tasks (filtered by license_name via the task
YAML’s process_docs hook):
• kmmlu_pro_medical (1,205 items): physician, dentist, pharmacist, Korean medicine
doctor, Korean pharmacist. D Session-level metadata
Table 8 lists the 40 conference sessions used in the
main experiment (19 medical + 21 finance). ROW
= transcript line count (sentence-segmented STT
output, proxy for session length). E Phase 2 stage acceptance
The per-condition QA-count spread in Table 2
arises from the Phase 2 pipeline applied to Phase 1
candidates. Table 9 reports per-stage acceptance
rates on the conditions where Phase 2 is active
(Exp 1-2 and Exp 2). The combined effect keeps per-condition training-set sizes within a
∼6% band. F Full FT sanity per-cell results
Stage-wise delta ∆stage = Exp2 − Exp0
on the
training-domain-aligned subsets of KMMLU and
MMLU across 12 cells (3 models × 2 domains
× {KMMLU, MMLU}) is reported in Table 10. G Sample QA selection statistics and
type-stratified decomposition
Per-cell selection statistics for the 200-QA public
sample (counts, question-type composition, difficulty distribution) are released in the companion
repository. Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question
type, supporting the within-type analysis in §5. Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question
type, supporting the within-type analysis in §5. H All 72 LoRA SFT cells (raw scores)
Table 13 reports per-cell MCQA accuracies
(KMMLU overall, KMMLU medical, KMMLU
finance, MMLU overall) for all 72 main LoRA
SFT cells plus the 9 base models, multi-seed
means (n seeds per row). I KMMLU-Pro per-cell results
We evaluate all SFT cells on KMMLUPro (Son et al. , 2025) with license-aligned
filtering (Appendix C): medical-trained cells
on kmmlu_pro_medical (1,205 items, 5 license categories) and finance-trained cells on
kmmlu_pro_finance (801 items, 4 license
categories). 12 pp), consistent with the
benchmark- and model-family-conditional pattern
discussed in §6. J Per-judge LLM-as-judge breakdown
Table 16 reports each of the four judges’ overall
scores on the 200-QA public sample (med+fin averaged) and decomposes ∆2−0 into single-stage 4% of the total variance
to between-cell (model×domain) heterogeneity
rather than seed noise (Appendix N), so per-cell
effects reflect cell-level signal rather than seedlevel instability. 3 W-grid: Whisper-medium reproduction
On Korean conference audio, Whisper-medium’s
relative WER against the in-house transcript as
reference is 27. 4% of the total variance
to between-cell (model×domain) heterogeneity
rather than seed noise (Appendix N), so per-cell
effects reflect cell-level signal rather than seedlevel instability. 3 W-grid: Whisper-medium reproduction
On Korean conference audio, Whisper-medium’s
relative WER against the in-house transcript as
reference is 27. Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question
type, supporting the within-type analysis in §5. H All 72 LoRA SFT cells (raw scores)
Table 13 reports per-cell MCQA accuracies
(KMMLU overall, KMMLU medical, KMMLU
finance, MMLU overall) for all 72 main LoRA
SFT cells plus the 9 base models, multi-seed
means (n seeds per row). For the subset of cells with persample evaluation data preserved (8 cells covering high-signal multi-seed and Whisper STT
analysis runs; Appendix M, Table 20), we report bootstrap 95% CIs (100,000 resamples): typical width ±0. 5 pp on KMMLU and ±0. A mixed-effects
analysis attributes 99. 4% of the total variance
to between-cell (model×domain) heterogeneity
rather than seed noise (Appendix N), so per-cell
effects reflect cell-level signal rather than seedlevel instability. 8% of the public-sample QA
on average. (v) We release the 200-QA sample
plus a Phase-2-filtered variant for sampling sensitivity (Appendix O), all SFT checkpoints, judge
prompts, and analysis code. 16, so within-type improvement accounts for ∼86% of ∆2−0 and composition shift ∼ 14% (per-type n = 5–20, Table 12). On the Phase-2-filtered sensitivity sample, the 4-judge ∆2−0 matches the primary +0. 22 human on a
1–5 scale) while the cross-model mean MCQA
gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608. 8%; per-condition rates span 19. A fixed-keyword heuristic baseline recovers the
same per-model spread (Appendix P). Catastrophic forgetting. We additionally evaluated 8 SFT cells (12 evaluation runs: 4 multiseed + 4 legacy single-seed Whisper STT analysis cells) on KoBEST (Korean general-purpose
benchmark, 5 subtasks; Appendix Q, Table 24). 4);
GPT-4o is the exception (P
s
2 > Ps
0
by +0. All
five dimensions improve from Exp 0 to Exp 2 (Appendix R); per-judge breakdown in Appendix J,
Table 16. Evaluation spans five axes: downstream
MCQA (KMMLU, KMMLU-Pro, MMLU) after
SFT, cell-blind QA-quality scoring by 4 LLM
judges (Zheng et al. , 2023) on a 200-QA stratified
public sample, a blind 6-expert human evaluation
of QA quality on the same sample, STT-engine
swap via Whisper-medium (Radford et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Factorial Ablation for Stage Value Assessment: By performing a 2x2 factorial design, practitioners can
independently toggle
transcript refinement (Phase 0) and SFT data quality refinement (Phase 2), allowing them to determineeach stage’s marginal value unknown.
-
Domain-Specific Transfer Analysis: The study reveals that
improvements in QA data quality do not transfer uniformly into downstream MCQA gains,
suggesting that thepositive transfer concentrates on family↔domain aligned pairs,
which can guide targeted model fine-tuning strategies. -
Robustness to Upstream STT Errors: Swapping the STT engine to Whisper-medium shows a minimal aggregate downstream effect, confirming that this substitute is
a viable substitute for in-house STT at the aggregate level
and demonstratingpipeline robustness.
-
Format Mismatch Identification: The differential pattern in transfer is attributed to a
format mismatch,
where Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall, indicating that downstream tasks benefit most from quality refinement when it aligns with the task's required format. -
Difficulty Audit for Production Readiness: An
LLM-difficulty audit shows that even the two frontier LLMs admit unknown on ∼8% of QA on average,
providing alow-cost triage tool for new corpora
to quantify the value of session-grounded SFT data over open-web augmentation. -
Quality Signal Convergence: The quality improvement direction is consistent across different instruments, as both
LLM judges (four cross-provider models) and human experts (six raters)
confirm positive delta2-0 values, strengthening confidence in the measured quality gains beyond single-instrument bias. -
Model Family Conditional Tuning: The analysis shows that
Stage-wise SFT transfer is therefore conditional on family↔domain alignment,
implying that scaling a model requires considering both the specific refinement stage and the model's family/domain pairing for optimal performance.
Abstract
Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and three downstream MCQA benchmarks. Our central finding is that, under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. An LLM-difficulty audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.
Sources
- Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
- Robust Speech Recognition via Large-Scale Weak Supervision
- KMMLU: Measuring Massive Multitask Language Understanding in Korean
- From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems