A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer

arXiv:2608.20394 · cs.SD, cs.CL · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "A Factorial Ablation of a Speech-to-SFT Pipeline".

Jane: Here's the gist of the paper. Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage’s marginal value unknown <ref:2608.20394#pg2>.

Tom: Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains <ref:2608.20394#pg2>.

Jane: Factorial Design and Evaluation Scope The study designs a productionready speech-to-SFT pipeline featuring a 2 × 2 factorial design by independently toggling transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) <ref:2608.20394#pg2>.

Tom: This setup was applied to Korean medical and finance conference recordings, involving training nine LLMs across five families and varying sizes <ref:2608.20394#pg2>.

Paper summary: Jane: Okay, so looking at this paper "A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer," what's the big picture here for us?

Tom: Basically, the authors set up this test to see how two stages—transcript refinement and data quality refinement—affect downstream question answering when they run a fixed setup.

Lu: And they found that even though both phases improved the QA quality numbers, you couldn't see a significant lift across all models for the final MCQA scores.

Jane: So what does that imply about building these speech fine-tuning pipelines?

Tom: It implies we can’t just optimize every step in isolation; you have to look at how those steps interact with the specific format of your ultimate task.

Lu: The paper is telling us that phase zero refinement, which deals with transcript accuracy, seems to be a stronger contributor to initial quality improvement than the question refinement stage does.

Jane: It suggests that if you're working on a speech-to-SFT pipeline, focusing on cleaning up the transcription might give you more immediate data quality gains.

Tom: And we have to be careful because those gains might not translate directly into better performance on downstream tasks unless the SFT data is also shaped correctly for what those tasks demand.

Lu: It’s a reminder that family or domain alignment is what actually drives positive transfer, not just any uniform improvement across the board.

Jane: So, in simple terms, this paper says that getting your input data right is only half the battle; you also need to make sure the data structure matches what your final benchmark needs.

Conclusion: Tom: So we've seen how these researchers looked at a speech fine-tuning pipeline and ran a factorial test, right? They were looking at two main things: changing the transcript refinement stage and changing the quality refinement stage for their training data.

Jane: Exactly. The main finding is that improving the quality of your input data doesn't automatically mean you get better results on those final downstream question answering tasks.

Lu: That’s what they found, specifically that improvements in QA data quality don't transfer uniformly into gains on the final MCQA scores across different models.

Meng: So, if I’m trying to ship a model fast, and I spend time making my initial training data super clean, does that actually help my deployment performance? What is the actual measurable difference here?

Tom: It seems like the structure of the data matters more than just having cleaner data points. They found that one stage—the transcript refinement—was a stronger driver for initial quality improvement than the second stage.

Jane: That suggests you might want to focus your energy on making sure your speech-to-text transcription is as accurate as possible before you start fine-tuning the language models.

Lu: And they pointed out that the final question type composition shifts toward more explanatory questions when you use that second refinement stage, but those changes don't translate into better performance on factoid recall tasks.

Tom: It’s like they’re moving the goalposts. The data gets better in one direction, but it doesn't help with what the final evaluation is actually testing for.

Jane: So what does this mean for someone building these complex speech systems? It means you need to look at how the data format matches what your final task requires.

Meng: From a deployment standpoint, if the transfer isn't uniform, then spending time on every single pipeline tweak might just be wasted effort unless you have a very specific downstream goal in mind.

Tom: Yeah, so while these two stages both added some quality to the data, they didn't provide that big lift across all models.

Jane: It highlights that family alignment or domain matching is still the most important factor for getting those real gains on performance.

Lu: The study also showed that while one stage contributed more to initial quality, the overall cross-cell effect wasn't statistically significant, emphasizing the importance of that specific alignment between training groups and their target domains.

Tom: So what we’re seeing is a trade-off where you can improve individual parts, but you don't guarantee a better final score without hitting those specific alignment targets.

Jane: It gives us a clearer picture of where to spend our time next when building these speech systems.

Wonsup Shin, Jingu Kim

cs.SD, cs.CL

Submitted: 2026-07-01

Updated: 2026-11-10

Comments: 20 pages, 2 figures; v2: added an evaluation prompt-format analysis and removed an unsupported claim

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 88/100

The gist: Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge…

Key concepts

Factorial Design
This experimental setup used a 2x2 design by independently toggling two pipeline stages: transcript refinement and SFT data quality refinement. This allowed researchers to see how changing one stage affects the final outcome, helping to isolate the specific impact of each step.
QA Quality Improvement Dynamics
The study measured how much the quality of the QA data improved after applying different pipeline stages. It found that both LLM judges and human experts confirmed improvements in QA quality, but these gains were not evenly distributed across all downstream tasks.
Downstream Transfer Mismatch
Improvements in QA data did not lead to consistent gains in downstream multiple-choice question answering (MCQA) benchmarks. This suggests a mismatch: Phase 2 changes the data composition (making questions more explanatory), but the MCQA task primarily tests factoid recall, leading to poor transfer.
Family-Domain Alignment
Positive performance transfer was only observed when models were aligned with their specific training family and domain. This indicates that uniform improvement across all models is not possible; instead, performance gains are highly dependent on matching the model's background to the specific task's context.

Terminology

Summary

Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage’s marginal value unknown <ref:2608. p. 0

Where the paper says it

com Jingu Kim Flitto jin@flitto. com Abstract Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage’s marginal value unknown.

20394#pg2>. Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains <ref:2608. p. 0
Where the paper says it

4B– 70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and 3 downstream MCQA benchmarks. Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains.

20394#pg2>.

Factorial Design and Evaluation Scope

The study designs a productionready speech-to-SFT pipeline featuring a 2 × 2 factorial design by independently toggling transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) <ref:2608.20394#pg2>. This setup was applied to Korean medical and finance conference recordings, involving training nine LLMs across five families and varying sizes <ref:2608. p. 0

Where the paper says it

We design a productionready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2 × 2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.

20394#pg2>. The evaluation spans four axes, including downstream MCQA benchmarks, cell-blind QA-quality scoring by four LLM judges, a blind six-expert human evaluation, STT-engine swap via Whisper-medium, and an LLM-difficulty audit <ref:2608. p. 0
Where the paper says it

Evaluation spans five axes: downstream MCQA (KMMLU, KMMLU-Pro, MMLU) after SFT, cell-blind QA-quality scoring by 4 LLM judges (Zheng et al. , 2023) on a 200-QA stratified public sample, a blind 6-expert human evaluation of QA quality on the same sample, STT-engine swap via Whisper-medium (Radford et al.

20394#pg2>.

QA Quality Improvement Dynamics

Improvements in QA data quality do not transfer uniformly into downstream MCQA gains <ref:2608.20394#pg2>. Specifically, the 4-judge LLM and 6-expert human evaluations both confirm robust QA quality improvement (∆2−0 = +0. p. 0

Where the paper says it

Main contributions. (i) Under a fixed, standard SFT recipe (§4), improvements in QA data quality do not transfer uniformly into downstream MCQA gains: 4-judge LLM and 6-expert human evaluations both confirm robust quality improvement (+0.

18 and +0.22) <ref:2608. p. 0
Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. The single-stage decomposition shows that Phase 0 is the stronger contributor to QA quality (P s 0 = +0. p. 3
Where the paper says it

23 (95% CI [+0. Phase 0 is the stronger single-stage contributor (P s 0 = +0.

09 vs P s 2 = +0.03) <ref:2608. p. 0
Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. Furthermore, Phase 2 shifts questiontype composition toward more explanatory questions, with Exp 2 being explanatory-dominant (+10 pp vs. p. 3
Where the paper says it

Phase-2 mechanism: composition shift and within-type quality. Phase 2 shifts questiontype composition toward more explanatory questions (Appendix G, Table 11): Exp 2 is explanatory-dominant (+10 pp vs.

Exp 0) <ref:2608. p. 0
Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>.

Downstream Transfer and Format Mismatch

The differential pattern observed in QA quality does not translate into significant cross-model mean MCQA gains, suggesting a format mismatch where Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall <ref:2608.20394#pg2>. Positive transfer concentrates on family↔domain aligned pairs <ref:2608. p. 0

Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. The analysis of question-type composition shows that within-type improvement accounts for approximately 86% of the total QA quality gain, while composition shift accounts for about 14% <ref:2608. p. 3
Where the paper says it

Typestratified analysis (Appendix G) shows withintype gains positive on all 4 types; re-weighting Exp 2 to Exp 0’s composition yields typecontrolled ∆2−0 ≈ +0. 16, so within-type improvement accounts for ∼86% of ∆2−0 and composition shift ∼ 14% (per-type n = 5–20, Table 12).

20394#pg2>.

Pipeline Robustness and Limitations

The study confirms pipeline robustness through an STT-engine swap to Whisper-medium, which showed minimal aggregate downstream effect (mean absolute KMMLU difference ≤ 1.32 pp) <ref:2608. p. 0

Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. The LLM-difficulty audit revealed that frontier LLMs cannot confidently answer approximately 7. p. 1
Where the paper says it

(iv) An LLM-difficulty audit shows that even the two frontier LLMs (Opus 4. 4) cannot confidently answer 7.

8% of the public-sample QA on average <ref:2608. p. 1
Where the paper says it

4) cannot confidently answer 7. 8% of the public-sample QA on average.

20394#pg2>. Limitations include relying on n = 2 seeds for main LoRA cells, where seed variance dominates per-sample bootstrap variance <ref:2608. p. 6
Where the paper says it

8 pp on MMLU; seed variance (n = 2 per cell, worst-case σ= 2. 87 pp) dominates per-sample bootstrap variance.

20394#pg2>. The overall cross-cell mean MCQA effect was not statistically distinguishable from zero, suggesting that family↔domain alignment is the key factor rather than a uniform improvement <ref:2608. p. 5
Where the paper says it

Under our standard SFT recipe, 4- judge LLM and 6-expert human evaluations both confirm robust QA quality improvement (∆2−0 = +0. 22), yet the cross-cell mean MCQA ∆2−0 is not significant; positive transfer concentrates on family↔domain aligned cells.

20394#pg2>.

Ethics and Release of Artifacts

The authors released the 200-QA sample plus a Phase-2-filtered variant, all SFT checkpoints, judge prompts, and analysis code for reproducibility <ref:2608.20394#pg2>. Data collection involved 40 conference sessions from Korean medical and finance domains with speaker consent <ref:2608. p. 7

Where the paper says it

Ethics Statement Data collection and consent. The 40 conference sessions used in this work are a subset of Korean-language session data collected through an AI-mediated interpretation service, spanning academic conferences, professional seminars, and industry forums in the medical and finance sectors (2024–2026); speakers consented to research and non-commercial use.

20394#pg2>. The human evaluators were recruited externally through referral, ensuring no institutional relationship with the authors <ref:2608. p. 7
Where the paper says it

The six human evaluators used in our blind expert evaluation, three medical and three finance, were recruited externally through referral from in-house contacts. No evaluator had an employment relationship with the authors or their institution, nor participated in pipeline design or model selection.

20394#pg2>. The released artifacts should respect the consent scope of research and non-commercial use <ref:2608. p. 7
Where the paper says it

The full pipeline implementation is available upon reasonable request. Downstream use of the released artifacts should respect the consent scope (research and non-commercial use).

20394#pg2>.

References

Amazon AGI. 2025. SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning. p. 7

Where the paper says it

References Amazon AGI. SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning.

In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) <ref:2608. p. 7
Where the paper says it

SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).

20394#pg2>. Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. p. 7
Where the paper says it

In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra.

2024. Cosmopedia v0. p. 1
Where the paper says it

, 2023), Evol-Instruct (Xu et al. , 2024), and Cosmopedia (Ben Allal et al.

1 <ref:2608. p. 0
Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, and 1 others. p. 7
Where the paper says it

co / datasets/HuggingFaceTB/cosmopedia. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, and 1 others.

2024. LoRA learns less and forgets less <ref:2608. p. 1
Where the paper says it

, 2022), for parameterefficient SFT; Biderman et al. (2024) recently showed that LoRA “learns less and forgets less” than full fine-tuning, motivating our Full FT sanity check (§5.

20394#pg2>. Joya Chen and 1 others. p. 7
Where the paper says it

Transactions on Machine Learning Research (TMLR). Joya Chen and 1 others.

2025. LiveCC: Learning video LLM with streaming speech transcription at scale <ref:2608. p. 7
Where the paper says it

Joya Chen and 1 others. LiveCC: Learning video LLM with streaming speech transcription at scale.

20394#pg2>. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. p. 7
Where the paper says it

In Conference on Computer Vision and Pattern Recognition (CVPR). Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.

2023. QLoRA: Efficient finetuning of quantized LLMs <ref:2608. p. 7
Where the paper says it

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs.

20394#pg2>. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. p. 7
Where the paper says it

In Advances in Neural Information Processing Systems (NeurIPS). Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

2021. Measuring massive multitask language understanding <ref:2608. p. 7
Where the paper says it

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.

20394#pg2>. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. p. 7
Where the paper says it

In International Conference on Learning Representations. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.

2022. LoRA: Low-rank adaptation of large language models <ref:2608. p. 7
Where the paper says it

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.

20394#pg2>. Shuyue Jia and 1 others. p. 7
Where the paper says it

In International Conference on Learning Representations (ICLR). Shuyue Jia and 1 others.

2025. PodGPT: An audioaugmented large language model for research and education <ref:2608. p. 7
Where the paper says it

Shuyue Jia and 1 others. PodGPT: An audioaugmented large language model for research and education.

20394#pg2>. Jing Pan and 1 others. p. 8
Where the paper says it

Preprint, arXiv:2402. Jing Pan and 1 others.

2024. COSMIC: Data efficient instruction-tuning for speech in-context learning <ref:2608. p. 8
Where the paper says it

Jing Pan and 1 others. COSMIC: Data efficient instruction-tuning for speech in-context learning.

20394#pg2>. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. p. 8
Where the paper says it

In Interspeech. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.

2022. Robust speech recognition via large-scale weak supervision <ref:2608. p. 8
Where the paper says it

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision.

20394#pg2>. Guijin Son and 1 others. p. 8
Where the paper says it

Preprint, arXiv:2402. Guijin Son and 1 others.

2025. From KMMLU-redux to pro: A professional Korean benchmark suite for LLM evaluation <ref:2608. p. 8
Where the paper says it

Guijin Son and 1 others. From KMMLU-redux to pro: A professional Korean benchmark suite for LLM evaluation.

20394#pg2>. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. p. 8
Where the paper says it

Preprint, arXiv:2507. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A.

Smith, Daniel Khashabi, and Hannaneh Hajishirzi. p. 8
Where the paper says it

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

2023. Self-instruct: Aligning language models with self-generated instructions <ref:2608. p. 8
Where the paper says it

Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.

20394#pg2>. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. p. 8
Where the paper says it

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.

2024. WizardLM: Empowering large pretrained language models to follow complex instructions <ref:2608. p. 8
Where the paper says it

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large pretrained language models to follow complex instructions.

20394#pg2>. Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. p. 8
Where the paper says it

In International Conference on Learning Representations (ICLR). Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto.

2025. Synthetic continued pretraining <ref:2608. p. 8
Where the paper says it

Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic continued pretraining.

20394#pg2>. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. p. 8
Where the paper says it

In International Conference on Learning Representations (ICLR). Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P.

Xing, Hao Zhang, Joseph E. p. 8
Where the paper says it

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E.

Gonzalez, and Ion Stoica. p. 8
Where the paper says it

Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.

2023. Judging LLM-as-a-judge with MT-bench and chatbot arena <ref:2608. p. 8
Where the paper says it

Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena.

20394#pg2>. A Per-phase token costs Per-QA token-cost estimates (USD; OpenRouter unified pricing, snapshot 2026-05) <ref:2608. p. 8
Where the paper says it

In Advances in Neural Information Processing Systems (NeurIPS). A Per-phase token costs Per-QA token-cost estimates (USD; OpenRouter unified pricing, snapshot 2026-05), measured on a representative log sub-sample: • Phase 0: 0.

20394#pg2>. B Seed inventory Table 7 lists the seed configuration used in every experiment reported in this paper <ref:2608. p. 8
Where the paper says it

050 (analyze + augment + strategize + generate) • Phase 2: 0. 013 (judge + refine ≤ 2 + dedup) B Seed inventory Table 7 lists the seed configuration used in every experiment reported in this paper.

20394#pg2>. C KMMLU and KMMLU-Pro domain mapping KMMLU 45 subjects → domain proxies used in §5. p. 8
Where the paper says it

013 (judge + refine ≤ 2 + dedup) B Seed inventory Table 7 lists the seed configuration used in every experiment reported in this paper. C KMMLU and KMMLU-Pro domain mapping KMMLU 45 subjects → domain proxies used in §5.

2 such as health, biology, chemistry, psychology <ref:2608. p. 8
Where the paper says it

C KMMLU and KMMLU-Pro domain mapping KMMLU 45 subjects → domain proxies used in §5. 2: • KMMLU-med (medical-relevant, 4 subjects): health, biology, chemistry, psychology.

20394#pg2>. MMLU 57 subjects → domain proxies used in the Full FT sanity (§5. p. 8
Where the paper says it

• KMMLU-fin (finance-relevant, 6 subjects): economics, accounting, taxation, management, marketing, real_estate. MMLU 57 subjects → domain proxies used in the Full FT sanity (§5.

5) including clinical knowledge and medical genetics <ref:2608. p. 8
Where the paper says it

MMLU 57 subjects → domain proxies used in the Full FT sanity (§5. 5, Appendix F): • MMLU-med (8 subjects): clinical_knowledge, medical_genetics, anatomy, college_medicine, professional_medicine, college_biology, nutrition, virology.

20394#pg2>. KMMLU-Pro license categories → domain tasks filtered by license name via the task YAML’s process docs hook <ref:2608. p. 8
Where the paper says it

• MMLU-fin (7 subjects): econometrics, high_school_macroeconomics, high_school_microeconomics, professional_accounting, business_ethics, management, marketing. KMMLU-Pro license categories → domain tasks (filtered by license_name via the task YAML’s process_docs hook): • kmmlu_pro_medical (1,205 items): physician, dentist, pharmacist, Korean medicine doctor, Korean pharmacist.

20394#pg2>. D Session-level metadata Table 8 lists the 40 conference sessions used in the main experiment, including transcript line count <ref:2608. p. 9
Where the paper says it

D Session-level metadata Table 8 lists the 40 conference sessions used in the main experiment (19 medical + 21 finance). ROW = transcript line count (sentence-segmented STT output, proxy for session length).

20394#pg2>. E Phase 2 stage acceptance rates from pipeline logs Table 9 reports per-stage acceptance rates on the conditions where Phase 2 is active <ref:2608. p. 9
Where the paper says it

E Phase 2 stage acceptance The per-condition QA-count spread in Table 2 arises from the Phase 2 pipeline applied to Phase 1 candidates. Table 9 reports per-stage acceptance rates on the conditions where Phase 2 is active (Exp 1-2 and Exp 2).

20394#pg2>. F Full FT sanity per-cell results Stage-wise delta ∆stage = Exp2 − Exp0 on the training-domain-aligned subsets of KMMLU and MMLU across 12 cells <ref:2608. p. 9
Where the paper says it

The combined effect keeps per-condition training-set sizes within a ∼6% band. F Full FT sanity per-cell results Stage-wise delta ∆stage = Exp2 − Exp0 on the training-domain-aligned subsets of KMMLU and MMLU across 12 cells (3 models × 2 domains × {KMMLU, MMLU}) is reported in Table 10.

20394#pg2>. G Sample QA selection statistics and type-stratified decomposition Table 11 shows the question-type composition shift across conditions <ref:2608. p. 9
Where the paper says it

G Sample QA selection statistics and type-stratified decomposition Per-cell selection statistics for the 200-QA public sample (counts, question-type composition, difficulty distribution) are released in the companion repository. Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question type, supporting the within-type analysis in §5.

20394#pg2>. H All 72 LoRA SFT cells (raw scores) Table 13 reports per-cell MCQA accuracies across all 9 models <ref:2608. p. 9
Where the paper says it

Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question type, supporting the within-type analysis in §5. H All 72 LoRA SFT cells (raw scores) Table 13 reports per-cell MCQA accuracies (KMMLU overall, KMMLU medical, KMMLU finance, MMLU overall) for all 72 main LoRA SFT cells plus the 9 base models, multi-seed means (n seeds per row).

20394#pg2>. I KMMLU-Pro per-cell results We evaluate all SFT cells on KMMLUPro with license-aligned filtering <ref:2608. p. 9
Where the paper says it

I KMMLU-Pro per-cell results We evaluate all SFT cells on KMMLUPro (Son et al. , 2025) with license-aligned filtering (Appendix C): medical-trained cells on kmmlu_pro_medical (1,205 items, 5 license categories) and finance-trained cells on kmmlu_pro_finance (801 items, 4 license categories).

20394#pg2>. J Per-judge LLM-as-judge breakdown Table 16 reports each of the four judges’ overall scores on the 200-QA public sample <ref:2608. p. 9
Where the paper says it

12 pp), consistent with the benchmark- and model-family-conditional pattern discussed in §6. J Per-judge LLM-as-judge breakdown Table 16 reports each of the four judges’ overall scores on the 200-QA public sample (med+fin averaged) and decomposes ∆2−0 into single-stage

20394#pg2>. K Transcript-level WER measurement (Whispermedium vs. p. 4
Where the paper says it

4% of the total variance to between-cell (model×domain) heterogeneity rather than seed noise (Appendix N), so per-cell effects reflect cell-level signal rather than seedlevel instability. 3 W-grid: Whisper-medium reproduction On Korean conference audio, Whisper-medium’s relative WER against the in-house transcript as reference is 27.

in-house STT) We measure word and character error rate (WER, CER) of Whisper-medium against the in-house STT reference <ref:2608. p. 4
Where the paper says it

4% of the total variance to between-cell (model×domain) heterogeneity rather than seed noise (Appendix N), so per-cell effects reflect cell-level signal rather than seedlevel instability. 3 W-grid: Whisper-medium reproduction On Korean conference audio, Whisper-medium’s relative WER against the in-house transcript as reference is 27.

20394#pg2>. L W-grid per-cell results Table 18 reports the W-grid per-cell accuracies and per-cell ∆i = Wi − inhousei versus the matched inhouse STT cells <ref:2608. p. 9
Where the paper says it

Table 11 shows the question-type composition shift across conditions, and Table 12 reports per-item 4-judge means stratified by question type, supporting the within-type analysis in §5. H All 72 LoRA SFT cells (raw scores) Table 13 reports per-cell MCQA accuracies (KMMLU overall, KMMLU medical, KMMLU finance, MMLU overall) for all 72 main LoRA SFT cells plus the 9 base models, multi-seed means (n seeds per row).

20394#pg2>. M Bootstrap 95% CIs For each cell with per-sample evaluation data (n = 35030 KMMLU items, n = 14042 MMLU items) we ran 100,000 bootstrap resamples <ref:2608. p. 6
Where the paper says it

For the subset of cells with persample evaluation data preserved (8 cells covering high-signal multi-seed and Whisper STT analysis runs; Appendix M, Table 20), we report bootstrap 95% CIs (100,000 resamples): typical width ±0. 5 pp on KMMLU and ±0.

20394#pg2>. N Mixed-effects variance decomposition To complement the paired bootstrap on cross-cell means we fit a linear mixed-effects model on all 72 observations <ref:2608. p. 4
Where the paper says it

A mixed-effects analysis attributes 99. 4% of the total variance to between-cell (model×domain) heterogeneity rather than seed noise (Appendix N), so per-cell effects reflect cell-level signal rather than seedlevel instability.

20394#pg2>. O Sampling-design sensitivity Phase-2-filtered subset The result on the filtered subset (Table 22) is ∆2−0 = +0. p. 1
Where the paper says it

8% of the public-sample QA on average. (v) We release the 200-QA sample plus a Phase-2-filtered variant for sampling sensitivity (Appendix O), all SFT checkpoints, judge prompts, and analysis code.

19 matches the primary 200-QA sample’s +0. p. 3
Where the paper says it

16, so within-type improvement accounts for ∼86% of ∆2−0 and composition shift ∼ 14% (per-type n = 5–20, Table 12). On the Phase-2-filtered sensitivity sample, the 4-judge ∆2−0 matches the primary +0.

18 <ref:2608. p. 0
Where the paper says it

22 human on a 1–5 scale) while the cross-model mean MCQA gain is not significant; positive transfer concentrates on family↔domain aligned pairs. This differential pattern is consistent with a format misarXiv:2608.

20394#pg2>. P Keyword-heuristic for unknown-admission detection A complementary keyword-heuristic baseline uses a fixed list of 8 Korean substrings drawn from common no-information phrases <ref:2608. p. 5
Where the paper says it

8%; per-condition rates span 19. A fixed-keyword heuristic baseline recovers the same per-model spread (Appendix P).

20394#pg2>. Q KoBEST forgetting evaluation Eight SFT cells were evaluated on KoBEST (5 subtasks: boolq, copa, hellaswag, sentineg, wic) <ref:2608. p. 6
Where the paper says it

Catastrophic forgetting. We additionally evaluated 8 SFT cells (12 evaluation runs: 4 multiseed + 4 legacy single-seed Whisper STT analysis cells) on KoBEST (Korean general-purpose benchmark, 5 subtasks; Appendix Q, Table 24).

20394#pg2>. R Per-dimension LLM-judge breakdown Table 25 reports each of the five dimensions show positive ∆2−0 from Exp 0 to Exp 2 <ref:2608. p. 3
Where the paper says it

4); GPT-4o is the exception (P s 2 > Ps 0 by +0. All five dimensions improve from Exp 0 to Exp 2 (Appendix R); per-judge breakdown in Appendix J, Table 16.

20394#pg2>. S Human evaluator profile and detailed results We complement the LLM-judge QA-quality evaluation with a blind 6-expert evaluation on the same public-sample QA <ref:2608. p. 0
Where the paper says it

Evaluation spans five axes: downstream MCQA (KMMLU, KMMLU-Pro, MMLU) after SFT, cell-blind QA-quality scoring by 4 LLM judges (Zheng et al. , 2023) on a 200-QA stratified public sample, a blind 6-expert human evaluation of QA quality on the same sample, STT-engine swap via Whisper-medium (Radford et al.

20394#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Factorial Ablation for Stage Value Assessment: By performing a 2x2 factorial design, practitioners can independently toggle transcript refinement (Phase 0) and SFT data quality refinement (Phase 2), allowing them to determine each stage’s marginal value unknown.

  2. Domain-Specific Transfer Analysis: The study reveals that improvements in QA data quality do not transfer uniformly into downstream MCQA gains, suggesting that the positive transfer concentrates on family↔domain aligned pairs, which can guide targeted model fine-tuning strategies.

  3. Robustness to Upstream STT Errors: Swapping the STT engine to Whisper-medium shows a minimal aggregate downstream effect, confirming that this substitute is a viable substitute for in-house STT at the aggregate level and demonstrating pipeline robustness.

  4. Format Mismatch Identification: The differential pattern in transfer is attributed to a format mismatch, where Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall, indicating that downstream tasks benefit most from quality refinement when it aligns with the task's required format.

  5. Difficulty Audit for Production Readiness: An LLM-difficulty audit shows that even the two frontier LLMs admit unknown on ∼8% of QA on average, providing a low-cost triage tool for new corpora to quantify the value of session-grounded SFT data over open-web augmentation.

  6. Quality Signal Convergence: The quality improvement direction is consistent across different instruments, as both LLM judges (four cross-provider models) and human experts (six raters) confirm positive delta2-0 values, strengthening confidence in the measured quality gains beyond single-instrument bias.

  7. Model Family Conditional Tuning: The analysis shows that Stage-wise SFT transfer is therefore conditional on family↔domain alignment, implying that scaling a model requires considering both the specific refinement stage and the model's family/domain pairing for optimal performance.

Abstract

Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and three downstream MCQA benchmarks. Our central finding is that, under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. An LLM-difficulty audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.

Sources

Related papers