A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules
cs.SE, cs.CL, cs.LG
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/JetXu-LLM/kestrel-capture-context
Terminology
Sources
- Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
- Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Shortcut Learning in Deep Neural Networks
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- LoRA+: Efficient Low Rank Adaptation of Large Models
- LoRA: Low-Rank Adaptation of Large Language Models
- Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
- MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
- Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing
- Unifying distillation and privileged information
- Propagating Knowledge Updates to LMs Through Distillation
- $\textit{New News}$: System-2 Fine-tuning for Robust Integration of New Knowledge
- Privileged Information Distillation for Language Models
- Measuring and Narrowing the Compositionality Gap in Language Models
- Self-Distillation Enables Continual Learning
- Learning by Distilling Context
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters