Synthetic Data Characterization via Training Dynamics

arXiv:2609.39447 · cs.CL, cs.LG · Submitted 2026-09-30 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: Accepted at Findings of EMNLP 2026

Code: https://github.com/irenelago/SynthTD

License: http://creativecommons.org/licenses/by/4.0/

The gist: Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks.

Terminology

Abstract

Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.

Sources

Related papers