From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from observational equivalence, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply transfer invariance, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as Executable Causal Intervention for Demonstrations Distillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions (do-operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.