Beyond Transfer Accuracy: Mechanism-Guided Controlled Adaptation for Low-Resource Languages

arXiv:2601.08146 · cs.CL, cs.AI, cs.LG · Submitted 2026-01-13 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-01-13

Updated: 2026-09-02

Comments: Accepted as a Findings paper at EMNLP 2026

Project page: https://transformerlensorg.github.io/TransformerLens

License: http://creativecommons.org/licenses/by-sa/4.0/

The gist: Existing circuit discovery methods rely on templated tasks with clean counterfactuals, limiting their use on diverse natural text.

Terminology

Abstract

Existing circuit discovery methods rely on templated tasks with clean counterfactuals, limiting their use on diverse natural text. We adapt Contextual Decomposition for Transformers (CD-T) for unstructured settings via label-balanced activation means and task-directional relevance scoring, enabling counterfactual-free circuit discovery. We leverage the discovered circuits for Circuit-Targeted Supervised Fine-Tuning (CT-SFT), restricting parameter updates to task-relevant heads and LayerNorm. Experiments on NusaX cross-lingual sentiment transfer show that CT-SFT is highly competitive for low-resource adaptation. While non-circuit sparse updates and full fine-tuning sometimes match target accuracy through capacity recruitment, CT-SFT most consistently avoids catastrophic forgetting, preserving source-language and related-task performance. Extensions to XNLI support the source-retention and intervention findings on a harder task and two model families, showing that circuit-targeted adaptation provides a more controlled, intervention-supported alternative to global fine-tuning.

Sources

Related papers