Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

This paper investigates whether large language models (LLMs) contain domain-specific parametric shells—concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others. The authors apply a uniform causal methodology across two domain granularities, three model families (1.5B to 7B parameters), and eight domains.

Models and domains: The study evaluates three instruction-tuned decoder-only models: Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct, and Mistral-7B-Instruct-v0.3, containing 250,880, 229,376, and 458,752 FFN intermediate neurons respectively (939,008 combined). Two domain granularities are studied: (1) four academic subject categories derived from MMLU, ARC-Challenge, and OpenBookQA (quantitative reasoning, biomedical, humanities/social sciences, elementary science), and (2) four language/modality domains (English from WikiText-2, code from MBPP, Spanish and Chinese from OPUS-100).

Methodology: For each domain, the authors capture mean pre-activation magnitudes across the FFN intermediate dimension, identify domain-selective neurons via a share-based metric with backbone exclusion (excluding the top 0.1% of neurons by global mean activation), and measure causal damage matrices net of a count-matched random baseline. Domain selectivity uses a max-share threshold of 0.60. Causal masking zeroes intermediate activations at shell positions during inference, measuring NLL change (for language) or accuracy change (for subjects) net of random baseline.

Key findings:

1. Subject-level domains show zero parametric modularity: At the academic subject level, zero neurons exceed 60% domain selectivity across all 939,008 combined FFN neurons. The selectivity histogram is unimodal and rapidly decaying, with the 99th percentile reaching at most 0.388. Causal damage matrices are flat, with diagonal means of 0.121–0.181 accuracy points and off-diagonal means of 0.031–0.076, with no model exceeding a diagonal-to-off-diagonal ratio of 2.4. This holds despite domain identity being linearly decodable above 85% accuracy via logistic regression probes, demonstrating a dissociation between representational geometry and parametric organization.

2. Language-level domains show concentrated parametric shells: The fraction of neurons exceeding 0.60 max-share is 1.14%, 0.65%, and 0.84% for Qwen2.5-1.5B, Llama-3.2-3B, and Mistral-7B respectively. Damage matrices are near-perfectly diagonal with diagonal means of 1.82, 2.19, and 3.37 nats; off-diagonal means are −0.012, −0.060, and +0.006 nats. The diagonal-to-off-diagonal ratio ranges from 37:1 to above 500:1 (up to 595:1). The largest single entry is the Chinese diagonal in Mistral-7B at +8.58 nats. Shell neuron sets are essentially disjoint with maximum IoU of 0.0029 and most pairs below 0.001.

3. Pareto analysis of shell concentration: Damage accumulates rapidly between ρ=0.005 and ρ=0.02 and saturates by ρ=0.05 for all language domains. At ρ=0.02, damage represents on average 72% of the maximum observed. For Chinese in Mistral-7B, ρ=0.005 (2,293 neurons) already produces +6.09 nats while the curve saturates at +9.03 nats by ρ=0.05. The authors provide the first systematic Pareto analysis showing that 2% of neurons carries approximately 94% of maximum domain-specific capability loss across all four language domains and all three model families.

4. Code shells carry mathematical reasoning: Masking code-selective neurons reduces GSM8K mathematical reasoning accuracy by 16–24 percentage points across all models (−17.3, −24.0, and −16.0 for Qwen2.5-1.5B, Llama-3.2-3B, and Mistral-7B respectively). Masking Spanish or Chinese neurons leaves mathematical reasoning at or below random. The Chinese result varies by model in a pattern consistent with pretraining composition: Mistral-7B shows near-zero effect (+0.7pp), Qwen2.5-1.5B shows −1.3pp, and Llama-3.2-3B shows −10.0pp.

5. Scale analysis: The fraction of neurons exceeding 0.60 max-share is non-monotonic with scale (1.14%, 0.65%, 0.84%), but diagonal mean dNLL increases monotonically (1.82, 2.19, 3.37 nats across 1.5B to 7B parameters). Larger models develop more functionally load-bearing shells independent of the proportion of selective neurons.

6. Spatial organization: Spanish and Chinese neurons concentrate strongly in final network layers across all three model families, while English and code neurons distribute more uniformly. This final-layer concentration differs from the U-shaped distribution reported for base models, consistent with instruction tuning consolidating language-specific computation into the vocabulary projection stage.

7. Spatial interleaving precludes selective quantization: Under per-group asymmetric INT4 quantization with group size g=128, the probability that any group of 128 consecutive neurons is entirely core evaluates to 2.2×10−8 for the highest-density model. Zero qualifying groups are found in any model. Uniform INT4 produces language-uniform increases of +0.028–+0.081 nats with no language showing systematically larger degradation.

The granularity principle: The results converge on a single organizing principle: parametric shells form where and only where training data was modular at the token level. Language domains produce shells because English, Spanish, Chinese, and code occupy largely non-overlapping regions of the token distribution during pretraining, creating systematic gradient pressure toward domain-selective neurons. Subject domains do not because questions about different academic subjects are written in English with similar syntactic structures and overlapping vocabulary, producing no systematic pressure toward domain-selective neurons. Shell purity tracks training data composition—a shell learned on purely linguistic content carries only linguistic function, while a shell learned on thematically diverse content carries a corresponding mixture.

Limitations: The analysis covers three instruction-tuned model families in the 1.5B–7B range; whether the granularity principle holds for base models, larger models, or substantially different architectures is not established. The layer distribution comparison with prior work is indirect. The activation-magnitude identification method may not surface all functionally relevant neurons. The subject-level null covers four coarse categories; finer-grained partitions might reveal structure at intermediate granularities.

Improvements to AI systems based on this paper:

1. Domain-adaptive pruning and compression: Build a pruning pipeline that identifies and removes language-specific parametric shells (e.g., Spanish or Chinese neurons) when deploying a model for a monolingual application, reducing memory and inference cost by up to 2% of FFN neurons without degrading the target language. For multilingual serving, keep shells intact but apply selective quantization only to non-shell regions, avoiding the observed INT4 degradation risk.

2. Targeted model editing for language control: Implement a language isolation module that can mask or amplify specific language shells (e.g., Chinese in Mistral-7B) at inference time, enabling on-the-fly language switching, suppression of unwanted language leakage in code generation, or controlled code-mixing without retraining.

3. Task-transfer risk assessment: Before fine-tuning a model on a new domain, use the granularity principle to predict whether the new task will form a parametric shell (token-level modularity) or not (subject-level similarity). This allows preemptive allocation of capacity or early stopping if the target domain is likely to interfere with existing shells.

4. Explainable failure diagnosis: When a model fails on a specific language or code task, probe the activation magnitudes of the corresponding shell neurons. If damage is concentrated there, apply targeted fine-tuning or neuron restoration to that shell rather than global retraining, reducing compute and data requirements.

5. Data composition auditing for pretraining: Use the shell-purity metric as a diagnostic tool during pretraining to detect whether new training data introduces token-level modularity (e.g., adding a new language) or merely broadens existing representations. This enables dynamic adjustment of data mixing ratios to prevent unintended shell formation or to deliberately create shells for high-priority languages.

6. Cross-lingual reasoning enhancement: Leverage the finding that code shells carry mathematical reasoning to design a code-priming inference strategy—briefly activating code-selective neurons (via a small positive bias) before solving math problems, potentially improving GSM8K accuracy by up to 24 percentage points without additional training.

7. Layer-aware architecture optimization: Given the final-layer concentration of language shells, redesign the final transformer layers (e.g., vocabulary projection) with higher neuron capacity or separate per-language output heads, while compressing earlier layers, to improve multilingual efficiency and reduce memory footprint.

8. Safety and bias mitigation: For models with known language-specific biases (e.g., harmful outputs in one language), identify and selectively mask the corresponding language shell during sensitive queries, providing a targeted, reversible intervention without affecting other languages or general capabilities.