Breaking Information Islands in Sparse Tuning via Small-World Connectivity
cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Comments: Accepted at NeurIPS 2026. 23 pages, 6 figures
Code: https://github.com/xianchaoguan/SWIFT
License: http://creativecommons.org/licenses/by/4.0/
The gist: Sparse tuning is widely used to adapt large language models due to its parameter efficiency.
Terminology
Abstract
Sparse tuning is widely used to adapt large language models due to its parameter efficiency. However, its effectiveness depends not only on the number of trainable parameters, but also on the feature-interaction topology of the sparse update's structural support. Limited or uneven connectivity in this support may isolate subsets of feature dimensions, forming information islands that restrict direct task-specific interactions through the trainable update. To address this issue, we propose Small-World Induced Fine-Tuning (SWIFT), a sparse tuning framework inspired by small-world connectivity. SWIFT combines block-diagonal sparse updates for local feature mixing with a fixed deterministic scattering permutation that routes the mixed outputs across feature groups without additional trainable parameters. We further show theoretically that certain random sparse-update models and block-diagonal updates can exhibit information islands, whereas a sufficiently scattering permutation guarantees positive block-level expansion when the trainable block-diagonal factor consists of equal-sized, structurally dense square blocks. Extensive experiments on commonsense reasoning, natural language generation, and image classification demonstrate that SWIFT outperforms the evaluated parameter-efficient fine-tuning (PEFT) baselines on average and can match or surpass full fine-tuning. Implementation is available at https://github.com/xianchaoguan/SWIFT
Sources
- Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs
- Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention
- Qwen2.5 Technical Report
- Instruction Tuning for Large Language Models: A Survey
- The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
- Scaling Laws for Neural Language Models
- LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
- Parameter-Efficient Fine-Tuning for Foundation Models
- AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
- LoRA+: Efficient Low Rank Adaptation of Large Models
- RaSA: Rank-Sharing Low-Rank Adaptation
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models
- MoRe Fine-Tuning with 10x Fewer Parameters
- DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
- Transformers are Graph Neural Networks
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
- The Expressive Power of Low-Rank Adaptation
- Scaling Sparse Fine-Tuning to Large Language Models
- The Llama 3 Herd of Models
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents