Locking Pretrained Weights via Deep Low-Rank Residual Distillation
cs.LG
Submitted: 2026-05-11
Updated: 2026-10-15
Comments: Accepted at NeurIPS 2026. Code: https://github.com/apple-aiml-research/ml-dlr-lock
Code: https://github.com/apple-aiml-research/ml-dlr-lock
License: http://creativecommons.org/licenses/by/4.0/
The gist: The quality of open-weight language models has dramatically improved in recent years.
Terminology
Abstract
The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardware and software platforms. They also allow for more open research and testing, to the extent that users can use them as checkpoints, fine-tune them according to their needs, and potentially redistribute them. In some cases, however, concerns on modifying these weights towards unauthorized uses may outweigh the pros of giving users such freedom. Defending against such adaptation is non-trivial: since an adaptive attacker can observe all weights and architectures by definition, they can reverse simple structural defenses, and use optimization to defeat the simplest locking mechanisms. In this work, we exploit the inference-training asymmetry of automatic differentiation as a novel defense axis. We propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network (DLR-Net) of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation. DLR-Nets are efficiently trained via module-wise regression. We show that, beyond this memory overhead, DLR-Lock results in architectural mismatches that complicate the optimization landscape of standard fine-tuning, and a backward pass that incurs disproportionately more overhead than the forward pass. Our defense succeeds in withstanding adaptive attackers with full knowledge of the defense strategy while preserving the original model's capabilities. Experiments on LLMs validate these claims.
Sources
- Distillation Scaling Laws
- Training Deep Nets with Sublinear Memory Cost
- Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
- The Llama 3 Herd of Models
- Distilling the Knowledge in a Neural Network
- On the Societal Impact of Open Foundation Models
- m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
- Eight Methods to Evaluate Robust Unlearning in LLMs
- Limits of Convergence-Rate Control for Open-Weight Safety
- GLU Variants Improve Transformer
- Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
- Towards Building Non-Fine-Tunable Foundation Models
- Qwen3 Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks