AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning
AEGIS addresses severe gradient erosion when fine-tuning vision-language models for robotics by injecting action gradients into cross-entropy trained backbones. It uses a buffer-free, layer-wise orthogonal projection framework to surgically modify task gradients. This preserves the pre-trained visual knowledge manifold by ensuring the new gradients remain orthogonal to the learned semantic structure without needing extra data.