Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
Low-rank adaptation methods leave adapter placement open. This work introduces PAGE, a probe to find where model sensitivity is concentrated. It found that adaptation sensitivity is highly focused on a single shallow FFN down-projection across different models and tasks. DomLoRA places only one adapter at this dominant module, showing it outperforms standard LoRA using less than 0.7% of trainable parameters.