PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

arXiv:2610.14558 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging".

Jane: The gist The proposed Pretrained-Guided Shared Basis (PGSB) is a two-stage framework for LoRA model merging that first identifies shared directions among task-specific LoRA updates and then re-parameterizes these directions…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, to summarize what the Pretrained-Guided Shared Basis paper claims, it’s a two-stage process. First, they do something called Task-shared Subspace Extraction or TSE to identify those shared directions among all the task updates by concatenating them and using singular value decomposition.

Jane: That sounds like they are trying to find a common subspace where all the task updates live together, which is smart because it groups similar tasks together. How does that first step work technically?

Tom: They concatenate all those individual task-specific LoRA updates and then apply singular value decomposition to extract a shared subspace, which gives them this unified multi-task basis that captures the common directions relevant to different tasks.

Lu: And to get the row space structure, they actually apply SVD again but on the concatenated transposed updates. That’s a clever trick because it lets them get both support matrices from the same singular value decomposition construction.

Meng: Okay, so they find this shared subspace first, and then in the second stage, Pretrained-coupled Coordinate Refinement or PCR, they refine that basis using the geometric structure of the pretrained weights to make sure it doesn't conflict with how those original weights are already set up.

Jane: That refinement step sounds crucial. So why is coupling with the pretrained weight geometry so necessary for this merging process? What happens if you just stick to a mathematically derived shared subspace without looking at the original model structure?

Lalam: The coupling quantifies it by taking T equal to sigma LG sigma R, which represents the pretrained-weight coupling after factoring in how important those task-shared directions actually are. They then compute the SVD of that T matrix to concentrate the dominant interactions on diagonal coordinates.

Tom: So they end up with these refined bases QL and QR that keep the original task-shared subspaces but give you a coordinate system that’s specifically tuned by the geometry of the pretrained weight structure.

Conclusion: Tom: So to wrap up on PGSB: this framework addresses those two big hurdles we talked about—finding shared directions and making them compatible with the pretrained weight geometry—by using TSE and then PCR.

Jane: It seems like they’ve shown that you need both the shared support construction and that refinement step, because just having a shared subspace isn't enough; it has to fit into the model structure itself.

Lu: The authors show that this two-stage approach consistently outperforms strong baselines on vision and language benchmarks, and they did an ablation study confirming that PCR gives them additional gains beyond just constructing the shared support.

Meng: From a deployment standpoint, the efficiency is also important. They’ve managed to avoid SVD on massive concatenated update matrices by using compact low-rank factors in their implementation, which means the preprocessing complexity is O(n 2Tr + T 3r three) <ref:2610.14558#pg1>. That keeps things manageable for large models.

Lalam: And they show that this method doesn't rely on a fixed or uniform LoRA rank, meaning it can be effective across diverse rank configurations, which is important for real-world scenarios where you don't know the optimal rank beforehand.

Tom: So what does this mean for the future? It suggests that for effective LoRA merging, we need both finding those shared task directions and making sure those directions align with how the pretrained model was originally trained.

Jane: And it points toward a future where model merging isn't just about picking the best update from a list, but about intelligently structuring those updates based on both their similarity to each other and their relationship to the core model.

Muqing Liu, Chongjie Si, Zhuoya Liu, Yuheng Jia

Chien-Shiung Wu College, Southeast University · MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Comments: NeurIPS 2026

Code: https://github.com/Kitschnada/Pretrained-Basis

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The proposed Pretrained-Guided Shared Basis (PGSB) is a two-stage framework for LoRA model merging that first identifies shared directions among task-specific LoRA updates and then

Key concepts

Task-shared Subspace Extraction (TSE): This stage identifies common directions among different task-specific LoRA updates. It achieves this by concatenating all updates and using Singular Value Decomposition (SVD) on the resulting matrices. This extracts a unified basis that captures the shared, task-relevant directions present across multiple tasks.
Pretrained-coupled Coordinate Refinement (PCR): This stage refines the initial shared basis by incorporating the geometric structure of the pretrained weights. It measures how much the pretrained weight couples with these task-shared directions and then uses SVD on this coupling matrix to concentrate dominant interactions onto diagonal coordinates, resulting in a more compatible coordinate system.
Pretrained-compatible Basis: This is the final output of PGSB, consisting of refined bases QL and QR. This basis preserves the original shared task subspaces while providing a coordinate system specifically tailored by the geometric structure of the pretrained model. It ensures that subsequent merging operations are performed in a space that minimizes conflicts with the original weight space.
Merging and Reconstruction: Once the refined bases are obtained, all task-specific LoRA updates are mapped into this new coordinate space. A merge rule is then applied in this simplified coordinate system to get a merged update, which is finally mapped back to the original parameter space using the refined basis for reconstruction.

Terminology

Model Merging — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed Pretrained-Guided Shared Basis (PGSB) is a two-stage framework for LoRA model merging that first identifies shared directions among task-specific LoRA updates and then re-parameterizes these directions using their interaction with the pretrained weights to produce a more suitable basis for merging.

How it works

The PGSB framework consists of two stages: Task-shared Subspace Extraction (TSE) and Pretrained-coupled Coordinate Refinement (PCR), which accordingly determines the shared task directions and their pretrained-compatible merge coordinates. In the first stage, PGSB concatenates all task-specific updates and applies singular value decomposition to extract a shared subspace, yielding a unified multi-task basis that captures common task-relevant directions<ref:2610.14558#pg4>. To extract the shared row-space structure, PGSB equivalently applies SVD to the concatenated transposed updates<ref:2610.14558#pg4>.

Task-shared Subspace Extraction (TSE)

PGSB first constructs a shared pair of support matrices for representing multiple LoRA updates<ref:2610.14558#pg4>. Specifically, PGSB first concatenates all task-specific updates and applies singular value decomposition (SVD)<ref:2610.14558#pg4>. The resulting left support matrix UL captures the shared column-space structure of task updates and is used as the left support matrix<ref:2610.14558#pg4>. To extract the shared row-space structure, PGSB equivalently applies SVD to the concatenated transposed updates<ref:2610.14558#pg4>. The transposition is introduced because the row space of ∆Wt becomes the column space of ∆W⊤t, allowing both support matrices to be obtained through the same SVD-based construction<ref:2610.14558#pg4>.

Pretrained-coupled Coordinate Refinement (PCR)

In the second stage, PGSB refines this basis using the geometric structure of the pretrained weights, producing a pretrained-compatible multi-task basis that reduces conflicts with the original weight space<ref:2610.14558#pg4>. The refinement involves projecting the pretrained weight onto the task-shared supports to measure how the pretrained weight is expressed under these coordinates<ref:2610.14558#pg4>. This coupling is quantified by T = ΣLGΣR, which represents the pretrained-weight coupling after taking into account the relative importance of task-shared directions<ref:2610.14558#pg4>. To reduce coordinate mixing, PGSB computes the SVD of T to concentrate dominant interactions on diagonal coordinates<ref:2610.14558#pg4>. This results in the refined bases QL and QR, which preserve the original task-shared subspaces while providing a coordinate system refined by the pretrained weight geometry<ref:2610.14558#pg4>.

Merging and Reconstruction

After obtaining the refined bases QL and QR, PGSB maps all task-specific LoRA updates into the same pretrained-compatible coordinate space<ref:2610.14558#pg4>. This gives a set of task matrices Ct = Q⊤L∆WtQR, where each Ct represents the corresponding update under the shared task subspace and the refined merge coordinate<ref:2610.14558#pg4>. Any merge rule M can then be applied in this coordinate space to obtain C⋆ = M(C1, C2,..., CT), and finally, the merged coordinate matrix is mapped back to the original parameter space as ∆W⋆ = QLC⋆Q⊤R<ref:2610.14558#pg4>.

Key Contributions and Results

The paper shows that effective LoRA merging requires both constructing a shared multi-task basis and ensuring its compatibility with the pretrained weight geometry<ref:2610.14558#pg3>. PGSB is proposed as a two-stage framework that first extracts a task-shared subspace from concatenated LoRA updates and then refines the resulting basis using the geometric structure of pretrained weights<ref:2610.14558#pg3>. Extensive experiments on vision and language benchmarks show that PGSB consistently outperforms strong baselines and achieves state-of-the-art results<ref:2610.14558#pg4>. The ablation study confirms the necessity of both shared support construction and pretrained-guided coordinate refinement, showing that PCR provides additional gains beyond support construction by refining the coordinates inside the same support<ref:2610.14558#pg4>. Furthermore, PGSB better preserves the transfer ability of the pretrained model after merging<ref:2610.14558#pg10>. The study also demonstrates that PGSB does not rely on a fixed or uniform LoRA rank and remains effective across diverse rank configurations<ref:2610.14558#pg10>.

Efficiency and Extensibility

The efficient implementation of PGSB avoids SVD on concatenated full-size update matrices by using compact low-rank factors, leading to a preprocessing complexity of O(n 2Tr + T 3r 3)<ref:2610.14558#pg19>. Caching the shared bases and PCR results substantially reduces merging time, achieving a speedup of 6.27×–30.66× for ViT-B/32<ref:2610.14558#pg22>. The framework is also extensible, as PCR can be applied to prior alignment spaces like KnOTS and Core Space to obtain hybrid variants that improve performance<ref:2610.14558#pg10>. PGSB achieves the highest average performance of 98.38 on additional visual tasks after TIES merging<ref:2610.14558#pg10>.

Conclusion

In this paper, we studied LoRA model merging as a scalable route toward building unified multitask models from parameter-efficient adaptations<ref:2610.14558#pg4>. We identified two key challenges that distinguish LoRA merging from conventional merging of fully fine-tuned models: extracting shared task directions from diverse low-rank updates, and representing these directions in coordinates compatible with the pretrained weight geometry<ref:2610.14558#pg4>. To address these challenges, we proposed PGSB, a two-stage framework consisting of Task-shared Subspace Extraction and Pretrained-coupled Coordinate Refinement<ref:2610.14558#pg4>. PGSB first constructs a shared subspace from concatenated LoRA updates, and then refines the merge coordinates using the geometric structure of the pretrained weights<ref:2610.14558#pg4>. This yields a pretrained-compatible basis that better supports coordinate-sensitive merging and improves multitask adaptation<ref:2610.14558#pg4>. Extensive experiments on vision and language benchmarks demonstrate that PGSB consistently outperforms strong baselines<ref:2610.14558#pg10>. Further analyses confirm the necessity of both shared support construction and pretrained-guided coordinate refinement, show that PGSB better preserves the transfer ability of the pretrained model, and demonstrate the robustness of PGSB across diverse rank configurations<ref:2610.14558#pg10>. The proposed framework is effective across different visual backbones and heterogeneous language task collections<ref:2610.14558#pg10>. The results indicate that the proposed merging space is not specific to visual encoders, but also transfers effectively to large language models<ref:2610.14558#pg10>. The proposed method is highly effective when task updates exhibit coherent shared structures and the downstream merge operator is sensitive to the coordinate representation<ref:2610.14558#pg10>.

reference:2610.14558 pg3,five or more words copied word for word from that page

reference:2610.

Improvements for AI systems

Bold header: Pretrained-Guided Shared Basis (PGSB) Framework Implementation

The improved system can perform merging PEFT-based adaptations by using PGSB's two-stage framework: Task-shared Subspace Extraction (TSE) and Pretrained-coupled Coordinate Refinement (PCR), which ensures the resulting merged model has a pretrained-compatible basis.

Bold header: Enhanced Multi-Task Adaptation

The system can achieve superior performance across diverse tasks by applying PGSB's merging capabilities, as evidenced by results showing it consistently outperforms baselines on vision and language benchmarks.

Bold header: Robustness Across Rank Configurations

The improved system maintains high accuracy regardless of the LoRA rank configuration, as shown in Table 9, demonstrating effectiveness for homogeneous, heterogeneous, and adaptively allocated ranks.

Abstract

Large pretrained models have achieved remarkable success across diverse domains, yet adapting them to multiple downstream tasks remains challenging: full fine-tuning is costly, while maintaining separate task-specific models is storage-inefficient and fails to produce a unified multi-task model. Low-Rank Adaptation (LoRA) offers an efficient alternative for Model Merging by representing each task adaptation as a low-rank update over frozen pretrained weights. However, existing LoRA merging methods often treat task-specific updates as directly comparable objects and merge them in their original low-rank parameter spaces. This overlooks two key factors: which update directions provide reliable shared support across tasks, and how these directions should be re-parameterized with respect to the pretrained weights before merging. To address this limitation, we propose Pretrained-Guided Shared Basis (PGSB), a two-stage framework for LoRA model merging. PGSB first identifies shared directions among task-specific LoRA updates, and then re-parameterizes these directions using their interaction with the pretrained weights, producing a more suitable basis for merging. Extensive experiments demonstrate that PGSB consistently outperforms state-of-the-art baselines across multiple benchmarks. Codes are available at https://github.com/Kitschnada/Pretrained-Basis.