DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion

The gist: DeKoR proposes an efficient framework for GNN and LLM knowledge fusion by decomposing LLM knowledge into local and global views, distilling them into lightweight experts, and using a node-adaptive router to selectively combine their predictions.

Problem Addressed

Existing GNN–LLM methods face two main challenges: a mismatched knowledge space between LLMs and GNNs where representation gaps are consistently larger than those between GCN and GCNII across five datasets, and limited gains under high computational cost because many methods require LLM inference for downstream prediction, which introduces substantial computational overhead

DeKoR Framework Overview

DeKoR decouples knowledge transfer from knowledge fusion by first transforming LLM knowledge into lightweight representations and then performing fusion on the transferred knowledge. The framework involves several key stages:

** DeKoR decomposes LLM knowledge into two complementary views: local knowledge that preserves fine-grained node information and global knowledge that characterizes each node through its relations to global prototypes These decomposed forms are then distilled into lightweight experts, which do not require LLM inference during deployment. Finally, a node-adaptive router learns how much each node should rely on each expert to combine their predictions The LLM is used only as an offline teacher, allowing the framework to retain useful LLM knowledge without requiring LLM inference during deployment **

Knowledge Decomposition and Distillation

The paper characterizes how to decompose semantic knowledge at different granularities through Proposition 1, which states that the prototype-induced representation captures global semantic structure while the original representation preserves fine-grained node-specific semantics. In DeKoR, this is achieved by applying K-means to node representations to obtain K global semantic prototypes and computing similarity with these prototypes to form the global view Q. The local view Z preserves the original LLM representations from the three different context scales (ego, one-hop, and two-hop prompts).

The knowledge is then distilled into lightweight modules using distillation objectives to preserve these views. For the local semantic expert, a symmetric InfoNCE loss Lnce is introduced to increase the lower bound on the mutual information shared between the local semantic expert and the LLM representation. For the global semantic view, a KL divergence loss Lglo is used to measure its discrepancy from the prototype-association distribution, ensuring that as Lglo decreases, it tightens an upper bound on this discrepancy

Node-Adaptive Routing

After distillation, the framework moves to the routing stage where frozen local and global semantic experts are used to obtain prediction logits. A node-adaptive router phi determines node-specific weights alpha v by assigning weights to the two experts based on structural representations H, local predictions y loc v, and global predictions y glo v. The final prediction is a linear combination of these expert outputs: yˆv = α loc v y loc v + α glo v y glo v A balance loss Lbal is included to encourage balanced utilization of the experts during routing, defined as Lbal = sum over m in each expert of (average routing weight - 1/2) squared

Experimental Validation

Extensive experiments on multiple benchmarks demonstrate that DeKoR consistently improves performance across node and subgraph classification tasks while substantially reducing inference time and GPU memory usage compared with most GNN–LLM baselines. DeKoR achieves an average accuracy of 85.78% across the five datasets, exceeding the strongest baseline, GLANCE (84.58%), by 1.20% Furthermore, in the ablation study, removing any of the key components—the router, local distillation loss Lloc, or global distillation loss Lglo—leads to a performance drop across all datasets The efficiency analysis shows that DeKoR requires substantially less GPU memory than most GNN–LLM baselines while maintaining strong predictive performance

Conclusion

DeKoR provides a practical way to combine the complementary capabilities of GNNs and LLMs by transferring and reorganizing LLM knowledge before fusion, allowing useful LLM knowledge to be retained while removing the LLM from downstream inference The results demonstrate that this design provides consistent improvements in both predictive performance and computational efficiency

AI USE STATEMENT

In this work, we used generative AI tools to improve the clarity, organization, and readability of the manuscript and to support literature retrieval and discovery All AI-assisted suggestions were manually reviewed and verified by three authors against the original experiments and cited literature Generative AI was not used to generate experimental results or make scientific conclusions We take full responsibility for the final content of this work.

REFERENCES

Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiang Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. IEEE Transactions on Big Data, 2025

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725–1735. PMLR, 2020

Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.08170, 2024

Imre Csiszar. On information-type measure of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar., 2:299–318, 1967

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805, 2025

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017

Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning. In International conference on learning representations, volume 2024, pp. 5711–5732, 2024a

Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876–132907, 2024b

Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs.

1. Bold Header: Knowledge Decomposition via Local and Global Views

This decomposition allows DeKoR to decompose LLM knowledge into complementary local and global views, capturing node-level semantics and prototype-based global relations, respectively. This structured decomposition ensures that the model captures both fine-grained node information (local knowledge) and broader relational context (global knowledge) simultaneously.

2. Bold Header: Lightweight Expert Distillation

The framework uses knowledge distillation to transfer the knowledge into lightweight modules, specifically optimizing objectives where the InfoNCE objective preserves local semantic knowledge and the KL divergence loss preserves the global prototype distribution. This creates efficient experts that can be used during inference without requiring full LLM inference.

3. Bold Header: Node-Adaptive Fusion Routing

A node-adaptive router learns how much each node should rely on each expert, allowing for dynamic prediction combination via the formula yˆv = αlocv ylocv + αglo v yglo v. This mechanism enables the model to leverage LLM knowledge selectively based on node characteristics, as shown in the routing analysis where the router favors the local semantic expert in the Local Only group but gives more weight to the global semantic expert in the Global Only group.

4. Bold Header: Efficient Knowledge Transfer

DeKoR addresses computational overhead by stating, The LLM is used only as an offline teacher, allowing DeKoR to retain useful LLM knowledge without requiring LLM inference during deployment. This allows for substantially less GPU memory than most GNN–LLM baselines and significantly lower inference times compared to methods like GraphGPT.

5. Bold Header: Robustness Across Backbones

The method demonstrates robustness by showing that DeKoR improves accuracy over each corresponding LLM-only baseline on all three datasets, suggesting it effectively coordinates the complementary knowledge from GNNs and LLMs while maintaining robustness to changes in backbones.