FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.
Ruth: We're looking at the paper: "FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning".
Warren: Here's the gist of the paper. Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context Benchmark Construction The paper presents FinReflectKG-MultiHop as a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024).
Ruth: Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), the authors generate financial analyst style questions with exact supporting evidence from the KG The two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity.
Warren: Evidence Retrieval Scenarios The study evaluates three controlled retrieval scenarios for each question: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations & distractors.
Ruth: This contrasts KG-guided precise retrieval against the pagewindow setting, which reflects the traditional vector retrieval paradigm.
Paper summary: Ruth: We just covered how this paper sets up this benchmark called FinMultiHopQA. The core thesis is that multi hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem because the relevant facts are dispersed across sections, filings, companies, and years.
Warren: That’s right. The authors argue that without precise knowledge graph guided selection of relevant context, even strong reasoning models either fail to answer or they consume an excessive amount of tokens just navigating that noisy context.
Iris: They propose a solution by using FinReflectKG, which is a temporally indexed financial knowledge graph linking audited triples to source chunks from S and P one hundred filings from twenty twenty two through twenty twenty four <ref:2510.02906#pg1,audited triples to source chunks from S>.
Ruth: The benchmark mines frequent two to three hop subgraph patterns across different sectors using GICS taxonomy to generate financial analyst style questions that have exact supporting evidence pulled right out of the knowledge graph.
Warren: They also detail a two phase prompt driven pipeline where they first create question answer pairs using specific pattern prompts and then apply a multi criteria quality control evaluation to ensure the QA pairs meet high standards for multi hop fidelity and temporal precision.
Iris: The dataset design is rich, containing one hundred seventy-five million triplets across seven hundred forty three S and P five hundred companies spanning twenty fourteen through twenty twenty four <ref:2510.02906#pg3>.
Ruth: That richness comes from a comprehensive financial knowledge graph schema derived from that FinReflectKG dataset which includes two dozen entity types like organizational information and risk factors, plus two dozen relationship types like operates in or negatively impacts.
Warren: The goal here is to capture the most relevant analytical insights that financial professionals would actually seek by generating these specific two hop and three hop traversal patterns tailored for SEC ten K filings <ref:2510.02906#pg2>.
Iris: And they use a large language model, specifically Qwen/Qwen3-235B-A22B, to generate these patterns based on that rich ontology of entities and relationships <ref:2510.02906#pg3,Qwen/Qwen3-235B-A22B, to generate>.
Ruth: So what this means is they are creating a really controlled environment to see how much better structured evidence retrieval is compared to just feeding the LLM raw text or a standard vector search.
Warren: They test this with three protocols: exact KG linked facts, relevant text windowed context, and relevant windowed context with some distractors.
Iris: That contrast between the precise KG path and the plain contexts reflects realistic semantic search retrieval scenarios that people actually face when they are trying to answer complex financial queries.
Conclusion: Ruth: So we’ve looked at how this paper, FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning, sets up this rigorous testing ground. The overall thesis is that using a knowledge graph to guide retrieval dramatically cuts down on the context noise that plagues multi hop reasoning tasks.
Warren: The authors claim that by mining those frequent two to three hop subgraph patterns, they are creating questions and answers with exact supporting evidence pulled straight from the knowledge graph structure rather than just relying on surrounding text.
Iris: And when they run the empirical results, they find that KG linked evidence improves correctness by about twenty four percent and reduces token usage by about eighty four point five percent compared to the pagewindow setting.
Ruth: That efficiency gain is significant, especially because it means models can focus their reasoning on composing already retrieved facts instead of wasting tokens just trying to connect dispersed evidence across time and company boundaries.
Warren: The benchmark itself is a curated subset containing five hundred fifty five multi hop QA pairs that cover cross company and inter year scenarios, which gives researchers a reproducible way to test these structured retrieval methods.
Iris: They also point out their limitation: the method relies heavily on the quality of the knowledge graph structure derived from SEC ten K filings, which means if that graph is incomplete or inaccurate, the benchmark might not fully capture all real world financial complexities <ref:2510.02906#pg2>.
Ruth: That’s a fair caveat. It shows that while this structured approach is very effective for getting high accuracy and saving tokens in a controlled setting, its performance still depends on how well that initial knowledge graph has been constructed.
Domyn
q-fin.CP, cs.AI
Submitted: 2025-10-03
Updated: 2026-11-04
Comments: 8 pages, 4 figures. Major revision of v1 with an expanded benchmark of 15,600 QA pairs, enhanced KG-driven question generation, additional evidence evaluation protocols, broader model evaluations, and expert quality validation
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 92/100
The gist: Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies…
Key concepts
- FinReflectKG
- This is a temporally indexed financial knowledge graph that connects specific audited facts (triples) to relevant text chunks found within S&P 100 company filings spanning the years 2022 to 2024. It serves as the structured source of evidence for financial questions.
- Multi-hop Reasoning
- This refers to answering complex financial questions that require connecting multiple pieces of information spread across different documents, years, or companies. The KG structure is designed specifically to facilitate these connections by linking disparate facts together in a chain.
- KG-guided Precise Retrieval
- This retrieval method uses the knowledge graph structure to find the exact path needed to support an answer, ensuring only highly relevant evidence is retrieved. This contrasts with standard methods that search for general text spans, focusing on structured connections rather than just textual proximity.
- Token Efficiency
- This measures how much input text (tokens) an LLM needs to process a question and its supporting evidence. The study found that using KG-guided retrieval dramatically lowers this requirement—by about 84.5% compared to page-window methods—making reasoning more cost-effective.
Terminology
Summary
Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context
Benchmark Construction
The paper presents FinReflectKG-MultiHop as a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024). Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), the authors generate financial analyst style questions with exact supporting evidence from the KG The two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity. p. 0 Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy),
we generate financial analyst style questions with exact supporting evidence from
the KG. A two-phase pipeline first creates QA pairs via pattern-specific prompts,
followed by a multi-criteria quality control evaluation to ensure QA validity.Where the paper says it
Evidence Retrieval Scenarios
The study evaluates three controlled retrieval scenarios for each question: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations & distractors. This contrasts KG-guided precise retrieval against the pagewindow setting, which reflects the traditional vector retrieval paradigm. p. 0 Across both reasoning and
non-reasoning models, KG-guided precise retrieval yields substantial gains on the
FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores
by ∼ 24% while reducing token utilization by ∼ 84. 5% compared to the pagewindow setting, which reflects the traditional vector retrieval paradigm.Where the paper says it
Performance Gains
Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains on the FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores by ∼ 24% while reducing token utilization by ∼ 84.5% compared to the pagewindow setting. p. 0 Across both reasoning and
non-reasoning models, KG-guided precise retrieval yields substantial gains on the
FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores
by ∼ 24% while reducing token utilization by ∼ 84. 5% compared to the pagewindow setting, which reflects the traditional vector retrieval paradigm. 5% compared to the pagewindow setting, which reflects the traditional vector retrieval paradigm. Spanning
intra-document, inter-year, and cross-company scopes, our work underscores the
pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop
financial QA.Where the paper says it
Where the paper says it
Model Comparison and Findings
Results show that KG-linked evidence consistently yields the highest LLM-Judge and BERTScore values across all models. On average, KG-linked improves LLM-Judge score by ∼ 24% over Page-window while using ∼ 84. p. 4 Across all models, KG-linked evidence consistently yields the highest LLM-Judge
and BERTScore values, demonstrating that structured KG-linked evidence reduces retrieval
noise and enables more accurate reasoning for answering financial multihop questions. On
average, KG-linked improves LLM-Judge score by ∼ 24% over Page-window while using
∼ 84. On
average, KG-linked improves LLM-Judge score by ∼ 24% over Page-window while using
∼ 84. 5% fewer input tokens, detailed summary in Appendix Tables 10, 11. 5% fewer input tokens, detailed summary in Appendix Tables 10, 11. The improvement is
more pronounced for smaller models, emphasizing the importance of efficient retrieval for limitedcapacity LLMs.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Future Work
The paper concludes that KG-guided, minimally noisy evidence consistently outperforms page-window style retrieval in both correctness and token efficiency across open-source reasoning and nonreasoning models. The benchmark includes a curated subset of 555 multi-hop QA pairs covering cross-company and inter-year scenarios, providing a controlled and realistic testbed for grounded financial reasoning. p. 7 The trends observed
are consistent across both evaluators, reinforcing the reliability of our findings and demonstrating
that evidence retrieval strategy, rather than judge-specific behavior, drives performance differences. Our benchmark includes a curated subset of 555 multi-hop QA pairs covering cross-company and
inter-year scenarios, providing a controlled and realistic testbed for grounded financial reasoning. Looking ahead, we plan to diversify and further calibrate LLM judges by incorporating additional
proprietary and open-source models to reduce evaluator-specific biases. We also aim to expand
model coverage to systematically assess the capabilities of closed-source reasoning systems, extend
dataset coverage with more cross-company and multi-year queries, and increase expert audits to
produce a larger, manually validated benchmark.Where the paper says it
Where the paper says it
References
[1] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data. p. 7 References
[1] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon,
Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data.Where the paper says it
[2] Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. p. 7 [2] Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering.Where the paper says it
[3] Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. p. 7 [3] Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie
Vidgen. Financebench: A new benchmark for financial question answering. Financebench: A new benchmark for financial question answering. arXiv preprint
arXiv:2311.Where the paper says it
Where the paper says it
[4] Zhiyu Zhu, Yifeng Wang, Chao-Chun Hsu, Yuan-Kuei Wu, and Qingyu Zhou. Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models. p. 7 [4] Zhiyu Zhu, Yifeng Wang, Chao-Chun Hsu, Yuan-Kuei Wu, and Qingyu Zhou. Fanoutqa:
A multi-hop, multi-document question answering benchmark for large language models.Where the paper says it
[5] Siddharth Sahai, Akshay Chaturvedi, and Rushang Karia. Multi-document financial question answering using llms. p. 7 [5] Siddharth Sahai, Akshay Chaturvedi, and Rushang Karia. Multi-document financial question
answering using llms. Multi-document financial question
answering using llms. arXiv preprint arXiv:2411.Where the paper says it
Where the paper says it
[6] Abhinav Arun, Fabrizio Dimino, Tejas Prakash Agarwal, Bhaskarjit Sarmah, and Stefano Pasquali. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs, 2025. p. 7 [6] Abhinav Arun, Fabrizio Dimino, Tejas Prakash Agarwal, Bhaskarjit Sarmah, and Stefano
Pasquali. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. URL https://arxiv. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. URL https://arxiv. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. URL https://arxiv. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. URL https://arxiv. Finreflectkg: Agentic construction and evaluation of financial knowledge graphs,
2025. URL https://arxiv. URL https://arxiv. org/abs/2508. org/abs/2508. Dataset available at: https://
huggingface.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[7] Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. Docfinqa: A long-context financial reasoning dataset, 2024. p. 7 [7] Varshini Reddy, Rik Koncel-Kedziorski, Viet Dac Lai, Michael Krumdick, Charles Lovering,
and Chris Tanner. Docfinqa: A long-context financial reasoning dataset, 2024. URL https:
//arxiv. org/abs/2401.Where the paper says it
Where the paper says it
[8] Yixuan Tang, Yi Yang, Ahmed Masry, Mor Geva, Dipanjan Das, Avijit Mitra, Dragomir Radev, and Ben Zhou. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. p. 7 [8] Yixuan Tang, Yi Yang, Ahmed Masry, Mor Geva, Dipanjan Das, Avijit Mitra, Dragomir Radev,
and Ben Zhou. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop
queries. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop
queries. arXiv preprint arXiv:2401.Where the paper says it
Where the paper says it
[9] Yash Sharma, Shubham Mehta, Kartik Sachdeva, and Srinivasan Parthasarathy. Biohopr: A benchmark for multi-hop, multi-answer reasoning in biomedical domain. p. 7 [9] Yash Sharma, Shubham Mehta, Kartik Sachdeva, and Srinivasan Parthasarathy. Biohopr: A
benchmark for multi-hop, multi-answer reasoning in biomedical domain.Where the paper says it
[10] Nikita Tatarinov, Vidhyakshaya Kannan, Haricharana Srinivasa, Arnav Raj, Harpreet Singh Anand, Varun Singh, Aditya Luthra, Ravij Lade, Agam Shah, and Sudheer Chava. Kg-qagen: A knowledge-graph-based framework for systematic question generation and long-context llm evaluation, 2025. p. 7 [10] Nikita Tatarinov, Vidhyakshaya Kannan, Haricharana Srinivasa, Arnav Raj, Harpreet Singh
Anand, Varun Singh, Aditya Luthra, Ravij Lade, Agam Shah, and Sudheer Chava. Kg-qagen:
A knowledge-graph-based framework for systematic question generation and long-context llm
evaluation, 2025. URL https://arxiv. org/abs/2505.Where the paper says it
Where the paper says it
[11] Lavanya Gupta, Saket Sharma, and Yiyun Zhao. Systematic evaluation of long-context llms on financial concepts. p. 1 Recent systematic evaluation by Gupta et
al. [11] revealed brittleness in long-context LLMs on financial concepts, showing performance degradation even on simple tasks as context length increases, highlighting the need
for robust multi-hop evaluation frameworks. Systematic evaluation of long-context llms on
financial concepts. In Proceedings of the 2024 Conference on Empirical Methods in Natural
Language Processing: Industry Track, page 1163–1175.Where the paper says it
Where the paper says it
[12] Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming Zhang, Xiangru Tang, Chen Zhao, and Arman Cohan. Findver: Explainable claim verification over long and hybrid-content financial documents, 2024. p. 8 [12] Yilun Zhao, Yitao Long, Yuru Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Yiming
Zhang, Xiangru Tang, Chen Zhao, and Arman Cohan. Findver: Explainable claim verification
over long and hybrid-content financial documents, 2024. URL https://arxiv. org/abs/
2411.Where the paper says it
Where the paper says it
[13] Xiaolu Lu, Soumajit Pramanik, Rishiraj Saha Roy, Abdalghani Abujabal, Yafang Wang, and Gerhard Weikum. Answering complex questions by joining multi-document evidence with quasi knowledge graphs. p. 8 [13] Xiaolu Lu, Soumajit Pramanik, Rishiraj Saha Roy, Abdalghani Abujabal, Yafang Wang, and
Gerhard Weikum. Answering complex questions by joining multi-document evidence with
quasi knowledge graphs.Where the paper says it
[14] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Multi-hop knowledge graph reasoning with reward shaping. p. 8 [14] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Multi-hop knowledge graph reasoning
with reward shaping.Where the paper says it
[15] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021. p. 8 [15] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced
bert with disentangled attention, 2021. URL https://arxiv. org/abs/2006.Where the paper says it
Where the paper says it
[16] Qwen3-32B Reasoning 8.23 8.71 2069 703
[17] Qwen3-8B Reasoning 8.03 8.18 2069 814
[18] GPT-OSS-120B KG-linked Page-window % Improvement GPT-OSS-120B 7.12 13.6%
[19] Qwen3-32B Page-window KG-linked % Savings Qwen3 -8B 84.8%
[20] GPT-OSS-120B Input token savings GPT-OSS-120B 84.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
KG-guided evidence selection reduces token consumption by ∼ 84.5% compared to page window settings, allowing models to focus reasoning on
composing already retrieved facts.
-
The system can perform multi-hop financial QA over 10-K filings of S&P 100 companies, spanning
intra-document, inter-year, and cross-company relationships.
-
Models can generate analyst style questions by mining frequent
2-3 hop subgraph patterns across sectors (via GICS taxonomy)
with exact supporting evidence from the Knowledge Graph. -
The system can utilize a two phase pipeline involving pattern prompts and a
multi-criteria quality control evaluation to ensure QA validity
before testing retrieval scenarios like KG-linked paths (S1), text-only page windows (S2), and randomized distractors (S3). -
The improved system exhibits accuracy gains by ∼ 24% on the FinReflectKG - MultiHop QA benchmark dataset when using KG-guided precise retrieval.
-
The system can distinguish between intra-document, inter-year, and cross-company relationships during evaluation, showing that
intra-document questions consistently achieve the highest correctness scores.
Abstract
Multi-hop financial question answering often requires composing evidence distributed across filing sections, fiscal years, and peer companies. Prior financial QA benchmarks address parts of this problem individually, but none jointly support controlled multi-hop taxonomies, document topologies, motif-level reproducibility, and provenance-linked auditability at scale. We introduce FinMultiHopQA, a benchmark of 15,600 qualitative and quantitative questions, grounded in FinReflectKG, a temporally indexed financial knowledge graph derived from SEC 10-K filings with 17.5M+ triples. The pipeline instantiates statistically salient 2-3 hop typed KG motifs across intra-document, inter-year, and cross-company regimes. Beyond closing this gap, the same construction lets us hold each question's underlying evidence constant while varying only its presentation - embedding it in noisy surrounding context or changing its order - giving a controlled instrument to study a secondary question at scale: using six such presentations, we test whether reasoning LLMs' performance depends on evidence structure. Results confirm this: despite their reasoning capacity, judged correctness remains consistently lower under less-structured presentations of the same evidence - most sharply for quantitative cross-company reasoning and 3-hop branch motifs - indicating that reasoning alone does not fully offset how evidence is structured; KG-linked presentation also uses fewer input tokens by design. We release the benchmark, provenance annotations, model outputs, evaluation records, and supporting technical documentation.
Sources
- FinanceBench: A New Benchmark for Financial Question Answering
- Multi-Document Financial Question Answering using LLMs
- FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs
- DocFinQA: A Long-Context Financial Reasoning Dataset
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
- KG-MuLQA: A Framework for KG-based Multi-Level QA Extraction and Long-Context LLM Evaluation
- FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Related papers
- AFNO: Universal Certifiably-Arbitrage-Free Operator Learning Framework for European Call-Price Surfaces
- Diffusion models for dynamic volatility surface generation and data-driven hedging
- Rethinking Synthetic Scenario Realism: Compatibility, Not Fidelity, Drives Hedging Performance
- CosmosAlpha: Multi-Relational Graph Search with Pareto-Guided Exploration for LLM-Driven Alpha Mining
- Diversification Across Predictors: Heterogeneous Ensemble Learning for Cross-Sectional Return Prediction
- DSINC beyond Heston: the 3/2 and other non-affine stochastic-volatility models