FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning

summary

The gist

Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies…

In short

The research introduced FinReflectKG-MultiHop, a benchmark based on a financial knowledge graph linking audited triples to S&P 100 filings from 2022-2024. It tests how grounding answers using precise KG paths outperforms traditional text-only retrieval methods. Results show KG guidance significantly boosts accuracy while drastically reducing the number of input tokens needed by LLMs for complex financial questions.

Key concepts

FinReflectKG
This is a temporally indexed financial knowledge graph that connects specific audited facts (triples) to relevant text chunks found within S&P 100 company filings spanning the years 2022 to 2024. It serves as the structured source of evidence for financial questions.
Multi-hop Reasoning
This refers to answering complex financial questions that require connecting multiple pieces of information spread across different documents, years, or companies. The KG structure is designed specifically to facilitate these connections by linking disparate facts together in a chain.
KG-guided Precise Retrieval
This retrieval method uses the knowledge graph structure to find the exact path needed to support an answer, ensuring only highly relevant evidence is retrieved. This contrasts with standard methods that search for general text spans, focusing on structured connections rather than just textual proximity.
Token Efficiency
This measures how much input text (tokens) an LLM needs to process a question and its supporting evidence. The study found that using KG-guided retrieval dramatically lowers this requirement—by about 84.5% compared to page-window methods—making reasoning more cost-effective.

Terminology used across episodes

This episode discusses

The paper

FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning · Read on arXiv

Domyn

Multi-hop financial question answering often requires composing evidence distributed across filing sections, fiscal years, and peer companies. Prior financial QA benchmarks address parts of this problem individually, but none jointly support controlled multi-hop taxonomies, document topologies, motif-level reproducibility, and provenance-linked auditability at scale. We introduce FinMultiHopQA, a benchmark of 15,600 qualitative and quantitative questions, grounded in FinReflectKG, a temporally indexed financial knowledge graph derived from SEC 10-K filings with 17.5M+ triples. The pipeline instantiates statistically salient 2-3 hop typed KG motifs across intra-document, inter-year, and cross-company regimes. Beyond closing this gap, the same construction lets us hold each question's underlying evidence constant while varying only its presentation - embedding it in noisy surrounding context or changing its order - giving a controlled instrument to study a secondary question at scale: using six such presentations, we test whether reasoning LLMs' performance depends on evidence structure. Results confirm this: despite their reasoning capacity, judged correctness remains consistently lower under less-structured presentations of the same evidence - most sharply for quantitative cross-company reasoning and 3-hop branch motifs - indicating that reasoning alone does not fully offset how evidence is structured; KG-linked presentation also uses fewer input tokens by design. We release the benchmark, provenance annotations, model outputs, evaluation records, and supporting technical documentation.

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: We're looking at the paper: "FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning".

Warren: Here's the gist of the paper. Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context Benchmark Construction The paper presents FinReflectKG-MultiHop as a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P 100 filings (2022-2024).

Ruth: Mining frequent 2-3 hop subgraph patterns across sectors (via GICS taxonomy), the authors generate financial analyst style questions with exact supporting evidence from the KG The two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity.

Warren: Evidence Retrieval Scenarios The study evaluates three controlled retrieval scenarios for each question: (S1) precise KG-linked paths; (S2) text-only page windows centered on relevant text spans; and (S3) relevant page windows with randomizations & distractors.

Ruth: This contrasts KG-guided precise retrieval against the pagewindow setting, which reflects the traditional vector retrieval paradigm.

Paper summary: Ruth: We just covered how this paper sets up this benchmark called FinMultiHopQA. The core thesis is that multi hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem because the relevant facts are dispersed across sections, filings, companies, and years.

Warren: That’s right. The authors argue that without precise knowledge graph guided selection of relevant context, even strong reasoning models either fail to answer or they consume an excessive amount of tokens just navigating that noisy context.

Iris: They propose a solution by using FinReflectKG, which is a temporally indexed financial knowledge graph linking audited triples to source chunks from S and P one hundred filings from twenty twenty two through twenty twenty four <ref:2510.02906#pg1,audited triples to source chunks from S>.

Ruth: The benchmark mines frequent two to three hop subgraph patterns across different sectors using GICS taxonomy to generate financial analyst style questions that have exact supporting evidence pulled right out of the knowledge graph.

Warren: They also detail a two phase prompt driven pipeline where they first create question answer pairs using specific pattern prompts and then apply a multi criteria quality control evaluation to ensure the QA pairs meet high standards for multi hop fidelity and temporal precision.

Iris: The dataset design is rich, containing one hundred seventy-five million triplets across seven hundred forty three S and P five hundred companies spanning twenty fourteen through twenty twenty four <ref:2510.02906#pg3>.

Ruth: That richness comes from a comprehensive financial knowledge graph schema derived from that FinReflectKG dataset which includes two dozen entity types like organizational information and risk factors, plus two dozen relationship types like operates in or negatively impacts.

Warren: The goal here is to capture the most relevant analytical insights that financial professionals would actually seek by generating these specific two hop and three hop traversal patterns tailored for SEC ten K filings <ref:2510.02906#pg2>.

Iris: And they use a large language model, specifically Qwen/Qwen3-235B-A22B, to generate these patterns based on that rich ontology of entities and relationships <ref:2510.02906#pg3,Qwen/Qwen3-235B-A22B, to generate>.

Ruth: So what this means is they are creating a really controlled environment to see how much better structured evidence retrieval is compared to just feeding the LLM raw text or a standard vector search.

Warren: They test this with three protocols: exact KG linked facts, relevant text windowed context, and relevant windowed context with some distractors.

Iris: That contrast between the precise KG path and the plain contexts reflects realistic semantic search retrieval scenarios that people actually face when they are trying to answer complex financial queries.

Conclusion: Ruth: So we’ve looked at how this paper, FinMultiHopQA: A KG-Grounded Multi-Hop Financial QA Benchmark for Evidence-Driven Reasoning, sets up this rigorous testing ground. The overall thesis is that using a knowledge graph to guide retrieval dramatically cuts down on the context noise that plagues multi hop reasoning tasks.

Warren: The authors claim that by mining those frequent two to three hop subgraph patterns, they are creating questions and answers with exact supporting evidence pulled straight from the knowledge graph structure rather than just relying on surrounding text.

Iris: And when they run the empirical results, they find that KG linked evidence improves correctness by about twenty four percent and reduces token usage by about eighty four point five percent compared to the pagewindow setting.

Ruth: That efficiency gain is significant, especially because it means models can focus their reasoning on composing already retrieved facts instead of wasting tokens just trying to connect dispersed evidence across time and company boundaries.

Warren: The benchmark itself is a curated subset containing five hundred fifty five multi hop QA pairs that cover cross company and inter year scenarios, which gives researchers a reproducible way to test these structured retrieval methods.

Iris: They also point out their limitation: the method relies heavily on the quality of the knowledge graph structure derived from SEC ten K filings, which means if that graph is incomplete or inaccurate, the benchmark might not fully capture all real world financial complexities <ref:2510.02906#pg2>.

Ruth: That’s a fair caveat. It shows that while this structured approach is very effective for getting high accuracy and saving tokens in a controlled setting, its performance still depends on how well that initial knowledge graph has been constructed.

More episodes

← Home