KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs

KGCaRe is a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated knowledge graphs (KGs) to answer complex conditional questions. The paper hypothesizes that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks.

The approach constructs a KG from documents using a multi-prompt extraction strategy and stores it in a Neo4j graph database. Simultaneously, documents are embedded into a FAISS vector store for neural retrieval. During inference, KGCaRe performs "innovative iterative graph traversal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traversal does not provide satisfactory context to generate the answer." The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are fed into custom KGCaRe prompts to generate answers with explanations.

The KG construction uses four progressive prompt stages: (1) contextual understanding and initial entity-relation extraction, (2) conditional and alternative scenario expansion, (3) refinement and logical graph expansion, and (4) output normalization and graph ingestion. The traversal algorithm (Algorithm 1) initializes topic entities from the question, uses partial matching at depth 1 and exact matching at subsequent depths, prunes triples via an LLM, and dynamically updates clue entities and memory until sufficient context is gathered or maximum depth is reached.

The paper evaluates KGCaRe on two datasets: ConditionalQA (271 development QA pairs from UK policy documents) and HotpotQA (500 QA pairs from Wikipedia). Results show KGCaRe consistently outperforms existing baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o.

Key results include: On ConditionalQA with Mistral, KGCaRe achieves an average F1 of 57.89 compared to HybridContextQA's 45.17 and Think-on-Graph's 16.24. With GPT-4o, KGCaRe achieves 67.55 average F1 versus HybridContextQA's 63.71. On HotpotQA, KGCaRe achieves the highest F1 scores across all categories for all models, with GPT-4o reaching 80.21 average F1. An ablation study on 106 Yes/No questions with single definitive answers shows KGCaRe consistently outperforms baselines, achieving 88.67 F1 with GPT-4o.

The paper also demonstrates explainability through KG traversal paths, showing how the approach retrieves relevant triples across multiple depths and generates answers with conditions and explanations. The authors conclude that combining structured and unstructured context, and reasoning over both, yields superior performance on complex, conditional QA tasks, regardless of the underlying LLM's capacity. The software pipeline and prompts are publicly released at https://github.com/GhanshyamVerma/KGCaRe.

Improvements to AI Systems:

1. Hybrid Retrieval-Augmented Generation (RAG) with Dual Knowledge Sources

• Integrate both unstructured text (via vector embeddings) and structured knowledge (via LLM-generated knowledge graphs) into the retrieval pipeline.

• The improved system can answer complex conditional questions (e.g., What benefits apply if the applicant is over 65 and lives alone?) by jointly reasoning over passages and graph paths, reducing hallucination and improving factual grounding.

2. Iterative, LLM-Guided Graph Traversal with Dynamic Clue Expansion

• Implement an adaptive traversal algorithm (Algorithm 1) that starts from topic entities, uses partial matching at depth 1 and exact matching later, prunes irrelevant triples via LLM judgment, and re-traverses with newly discovered clue entities if initial context is insufficient.

• The improved system can handle multi-hop reasoning tasks (e.g., HotpotQA) by exploring alternative paths, avoiding premature termination, and self-correcting when the first retrieval path is incomplete.

3. Progressive Multi-Stage Knowledge Graph Construction

• Use four sequential prompt stages: (a) initial entity-relation extraction, (b) conditional/alternative scenario expansion, (c) logical refinement, and (d) normalization for graph ingestion.

• The improved system can build richer, more accurate KGs from documents, capturing conditional logic (e.g., if-then rules, exceptions, and nested conditions) that standard extraction misses, leading to better handling of nuanced policy or legal texts.

4. Unified Prompting with Both Graph Paths and Text Passages

• Feed extracted triples (in path form) alongside semantically retrieved passages into a single custom prompt for answer generation with explanations.

• The improved system can produce answers that cite both structured relationships (e.g., Entity A → relation → Entity B) and supporting text, improving transparency and user trust in AI-generated responses.

5. Cross-LLM Robustness and Performance Gains

• Apply the KGCaRe architecture to any LLM (Mistral, GPT-4o, etc.) without fine-tuning, leveraging the model’s own reasoning for graph traversal and pruning.

• The improved system consistently outperforms vanilla RAG, graph-only, and text-only baselines (e.g., +12.7 F1 over HybridContextQA on ConditionalQA with Mistral; +8.6 F1 on HotpotQA with GPT-4o), making it suitable for high-stakes domains like policy analysis, legal QA, and medical guideline interpretation.

6. Explainable Answer Generation via Traversal Paths

• Output the actual KG traversal path (e.g., Entity A → condition → Entity B → exception → Entity C) as part of the answer explanation.

• The improved system can justify its reasoning step-by-step, enabling users to verify logical chains and detect potential errors—critical for auditability in regulated industries.

7. Ablation-Driven Precision for Yes/No Conditional Queries

• Use the ablation study’s insight (88.67 F1 on single-definitive-answer questions) to optimize the system for binary classification tasks with conditional constraints.

• The improved system can reliably determine eligibility, compliance, or entitlement (e.g., Is the applicant eligible for housing aid if they are a part-time student?) by combining graph-based condition checking with textual evidence.

8. Open-Source, Reproducible Pipeline Integration

• Adopt the released software and prompts (from the GitHub repo) to quickly deploy the system in existing RAG frameworks.

• The improved system can be extended to new domains by simply swapping documents and re-running the KG construction stages, without manual ontology design or extensive feature engineering.

Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a challenge, particularly in domain-specific contexts where general-purpose LLMs and RAG tend to underperform. We hypothesize that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks. To test this, we propose KGCaRe, a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated KGs. KGCaRe constructs a KG from documents using a multi-prompt extraction strategy and stores it in a graph database. Simultaneously, the documents are embedded into a vector store to enable neural retrieval. KGCaRe performs innovative iterative graph traversal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traversal does not provide satisfactory context to generate the answer. The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are then fed into custom KGCaRe prompts to generate answers to the complex conditional questions with explanations. We evaluate KGCaRe on two complex conditional QA datasets. Our results on these datasets show that KGCaRe consistently outperforms existing baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. We publicly release the software pipeline that we developed to implement the proposed KGCaRe approach.