IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

arXiv:2608.04703 · cs.CL · Submitted 2026-08-05 · Read on arXiv

Shahd Gaben, Heba Sbahi, Samer Rashwani, Abdessalam Bouchekif, Mutaz Al-Khatib, Emad Mohamed, Somaya Eltanbouly, Mohammed Ghaly

cs.CL

Submitted: 2026-08-05

Comments: Includes supplementary materials. Submitted to the Journal of Scientific Data. Data and code are publicly available

Code: https://github.com/GabenS99/IslamicTurathBench_Evaluation

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 41× in this paper · explained in 2 episodes
• RAG — 10× in this paper · explained in 3 episodes
• Calibration — 8× in this paper · explained in 9 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Benchmarking — 3× in this paper · explained in 4 episodes
• Clarity — 3× in this paper · explained in 3 episodes
• Core — 3× in this paper · explained in 2 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 4 episodes
• Language Models (LLMs) — 3× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 3× in this paper · explained in 9 episodes
• Retrieval-Augmented Generation — 3× in this paper
• Commentary — 2× in this paper · explained in 2 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Abstention — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Composition — 1× in this paper · explained in 2 episodes
• Evaluation Framework — 1× in this paper · explained in 3 episodes
• Evaluation Protocol — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 8 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Multi-Agent Architecture — 1× in this paper · explained in 2 episodes
• Performance Gap — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Retrieval-Augmented Generation (RAG) — 1× in this paper · explained in 21 episodes
• Unified Framework — 1× in this paper · explained in 6 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• On the Opportunities and Risks of Foundation Models
• GPT-4 Technical Report
• IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditions
• IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge
• MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs
• Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions
• Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data
• DeepSeek-V3 Technical Report
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• ALLaM: Large Language Models for Arabic and English
• Fanar 2.0: Arabic Generative AI Stack
• Fanar-Sadiq: A Multi-Agent Architecture for Grounded Islamic QA
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering