IntelliAudit: Using Large Language Models to Evaluate Audit Controls
Allison Wilson, Sina Moradi Sabet, Diar Shakimov, Panteha Shahrivar, Mohammad Reza Bagheri, Dean Konenkamp, Mohammad A. Tayebi
cs.AI, cs.CL, cs.CR, cs.HC, cs.MA
Submitted: 2026-08-07
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls.
Terminology
Abstract
IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls. This judgment is difficult to automate because relevant evidence is distributed across policies, records, spreadsheets, and operational artifacts, and because audit conclusions depend on evidentiary sufficiency rather than keyword matching. We present IntelliAudit, a retrieval-grounded multi-agent system for IT audit evidence evaluation. Given a control and an evidence corpus, IntelliAudit retrieves relevant artifacts, generates an evidence-grounded assessment, challenges adverse findings, adjudicates disagreements, and produces an auditor-facing recommendation with cited evidence, rationale, missing-evidence analysis, and remediation guidance. We instantiate IntelliAudit on ISO/IEC 27001 and evaluate it across multiple simulated organizations using expert auditor review and audit-readiness user feedback. The evaluation shows that IntelliAudit can support control interpretation, evidence-grounded reasoning, and audit-preparation workflows, while also revealing the importance of human oversight for calibrating sufficiency judgments and correcting overly permissive recommendations. These results suggest that retrieval-grounded multi-agent systems can assist audit evidence review, but should remain decision-support tools rather than autonomous certification systems.
Sources
- Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
- RAGulating Compliance: A Multi-Agent Knowledge Graph for Regulatory QA
- LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop
- Automating Security Audit Using Large Language Model based Agent: An Exploration Experiment
- GraphCompliance: Structure-Aware Regulatory Compliance through Policy-Context Graph Alignment
- Improving Factuality and Reasoning in Language Models through Multiagent Debate
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- From COBIT to ISO 42001: Evaluating Cybersecurity Frameworks for Opportunities, Risks, and Regulatory Compliance in Commercializing Large Language Models
- Supporting Human-AI Collaboration in Auditing LLMs with LLMs
- Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
- Multi-Agent Collaboration Mechanisms: A Survey of LLMs
- Automating Financial Statement Audits with Large Language Models
- AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification
- FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs
- LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
- Smart Audit System Empowered by LLM
- LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems