PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs

PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs

Abstract Summary:

Institutional policies are written in natural language, but verifying compliance increasingly demands executable constraints, leaving organisations to bridge that gap by hand. The paper presents PolicyKG, an agentic LLM pipeline that translates institutional policy documents into SHACL-validated knowledge graphs via a first-order deontic intermediate representation, orchestrated as four self-correcting stages over a LangGraph state machine. The main technical contribution is the Corpus Adapter, a configuration-driven vocabulary-grounding layer that constrains LLM-generated predicates to a target ontology and enables cross-domain transfer by swapping registries rather than retraining models. Evaluated on the full Asian Institute of Technology Policies & Procedures corpus (1,663 sentences yielding 443 rules), PolicyKG achieves 86.9% deontic-type classification accuracy (Cohen’s κ =.709, corroborated by a three-annotator IRR with Fleiss’ κ =.844, item-level bootstrap 95% CI [.67,.97]) and SHACL shape correctness of F1 =.866, 95% CI [.79,.93], on a 69-shape evaluation subset of an ontology-grounded reference set. End-to-end formalisation yield is 79.2% (351/443) via the FOL path; the remaining 20.8% fall back to a direct NL→SHACL path whose failure modes are predominantly structural (long compound conjunctions, embedded bullets), not expressiveness-bounded. A two-pass audit (an automated regex checklist over all 443 rules plus a first-author manual review of the 92 FOL-fallback cases where higher-order constructs could otherwise hide) flagged zero second- or higher-order constructs (exact Clopper–Pearson upper 95% CI on the true HOL requirement rate: 0.67%). A controlled cross-domain experiment on N = 15 GDPR rules raises exact property alignment from 1/15 to 11/15 with the Corpus Adapter (Fisher’s exact p <.001; Cohen’s h = 1.53, large effect). External validation on the LexDeMod lease-contract benchmark (N = 200) drops Macro F1 from.810 to.370, with permission F1 =.038; the failure mechanism is a vocabulary-prior mismatch (lease English uses shall be entitled for permission) precisely of the class the Corpus Adapter is designed to repair by registry swap.

Introduction Summary:

Institutional policies governing students, staff, and operations are written as natural-language documents and read by humans, yet verifying compliance is increasingly automated. The gap between human-authored policy and machine-checked compliance is bridged today by manual interpretation, which does not scale, does not survive policy revisions, and offers no audit trail. Three lines of research address this: policy and legal NLP (rule-extraction systems reporting accuracies between 75% and 85%), deontic logic (operators of obligation, permission, and prohibition), and SHACL (W3C Shapes Constraint Language for validation constraints over RDF knowledge graphs). What is missing is an empirically validated, end-to-end pipeline connecting natural-language policy to executable SHACL, together with an answer to whether first-order logic with deontic extensions is sufficient for institutional policy. PolicyKG runs four orchestrated stages: document extraction, deontic rule classification, FOL+deontic formalisation, and SHACL translation, coordinated by a LangGraph state machine with per-stage validators triggering self-correction. The Corpus Adapter is a configuration-driven vocabulary-grounding layer enabling cross-domain transfer by swapping registries.

Contributions:

• An end-to-end agentic LLM pipeline (NL→FOL+deontic→SHACL) validated on a real 1,663-sentence institutional corpus.

• The Corpus Adapter, a vocabulary-grounding pattern enabling cross-domain transfer (GDPR experiment raises exact property alignment from 1/15 to 11/15, Fisher’s exact p <.001; Cohen’s h = 1.53).

• A two-pass empirical audit finding that first-order deontic logic is observationally sufficient on this corpus: automated regex checklist flags 0 of 443 rules with second- or higher-order construct patterns, and manual review of 92 FOL-fallback rules confirms 0/92 genuine HOL cases (exact Clopper–Pearson upper 95% CI on the true HOL rate: 0.67%).

• An open evaluation: three-annotator gold (Fleiss’ κ =.844, item-level bootstrap 95% CI [.67,.97]), LexDeMod external validation, and fixed-seed reproducibility yielding hash-identical SHACL outputs.

Related Work Summary:

Pattern-based extraction (Maxwell and Antón) reported around 77% accuracy on privacy policies; Brodie et al. reached 82% on financial regulation using SVM classifiers; hybrids of pattern matching and ontology grounding reached 85% accuracy on legal text. The LLM era produced benchmarks like LexGLUE, LegalBench, and PAPEL (80% F1 on clause labelling) but few end-to-end systems producing executable SHACL. Deontic logic provides obligation, permission, and prohibition operators; Governatori and Rotolo argued theoretically that first-order deontic logic suffices for normative systems. SHACL enables declarative validation of RDF knowledge graphs; Palmirani and Governatori argued theoretically that SHACL has adequate expressiveness for institutional policy formalisation but lacked empirical validation. Schema-constrained and grammar-guided generation addresses syntactic structured output; RAG and GraphRAG ground LLM outputs in external knowledge but assume the target graph exists. The Corpus Adapter addresses the semantic gap of ensuring LLM-generated predicates align with a target ontology.

Preliminaries Summary:

First-order deontic logic extends FOL with three modal operators: O(φ) for obligatory, P(φ) for permitted, and F(φ) for forbidden, with F(φ) ≡ O(¬φ), adopting Standard Deontic Logic axioms including O(φ) → P(φ). A worked example: Students must register before classes begin formalises as ∀x Student(x) → O(RegisterBefore(x, Classes)). SHACL constraints validate RDF graphs via NodeShape targeting a class and PropertyShape constraining predicates with cardinality, value-type, or pattern restrictions, with severity levels (sh:Violation, sh:Warning, sh:Info). The translation gap: naive LLM prompting to emit SHACL directly fails along three axes—vocabulary hallucination (invented property names), lost deontic semantics (modality collapses into uniform constraint phrasing), and predicate-quality drift (lexically divergent predicates across re-runs). These motivate the FOL+deontic intermediate layer and the Corpus Adapter.

Pipeline Summary:

PolicyKG is structured as four sequential stages: Stage 1 normalises policy PDFs into sentences (pdfplumber extraction, policy-aware regex splitter, conservative simplification pass preserving semantics dual-checked by LLM self-assessment and Sentence-BERT cosine similarity ≥ 0.85 accepting 97.9% of rules); Stage 2 classifies each sentence as Obligation, Permission, Prohibition, Non-rule using zero-shot prompting with confidence scores; Stage 3 lifts each classified rule to an FOL+deontic formula with exactly one top-level operator from O, P, F using modal-cue mappings (must, shall → O; may → P; must not, shall not → F), constrained to structured JSON schema, with validation and one retry on rejection; Stage 4 maps each formula to a SHACL shape via deterministic rule-driven translation (universal-obligation becomes NodeShape with sh:minCount 1 and sh:Violation; permissions become Info-level shapes; prohibitions use sh:maxCount 0 with sh:Violation). The Corpus Adapter is a thin configuration-driven layer with three parts: (i) a vocabulary registry enumerating classes, properties, and synonyms from the target ontology; (ii) a constrained-selection prompt at Stage 4 reframing predicate emission as selection rather than open generation; (iii) a post-hoc canonicaliser mapping free-form predicates onto registry matches via normalisation plus fuzzy fallback, triggering Stage 3 repair on reject. Retargeting to a new domain means writing a new registry; nothing else changes. Conditional exceptions are recorded as explicit deontic:overrides relations between shapes (12 emitted in the AIT corpus).

Experimental Setup Summary:

The primary corpus D1 is built from five AIT Policies & Procedures documents (81 pages, spanning credit/financial, accommodation, grievance, ethics, and general-conduct domains). Stage 1 extracts 1,663 sentences; a deontic-marker prefilter retains 461 candidates; Stage 2 yields 443 classified rules. The gold standard is built in two passes: Pass 1 labels every sentence (primary annotator); Pass 2 has two additional annotators (Kittipat with Law/Legal Studies background; Mayuree with Linguistics background) independently re-annotate a stratified 50-item sample (28 O / 12 P / 10 F) blind to model outputs. The IRR sample yields Fleiss’ κ =.844 (almost perfect; 10,000-iteration item-level bootstrap 95% CI [.673,.967]), with pairwise Cohen’s κ values of.833,.868, and.832. A post-hoc disagreement audit reclassified 12 systematically mislabelled permissions, yielding final distribution: 308 obligations (69.5%), 75 permissions (16.9%), and 60 prohibitions (13.5%). External datasets: D2 LexDeMod (200 lease-contract clauses) and D3 GDPR portability set (15 rule-bearing sentences with hand-curated property targets). All LLM calls use Mistral 7B Instruct served locally by Ollama on NVIDIA A100 (40 GB) with near-deterministic decoding (temperature = 0.1, fixed seed, 2,048 max output tokens). Mistral was selected from a five-model pilot (3.8–70 B parameters) for zero parse-error rate and inference stability, not highest raw accuracy. Five metrics: M1 binary rule detection, M2 deontic-type accuracy, M3 FOL semantic predicate quality, M4 SHACL shape correctness, M5 output stability. Reproducibility: all numbers correspond to repository commit 546dee6f, pipeline version 2.1-final-defense, Ollama model digest 6577803aa9a0, random seed 42; repeated runs produce hash-identical SHACL outputs across all 443 shapes on the same host; independent verification on RTX 4060 produces counts within ±2 pp.

Results Summary:

RQ1: On the full D1 corpus (N = 443 rules), Mistral 7B Instruct achieves 86.9% deontic-type accuracy with Cohen’s κ =.709 (substantial agreement) and Macro F1 =.810, above the 75–85% range reported for prior approaches. Per-class F1:.915 for obligations (290/308 correct),.784 for permissions (49/75),.730 for prohibitions (46/60). The class-specific weakness is recall on permissions (65.3%): 22 of 75 permission rules reclassified as obligations due to epistemic-versus-deontic may ambiguity. On the IRR sample (N = 50), the model labels 42/50 sentences in agreement with majority-vote human gold (κ =.629); human-only Fleiss’ κ =.844. Binary rule detection reaches F1 =.963 at 93.81% accuracy with κ =.764, against regex baseline at F1 =.898 (+.065 F1; Cohen’s h ≈ 0.34). A learned in-domain baseline (NumPy-only multinomial logistic regression on TF bag-of-words plus six deontic-marker features, stratified 5-fold CV) reaches accuracy 90.3% / κ =.799 / Macro F1 =.863, outperforming the zero-shot LLM on D1 as expected since it consumes in-domain labels. Head-to-head open-model comparison (N = 50): Llama 3.1 8B and Gemma 2 9B both reach 88% accuracy, 4 pp above Mistral, but differences are not statistically significant by exact McNemar test (p =.77 and.69).

RQ2: A two-pass audit of all 443 D1 rules (automated regex checklist plus first-author manual review of 92 FOL-fallback cases) matched zero second-order or higher-order construct patterns. The regex checklist scans for six canonical SOL/HOL signatures: predicate quantification, set quantification, function variables, lambda expressions, recursive/transitive-closure definitions, and meta-policy quantification—all zero matches. The manual review confirmed every one of the 92 fallbacks is expressible in FOL, with failures attributable to LLM engineering limits (long compound sentences, embedded bullets, multiple negations) rather than expressiveness requirements. Given 0 confirmed HOL cases in 443 rules, the exact one-sided Clopper–Pearson upper 95% CI on the true HOL requirement rate is 1 − 0.051/443 ≈ 0.67%. End-to-end pipeline yield is 79.2% (351/443); across the 351 well-formed formulae, M3 semantic predicate quality is 100% after retry-and-backfill. Of the 92 fallback rules, 77.2% fall into four structural categories: compound and/or-joined (28), very long (14), multi-clause complex (14), bulleted compound (10); zero fallbacks attributed to higher-order construct requirements.

RQ3: Shape correctness reported at four coverage tiers: T1 syntactic validity (401/443 = 90.5%, Wilson 95% CI [.87,.93]); T2 structural correctness (351/351 = 100% on FOL-mediated subset); T3 deontic-severity match against gold (330/351 = 94.0%, [.91,.96]); T4 behavioural M4 F1 against Pos/Neg entities (69-shape subset, F1 =.866, [.79,.93], Precision.977, Recall.778). A static ablation isolates vocabulary-grounding effect: exact vocabulary alignment rises from 174/351 (49.6%) with raw LLM-emitted predicates to 304/351 (86.6%) after adapter normalisation (+37.0 percentage points).

Cross-Domain Portability (GDPR): Retargeting PolicyKG to GDPR by swapping the AIT registry for a GDPR-derived registry improves exact property-name alignment from 1/15 (Wilson 95% CI [.01,.30]) without the adapter to 11/15 ([.48,.89]) with it; risk difference +66.7 percentage points (Fisher’s exact p <.001; Cohen’s h = 1.53, large effect). Comparison to RAG: no-grounding 0/15; RAG (top-10 ontology properties via SentenceBERT cosine similarity) 9/15 (60.0%, +60.0 pp, Cohen’s h = 1.77, Fisher p =.0007); Corpus Adapter 11/15 (73.3%, +13.3 pp over RAG, Cohen’s h = 0.28, Fisher p =.70, not statistically significant at N = 15). The honest reading: RAG closes most of the accuracy gap (60 pp of 73 pp total); the Corpus Adapter’s incremental accuracy gain over RAG is small on this sample and cannot be certified at p <.05. The Corpus Adapter’s remaining advantage is in deployment properties: static YAML artefact (no embedding-model dependency, no top-K tuning, byte-level determinism), human-auditable, config-file swap for retargeting.

Component Ablations: Six components separated into three buckets. Structurally necessary: pre-filter (required for throughput; without it, 2-hour budget expires before run completes), may disambiguation filter (disabling admits 56 additional candidates but LLM–regex agreement drops 4.2 pp), NL→SHACL fallback (disabling drops 82 rules from final shape set, 18.5% coverage loss). Empirically marginal: prefilter hints, reclassify second-opinion pass, FOL retry loop (all show swings smaller than natural rule-count variation between runs, ±3 rules, agreement within ±0.5 pp).

Output Stability and External Validation: Repeated pipeline runs with fixed seed and pinned model tags produce SHACL outputs hash-identical to the first run across all 443 shapes. On LexDeMod (D2: 200 lease-contract clauses), the same classifier drops from D1’s Macro F1 =.810 to Macro F1 =.370 (obligation F1 =.569; permission F1 =.038). The failure mechanism: lease contracts express permission as Tenant shall be entitled to... , whose obligation-style shall cue dominates the LLM’s prompted heuristic, causing permissions to be absorbed into the obligation class—a vocabulary-prior mismatch, not an expressiveness or reasoning failure, precisely the class the Corpus Adapter is designed to address.

Discussion Summary:

Three surprising results: (1) The learned baseline beat the zero-shot LLM in-domain (Macro F1 =.863 vs.810) but not out-of-domain (on LexDeMod, LR yields.360 vs LLM’s.370), indicating the LR was learning AIT-specific vocabulary distribution rather than deontic logic; (2) The permission-class collapse on LexDeMod (F1 =.038) was larger than anticipated, caused by the shall be entitled phrase being read as obligation—a vocabulary-prior mismatch; (3) The 0/443 SOL/HOL audit finding was cleaner than expected—even six exceptional rules that looked meta-policy resolved to first-order predicates over an unquantified exception set. Comparison to prior work: Pol…

Based on this paper, here are specific improvements to AI systems and what the improved systems can do:

Improvement: Implement a configuration-driven vocabulary registry that constrains LLM output to predefined ontologies. Replace open-ended predicate generation with constrained selection from a registry containing classes, properties, and synonyms.

Capability: The improved system can generate knowledge graphs with 86.6% exact vocabulary alignment (vs. 49.6% without grounding) and transfer across domains by swapping a YAML config file rather than retraining—demonstrated by improving GDPR property alignment from 1/15 to 11/15 (Fisher’s exact p <.001).

Improvement: Build a four-stage pipeline (extraction → classification → formalization → translation) with per-stage validators that trigger retry loops, rather than single-pass generation.

Improvement: Insert a first-order deontic logic (FOL+deontic) intermediate representation between natural language and executable constraints (SHACL), rather than direct NL→constraint translation.

Improvement: Add a diagnostic component that identifies when LLM performance drops due to vocabulary-prior mismatches (e.g., shall be entitled meaning permission in lease contracts vs. obligation in institutional policy) versus actual reasoning failures.

Improvement: Use rule-driven translation from formal logic to executable constraints (e.g., universal-obligation → NodeShape with sh:minCount 1; prohibition → sh:maxCount 0) instead of LLM-generated constraints.

Improvement: Implement automated regex-based checks for higher-order logic signatures (predicate quantification, lambda expressions, transitive closure) plus manual review of fallback cases.

Improvement: Design the grounding layer as a static, human-auditable YAML artifact with no embedding-model dependency, no top-K tuning, and byte-level determinism.

Improvement: Add explicit disambiguation filters for ambiguous modal cues (e.g., epistemic vs. deontic may) and confidence scoring per classification.

Improvement: Implement a direct NL→SHACL fallback for cases where FOL formalization fails, with automated categorization of failure modes (compound conjunctions, embedded bullets, multi-clause structures).

Improvement: Build the pipeline to retain full provenance: source PDF spans → extracted sentences → FOL formulas → SHACL shapes, with side-by-side inspection.