From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems
cs.AI, cs.DC, cs.LG, cs.SI, q-fin.CP
Submitted: 2026-05-11
Updated: 2026-08-26
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Deploying machine learning in regulated financial environments -- credit risk, fraud detection, and anti-money laundering -- exposes critical vulnerabilities in algorithmic reproducibility.
Terminology
Abstract
Deploying machine learning in regulated financial environments -- credit risk, fraud detection, and anti-money laundering -- exposes critical vulnerabilities in algorithmic reproducibility. While early financial ML addressed statistical challenges such as backtest overfitting, deep neural networks and Generative AI have introduced mechanical nondeterminism rooted in hardware and architecture. This survey provides a systems perspective on reproducibility failures across three modalities now dominant in financial AI: tabular models (post-hoc explanation variance), graph networks (stochastic sampling and temporal asynchrony), and LLM-based agentic workflows (batch-dependent divergence and trajectory drift). We supplement the literature analysis with first-party experiments on public financial datasets -- quantifying explanation rank instability in credit scoring, prediction flip rates in GNN-based fraud detection, and tensor-parallel-induced output divergence in LLM entity extraction. We propose a layered evaluation framework linking modality-specific metrics (RBO, D cos, TDI, PSD) to audit readiness, and report where these measures overlap rather than complement one another.
Sources
- On the Reproducibility of Neural Network Predictions
- Graph Neural Networks for Financial Fraud Detection: A Review
- Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
- Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
- LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
- Pitfalls of Graph Neural Network Evaluation
- Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models
- LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows
- Efficient Guided Generation for Large Language Models
- SHAP Stability in Credit Risk Management: A Case Study in Credit Card Default Model
- Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch
- STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability
- AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems
- Network Analytics for Anti-Money Laundering -- A Systematic Literature Review and Experimental Evaluation
- Normalisation and Initialisation Strategies for Graph Neural Networks in Blockchain Anomaly Detection
- JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
- Estimating LLM Uncertainty with Evidence
- Towards Modeling Data Quality and Machine Learning Model Performance
- SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems