Observational Policy Ranking for SMB Financial Guidance from Multi-Action Accounting Logs

Summary

This paper introduces an observational policy-ranking framework for providing financial guidance to small and medium-sized businesses (SMBs) using multi-action accounting logs. The authors formulate the task as follows: at a prospective decision boundary, a policy observes information available through month t+0, selects one category from a fixed 34-category catalog, and targets a subsequent financial KPI. Historical categories are "ledger-derived business-change proxies detected during month t+1; outcomes are measured from t+2 through t+12. The action month is excluded from both state and outcome, producing a clean prospective timeline for every company-month decision."

The dataset comprises 85,078 company-month observations from 7,505 firms, with a median of 12 decision months per firm. The policy state includes a narrative embedding (median approximately 23,000 characters) encoded into 768 dimensions using Qwen3Embedding-0.6B, plus a 44-dimensional binary history block. The action catalog consists of 34 categories grouped into ten operational domains including Revenue/Sales, Marketing, Subscriptions, Vendor/Purchasing, Cost Reduction, Payroll/Contractors, Cash Flow/Financing, Operations/Tools, and Accounting/Compliance.

Three properties make the task difficult: "(1) category occurrence is strongly state dependent, creating systematic selection; (2) multiple categories can occur in the same month, complicating focal credit assignment; and (3) observational support varies sharply across the 34 categories, making the policy argmax sensitive to rare-category noise."

The authors introduce Covariate-Adjusted Residual Policy Learning (CAR-PL), an action-wise R-learner that operates directly on multi-hot logs and regularizes selection by observational support. CAR-PL uses five-fold company-grouped cross-fitting to estimate reward and propensity nuisances, then fits one effect head per category via weighted regression with pseudo-outcome and weights based on residuals. A support multiplier δ̂a shr(x) = δ̂a(x) × (na / (na + λsup)) with λsup = 2000 is applied to stabilize selection among unevenly supported categories.

The paper compares CAR-PL against several baselines: an uplift T-Learner (LightGBM regressors), a conservative contextual value model (CQL-style Q-MLP with α=0.01), a zero-shot LLM (Claude Opus 4.6), a modal constant policy, and a frequency-weighted random reference. All policies are evaluated on company-disjoint held-out test firms using a shared model-assisted scoring rule that combines an outcome-model reference contrast with a bounded propensity-weighted residual term.

Key results from Table 4 show KPI-specific rankings:

• Gross Profit: CAR-PL has the highest point estimate (0.084, CI [0.056, 0.110]), followed by T-Learner (0.074). Their paired difference is 0.010 with 95% CI [−0.008, 0.028], so they are not statistically separated.

• Revenue: T-Learner has the highest point estimate (0.085, CI [0.061, 0.110]), followed by CAR-PL (0.079). Paired difference is −0.006 with CI [−0.025, 0.012], again not statistically separated.

• Quick Ratio: Contextual value model has the highest point estimate (0.062, CI [0.019, 0.101]), with positive paired margins over T-Learner and random reference.

CAR-PL has positive paired margins over the contextual value model on Gross Profit (0.037, [0.016, 0.056]) and Revenue (0.020, [0.002, 0.039]), and over random reference on both growth KPIs. The modal constant policy scores below the highest-scoring learned policy on every KPI.

Regarding policy behavior, CAR-PL selects 33–34 categories and produces less concentrated selections across the catalog. Specifically, CAR-PL uses 33–34 categories with top-category shares below 19% on all three KPIs, while the T-Learner is more concentrated (22 categories, top-1 shares 25.2% on Gross Profit and 30.1% on Revenue). The zero-shot LLM assigns 84.2% of its Revenue recommendations to one broad customer-growth category. CAR-PL and T-Learner agree on only 12.8% of Gross Profit states, 14.2% of Revenue states, and 10.7% of Quick Ratio states, showing that comparable aggregate scores can arise from markedly different state-to-category mappings.

Robustness analyses show that outcome-model-only scoring retains the same KPI-level point-estimate leader or top pair, and category rankings remain similar when the all-zero treatment reference is replaced by the most common training co-action pattern (Spearman ρ = 0.884–0.971, with top-ranked category unchanged for every KPI). Propensity diagnostics show macro AUROC 0.819, macro AUPRC 0.423/0.428, ECE 0.036, and treated-side clip rate 4.8%. Residual-correction saturation is 2.6–5.5% on growth KPIs and 6.6–12.9% on Quick Ratio.

The authors conclude that the findings support objective-specific ranking of SMB financial guidance from multi-action accounting logs and position CAR-PL as a practical support-aware method for ranking guidance over large observational action catalogs. Limitations include reliance on observational data, measured covariates, overlap assumptions, and the need for prospective validation before operational deployment.

Improvements to AI Systems:

1. Support-Aware Policy Regularization for Sparse Action Catalogs

• Improvement: Integrate CAR-PL’s support multiplier (δ̂a shr(x) = δ̂a(x) × (na / (na + λsup))) into any multi-action recommendation or decision-making AI that faces imbalanced action frequencies. This prevents over-selection of rare actions due to noisy effect estimates.

• Improved System: An AI that recommends among hundreds of possible actions (e.g., customer support interventions, marketing tactics, or clinical treatments) will now produce more stable and diverse recommendations, avoiding degenerate concentration on a single over-represented action while still leveraging rare but genuinely effective options.

2. Action-Wise R-Learner for Multi-Hot Observational Data

• Improvement: Adopt the covariate-adjusted residual learning framework to handle simultaneous, overlapping actions in historical logs. This decouples the effect of each action from co-occurring actions, enabling clean credit assignment.

• Improved System: An AI that learns from real-world logs where multiple interventions happen together (e.g., a user receives multiple app notifications, a firm takes multiple financial steps) can now estimate the marginal benefit of each action individually, leading to more accurate per-action value predictions and better sequential decision-making.

3. Company-Grouped Cross-Fitting for Nuisance Estimation

• Improvement: Use five-fold company-grouped cross-fitting (rather than random splits) to estimate propensity and reward models. This reduces leakage and overfitting when the same entity appears multiple times.

• Improved System: An AI that personalizes recommendations for repeat users or long-lived clients (e.g., SaaS accounts, patient records) will have more reliable uncertainty estimates and less optimistic performance on new, unseen entities, improving real-world generalization.

4. Narrative Embedding as State Representation

• Improvement: Encode long, unstructured text (e.g., 23,000-character business narratives) into a 768-dimensional embedding (via Qwen3Embedding-0.6B) and combine it with structured binary history. This provides a rich, prospective state representation that excludes the action month.

• Improved System: An AI that must make decisions from free-text logs (e.g., legal case notes, medical histories, or customer support transcripts) can now leverage both semantic context and structured temporal features, enabling more informed and context-aware recommendations.

5. Model-Assisted Scoring with Bounded Propensity-Weighted Residuals

• Improvement: Replace pure outcome-model or pure propensity-based evaluation with a hybrid scoring rule that combines an outcome-model reference contrast with a bounded propensity-weighted residual term. This reduces variance and bias in offline policy evaluation.

• Improved System: An AI that evaluates candidate policies or recommends actions without live A/B testing can now rank policies more reliably, especially when action assignment is non-random and outcome models are imperfect, leading to safer deployment of new strategies.

6. Objective-Specific Policy Ranking

• Improvement: Train and select policies per target KPI (e.g., Gross Profit vs. Revenue vs. Quick Ratio) rather than using a single universal policy. This is enabled by the framework’s ability to produce KPI-specific effect heads.

• Improved System: An AI that serves multiple business or clinical objectives can now dynamically switch its recommendation strategy based on the user’s primary goal (e.g., maximize profit vs. improve liquidity), offering tailored guidance that outperforms a one-size-fits-all approach.

7. Diagnostic-Aware Propensity Calibration

• Improvement: Incorporate propensity diagnostics (macro AUROC 0.819, ECE 0.036, clip rate 4.8%) into the training loop to detect and correct for selection bias in real time.

• Improved System: An AI that continuously learns from observational data can self-monitor its propensity model’s calibration and automatically adjust clipping thresholds or re-weighting, maintaining robustness as data distribution shifts over time.

8. Residual-Correction Saturation Monitoring

• Improvement: Track the residual-correction saturation (2.6–12.9% across KPIs) to decide when additional model complexity yields diminishing returns.

• Improved System: An AI that balances model accuracy against computational cost can automatically halt further training or feature engineering when the marginal gain from residual correction falls below a threshold, saving resources while preserving performance.

What the Improved AI System Can Do:

• Provide stable, diverse, and objective-specific recommendations from large, imbalanced action catalogs using only historical logs.

• Handle multi-action, multi-hot observational data without requiring randomized trials.

• Generalize to new companies, users, or patients by using company-grouped cross-fitting and narrative embeddings.

• Evaluate its own policies offline with low bias and variance, enabling safe deployment without live experimentation.

• Self-calibrate against selection bias and adapt to shifting data distributions.

• Optimize for multiple, potentially conflicting KPIs by switching decision strategies on demand.