Bayesian Evidence Fusion for Capability Tracking-A Simulation Study and Historical CI Pilot

arXiv:2611.14541 · stat.AP · Submitted 2026-10-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Bayesian Evidence Fusion for Capability Tracking-A Simulation Study and Historical CI Pilot".

Jane: Here's the gist of the paper. SaaS capability assessments can become stale between questionnaires while delivery telemetry offers incomplete and correlated evidence The proposed implementation augments that state with an organization-specific survey offset and updates it using the covariance of the available telemetry sources This work examines how a continuous technical capability estimate should respond to biased surveys, dependent telemetry, and missing observations Measurement Model Design The researchers represent three technical capability proxies by a continuous latent state and treat an aggregate questionnaire score as one noisy observation channel The proposed implementation augments that state with an organization-specific survey offset and updates it using the covariance of the available telemetry sources The entities in the model are defined as an organization, week, capability, observation source, and candidate action The fitted transition is a random walk with a slowly changing survey offset defined by equation (1) The stacked observation model is defined by equation (2), which represents a Gaussian model of an aggregate score Filtering and State Recovery The sequential measurement update predicts the augmented state using only prior information for each organization The innovation covariance and gain are calculated using equations (6) to determine the Kalman gain The update uses equation (7) to solve for the augmented state, retaining positive semidefiniteness numerically A specific method is used to select the process variance q from a fixed grid using preintervention validation observations The conditional 90% credible interval is calculated using equation (9) to provide uncertainty estimates Action Screening and Utility The simulator assigns each training organization to a control arm or one of three capability-specific shocks at week 64, with six organizations per arm The expected instantaneous utility is approximated by equation (12) using the logistic risk model and bootstrap effect draws Algorithm 2 implements a budgeted action screening procedure to select at most one action per organization under a common budget The conservative policy substitutes the tenth percentile of these risk reductions, resulting in regret that is 0.293 for the conservative policy Simulation Results and Limitations Across ten simulated cohorts, reference latent error was 0.1261, compared with 0.1452 for tuned moving averages The full model’s 90.3% state coverage is close to its 90% nominal level in this reference setting Incident prediction gives a more restrained result, with Twin’s Brier score being 0.2003 The main limitation is the absence of validated longitudinal SaaS capability measurements and real intervention outcomes The study does not establish that their combination alone meets the novelty threshold of a Transactions paper Historical Pilot Findings The historical CI pilot tests whether a simple project-disjoint, forward-time CI workflow can be executed on real records Logistic regression achieves a Brier score of 0.1289, close to random forests The pilot supports a reproducible data/availability workflow, but it does not favor a complex latent model and does not test the full Twin estimator The main limitation is the absence of validated longitudinal SaaS capability measurements and real intervention outcomes The present package supports reproducible investigation, while remaining a research draft requiring field evidence External Validity Concerns The principal limitation is that capability truth is simulated, as the three labels, unit loadings, survey offset law, intervention shocks, and incident mechanism were chosen by the study designer The estimator also receives a separate calibrated noise sample that must be matched by instrument studies before field use The model does not assess comprehensive SaaS business maturity, tenancy isolation, customer value, security effectiveness, or monetary savings At least three partner organizations would provide a starting point for a longitudinal feasibility study The literature audit includes established continuous assessment and Bayesian delivery systems A separate computational review found and corrected an interval-rank error and a policy-horizon mismatch before the retained runs The package is a reproducible research draft References The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods

1: S.

Tom: Aleem, R.

Jane: Batool, S.

Tom: Alkobaisi, F.

Paper summary: Jane: So looking at the whole study, "Bayesian Evidence Fusion for Capability Tracking—A Simulation Study and Historical CI Pilot," the authors are essentially showing how to fuse imperfect survey data with telemetry using a Gaussian state model to track capability over time.

Tom: The paper claims this combination helps filter capability proxies and survey offsets together, and it uses source covariance to handle correlated observations, which they found leads to a reference latent root mean squared error of zero point one two six one in their simulation setup <ref:2611.14541#pg1,reference latent root mean squared error>.

Lu: The main thing is that they test this framework across ten simulated cohorts spanning various conditions like missingness and noise, showing how the state recovery works under those challenging scenarios.

Meng: What this means for deployment is that you can get a more robust estimate of capability even when your standard data sources are messy or don't align perfectly with the surveys.

Lalam: It also shows that they can design budgeted action screening procedures to select at most one action per organization under a common budget, and the conservative policy results in a regret of zero point two nine three.

Jane: The paper's title suggests it’s about fusing different types of evidence—the survey and the telemetry—to get a better picture of how an organization’s technical capability is actually evolving.

Tom: And even though they have to be careful not to claim their synthetic state maps perfectly to real engineering capability, the simulation study itself provides observable latent recovery and policy regret.

Lu: The historical pilot with the TravisTorrent data offers a separate check on whether this can actually run on real records, even if it doesn't validate the underlying constructs of SaaS maturity.

Meng: So for someone just listening to this, it’s about taking messy, incomplete signals and using a Bayesian framework to try and track progress in an engineering process without getting totally lost in the noise of inconsistent data.

Conclusion: Tom: So we’ve been looking at how they fused those survey scores with telemetry data using this Gaussian state model. Now we need to talk about what that actually means for us, right?

Jane: Right. The title is "Bayesian Evidence Fusion for Capability Tracking—A Simulation Study and Historical CI Pilot." It sounds a bit heavy, but at its core, they’re trying to track a company's technical capability even when their measurement tools are inconsistent.

Lu: They use the idea that you have two noisy ways of looking at the same thing—the survey score and the live telemetry—and they combine them using covariance to get a better, single estimate of the true state.

Meng: From my side, I’m thinking about deployment. If this model works on simulated data, how sure are we that it won't just fail when we hit real production traffic? What breaks at scale?

Tom: That’s the million-dollar question for any engineer. The paper shows their simulation results are pretty close to their own error estimates, which is a good sign that the mechanics of the model work as intended in a controlled setting.

Jane: It’s about getting a better picture of reality when your data sources are messy. It moves away from just trusting one survey or one piece of telemetry alone.

Lu: Exactly. They aren't claiming this fixes everything, but they show that using this fused approach can give you more stable estimates for capability over time than just looking at the raw numbers separately.

Meng: But what about the real world? The paper admits their measurements are simulated, and they haven't shown what happens when a company actually implements these actions in a live environment with real business outcomes. That’s a big gap for me.

Tom: It is a gap. And that leads right into whether this kind of complex estimation is worth the effort when you don't have those long-term validation studies yet, or if we need to build more robust ways to test it first on actual production data streams.

stat.AP

Submitted: 2026-10-19

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: SaaS capability assessments can become stale between questionnaires while delivery telemetry offers incomplete and correlated evidence The proposed implementation augments that state with an…

Key concepts

Latent State
This is the underlying, unobservable technical capability of an organization. Researchers model this as a continuous variable that changes slowly over time. The goal is to estimate this hidden state accurately using both survey scores and real-time data.
Survey Offset
Since surveys are noisy and biased, this component adjusts the latent state estimate based on the specific organization's unique survey characteristics. It accounts for systematic errors in the questionnaire, helping to correct the raw survey score.
Kalman Gain
This is a mathematical tool used in sequential measurement updates to determine how much weight to give new information (telemetry) versus the existing estimate (the state). It helps filter out noise and optimally combine different data sources for better state recovery.
Action Screening
This involves simulating decisions under uncertainty. The model tests which actions (e.g., training interventions) are best to take at specific times based on the predicted capability trajectory, aiming to maximize expected utility while managing risk.

Terminology

Summary

SaaS capability assessments can become stale between questionnaires while delivery telemetry offers incomplete and correlated evidence The proposed implementation augments that state with an organization-specific survey offset and updates it using the covariance of the available telemetry sources This work examines how a continuous technical capability estimate should respond to biased surveys, dependent telemetry, and missing observations

Measurement Model Design

The researchers represent three technical capability proxies by a continuous latent state and treat an aggregate questionnaire score as one noisy observation channel The proposed implementation augments that state with an organization-specific survey offset and updates it using the covariance of the available telemetry sources The entities in the model are defined as an organization, week, capability, observation source, and candidate action The fitted transition is a random walk with a slowly changing survey offset defined by equation (1) The stacked observation model is defined by equation (2), which represents a Gaussian model of an aggregate score

Filtering and State Recovery

The sequential measurement update predicts the augmented state using only prior information for each organization The innovation covariance and gain are calculated using equations (6) to determine the Kalman gain The update uses equation (7) to solve for the augmented state, retaining positive semidefiniteness numerically A specific method is used to select the process variance q from a fixed grid using preintervention validation observations The conditional 90% credible interval is calculated using equation (9) to provide uncertainty estimates

Action Screening and Utility

The simulator assigns each training organization to a control arm or one of three capability-specific shocks at week 64, with six organizations per arm The expected instantaneous utility is approximated by equation (12) using the logistic risk model and bootstrap effect draws Algorithm 2 implements a budgeted action screening procedure to select at most one action per organization under a common budget The conservative policy substitutes the tenth percentile of these risk reductions, resulting in regret that is 0.293 for the conservative policy

Simulation Results and Limitations

Across ten simulated cohorts, reference latent error was 0.1261, compared with 0. p. 0

Where the paper says it

Reference latent root mean squared error is 0. 1261, compared with 0.

1452 for tuned moving averages The full model’s 90. p. 0
Where the paper says it

1261, compared with 0. 1452 for a tuned moving average; nominal ninety percent intervals cover 90.

3% state coverage is close to its 90% nominal level in this reference setting Incident prediction gives a more restrained result, with Twin’s Brier score being 0. p. 6
Where the paper says it

Incident prediction gives a more restrained result. Twin’s Brier score is 0.

2003 The main limitation is the absence of validated longitudinal SaaS capability measurements and real intervention outcomes The study does not establish that their combination alone meets the novelty threshold of a Transactions paper

Historical Pilot Findings

The historical CI pilot tests whether a simple project-disjoint, forward-time CI workflow can be executed on real records Logistic regression achieves a Brier score of 0.1289, close to random forests The pilot supports a reproducible data/availability workflow, but it does not favor a complex latent model and does not test the full Twin estimator The main limitation is the absence of validated longitudinal SaaS capability measurements and real intervention outcomes The present package supports reproducible investigation, while remaining a research draft requiring field evidence

External Validity Concerns

The principal limitation is that capability truth is simulated, as the three labels, unit loadings, survey offset law, intervention shocks, and incident mechanism were chosen by the study designer The estimator also receives a separate calibrated noise sample that must be matched by instrument studies before field use The model does not assess comprehensive SaaS business maturity, tenancy isolation, customer value, security effectiveness, or monetary savings At least three partner organizations would provide a starting point for a longitudinal feasibility study The literature audit includes established continuous assessment and Bayesian delivery systems A separate computational review found and corrected an interval-rank error and a policy-horizon mismatch before the retained runs The package is a reproducible research draft

References

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [1] S. Aleem, R. Batool, S. Alkobaisi, F. Ahmed, and A. Masood Khattak “SaaS application maturity assessment model” IEEE Access vol 12 pp 128 305–128 325, 2024 [Online] Available: https://doi. p. 0

Where the paper says it

A maturity label alone does not supply that distinction. The SaaS Application Maturity Assessment Model organizes questionnaire-based assessment around design, architecture, business performance, and the overall organization [1], [2].

org/10.1109/ACCESS. p. 1
Where the paper says it

This distinction follows the broader measurement-validity problem discussed by Borsboom et al. Data access sets the evidential scope.

2024.3455937

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [2] S. Alkobaisi, M. F. Safdar, P. Pałka, and N. A. Abu Ali “Artificial intelligence algorithms in asthma management: a review of data engineering, predictive models, and future implications” Applied Sciences vol 15 no 7 p 3609, 2025 [Online] Available: https://doi.org/10.1148/Applied Sciences.3609

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [3] O. H. Plant, J. van Hillegersberg, and A. Aldea “Design and validation of a capability measurement instrument for DevOps teams: A participatory action research approach” in Agile Processes in Software Engineering and Extreme Programming ser Lecture Notes in Business Information Processing vol 445 Springer 2022 pp 151–167 [Online] Available: https://doi.org/10.1007/978-3-031-08169-9 10

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [4] J. A. V. M. K Jayakody and W. p. 3

Where the paper says it

Study Setting or evidence Established focus Remaining requirement for this study Aleem et al. [1], 2024 SaaS assessment model Questionnaire, dimensions, rating Joint dynamic evidence and independent outcome validity Jayakody et al.

M. J I Wijayanayake “DevOps maturity; a systematic literature review” in 2024 International Research Conference on Smart Computing and Systems Engineering SCSE 2024 pp 1–6 [Online] Available: https://doi. p. 3
Where the paper says it

[1], 2024 SaaS assessment model Questionnaire, dimensions, rating Joint dynamic evidence and independent outcome validity Jayakody et al. [4], 2024 Literature review DevOps maturity review Primary-content details not fully accessible here Plant et al.

org/10.1109/SCSE61872.2024.10550493

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [5] C. Marnewick and J. p. 1

Where the paper says it

Amaro et al. [7] further connect capabilities to outcome metrics, while Marnewick and Langerman [5] question the assumption that maturity progression alone secures performance.

Langerman “DevOps and organizational performance The fallacy of chasing maturity” IEEE Software vol 38 no 5 pp 48–55, 2021 [Online] Available: https://doi. p. 3
Where the paper says it

[19], 2025 Public-sector DevOps Context–capability coalignment Cross-context longitudinal validation Marnewick et al. [5], 2021 Organizational performance Limits of maturity pursuit Causal outcome evidence for an intervention Amaro et al.

org/10.1109/MS.2020.3023298

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [6] C. Stoiber, M. p. 1

Where the paper says it

Pooling organizations without accounting for such differences can conceal rather than resolve measurement problems. Stoiber et al.

Stöter, L. Englbrecht, S. Schönig, and B. Häckel “Keeping your maturity assessment alive: A method for the continuous tracking and assessment of organizational capabilities and maturity” Business & Information Systems Engineering vol 65 no 6 pp 703–721, 2023 [Online] Available: https://doi. p. 0
Where the paper says it

Continuous assessment itself is established prior work. The Continuous Maturity Assessment Method describes iterative tracking of organizational capabilities and has been evaluated in organizational settings [6].

org/10.1007/s12599-023-00805-y

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [7] R. Amaro, R. Pereira, and M. M da Silva “Capabilities and metrics in DevOps A design science study” Information & Management vol 60 no 5 p 103809, 2023 [Online] Available: https://doi. p. 3

Where the paper says it

[8], 2023 Multivocal review Capabilities and practices Calibrated observation-tocapability mapping Amaro et al. [7], 2023 Design science and interviews Capability–metric relationships Dynamic bias-aware measurement Amaro et al.

org/10.1016/j.im.2023.103809

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [8] —— “Capabilities and practices in DevOps A multivocal literature review” IEEE Transactions on Software Engineering vol 49 no 2 pp 883–901, 2023 [Online] Available: https://ieeexplore.ieee.org/document/9756241/

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [9] R. Amaro, R. Pereira, and M Mira da Silva “DevOps metrics and KPIs A multivocal literature review” ACM Computing Surveys vol 56 no 9 pp 1–41, 2024 article 231 [Online] Available: https://doi. p. 3

Where the paper says it

[1], 2024 SaaS assessment model Questionnaire, dimensions, rating Joint dynamic evidence and independent outcome validity Jayakody et al. [4], 2024 Literature review DevOps maturity review Primary-content details not fully accessible here Plant et al.

org/10.1145/3652508

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [10] B. Wilkes, A. M. P Milani, and M.-A Storey “A framework for automating the measurement of DevOps research and assessment DORA metrics” in 2023 IEEE International Conference on Software Maintenance and Evolution ICSME 2023 pp 62–72 [Online] Available: https://doi. p. 3

Where the paper says it

[9], 2024 Multivocal metric review Metrics and KPI definitions Target-setting measurement invariance Stoiber et al. [6], 2023 Organizational evaluations Continuous maturity assessment Explicit probabilistic source disagreement Wilkes et al.

org/10.1109/ICSME58846.2023.00018

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [11] B. Yu, S. H. Kim, S. Alkobaisi, W. D Bae, and T Bailey “The tornado model: Uncertainty model for continuously changing data” in International Conference on Database Systems for Advanced Applications Springer 2007 pp 624–636

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [15] D. Borsboom, G. p. 1

Where the paper says it

RESEARCH PAPER 2 is an assumption: it does not establish that a unit of the synthetic state corresponds to an empirically meaningful change in engineering capability. This distinction follows the broader measurement-validity problem discussed by Borsboom et al.

J. Mellenbergh, and J van Heerden “The Concept of Validity” Psychological Review vol 111 no 4 pp 1061–1071, 2004 [Online] Available: https://doi. p. 6
Where the paper says it

Training and validation labels are purged when their completion proxy falls within 24 hours of the next cutoff, preventing boundary-label leakage. Held-out projects retain their own previously available histories, which is appropriate for online prediction but is not a cold-history experiment.

org/10.3778/0033-295x.111.4.no6

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [30] B. Callaway and P. H Sant’Anna “Difference-in-Differences with multiple time periods” Journal of Econometrics vol 225 no 2 pp 200–230, 2021 [Online] Available: https://doi.org/10.1016/j.jeconom.2020.12.89

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [31] L. Sun and S. Abraham “Estimating dynamic treatment effects in event studies with heterogeneous treatment effects” Journal of Econometrics vol 225 no 2 pp 175–199, 2021 [Online] Available: https://doi. p. 2

Where the paper says it

For eventual field interventions, panel methods offer more defensible estimands than an unqualified before/after difference. Group-time treatment effects [30], heterogeneous event studies [31], robust imputation estimators [32], and sensitivity to parallel-trend violations [33] address distinct design problems.

org/10.1016/j.jeconom.2020.09.8

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [32] K. Borusyak, X. Jaravel, and J Spiess “Revisiting Event-Study Designs Robust and Efficient Estimation” Review of Economic Studies vol 91 no 6 pp 3253–3285, 2024 [Online] Available: https://doi. p. 2

Where the paper says it

For eventual field interventions, panel methods offer more defensible estimands than an unqualified before/after difference. Group-time treatment effects [30], heterogeneous event studies [31], robust imputation estimators [32], and sensitivity to parallel-trend violations [33] address distinct design problems.

org/10.1093/restud/rdae007

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [33] A. Rambachan and J Roth “A More Credible Approach to Parallel Trends” Review of Economic Studies vol 90 no 5 pp 2555–2591, 2023 [Online] Available: https://doi.org/10.1093/restud/rdad018

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [24] I. Gibbs and E Candes “Adaptive conformal inference under distribution shift” in Advances in Neural Information Processing Systems M Ranzato A Beygelzimer Y Dauphin P Liang J Vaughan Eds vol 34 Curran Associates Inc 2021 pp 1660–1672 [Online] Available: https://proceedings.neurips.cc/paper files/paper/2021/file/0d441de75945e5acbc865406fc9a2559-Paper. p. 0

Where the paper says it

RESEARCH PAPER 1 Bayesian Evidence Fusion for Capability Tracking—A Simulation Study and Historical CI Pilot Taimoor Ahmad Abstract—SaaS capability assessments can become stale between questionnaires, while delivery telemetry offers incomplete and correlated evidence. Continuous assessment and Bayesian delivery analysis already exist, leaving uncertainty about how biased surveys and missing observations should influence capability estimates.

pdf

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [25] A. N Angelopoulos, E. J Candès, and R J Tibshirani “Conformal PID Control for Time Series Prediction” in Advances in Neural Information Processing Systems vol 36 Curran Associates Inc 2023 pp 23 047–23 074 [Online] Available: https://doi. p. 2

Where the paper says it

ACI [24] and conformal PID control [25] supply recent sequential observation-interval comparators. The later online conformal framework [26] further highlights that adaptation rules and local guarantees differ.

org/10.52202/075280-1000

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [26] I. Gibbs and E J Candès “Conformal inference for online prediction with arbitrary distribution shifts” Journal of Machine Learning Research vol 25 no 162 pp 1–36, 2024 [Online] Available: http://jmlr. p. 6

Where the paper says it

Training and validation labels are purged when their completion proxy falls within 24 hours of the next cutoff, preventing boundary-label leakage. Held-out projects retain their own previously available histories, which is appropriate for online prediction but is not a cold-history experiment.

org/papers/v25/22-1218.html

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [27] D. B Rubin “Inference and missing data” Biometrika vol 63 no 3 pp 581–592, 1976 [Online] Available: https://doi. p. 6

Where the paper says it

Training and validation labels are purged when their completion proxy falls within 24 hours of the next cutoff, preventing boundary-label leakage. Held-out projects retain their own previously available histories, which is appropriate for online prediction but is not a cold-history experiment.

org/10.1093/biomet/63.3.581

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [28] W. Miao, P Ding, and Z Geng “Identifiability of Normal and Normal Mixture Models with Nonignorable Missing Data” Journal of the American Statistical Association vol 111 no 516 pp 1673–1683, 2016 [Online] Available: https://doi. p. 2

Where the paper says it

Ignoring a missing observation is justified only under assumptions on the observation process [27]. Informative telemetry loss can violate those assumptions, and even parametric nonignorable missing-data models have identification limitations [28].

org/10.8080/0162-Astat.2024.49

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [29] R. F Barber, E J Candès, A Ramdas, and R J Tibshirani “Conformal prediction beyond exchangeability” The Annals of Statistics vol 51 no 2 pp 816–845, 2023 [Online] Available: https://doi. p. 6

Where the paper says it

Training and validation labels are purged when their completion proxy falls within 24 hours of the next cutoff, preventing boundary-label leakage. Held-out projects retain their own previously available histories, which is appropriate for online prediction but is not a cold-history experiment.

org/10.1214/23-aos2276

The references listed support the study's context and related work on capability assessment, telemetry, and Bayesian methods [34] S. Athey and S Wager “Policy Learning With Observational Data” Econometrica vol 89 no 1 pp 133–161, 2021 [Online] Available: https://doi. p. 2

Where the paper says it

Group-time treatment effects [30], heterogeneous event studies [31], robust imputation estimators [32], and sensitivity to parallel-trend violations [33] address distinct design problems. Observational policy learning [34] requires appropriate identification and overlap, while augmented synthetic controls [35] impose assumptions about untreated outcomes.

org/10.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Create an anchored Gaussian state model that jointly filters capability proxies and persistent survey offsets to improve latent recovery in continuous assessment systems. This system can distinguish between changes in engineering capability and changes in measurement by treating a biased survey as an unbiased observation channel when properly modeled with a persistent offset.

  2. Implement covariance aware updates where the estimation of the augmented state is updated using the covariance of available telemetry sources, preventing correlated observations from being counted independently. This allows for more accurate assessment when evidence sources overlap or disappear during difficult periods.

  3. Develop a decision procedure that estimates action effects from randomized synthetic cohorts to screen interventions under a common budget. This screening procedure can select at most one action per organization by approximating expected instantaneous utility using state draws and estimated shock values, providing an inspectable uncertainty-aware integration and evaluation boundary.

  4. Use adaptive conformal inference or adapted conformal PID controllers when evaluating observed telemetry to manage calibration limitations exposed by noise shifts. This allows the system to adjust its prediction intervals based on observation conditions, such as increased noise, rather than relying on fixed coverage levels which may become unreliable during distribution shifts.

Abstract

SaaS capability assessments can become stale between questionnaires, while delivery telemetry offers incomplete and correlated evidence. Continuous assessment and Bayesian delivery analysis already exist, leaving uncertainty about how biased surveys and missing observations should influence capability estimates.We implement SAAS-MATURITY-TWIN, an anchored Gaussian state model that jointly filters capability proxies and persistent survey offsets. Source covariance prevents correlated observations from being counted independently. Conditional uncertainty supports forecasting and budgeted intervention screening, with action effects estimated from randomized synthetic cohorts. Ten seeds generate forty paired worlds spanning reference observations, extensive missingness, informative missingness, and increased noise. Reference latent root mean squared error is 0.1261, compared with 0.1452 for a tuned moving average; nominal ninety percent intervals cover 90.3 percent. Improved state recovery produces only small incident forecasting differences. Adaptive conformal inference and an adapted conformal PID controller are evaluated separately for observed telemetry, where noise shifts expose calibration limitations. A licensed TravisTorrent pilot evaluates build prediction on 13 074 held out builds from 224 projects. Logistic regression achieves a Brier score of 0.1289, close to random forests. These results support reproducible methodological testing, but do not establish real organizational maturity, intervention effectiveness, or production readiness. Independent calibration and longitudinal validation remain necessary.

Related papers