MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

arXiv:2605.22949 · cs.LG, cs.MA · Submitted 2026-05-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination".

Jane: The gist The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So, what does MARGIN actually do in practice? It’s about making the coordinator interpret that raw confidence score differently based on how reliable that specific model has been recently.

Tom: Essentially, instead of trusting the reported confidence directly, you get a corrected score i,t which is clipped between zero and one using that effective calibration factor gamma eff.

Lu: The objective they are optimizing is this: given a corrected confidence p, the probability of getting the correct outcome o i,t should be approximately equal to p, which they write as Pr(o i,t = one i,t = p) about p (reference: <ref:2605.22949#pg3>).

Meng: So it's not about making the model predict better; it’s about making the decision-making process for coordinating agents more robust when those agents are unreliable or shifting their behavior.

Lalam: It’s a coordinator responder setting where each agent answers independently, and MARGIN changes only how the coordinator interprets that agent's confidence, not its response policy itself (reference: <ref:2605.22949#pg3>).

Jane: This is important because it means you can use models with different inherent calibration properties—some might be better at certain types of answers than others—and still get a more reliable collective decision.

Tom: And they tested this across code generation, question answering, and mathematics using an eighteen-model pool <ref:2605.22949#pg1,code generation, question answering, and mathematics using an 18-model pool>. It achieved lower post-shift expected calibration error, or ECE, compared to five other online baselines in two code generation transitions.

Lu: They also showed this adaptation works when the task distribution shifts—for example, in a question answering transition involving changes in subject and answer format (reference: <ref:2605.22949#pg3>).

Meng: But what about the cost? If you have to maintain these running estimates for every model band, how much overhead is that on the inference side?

The paper's summary: Tom: The main improvement they point to is that this runtime calibration method learns those model-specific confidence corrections from observed answer outcomes without retraining the models or needing a held-out calibration set.

Jane: That’s a big deal because traditional methods, like temperature scaling or histogram binning, require you to have that separate calibration data upfront.

Lu: MARGIN updates online from subsequent outcomes, which is what allows it to handle changing workloads better than those fixed post-hoc methods that can deteriorate under distribution shift (reference: <ref:2605.22949#pg2>).

Meng: So the ability to adapt calibration learned on an earlier workload to a changed task distribution is key here, and they showed it works for code generation transitions, and even for question answering shifts.

Lalam: They specifically showed that this correction improves the ranking of correct responses on three code generation benchmarks and boosts answer-selection accuracy by four point three and fourteen point zero percentage points on two of those benchmarks relative to uncalibrated confidence weighting (reference: <ref:2605.22949#pg1>).

Tom: And they also found that when you use this corrected confidence, the coordinator’s decisions are better, with pairwise resolution increasing by thirty-five point one three on HumanEval and thirty-one point three one on MBPP (reference: <ref:2605.22949#pg1>).

Jane: It means for someone just listening to the show, this translates to a system that is more reliable when different AI agents are working together, especially when those agents might be acting differently than they were trained on.

Lu: They also diagnosed a confidence inversion issue, showing that for some models on BigCodeBench, the model-mean confidence is negatively related to accuracy (reference: <ref:2605.22949#pg1>).

The paper's improvements: Tom: So to wrap up, MARGIN is a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set.

Jane: It’s useful because it lets the coordinator produce corrected scores to weight candidate answers in a collective decision, addressing the problem where self-reported confidence might mean different things across responders.

Lu: The adaptation under distribution shift is what I find most ambitious—being able to update those estimates as new evidence arrives, rather than just relying on a single calibration point from before the shift.

Meng: From an engineering standpoint, it works with correctness feedback for every response after scoring, but they also flag a limitation: the calibration history is only useful insofar as it remains relevant to the current workload.

Lalam: And they did say that generalization is limited by the archived cohort and abrupt workload transitions.

Tom: So we see that this adaptive correction helps reduce post-shift expected calibration error, particularly in coding transitions, but it’s an online system that needs to stay relevant to the current task distribution.

Jane: It suggests a path forward where coordination systems can dynamically adjust their reliance on individual model confidence based on real-time performance feedback.

Lu: I think the paper establishes a solid framework for this online, adaptive approach in multi-agent settings.

Meng: It’s an interesting direction, but we still need to figure out how to deploy this adaptation efficiently without adding too much inference time.

Lalam: The core idea of MARGIN is learning model-specific corrections through observed outcomes while keeping the models themselves static.

Conclusion: Tom: So MARGIN is about runtime confidence calibration for multi-agent foundation model coordination, learning those corrections from observed outcomes without retraining or a held-out calibration set.

Jane: That’s the main idea: you get a better sense of reliability for each AI agent when they are coordinating something complex, like code generation.

Lu: The method partitions confidence into bands and maintains running estimates for accuracy and mean confidence within those bands, using an exponentially weighted moving average to update them.

Meng: From a production standpoint, it means we can adjust the reported confidence scores in real-time based on what actually happens during the coordination process.

Lalam: It uses a per-band correction, which is just the ratio of observed accuracy to stated confidence, gamma i,k equals empirical accuracy divided by mean confidence.

Tom: And they blend those sparse band corrections toward a model-level estimate using an effective calibration factor gamma eff, which makes the final corrected confidence i,t clipped between zero and one.

Jane: The results are pretty strong. They saw lower post-shift expected calibration error, or ECE, than five other online baselines in two code generation transitions alone.

Lu: They also showed adaptation under distribution shift; the correction worked even when the task distribution changed, for instance in question answering transitions involving changes in subject and answer format.

Meng: I'm interested that it works across those different types of tasks—code, QA, math—using a single model pool of eighteen agents.

Tom: And they showed that this corrected confidence actually improves the ranking of correct responses on three code generation benchmarks and boosts answer-selection accuracy by four point three and fourteen point zero percentage points on two of those benchmarks relative to uncalibrated confidence weighting.

Jane: That means for a user, the AI agent coordinating their work is making better choices because they are using that corrected score instead of just the raw number they reported.

Lalam: Plus, they diagnosed a confidence inversion issue where model-mean confidence was negatively associated with accuracy on BigCodeBench.

Meng: What I’m watching is how much overhead this running update adds to the total inference cost when you scale it up to production.

Tom: It's an online system that needs to stay relevant to the current workload, and the study flags that generalization is limited by archived cohorts and abrupt workload transitions.

Lu: So they didn't claim it solves everything; it’s a correction tool for dynamic environments, not a universal calibration fix.

Jane: Anyway, MARGIN is a runtime confidence calibration for multi-agent foundation model coordination. It’s showing us how to make these complex AI teams more reliable when things change on the fly.

Tom: It's definitely something to keep an eye on as we build out more sophisticated coordination systems.

Joss Armstrong

Ericsson · Athlone

cs.LG, cs.MA

Submitted: 2026-05-21

Updated: 2026-10-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 81/100

The gist: The gist The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out

Key concepts

Confidence Bands: The method divides the entire range of possible model outputs (from 0 to 1) into several distinct intervals, or bands. Each band represents a specific confidence level where the model's performance is being tracked separately. This partitioning allows MARGIN to maintain unique calibration factors for each combination of model and band.
Empirical Accuracy Rate ($\hat{a}_{i,k}$): This is a running estimate that tracks how often a specific model ($a_i$) performs correctly when its stated confidence falls within a particular confidence band ($B_k$). It is updated using an exponentially weighted moving average (EWMA), meaning recent performance has more influence on the current rate than older data.
Per-Band Correction ($\gamma_{i,k}$): This correction factor is calculated as the ratio of the observed empirical accuracy rate to the mean confidence expressed by the model within that band. It essentially measures how much better or worse a model's stated confidence aligns with its actual correctness for that specific confidence range.
Effective Calibration Factor ($\gamma_{eff}$): This factor blends individual, sparse band corrections toward a single, overall model estimate. It uses a weighted average between the per-band correction and the general model calibration to produce a final, robust correction applied to the model's confidence at runtime.

Terminology

Calibration — 7× in this paper · explained in 7 episodes
Confidence Calibration — 2× in this paper · explained in 2 episodes
Foundation Model — 2× in this paper · explained in 3 episodes
Adaptation — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 9 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Online Adaptation — 1× in this paper · explained in 3 episodes

Summary

How it works

MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate The construction uses only model outputs, stated confidence, and subsequent correctness feedback without model internals, retraining, or a separate calibration dataset

The method partitions [0, 1] into K confidence bands and maintains a separate calibration factor for each (model, band) pair
For each model ai and band k, it maintains two running estimates: the empirical accuracy rate aˆi,k when its confidence falls in band Bk and the mean confidence c¯i,k expressed by model ai within band Bk
These estimates are updated using an exponentially weighted moving average (EWMA) with a constant learning rate α
The per-band correction is defined as the ratio of observed accuracy to stated confidence: γi,k = aˆi,k/c¯i,k
Sparse (model, band) pairs are blended toward a model-level estimate using the effective calibration factor γeff i,k = ni,k / (ni,k + ks) γi,· + ks / (ni,k + ks)
The corrected confidence for model ai at time t is then calculated as c˜i,t = clip(γeff i,κ(ci,t) ci,t, 0, 1)

Evaluation and Results

The evaluation covers code generation, question answering, and mathematics using an 18-model pool MARGIN achieves lower postshift expected calibration error (ECE) than all five online baselines in two code-generation transitions and than four in a question-answering transition In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting

Coordination and Answer Selection

MARGIN improves the ranking of correct responses on the three evaluated code-generation benchmarks and increases answer-selection accuracy on two Separate cold-start experiments assess whether corrected confidence improves the coordinator’s decisions, showing lower ECE on all three benchmarks Pairwise resolution increases by 35.13, 31.31, and 20.99 percentage points on HumanEval, MBPP, and BigCodeBench respectively

Adaptation and Sensitivity

The central comparison concerns calibration learned on an earlier workload; MARGIN has lower post-shift ECE than all five baselines in the coding transitions The observed adaptation result extends to a question-answering benchmark transition involving changes in subject and answer format, alongside the two coding-source transitions Parameter sensitivity analysis shows that α = 0.08 improves on 0.04 for the HE→BCB shift values but worsens the same-task hardening values

Limitations

The study concludes that calibration history is useful only insofar as it remains relevant to the workload, and MARGIN’s lower post-shift ECE against the tested online baselines in the coding transitions supports its use as an adaptive correction in these settings Generalisation is limited by the archived cohort, retained task support, and abrupt workload transitions The study does not establish that an ensemble is preferable to deploying a known strongest model at lower inference cost Correctness feedback is the main operational condition for the primary experiments

The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set. The gist The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set

The method partitions [0, 1] into K confidence bands and maintains a separate calibration factor for each (model, band) pair
For each model ai and band k, it maintains two running estimates: the empirical accuracy rate aˆi,k when its confidence falls in band Bk and the mean confidence c¯i,k expressed by model ai within band Bk
These estimates are updated using an exponentially weighted moving average (EWMA) with a constant learning rate α
The per-band correction is defined as the ratio of observed accuracy to stated confidence: γi,k = aˆi,k/c¯i,k
Sparse (model, band) pairs are blended toward a model-level estimate using the effective calibration factor γeff i,k = ni,k / (ni,k + ks) γi· + ks / (ni,k + ks)
The corrected confidence for model ai at time t is then calculated as c˜i,t = clip(γeff i,κ(ci,t) ci,t, 0, 1)

Adaptation and Sensitivity

The central comparison concerns calibration learned on an earlier workload; MARGIN has lower post-shift ECE than all five online baselines in the coding transitions The observed adaptation result extends to a question-answering benchmark transition involving changes in subject and answer format, alongside the two coding-source transitions Parameter sensitivity analysis shows that α = 0.08 improves on 0.04 for the HE→BCB shift values but worsens the same-task hardening values

The gist The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set. The gist The MARGIN method presents a runtime calibration technique that learns model-specific confidence corrections from observed answer outcomes without retraining models or requiring a held-out calibration set

The method partitions [0, 1] into K confidence bands and maintains a separate calibration factor for each (model, band) pair
For each model ai and band k, it maintains two running estimates: the empirical accuracy rate aˆi,k when its confidence falls in band Bk and the mean confidence c¯i,k expressed by model ai within band Bk
These estimates are updated using an exponentially weighted moving average (EWMA) with a constant learning rate α

Improvements for AI systems

Confidence-Band Runtime Calibration for Heterogeneous Coordination: MARGIN learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. This allows a coordinator to produce corrected scores weight candidate answers in a collective decision by adjusting reported confidence based on observed reliability, addressing the issue where self-reported confidence may have different meanings across responders and changing workloads.
Adaptation Under Distribution Shift: The system can adapt its calibration when working with changing task distributions, as MARGIN has lower post-shift expected calibration error (ECE) than all five online baselines in two code-generation transitions and than four in a question-answering transition when tested on matched feedback.
Improved Answer Selection Accuracy: By using corrected scores, the system can improve coordination outcomes; specifically, MARGIN shows an increase of answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting in code-generation coordination experiments.
Diagnosis of Confidence Inversion: The system provides a diagnostic tool to identify when raw confidence is misleading, as demonstrated by the finding that model-mean confidence is negatively associated with accuracy on BigCodeBench, and its ability to identify confidence inversion across the heterogeneous pool.