Acceptance Cards:A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Acceptance Cards:A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims".
Elias: Here's the gist of the paper. Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does not transfer <ref:2605.10575#pg2> The paper introduces Acceptance Cards: an evaluation protocol, a documentation object, an executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims <ref:2605.10575#pg3> This protocol checks statistical reliability, fresh semantic generalization, mechanism alignment, and cross-task transfer before treating a gap reduction as a full-card pass <ref:2605.10575#pg4> The resulting audit on Gemma-2-2B-it shows that SafeLoRA fails the full-card pass under strict mechanism coding and still fails three of four under a permissive shrinkage relabel <ref:2605.10575#pg5> The Acceptance Standard A minimum acceptance standard for this claim is defined as a fine-tuning-time recipe reducing an installed gap in a way that is statistically reliable, semantically fresh, mechanism-consistent, and transferable <ref:2605.10575#pg3> This standard requires four diagnostic gates to pass strictly for a full-card pass <ref:2605.10575#pg4> The four diagnostics are Diagnostic (a) catching finite-split and template-level noise, Diagnostic (b) catching subject- or prompt-split artifacts, Diagnostic (c) checking mechanism class consistency via the parameter-space signature ρAT, and Diagnostic (d) testing transfer to a second installable task <ref:2605.10575#pg4> The conjunction rule requires all four diagnostics to pass strictly for a full-card pass <ref:2605.10575#pg4> The Four Diagnostic Gates The paper defines the thresholds for each diagnostic gate, which are conservative and designed to block specific over-readings <ref:2605.10575#pg4> Diagnostic (a) uses Question-Clustered Bootstrap, requiring the upper 95% CI on ∆gap to be below zero <ref:2605.10575#pg4> Diagnostic (b) requires both the paired and independent signed checks to hold, demanding that ∆gap + 2SEpaired Diagnostic (c) uses the parameter-space signature ρAT, where shrinkage passes if ρAT ≥ 0.6 and attack-targeted passes if ρAT Diagnostic (d) tests cross-task transfer, requiring conditions like ΓP > 0 and ΓX − ΓP ≤ 2SEcombined to pass <ref:2605.10575#pg4> Audit Results and Verdicts The audit tests whether available defense evidence supports the full-card pass, not whether any recipe is deployable <ref:2605.10575#pg4> The audit results in a 46-cell matrix where no cell satisfies the strict conjunction for a full-card pass <ref:2605.10575#pg5> The closest family is AC-AdamW, which passes (a) and (c) but fails (b) on the fresh-subject threshold and lacks a strict (d) pass <ref:2605.10575#pg5> The paper reports four verdict labels: full-card pass, near miss, missing evidence, and undefined <ref:2605.10575#pg4> Mechanism Diagnostic The parameter-space signature ρAT is introduced as the new mechanism-class diagnostic among the four <ref:2605.10575#pg5> The pass rule for (c) states that a cell’s claimed class matches the signature in (3): shrinkage passes if ρAT ≥ 0.6, while attack-targeted passes if ρAT The diagnostic flags claimed attack-aware updates whose parameter-space signature aligns with AdamW shrinkage <ref:2605.10575#pg5> Limitations and Intended Use The audit has empirical limits, as it is primarily based on Gemma-2-2B and uses cross-architecture runs as spot-checks rather than a scaling proof <ref:2605.10575#pg4> The standard is meant to reduce overclaiming and make partial results reportable, scoping SafeLoRA’s verdict to this installed-gap audit on Gemma-2-2B-it rather than a global judgment <ref:2605.10575#pg5> The paper emphasizes that missing diagnostics block the full-card label but are not counted as empirical evidence of failure on that gate <ref:2605.10575#pg5> Safe fine-tuning defense claims should not rest on held-out gap reductions alone <ref:2605.10575#pg5> Acceptance Cards type evidence for one claim under four diagnostics, and bundle the protocol with a documentation object and an executable audit package <ref:2605.10575#pg5> Partial passes remain reportable evidence; missing diagnostics are missing evidence, not implicit success <ref:2605.10575#pg5> The closest family is a near miss with deployment-accuracy cost, and SafeLoRA does not satisfy the full-card pass under the installed-gap protocol <ref:2605.10575#pg5> The standard is strict but satisfiable, as each gate targets a separate failure mode <ref:2605.10575#pg5> The gap between a 15.6pp training-distribution reduction and a 5.2pp fresh-domain reduction is itself informative <ref:2605.10575#pg5> The standard is built to detect the generalization failure that closes the gap on trained topics without addressing the underlying routing mechanism <ref:2605.
Title and authors: Nadia: Let's talk about the title and who wrote this thing: "Acceptance Cards: A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims". It’s a pretty descriptive name, suggesting they are building a formal system of evaluation rather than just another benchmark.
Elias: I think that title tells you right away that this isn't just about measuring the final gap reduction on a held-out set; it’s about creating an entire protocol—documentation, an audit package, and a standard—for how we assess those claims.
Priya: From my side, I’m focused on what that means for data trust. If you have a defense claim, you need to know exactly what evidence is required to back it up before you treat it as valid data. This sounds like the kind of formal structure needed when dealing with complex AI artifacts.
Nadia: Right. They are defining this minimum acceptance standard as reducing the installed gap in a statistically reliable way that is also semantically fresh, mechanism consistent, and transferable across different tasks or models.
Elias: And they validate this standard on Gemma-two-2B-it, while using other models like Qwen2 point 5-1 point 5B-Instruct, Llama three 8B, and Phi three mini as spot checks to see if the validation holds up across different architectures <ref:2605.10575#pg2,Qwen2.5-1.5B-Instruct, Llama 3 8B, and Phi 3 mini>.
Priya: I wonder how those cross architecture runs play into this. If a defense works on one model but fails the transfer test on another, does that invalidate the whole claim? It seems they are testing for a kind of robustness that goes beyond just one model family.
Nadia: They are testing for mechanism alignment and cross-task transfer as part of those four gates, so they’re explicitly looking to see if the defense works generally or just in a very specific setup.
Elias: The contribution isn't a new defense itself, which is important because it keeps the focus on evaluation rather than just another claim to be defended. It’s an evaluation protocol and an evidential standard for a narrow claim, not a general safe fine-tuning benchmark.
Priya: So, if I’m building a system that claims it has this defense, I can use these Acceptance Cards as the template for how I document my evidence instead of just presenting the raw gap number in isolation.
Nadia: Exactly. It gives you a clear structure for what needs to be measured and what remains missing when you evaluate those defense claims in practice.
The paper's summary: Nadia: So, to summarize the main idea of "Acceptance Cards: A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims," they are proposing a concrete set of tests to validate if a claim about a fine-tuning defense is actually sound.
Elias: The summary boils down to this: before you treat a recipe that reduces the gap as proof of safety, you have to pass four diagnostics: statistical reliability, fresh semantic generalization, mechanism consistency, and cross-task transfer.
Priya: They are saying that just seeing a smaller installed gap isn't enough because that reduction could be due to noise or artifacts. The paper argues that you need to prove the reduction is real in multiple ways simultaneously before you give it credit.
Nadia: That’s right, and they define the installation gap as the difference between the deployment framed rate and the evaluation framed rate, which is then compared to a matched baseline of zero. They use that delta gap to test if a recipe has reduced that installed gap in a way that isn't just luck.
Elias: The four diagnostic gates are what enforce this: they require all four diagnostics to pass strictly for you to get the full card pass, which is the minimum acceptance standard they set.
Priya: I’m thinking about how those gates translate into real data checks. Diagnostic one handles things like template-level noise and finite split issues, while diagnostic two deals with subject or prompt split artifacts that can skew results.
Nadia: And diagnostic three is where the mechanism class check comes in, using the parameter space signature rhoAT to see if the update behaves like what you claim it is, whether it’s shrinkage or something else.
Elias: And diagnostic four is about checking cross-task transfer, demanding that conditions like gamma P are greater than zero and the difference between gamma X and gamma P is less than or equal to two standard errors combined.
Priya: It sounds like a lot of checks that tie together statistical rigor with semantic meaning and operational consistency. It’s trying to prevent you from just claiming a defense when you haven't actually proven it generalizes properly.
Nadia: And the summary emphasizes that the resulting audit matrix shows that under strict coding, SafeLoRA fails all four diagnostics, which highlights how hard these standards are to meet in practice.
The paper's improvements: Elias: One of the main points they make is that this protocol is designed to reduce overclaiming. They want to ensure that a small gap reduction isn't being reported when the underlying mechanism isn't actually what it seems.
Nadia: And they suggest using these verdict labels—like near miss or missing evidence—instead of just giving a single score. That prevents people from reporting minor successes when they haven't met one of the four criteria, like failing the fresh subject threshold.
Priya: That’s crucial because it forces transparency about where the defense falls short. Instead of hiding that generalization failure in a single metric, you have to report it explicitly as a missing diagnostic.
Elias: They also introduce this concept of separating gap reduction from deployment accuracy cost. They want to make sure that utility loss isn't hidden inside the success metric itself, which means you have to report the deployment-accuracy cost outside of those gates too.
Nadia: So, they are pushing users to impose an additional utility floor on top of whatever gap reduction they find, making sure that the overall trade-off is what matters.
Priya: That makes sense. It shifts the focus from just achieving a low gap number to managing the trade-off between performance gain and deployment cost in a more structured way.
Elias: And they are also being careful about how they frame their tests, noting that cross architecture runs on models like Qwen2 point 5-1 point 5B-Instruct, Llama three 8B, and Phi three mini are just spot checks, not a proof that this standard scales universally <ref:2605.10575#pg2,Qwen2.5-1.5B-Instruct, Llama 3 8B, and Phi 3 mini>.
Nadia: So the improvement is less about making one perfect defense and more about creating a framework to evaluate *any* defense claim with enough evidence to be considered credible for reporting.
Conclusion: Nadia: To wrap up, the paper "Acceptance Cards: A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims" is proposing this rigorous four-diagnostic protocol—statistical reliability, semantic freshness, mechanism consistency, and cross-task transfer—as the standard for evaluating safe fine-tuning defense claims.
Elias: The key implication is that simply claiming a gap reduction isn't enough; you have to prove that reduction happens reliably and in multiple ways across different aspects of the model's behavior.
Priya: And they are providing tools—the Acceptance Cards, the documentation object, and the executable audit package—so researchers can actually execute this evaluation protocol on their own claims instead of just relying on a single number.
Nadia: They show that even with this strict standard applied to SafeLoRA on Gemma-two-2B-it, it fails the full card pass under strict mechanism coding, which is important context for anyone trying to deploy defenses right now <ref:2605.10575#pg1,on Gemma-2-2B-it>.
Elias: The paper also makes it clear that partial passes are reportable evidence, but missing diagnostics are considered missing evidence rather than implicit success on that gate.
Priya: So the final word is that these cards help prevent overclaiming and make sure we're reporting on real generalization failures, not just small numbers on a single metric.
Nadia: It’s a strict standard, but it seems satisfiable because each of those four gates targets a separate failure mode, so you can fix one problem at a time.
Elias: The paper suggests that the gap between different types of reduction—like the training distribution reduction versus the fresh domain reduction—is itself informative data.
Priya: It’s about detecting that generalization failure when it closes on trained topics without actually addressing the underlying routing mechanism, which is a very specific kind of problem.
Nadia: So, if you’re looking at a defense claim, don't just look at the gap reduction; use the Acceptance Cards to ensure you’ve checked for reliability and transferability.
Elias: That way, you get a clearer picture of what your defense actually achieves and where its real weaknesses lie.
Phongsakon Mark Konrad, Toygar Tanyel, Serkan Ayvaz
Centre for Industrial Software, University of Southern Denmark · ProMake
cs.CR, cs.AI, cs.LG
Submitted: 2026-05-11
Updated: 2026-11-01
Comments: Because the validation and scope of the four-diagnostic pass rule (deployment capability, cross-task criterion, mechanism diagnostic) and the breadth of the empirical audit do not support the claims as stated. The SafeLoRA result is a narrow out-of-domain stress test, not a judgment of the method. The work is being revised
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 91/100
The gist: Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same reduction can come from sampling noise, subject artifacts, capability loss…
Key concepts
- Acceptance Standard
- This is the minimum requirement for a defense claim to be accepted. It demands that the fine-tuning recipe reduces the gap in a statistically reliable, semantically fresh, mechanism-consistent, and transferable manner. Passing this standard requires all four diagnostic gates to succeed strictly.
- Diagnostic (a) - Statistical Reliability
- This gate uses Question-Clustered Bootstrap to ensure the observed gap reduction is statistically sound. It requires the upper 95% confidence interval for the gap reduction ($\Delta$gap) to be below zero, confirming that the observed improvement is not due to random noise or finite-split artifacts.
- Diagnostic (c) - Mechanism Alignment ($ ho_{AT}$)
- This diagnostic checks if the fine-tuning mechanism aligns with its claimed class. It uses a parameter-space signature ($ ho_{AT}$) where shrinkage passes if $\rho_{AT} \ge 0.6$. This helps flag cases where an update claiming one behavior (like shrinkage) actually exhibits characteristics of another mechanism.
- Diagnostic (d) - Cross-Task Transfer
- This gate tests the generalization capability by requiring conditions like $\Gamma_P > 0$ and $|\Gamma_X - \Gamma_P| \le 2SE_{combined}$ to pass. This ensures the defense mechanism is robust enough to transfer its learned behavior successfully to a second, independent installable task.
Terminology
Summary
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does not transfer <ref:2605. p. 0 ACCEPTANCE CARDS:
A FOUR-DIAGNOSTIC STANDARD FOR SAFE FINE-TUNING
DEFENSE CLAIMS
A PREPRINT
Phongsakon Mark Konrad*1
, Toygar Tanyel2
, and Serkan Ayvaz1
1Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark
2ProMake, Newark, DE, USA
May 12, 2026
ABSTRACT
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same
reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does
not transfer. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an
executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. ACCEPTANCE CARDS:
A FOUR-DIAGNOSTIC STANDARD FOR SAFE FINE-TUNING
DEFENSE CLAIMS
A PREPRINT
Phongsakon Mark Konrad*1
, Toygar Tanyel2
, and Serkan Ayvaz1
1Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark
2ProMake, Newark, DE, USA
May 12, 2026
ABSTRACT
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same
reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does
not transfer. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an
executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an
executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. The protocol checks statistical reliability, fresh semantic generalization, mechanism alignment, and
cross-task transfer before treating a gap reduction as a full-card pass. Re-scored under this installedgap protocol, SafeLoRA fails the full-card pass on Gemma-2-2B-it: under strict mechanism-class
coding it fails all four diagnostics, and under a permissive shrinkage relabel it still fails three of
four. This is a narrow installed-gap audit on one model family, not a global judgment of SafeLoRA’s
effectiveness.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The Acceptance Standard
A minimum acceptance standard for this claim is defined as a fine-tuning-time recipe reducing an installed gap in a way that is statistically reliable, semantically fresh, mechanism-consistent, and transferable <ref:2605.10575#pg3> This standard requires four diagnostic gates to pass strictly for a full-card pass <ref:2605. p. 2 Satisfying the standard is
evidence for the tested claim, not a deployability certificate: the card requires deployment-accuracy cost to be reported
so that utility loss is not hidden inside the success metric, and a user may impose an additional utility floor on top of the
card. 2 Four Diagnostic Gates
Naive claim
held-out gap smaller
Acceptance card
(a) clustered CI excludes no effect
(b) fresh subjects clear signed SE
(c) ρAT matches claimed class
(d) transfer to installable task
Accept only if
all four pass
46-cell audit
no strict pass
closest: AC-AdamW
Deployment accuracy: reported cost Missing diagnostic: not a pass
Figure 1: The Acceptance Card standard. Diagnostic Blocks Statistic Pass rule Does not prove
(a) Bootstrap finite-split / template
noise
clustered CI for
∆gap
upper 95% CI < 0 fresh-subject generalization
(b) Fresh subjects subject-split artifacts paired and independent signed SE
both upper 2SE checks
< 0
mechanism or transfer
(c) ρAT mechanism-label
mismatch
update projection ratio in slice
claimed class matches
signature
efficacy
(d) Transfer primary-task-only
improvement
ΓP vs ΓX w/ comb. SE
installable cross-task
gap; comparable reduction
universal safety
Conjunction overstated headline all four pass strictly (a)∧(b)∧(c)∧(d) deployment safety
2 The
conjunction requires all four to pass strictly. Diagnostic Blocks Statistic Pass rule Does not prove
(a) Bootstrap finite-split / template
noise
clustered CI for
∆gap
upper 95% CI < 0 fresh-subject generalization
(b) Fresh subjects subject-split artifacts paired and independent signed SE
both upper 2SE checks
< 0
mechanism or transfer
(c) ρAT mechanism-label
mismatch
update projection ratio in slice
claimed class matches
signature
efficacy
(d) Transfer primary-task-only
improvement
ΓP vs ΓX w/ comb.Where the paper says it
Where the paper says it
Where the paper says it
The Four Diagnostic Gates
The paper defines the thresholds for each diagnostic gate, which are conservative and designed to block specific over-readings <ref:2605.10575#pg4> Diagnostic (a) uses Question-Clustered Bootstrap, requiring the upper 95% CI on ∆gap to be below zero <ref:2605. p. 1 The
conjunction requires all four to pass strictly. Diagnostic Blocks Statistic Pass rule Does not prove
(a) Bootstrap finite-split / template
noise
clustered CI for
∆gap
upper 95% CI < 0 fresh-subject generalization
(b) Fresh subjects subject-split artifacts paired and independent signed SE
both upper 2SE checks
< 0
mechanism or transfer
(c) ρAT mechanism-label
mismatch
update projection ratio in slice
claimed class matches
signature
efficacy
(d) Transfer primary-task-only
improvement
ΓP vs ΓX w/ comb. The
conjunction requires all four to pass strictly. Diagnostic Blocks Statistic Pass rule Does not prove
(a) Bootstrap finite-split / template
noise
clustered CI for
∆gap
upper 95% CI < 0 fresh-subject generalization
(b) Fresh subjects subject-split artifacts paired and independent signed SE
both upper 2SE checks
< 0
mechanism or transfer
(c) ρAT mechanism-label
mismatch
update projection ratio in slice
claimed class matches
signature
efficacy
(d) Transfer primary-task-only
improvement
ΓP vs ΓX w/ comb. Diagnostic Blocks Statistic Pass rule Does not prove
(a) Bootstrap finite-split / template
noise
clustered CI for
∆gap
upper 95% CI < 0 fresh-subject generalization
(b) Fresh subjects subject-split artifacts paired and independent signed SE
both upper 2SE checks
< 0
mechanism or transfer
(c) ρAT mechanism-label
mismatch
update projection ratio in slice
claimed class matches
signature
efficacy
(d) Transfer primary-task-only
improvement
ΓP vs ΓX w/ comb. SE
installable cross-task
gap; comparable reduction
universal safety
Conjunction overstated headline all four pass strictly (a)∧(b)∧(c)∧(d) deployment safety
2Where the paper says it
Where the paper says it
Where the paper says it
Audit Results and Verdicts
The audit tests whether available defense evidence supports the full-card pass, not whether any recipe is deployable <ref:2605.10575#pg4> The audit results in a 46-cell matrix where no cell satisfies the strict conjunction for a full-card pass <ref:2605. p. 4 3 (full-card pass, near miss, missing evidence, undefined) keep these cases
distinct. Across 46 audited cells, no cell with the required evidence satisfies the strict conjunction. In a 46-cell audit, no cell satisfies the strict conjunction. The closest family is a near
miss that passes reliability and mechanism checks where the required data are available, but fails the
fresh-subject threshold, lacks a strict transfer pass, and carries a measurable deployment-accuracy
cost. 2 Acceptance Standard
This section types evidence for one claim: that a fine-tuning-time recipe reduces an installed evaluation-to-deployment
gap. The standard defines four diagnostic gates, the threshold for each, a single conjunction rule, and four verdict labels
(full-card pass, near miss, missing evidence, undefined) used to type cells that do not satisfy the strict conjunction.Where the paper says it
Where the paper says it
Where the paper says it
Mechanism Diagnostic
The parameter-space signature ρAT is introduced as the new mechanism-class diagnostic among the four <ref:2605.10575#pg5> The pass rule for (c) states that a cell’s claimed class matches the signature in (3): shrinkage passes if ρAT ≥ 0. p. 3 (3)
Pass rule. A cell’s claimed class matches the signature in (3): shrinkage passes if ρAT ≥ 0. A pure
shrinkage of the base gradient keeps ρAT near its baseline value; an attack-targeted update suppresses the projection on
vA relative to vT and lowers ρAT . The diagnostic flags claimed attack-aware updates whose parameter-space signature
aligns with AdamW shrinkage.Where the paper says it
Where the paper says it
Limitations and Intended Use
The audit has empirical limits, as it is primarily based on Gemma-2-2B and uses cross-architecture runs as spot-checks rather than a scaling proof <ref:2605.10575#pg4> The standard is meant to reduce overclaiming and make partial results reportable, scoping SafeLoRA’s verdict to this installed-gap audit on Gemma-2-2B-it rather than a global judgment <ref:2605. p. 8 Broader Impacts. The standard is meant to reduce overclaiming and make partial results reportable; SafeLoRA’s
verdict is scoped to this installed-gap audit on Gemma-2-2B-it, not to the method in general. The 46-cell audit is an evidence-coverage audit rather than 46 complete four-gate failures. A
missing or undefined gate blocks the full-card label but is not counted as empirical evidence that the method fails that
gate; the four verdict labels in Section 2.Where the paper says it
Where the paper says it
Conclusion
Safe fine-tuning defense claims should not rest on held-out gap reductions alone <ref:2605.10575#pg5> Acceptance Cards type evidence for one claim under four diagnostics, and bundle the protocol with a documentation object and an executable audit package <ref:2605. p. 8 7 Conclusion
Safe fine-tuning defense claims should not rest on held-out gap reductions alone. Acceptance Cards type evidence for
one claim under four diagnostics, and bundle the protocol with a documentation object and an executable audit package. The closest family is a near miss with deployment-accuracy cost; SafeLoRA does not satisfy the full-card pass under
the installed-gap protocol. Partial passes remain reportable evidence; missing diagnostics are missing evidence, not
implicit success. Acceptance Cards type evidence for
one claim under four diagnostics, and bundle the protocol with a documentation object and an executable audit package. The closest family is a near miss with deployment-accuracy cost; SafeLoRA does not satisfy the full-card pass under
the installed-gap protocol. 9pp
cost
Strict conjunction (a)∧(b)∧(c)∧(d) near miss
The closest partial result is a shrinkage-class recipe that drops gap by 5pp on fresh items at a cost of 12pp in deployment
accuracy, without clearing the paired 2SE threshold on (b) and without a non-degenerate cross-task for (d). The standard
is strict but satisfiable: each gate targets a separate failure mode, and nothing in the definitions prevents a future defense
from passing all four. ACCEPTANCE CARDS:
A FOUR-DIAGNOSTIC STANDARD FOR SAFE FINE-TUNING
DEFENSE CLAIMS
A PREPRINT
Phongsakon Mark Konrad*1
, Toygar Tanyel2
, and Serkan Ayvaz1
1Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark
2ProMake, Newark, DE, USA
May 12, 2026
ABSTRACT
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same
reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does
not transfer. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an
executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. The gap between a 15. 6pp training-distribution reduction and a 5. 6pp training-distribution reduction and a 5. 2pp fresh-domain reduction is
itself informative: the defense closes the gap on trained topics without addressing the underlying routing mechanism,
exactly the generalization failure the standard is built to detect. 6pp training-distribution reduction and a 5. 2pp fresh-domain reduction is
itself informative: the defense closes the gap on trained topics without addressing the underlying routing mechanism,
exactly the generalization failure the standard is built to detect.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precise evidence reporting for fine-tuning defenses can be achieved by using Acceptance Cards which require a recipe to pass four diagnostic gates: statistical reliability, fresh semantic generalization, mechanism alignment, and cross-task transfer. This protocol blocks overclaiming by requiring all four diagnostics to pass strictly before a full-card pass is awarded.
-
A system can now rigorously audit defense claims by incorporating the parameter-space class-consistency signature rhoAT which classifies the mechanism class from the merged update without re-running the fine-tune. This allows researchers to identify cases where a published defense's update signs as shrinkage despite a non-shrinkage mechanism claim, as seen in SafeLoRA failing under strict mapping.
-
The system can differentiate between partial evidence and full-card labels by using verdict labels such as near miss or missing evidence instead of relying on a single headline score. This prevents the reporting of
small but real generalization
when the fresh subject threshold is not met, as noted in the paper's Claim-Language Rules. -
Deployment safety can be assessed by separating gap reduction from deployment accuracy cost, ensuring that utility loss is not hidden inside the success metric. The audit requires reporting a
deployment-accuracy cost of Xpp reported outside the gate,
which allows users to impose an additional utility floor on top of the card. -
The system can be trained to recognize when cross-task transfer evidence is undefined by construction, such as when a baseline gap is within two standard errors of zero, and report this as N/A rather than pass. This prevents the conversion of missing evidence into success on the only diagnostic that tests transfer, which is a failure mode the standard is built to prevent.
Abstract
Safe fine-tuning defenses are often endorsed on the basis of a held-out gap reduction, but the same reduction can come from sampling noise, subject artifacts, capability loss, or a mechanism that does not transfer. We introduce Acceptance Cards: an evaluation protocol, a documentation object, an executable audit package, and a claim-specific evidential standard for safe fine-tuning defense claims. The protocol checks statistical reliability, fresh semantic generalization, mechanism alignment, and cross-task transfer before treating a gap reduction as a full-card pass. Re-scored under this installed-gap protocol, SafeLoRA fails the full-card pass on Gemma-2-2B-it: under strict mechanism-class coding it fails all four diagnostics, and under a permissive shrinkage relabel it still fails three of four. This is a narrow installed-gap audit on one model family, not a global judgment of SafeLoRA's effectiveness. In a 46-cell audit, no cell satisfies the strict conjunction. The closest family is a near miss that passes reliability and mechanism checks where the required data are available, but fails the fresh-subject threshold, lacks a strict transfer pass, and carries a measurable deployment-accuracy cost.
Sources
- Gemma 2: Improving Open Language Models at a Practical Size
- Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- Qwen2.5 Technical Report
- The Llama 3 Herd of Models
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data