BioFirewall: A genome-writing-native governance layer for design-stage biosecurity screening of agentic AI

summary

The gist

BioFirewall is a rule-governed middleware designed as a genome-writing-native governance layer for design-stage biosecurity screening of agentic AI systems.

In short

BioFirewall is a rule-governed middleware that screens AI systems during their design stage by analyzing genome-writing plans for biosecurity risks. It classifies plans across five hazard axes—cargo, locus, edit type, germline, and scale—returning allow, flag-for-review, or refuse verdicts. This system provides evidence and tamper-proof records to ensure safety before synthesis.

Key concepts

Hazard Axes
These are the five specific dimensions BioFirewall uses to classify potential risks in a genome-writing plan: cargo (what is being modified), locus (where the modification occurs), edit type, germline impact, and scale of the operation. They allow for a comprehensive hazard assessment.
Genome-Writing-Native Governance Layer
This describes BioFirewall's core function: it sits directly between planning and synthesis to govern how AI systems write genetic code. It intercepts these plans to enforce security rules, making governance an intrinsic part of the design process rather than an afterthought.
Stratified Screen
The system uses a multi-layered screening approach. It combines a five-axis stratified screen with benchmark testing and red teaming analysis to ensure robust hazard detection. This layered method helps identify risks that standard sequence screening might miss.
Design-Stage Screening
BioFirewall operates at the critical design phase, before the actual synthesis of sequences begins. Its goal is to catch dangerous plans early in the development process, acting as a crucial guardrail between initial model refusal checks and final sequence screening.

Terminology used across episodes

This episode discusses

The paper

BioFirewall: A genome-writing-native governance layer for design-stage biosecurity screening of agentic AI · Read on arXiv

Anees Ahmed Mahaboob Ali, Radhakrishnan Delhibabu, Everette Jacob Remington Nelson

School of Bio Sciences and Technology, Vellore Institute of Technology · School of Computer Science and Engineering, Vellore Institute of Technology

AI design tools now plan genome-scale edits, and agentic systems execute those plans with progressively less human oversight. Biosecurity controls are limited to two points: refusal guardrails at the foundation model and sequence-identity screening at the synthesiser. The design stage between them, where the plan is specified, is governed by recommendations, not by any deployed system. We present BioFirewall, a rule-governed middleware that intercepts a genome-writing plan and returns allow, flag-for-review, or refuse across five genome-writing-native hazard axes: cargo, locus, edit type, germline and scale, with cited evidence, a signed design passport, a tamper-evident audit log, and tiered access. On a de-circularised benchmark of safe proxies scored against independent oracles, a function-aware cargo classifier reached a true-positive rate of 0.72 (95% CI 0.43 to 0.89) at a false-positive rate of 1.105%, the nearest attainable to 1%, on a held-out split. Language-model judges did not reliably screen these sequences: their engagement depended heavily on sequence class, so a rate conditioned on engagement does not measure screening performance. Under prompt injection, the blocking verdicts of the open-weight judges tested flipped to allow, while the deterministic screen remained invariant. None of the 288 legitimate plans was refused, giving a 95% upper bound on the false-refuse rate of 0.0103 for the evaluated corpus under an independence assumption, or 0.632 treating the three plan templates as independent units. On a held-out gene set, the locus axis was enriched for drivers of in vivo insertional oncogenesis (AUROC 0.605; odds ratio 3.34), and a session monitor intercepted the cross-call decomposition attacks it was built to detect. These results show design-stage governance is achievable within the evaluated scope. BioFirewall is open source under a permissive licence.

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: I'm Ines.

Marcus: And I'm Marcus, here with Yuki.

Ines: We're looking at the paper: "BioFirewall: A genome-writing-native governance layer for design-stage biosecurity screening of agentic AI".

Marcus: Here's the gist of the paper. BioFirewall is a rule-governed middleware designed as a genome-writing-native governance layer for design-stage biosecurity screening of agentic AI systems.

Ines: It intercepts genome-writing plans and classifies them across five hazard axes: cargo, locus, edit type, germline, and scale, returning one of three verdicts: allow, flag-for-review, or refuse.

Marcus: The system provides cited evidence and a signed design passport secured with an HMAC-SHA256 signature to ensure tamper evidence.

Ines: It also implements tiered access controls and addresses the complete set of design-stage guardrails specified by the Nuclear Threat Initiative (NTI), including input/output screening, signed metadata management, and managed access.

Paper summary: Ines: So we've heard that BioFirewall is this rule-governed layer designed for design-stage screening of agentic AI genome writing plans. What they are doing is reasoning about hazards that standard synthesizers just don't see.

Marcus: They claim their framework has a five-axis stratified screen, a de-circularized benchmark, and measurements of language models against baselines to show how well it works.

Yuki: And they also categorized red team analysis to test the system's resilience against different kinds of attacks.

Ines: The results show some specific numbers on the cargo axis classifier, where it achieved a true positive rate of zero point seven two with a ninety-five percent confidence interval between zero point four three and zero point eight nine when tested at a one percent false-positive rate on their de-circularized benchmark of safe proxies <ref:2608.20413#pg2,a true positive rate of 0.72>.

Marcus: That sounds like a solid result for the cargo axis, but they also showed that under prompt injection attacks, the open weight judges flipped their blocking verdict to allow in three out of six trials per channel, while the deterministic screen stayed the same.

Yuki: And interestingly, none of the two hundred eighty-eight legitimate plans from three templates were refused during testing, which means they have a certified ninety-five percent upper bound on the false refuse rate at zero point zero one zero three <ref:2608.20413#pg2,288 legitimate plans from three templates>.

Ines: They also looked at the locus axis and found that it was enriched for drivers of in vivo insertional oncogenesis, with an auroc of zero point six zero five and an odds ratio of three point three four when compared against a held out gene set, which they linked to some mouse model results <ref:2608.20413#pg2,was enriched for drivers of in vivo insertional oncogenesis>.

Marcus: While cargo and locus gave them quantified results, the edit type, germline, and scale axes are just described as mechanism grounded rule bases rather than specific statistical outcomes.

Yuki: So what this means for us is that they are trying to build something that actually works at this design stage, which is where things are currently pretty open.

Conclusion: Ines: So we've covered the details of BioFirewall, its five hazard axes, and those specific numbers on the cargo and locus axes. The authors are calling this design-stage governance achievable in practice.

Marcus: They released it open source along with a pre-registered, open data reproducible benchmark that sets a certified ceiling on the false refuse rate for anyone trying to use it.

Yuki: It seems like the implication is that we can start moving beyond just model layer refusals or sequence screening at the execution stage and build this kind of explicit governance before plans even get sent to the lab robots.

Ines: The title itself, "BioFirewall: A genome-writing-native governance layer for design-stage biosecurity screening of agentic AI", suggests they're building something deeply integrated into how these systems are designed from the ground up.

Marcus: It’s about moving the guardrails from being just a filter on the output to being a part of the initial plan formulation process itself, which is where most of this new risk is happening.

Yuki: It forces us to think about what kind of planning agents we are even allowing to design these things in the first place.

More episodes

← Home