Risk Ceilings and Development Deadlines: Pacing AI under Uncertain Safety Productivity
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Risk Ceilings and Development Deadlines".
Warren: Here's the gist of the paper. Proposals for pacing frontier Artificial Intelligence aim to establish both a ceiling on monthly catastrophe hazard and a deadline on development delay, contingent upon how productive safety research is.
Ruth: Achieving this ceiling necessitates maintaining a minimum stock of safety knowledge before the deadline, which in turn requires establishing a lower bound on productivity.
Warren: The analysis compares various pacing rules, including fixed allocation rules, hazard caps, and deadlines.
Ruth: The model partitions total compute among three areas: customer inference, capability research (advancing the frontier), and safety research (building knowledge).
Paper summary: Ruth: So, we're looking at this paper, "Risk Ceilings and Development Deadlines: Pacing AI under Uncertain Safety Productivity." The main idea is trying to figure out how to pace artificial intelligence development so that we can put a cap on monthly catastrophe hazard and also set a deadline for when the technology gets too far ahead of us.
Warren: Right, the core claim there is that achieving both of those promises depends entirely on how productive safety research actually is. It suggests you can't just set arbitrary rules; you need to know how much progress safety research makes so you can guarantee those limits.
Iris: And it sets up this division of compute among three areas: customer inference, capability research, and safety research. The hazard itself is split into state hazard, which is what we have running now, and step hazard, which is about moving to stronger models.
Ruth: Exactly. The paper argues that to hit that monthly catastrophe ceiling while also respecting a development deadline, you need a minimum stock of safety knowledge in place before you hit the deadline. That means there has to be a lower bound on how productive safety research can be.
Warren: And the model shows that feasibility hinges on finding this minimum productivity bound. If you can't rule out productivity being below this certain level, then no pacing rule can simultaneously guarantee both the hazard control and the deadline constraint.
Iris: The necessary condition for guaranteeing both promises is defined by an equation, which states that "The smallest productivity bound that satisfies condition twelve is the necessary bound" <ref:2610.11093#pg2>. It’s a necessary input for assessing feasibility before you even think about a specific strategy.
Ruth: That sounds like it puts a real constraint on the system before we look at the actual pacing rules, like fixed allocation or hazard caps, which they compare in detail. The abstract mentions that learning first and then replaying development comes within twenty-one percent of that bound <ref:2610.11093#pg2>.
Warren: That twenty-one percent figure is interesting because it suggests a specific way to calibrate things, which leaves eight percent catastrophe risk at the bound, against seventy-five percent at full speed <ref:2610.11093#pg1>.
Iris: From an econometrician's view, the paper connects this AI-risk regulation to environmental economics models where catastrophe probability depends on a stock the planner controls <ref:2610.11093#pg3>. It's using that framework to show how safety knowledge scales down the hazard.
Ruth: So, when we look at what this means for the world, it’s about turning an impossible universal promise into a conditional one. If you can demonstrate rules out productivity below that calculated bound, then a regulator can actually promise both the hazard limit and the deadline using a spare compute replay strategy <ref:2610.11093#pg1>.
Warren: That means demonstrating that minimum productivity level is the critical input for feasibility testing. If we can't rule out productivity below, say, zero point six nine, then we have to use that replay strategy with a specific cost involved <ref:2610.11093#pg1>.
Iris: The paper also notes that the optimal policy comparison depends on whether the hazard is coming from operating the current model or advancing to stronger models, which is called state versus step hazard <ref:2610.11093#pg3>. That distinction matters a lot for setting up those safety research functions.
Ruth: And that brings us into the conclusion section, where they talk about what this all means practically regarding regulation and timing strategies. They discuss how a regulator can proceed through three sequential steps: establishing feasibility, selecting a compliant timing strategy, and then comparing customer costs <ref:2610.11093#pg2>.
Warren: The title of the paper, "Risk Ceilings and Development Deadlines: Pacing AI under Uncertain Safety Productivity," really captures that tension between controlling risk and managing the pace of innovation. It’s not just about setting a cap; it’s about managing the timeline too.
Iris: In simpler terms, this work argues that to put both limits in place, you have to first figure out how productive safety research is expected to be <ref:2610.11093#pg2>. If that productivity can't be ruled out below a certain level, you can't make the universal guarantee.
Ruth: The implication for us is that we need concrete metrics on safety research output to design any effective pacing strategy. It moves the conversation away from just saying "we need more safety" to figuring out exactly how much knowledge needs to be built by a certain time <ref:2610.11093#pg3>.
Warren: And from a quantitative standpoint, the analysis shows that if we can prove that minimum productivity bound is met, then learning first and then replaying development preserves both the hazard control and the deadline at that associated service cost <ref:2610.11093#pg3>.
Iris: But we have to remember what the authors flag as a limitation for this analysis. They acknowledge that compute growth is driven by external factors, and measurements are treated as idealized, meaning safety research can be harder to distinguish from capability research <ref:2610.11093#pg3>.
Ruth: That's fair. And the authors treat the deadline as a fixed delay limit rather than a race outcome, which simplifies things for their model but is a key assumption we have to keep in mind when applying this to real-world scenarios <ref:2610.11093#pg2>.
Warren: So, what does this mean for the future of how we think about AI regulation? It seems like the focus has shifted from just reactive safety measures to proactively modeling the pace and productivity of the research itself <ref:2610.11093#pg2>.
Conclusion: Ruth: So we're wrapping up on this paper, "Risk Ceilings and Development Deadlines: Pacing AI under Uncertain Safety Productivity." It’s about how we can set limits on both the monthly catastrophe risk from AI and a hard deadline for when it gets too advanced.
Warren: Right, so the authors are looking at pacing something—AI development—and they’re trying to figure out if you can hit those two targets at the same time, or if one has to give way.
Iris: What they're really wrestling with is that whether you *can* guarantee both things depends entirely on how productive safety research actually is going to be in the future.
Ruth: Exactly. The core idea is that if we can figure out a minimum level of productivity for that safety research, then we can start making promises about those hazard ceilings and deadlines.
Warren: And if you can’t rule out that productivity dropping below a certain point, then no pacing strategy you pick will work to guarantee both the hazard control and the deadline.
Iris: They establish this necessary bound using some math—a specific equation—which basically sets the floor for what's possible before they even start choosing a strategy.
Ruth: So it boils down to this: first, you have to prove feasibility by finding that minimum productivity bound, then you pick a timing strategy like how much compute to allocate to safety research, and finally, you compare how much that costs the customer.
Warren: It suggests that if we can show we can rule out productivity dropping below zero point six nine, then a specific replay strategy might work for us at a certain cost of service.
Iris: But they’re clear about the limitations too—the way they measure things is idealized, and the deadline is treated more like a fixed delay than something that races to finish.
Ruth: So the big implication here is that we need concrete numbers on how fast safety research progresses, not just general hopes for improvement, if we want to actually promise these limits to the public.
Warren: Because if you don't nail down that productivity bound, all those pacing rules are just guesswork, and they might not actually stop the disaster or miss the deadline.
Iris: That brings us back to figuring out what those productivity metrics should even look like in practice.
Department of Economics, Texas A&M University
econ.GN, cs.CY, q-fin.EC
Submitted: 2026-10-08
Updated: 2026-11-05
Comments: v2: full-length version, with the supplemental appendix included as Appendices C-J; v1 was a short version of the same paper
Project page: https://leopoldaschenbrenner.github.io/xriskandgrowth/ExistentialR
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 77/100
The gist: Proposals for pacing frontier Artificial Intelligence aim to establish both a ceiling on monthly catastrophe hazard and a deadline on development delay…
Key concepts
- Hazard
- This refers to the risk associated with AI models. It's categorized into state hazard (operating current models) and step hazard (advancing to stronger models). Safety research aims to reduce this hazard by building knowledge.
- Productivity Bound
- This is a minimum required level of productivity needed for safety research. Determining this bound is crucial because it dictates whether it's possible to meet both the hazard ceiling and the development deadline simultaneously.
- Learn-then-Replay Policy
- This strategy involves dedicating all compute to safety research for at least one year (the deadline) before replaying the full development path with adjusted resource allocations. This method is proposed as a way to control hazard while respecting the time limit.
- Necessary Condition
- This is a mathematical requirement that must be satisfied for any pacing rule to guarantee both the hazard ceiling and the development deadline. If this condition isn't met, no pacing rule can simultaneously guarantee both promises.
Terminology
Summary
Proposals for pacing frontier Artificial Intelligence aim to establish both a ceiling on monthly catastrophe hazard and a deadline on development delay, contingent upon how productive safety research is. p. 0 Achieving this ceiling necessitates maintaining a minimum stock of safety knowledge before the deadline, which in turn requires establishing a lower bound on productivity. p. 0 The analysis compares various pacing rules, including fixed allocation rules, hazard caps, and deadlines. p. 2
The model partitions total compute among three areas: customer inference, capability research (advancing the frontier), and safety research (building knowledge). p. 5 Hazard is categorized as state hazard (operating current models) and step hazard (advancing to stronger models). p. 1 Safety research functions by building a stock of knowledge that progressively reduces the hazard associated with subsequent models. p. 9
Feasibility hinges on determining a minimum productivity bound, which is derived by bounding the time required to reach the specified ceiling while respecting the development deadline. p. 1 The necessary condition for guaranteeing both promises is defined by equation (12), stating that The smallest productivity bound that satisfies condition (12) is the necessary bound.
This implies that if productivity cannot be ruled out below this calculated bound, no pacing rule can simultaneously guarantee both hazard and deadline constraints.
The paper investigates several pacing strategies, such as compute floors, pauses, and R&D-model lags. p. 5 The learn-then-replay
policy specifically involves dedicating all compute to safety research for at least one year (the deadline) before replaying the full development path with adjusted resource allocations.
The conclusion is that a regulator can proceed through three sequential steps: establishing feasibility, selecting a compliant timing strategy, and comparing customer costs. p. 2 The necessary condition for guaranteeing both promises is that the necessary bound (Proposition 1(b)) must be met: A necessary condition for guaranteeing the deadline is then c bar at least h(1)e-alpha min sigma max gamma bar(T 0+L).
The analysis reveals that a hazard ceiling only loosely bounds catastrophe probability, and the optimal policy comparison depends on whether the hazard stems from state or step hazards. The productivity bound serves as the critical input for assessing feasibility; for instance, under normalized hazards, the necessary bound is 18
(in terms of relative productivity). A regulator who cannot rule out productivity below 0. p. 169 can promise both using a spare-compute replay strategy at a cost of 13 percent of five-year services. p. 1 This demonstrates that demonstrating rules out productivity below the replay bound transforms an impossible universal promise into a feasible conditional one, confirming the binding input is the productivity bound.
The analysis acknowledges limitations, noting that compute growth is exogenous and enforcement/measurement are idealized—for example, safety research is harder to distinguish from capability research. The deadline is treated as a fixed delay limit rather than a race outcome. p. 2 Ultimately, the paper concludes that if the replay condition (13) holds, learning first and then replaying preserves both hazard control and the deadline at the associated service cost. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement a safety floor rule that moves compute from customers to safety research while keeping capability research unchanged. This isolates one half of the compute floors and allows for dedicated safety knowledge accumulation before development resumes, as described in Rule 6 of Table 1.
-
Adopt a learning first then replay policy where the lab devotes all initial compute to safety research for a set time and then follows the full-speed development path with reduced research share. This strategy is shown to respect both the ceiling and the deadline when productivity is at its necessary bound, as detailed in Proposition 2.
-
Establish a regulator's commitment based on a necessary condition that requires a minimum productivity bound, which is calculated as c bar ≥ h(1)e negative amin σ bar max y bar (T0 + L). This ensures that both the risk ceiling and the development deadline can be guaranteed together at every productivity level above this bound.
-
Utilize a dynamic pause duration calculation based on equation G4 to determine the optimal pause length, which is either no pause or a full lead L, minimizing cumulative hazard within the feasible set of pauses under a fixed start date. This allows for precise timing decisions that balance hazard reduction against exposure cost.
-
Apply an adaptive speed limit mechanism where compute allocation adjusts dynamically to safety research as models improve, allowing the system to learn and then replay development with reduced compute expenditure over time. This mechanism is described in Rule 6 of Table 1 and is shown to be feasible at the necessary productivity bound.
Sources
- Governing Through the Cloud: The Intermediary Role of Compute Providers in AI Regulation
- Technology Speed Limits
- Computing Power and the Governance of Artificial Intelligence
- What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring
Related papers
- Huge If True: Learning from the Size of the Surprise
- The Rebundling of Freelance Work in the AI Age
- Dynamic investment and coordination in teams: Theory and experimental evidence
- Does Frontier AI Strengthen Cyber Defense? Evidence from Project Glasswing
- Generative AI at Work
- What is Work For? Labor, Politics, and Meaning in the Age of AI