Ranking Experiments under Sequential Sampling

arXiv:2608.19897 · econ.TH · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Ranking Experiments under Sequential Sampling".

Warren: Here's the gist of the paper. We compare statistical experiments when observations are inexpensive and can be acquired sequentially until the decision maker chooses to stop This paper compares statistical experiments when observations are inexpensive and can be acquired sequentially until the decision maker chooses to stop The main result shows that, for generic pairs, the two orders coincide and are both characterized by strict dominance of every pairwise Kullback–Leibler divergence Defining the Comparison Orders The paper introduces two orders to compare sequential experiments: small-cost decision dominance asks which of two equally priced experiments is eventually preferred in every decision problem as the per-observation cost vanishes large-budget stopping dominance asks which experiment can reproduce every terminal experiment attainable from the other under all sufficiently large expected sample budgets The authors define these orders based on how experiments compare across decision problems and source experiments, respectively.

Ruth: The Generic Equivalence Theorem The main theorem establishes that the two comparisons have the same generic answer, showing that for distinct states i and j, F large-budget stopping dominates G if and only if Iij (F) > Iij (G) for every ordered pair i!= j This is characterized by the statement that F small-cost decision dominates G if and only if Iij (F) > Iij (G) for every ordered pair i!= j The intuition is that directed pairwise KL divergence measures the rate at which an experiment accumulates evidence The Uniform Exact-Conversion Theorem The proof of the main theorem rests on a uniform exact-conversion theorem, which states that any finite-output stopping policy based on one experiment can be reproduced exactly using another The reproducing F-policy’s expected sample size satisfies E F i N ≤ ρ i(F → G)E G i T + CF,GqE G iT If every pairwise KL inequality is strict, then ρi(F → G) < 1 for every state; the resulting linear saving eventually dominates the square-root remainder Comparison with Fixed-Sample Orders The paper compares its endogenous-stopping orders with fixed-sample orders like Moscarini–Smith and Mu–Pomatto–Strack–Tamuz, showing that for KL-generic pairs, the MPST order implies the stopping order The distinction lies in the resource constraint: a deterministic horizon assigns the same number of observations to every sample path, whereas an expected-sample budget allows a policy to stop early on easy histories and continue on difficult ones One experiment yields at least as much value in every decision problem once observations are sufficiently cheap if and only if, under sufficiently large expected-sample budgets, it can reproduce every terminal experiment attainable from the other source Both comparisons are characterized by the same condition: the experiment has a strictly larger directed pairwise KL divergence for every ordered pair of states The quantities Iij (E) are the rates at which a source produces evidence for state i against state j, and ρi(F → G) is the resulting first-order conversion factor under state i Pruning redundant reasoning tokens The paper discusses connections to four literatures including comparison of sequential experiments, large-sample orders for repeated experiments, endogenous information acquisition, and sequential testing and experimental design The authors show that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders How the training signal is built The proof of Theorem 2 involves a uniform exact-conversion theorem which decomposes the expected source sample requirement into a first-order conversion term and a square-root remainder The key input is a uniform exact-conversion theorem Where the gains come from The gains come from the flexibility of endogenous stopping, which allows sampling effort to be shifted across histories: easy histories can terminate early while difficult histories receive more observations Under an expected-sample budget, what survives to first order is therefore the mean rate of evidence accumulation The exact-conversion theorem shows that this flexibility reduces the exact-production constraints to the directed pairwise KL divergences The final result is that E F i N ≤ ρ i(F → G)E G i T + CF,GqE G iT Where the gains come from The authors show that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders The key input is a uniform exact-conversion theorem Pruning redundant reasoning tokens

Paper summary: Iris: To summarize, the paper "Ranking Experiments under Sequential Sampling" compares statistical experiments when observations are inexpensive and acquired sequentially until a decision maker chooses to stop.

Ruth: The thesis is that they introduce two orders, small-cost decision dominance and large-budget stopping dominance, and they show that for generic pairs of experiments, these two orders actually coincide.

Warren: They characterize both comparisons by stating that there's a strict dominance of every single pairwise Kullback–Leibler divergence across all states.

Iris: The key step in proving this is a uniform exact-conversion theorem, which basically means any finite output stopping policy from one experiment can be perfectly reproduced using another.

Ruth: That conversion involves determining the first-order expected sample requirements based on those pairwise KL rates plus a square root remainder that is uniform across all policies.

Warren: This square root remainder is what accounts for the flexibility of endogenous stopping, letting you stop early on easy histories while continuing on difficult ones under an expected sample budget.

Iris: So, what they’re saying is that if those pairwise KL inequalities are strict, then this resulting linear saving eventually dominates the square root remainder term.

Ruth: This leads to the main result where one experiment yields at least as much value in every decision problem once observations are sufficiently cheap if and only if it can reproduce every terminal experiment attainable from the other under large expected sample budgets.

Warren: It ties this comparison back to things like sequential testing and experimental design, showing that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders.

Iris: They also compare these endogenous-stopping orders against fixed-sample orders, finding that for KL-generic pairs, the Moscarini–Smith order implies this stopping order.

Ruth: The main idea is that both comparisons are ultimately characterized by the same condition: the experiment has a strictly larger directed pairwise KL divergence for every ordered pair of states.

Warren: The quantities Iij E represent the rates at which a source produces evidence for state i against state j, and rho i F to G is that resulting first-order conversion factor under state i.

Iris: This gives us a strong statement about how the structure of information accumulation determines the relative quality of two experimental designs under sequential sampling.

Conclusion: Ruth: So, looking at "Ranking Experiments under Sequential Sampling," we see that the core finding is a convergence between two seemingly different ways of comparing experiments under sequential data acquisition.

Warren: The authors show that small-cost decision dominance and large-budget stopping dominance have the same generic answer for most pairs of experiments.

Iris: This convergence is driven by the strict dominance of every pairwise Kullback–Leibler divergence, which is a measure of how fast an experiment accumulates evidence against another state.

Ruth: It means that if you can afford enough data under a large expected sample budget, one experiment will be better than or equal to the other in every decision problem.

Warren: And this holds true because of the uniform exact-conversion theorem, which precisely quantifies the expected sample size needed for reproduction.

Iris: The implication is that for sequential sampling, you don't need to worry about complex stopping rules if you can afford a budget large enough to let the fundamental divergence rates do their work.

Ruth: It’s a statement about the power of information accumulation under sequential constraints, showing how flexibility in stopping rules translates directly into evidence accumulation rates.

Warren: This framework gives us a solid way to think about what survives when you have limited but flexible sampling resources, and it connects this idea to other areas like repeated experiments.

Iris: The paper does rely on the assumption that these pairwise KL inequalities are strict for the generic pairs they compare, which is something we should keep in mind for applying this result.

Department of Economics, Columbia University · Cowles Foundation for Research in Economics, Yale University

econ.TH

Submitted: 2026-08-20

Updated: 2026-11-03

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: Ranking Experiments under Sequential Sampling compares statistical experiments when observations are inexpensive and can be acquired sequentially until a decision maker chooses to stop.

Key concepts

Small-cost decision dominance
This order asks which of two equally priced experiments will be preferred in every decision problem as the cost per observation approaches zero. It focuses on long-term preference under minimal cost constraints.
Large-budget stopping dominance
This order determines which experiment can reproduce every possible terminal outcome achievable from the other, given a sufficiently large expected sample budget. It considers stopping rules based on expected sample sizes.
Directed pairwise KL divergence
This measures the rate at which an experiment accumulates evidence for one state against another. A strictly larger divergence means that experiment is better at distinguishing between those two states over time.
Uniform exact-conversion theorem
This theorem proves that any finite-output stopping policy based on one experiment can be perfectly replicated using another. It decomposes the required sample size into a first-order conversion term and a square-root remainder.

Terminology

Summary

Ranking Experiments under Sequential Sampling compares statistical experiments when observations are inexpensive and can be acquired sequentially until a decision maker chooses to stop. p. 0 The main result shows that, for generic pairs, the two orders coincide and are both characterized by strict dominance of every pairwise Kullback–Leibler divergence. p. 0

Definition of Comparison Orders

Two primary orders are introduced: small-cost decision dominance, which asks which experiment is eventually preferred in every decision problem as the per-observation cost vanishes. Conversely, large-budget stopping dominance asks which experiment can reproduce every terminal experiment attainable from the other under all sufficiently large expected sample budgets. p. 0 The paper establishes that for generic pairs, these two orders are equivalent and characterized by strict dominance of every pairwise Kullback–Leibler divergence. p. 0

Equivalence and Characterization Theorem

Theorem 1 states that for KL-generic pairs, the decision and stopping orders are equivalent, both being characterized by strict dominance of directed pairwise KL divergences. The intuition is that directed pairwise KL divergence measures the rate at which an experiment accumulates evidence. p. 7 The theorem establishes a uniform exact-conversion theorem, showing that any finite-output stopping policy based on one experiment can be reproduced exactly using another with first-order expected-sample requirements determined by pairwise KL rates and a square-root remainder. p. 0

Relationship to Existing Literature

The paper relates its findings to several fields, including the comparison of sequential experiments, large-sample orders for repeated experiments, endogenous information acquisition, and sequential testing and experimental design. It distinguishes its approach from fixed-sample orders like Moscarini–Smith and Mu–Pomatto–Strack–Tamuz by allowing the sample size to depend on the realized history through endogenous stopping. p. 4

Key Technical Results

The proof of equivalence relies on Theorem 2, which provides a uniform exact-conversion theorem where the expected source sample requirement is decomposed into a first-order conversion term and a square-root remainder. The stopping order implies pairwise KL dominance by showing that if F large-budget stopping dominates G, then for any fixed ordered pair i ̸= j, the inequality Iij (F) ≥ Iij (G) holds weakly. p. 9 The decision order implies pairwise KL dominance through a sequence of arguments involving nearly optimal policies and binary testing problems. p. 8

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Small-cost decision dominance for AI systems will involve comparing two equally priced experimental methods to determine which one eventually yields a weakly higher optimal value across every finite decision problem as observation costs vanish. This allows AI designers to select data acquisition strategies that are robust across different future operational scenarios when marginal data costs are low.

  2. Large-budget stopping dominance can be used to compare different source experiments by assessing which one can exactly reproduce every terminal experiment attainable from the other under a sufficiently large expected sample budget in every state. This is crucial for determining the most productive underlying data generation technology when a fixed number of samples is available and the decision maker has flexibility on when to stop.

  3. The system can be designed to prioritize experiments that exhibit a strictly larger directed pairwise KL divergence for every ordered pair of states, as this condition generically characterizes both small-cost decision dominance and large-budget stopping dominance. This allows the AI to select data sources that accumulate evidence faster in every direction, leading to a more robust and efficient learning process regardless of the specific state being true.

  4. The system can implement an exact conversion theorem that reproduces any finite output based on one experiment exactly using another, with a first-order expected sample requirement determined by pairwise KL rates and a square-root remainder that is uniform over policies. This ensures that complex, learned behaviors can be reliably achieved from simpler underlying data acquisition mechanisms while controlling the sample budget overhead.

  5. The AI can perform optimal sequential decision making under varying observation costs by using a stopping policy derived from an experiment that exhibits strict pairwise KL dominance, ensuring the resulting sequence of actions is weakly more valuable than any other policy for sufficiently small per-observation costs. This enables adaptive learning where the decision to continue or stop is based on the accumulation rate of evidence rather than a fixed sample size.

Sources

Related papers