Ranking Experiments under Sequential Sampling
summary
The gist
Ranking Experiments under Sequential Sampling compares statistical experiments when observations are inexpensive and can be acquired sequentially until a decision maker chooses to stop.
In short
The paper compares two ways to rank sequential experiments: small-cost decision dominance and large-budget stopping dominance. The main result shows these two orders coincide for generic pairs, both being characterized by strict dominance in pairwise Kullback–Leibler divergence. This means an experiment is superior if it accumulates evidence faster against every other possible state.
Key concepts
- Small-cost decision dominance
- This order asks which of two equally priced experiments will be preferred in every decision problem as the cost per observation approaches zero. It focuses on long-term preference under minimal cost constraints.
- Large-budget stopping dominance
- This order determines which experiment can reproduce every possible terminal outcome achievable from the other, given a sufficiently large expected sample budget. It considers stopping rules based on expected sample sizes.
- Directed pairwise KL divergence
- This measures the rate at which an experiment accumulates evidence for one state against another. A strictly larger divergence means that experiment is better at distinguishing between those two states over time.
- Uniform exact-conversion theorem
- This theorem proves that any finite-output stopping policy based on one experiment can be perfectly replicated using another. It decomposes the required sample size into a first-order conversion term and a square-root remainder.
Terminology used across episodes
This episode discusses
- Ranking Experiments under Sequential Sampling · Paper Radio
- The Cost of Optimally Acquired Information
- Comparisons of Sequential Experiments for Additively Separable Problems
The paper
Ranking Experiments under Sequential Sampling · Read on arXiv
Department of Economics, Columbia University · Cowles Foundation for Research in Economics, Yale University
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Ranking Experiments under Sequential Sampling".
Warren: Here's the gist of the paper. We compare statistical experiments when observations are inexpensive and can be acquired sequentially until the decision maker chooses to stop This paper compares statistical experiments when observations are inexpensive and can be acquired sequentially until the decision maker chooses to stop The main result shows that, for generic pairs, the two orders coincide and are both characterized by strict dominance of every pairwise Kullback–Leibler divergence Defining the Comparison Orders The paper introduces two orders to compare sequential experiments: small-cost decision dominance asks which of two equally priced experiments is eventually preferred in every decision problem as the per-observation cost vanishes large-budget stopping dominance asks which experiment can reproduce every terminal experiment attainable from the other under all sufficiently large expected sample budgets The authors define these orders based on how experiments compare across decision problems and source experiments, respectively.
Ruth: The Generic Equivalence Theorem The main theorem establishes that the two comparisons have the same generic answer, showing that for distinct states i and j, F large-budget stopping dominates G if and only if Iij (F) > Iij (G) for every ordered pair i!= j This is characterized by the statement that F small-cost decision dominates G if and only if Iij (F) > Iij (G) for every ordered pair i!= j The intuition is that directed pairwise KL divergence measures the rate at which an experiment accumulates evidence The Uniform Exact-Conversion Theorem The proof of the main theorem rests on a uniform exact-conversion theorem, which states that any finite-output stopping policy based on one experiment can be reproduced exactly using another The reproducing F-policy’s expected sample size satisfies E F i N ≤ ρ i(F → G)E G i T + CF,GqE G iT If every pairwise KL inequality is strict, then ρi(F → G) < 1 for every state; the resulting linear saving eventually dominates the square-root remainder Comparison with Fixed-Sample Orders The paper compares its endogenous-stopping orders with fixed-sample orders like Moscarini–Smith and Mu–Pomatto–Strack–Tamuz, showing that for KL-generic pairs, the MPST order implies the stopping order The distinction lies in the resource constraint: a deterministic horizon assigns the same number of observations to every sample path, whereas an expected-sample budget allows a policy to stop early on easy histories and continue on difficult ones One experiment yields at least as much value in every decision problem once observations are sufficiently cheap if and only if, under sufficiently large expected-sample budgets, it can reproduce every terminal experiment attainable from the other source Both comparisons are characterized by the same condition: the experiment has a strictly larger directed pairwise KL divergence for every ordered pair of states The quantities Iij (E) are the rates at which a source produces evidence for state i against state j, and ρi(F → G) is the resulting first-order conversion factor under state i Pruning redundant reasoning tokens The paper discusses connections to four literatures including comparison of sequential experiments, large-sample orders for repeated experiments, endogenous information acquisition, and sequential testing and experimental design The authors show that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders How the training signal is built The proof of Theorem 2 involves a uniform exact-conversion theorem which decomposes the expected source sample requirement into a first-order conversion term and a square-root remainder The key input is a uniform exact-conversion theorem Where the gains come from The gains come from the flexibility of endogenous stopping, which allows sampling effort to be shifted across histories: easy histories can terminate early while difficult histories receive more observations Under an expected-sample budget, what survives to first order is therefore the mean rate of evidence accumulation The exact-conversion theorem shows that this flexibility reduces the exact-production constraints to the directed pairwise KL divergences The final result is that E F i N ≤ ρ i(F → G)E G i T + CF,GqE G iT Where the gains come from The authors show that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders The key input is a uniform exact-conversion theorem Pruning redundant reasoning tokens
Paper summary: Iris: To summarize, the paper "Ranking Experiments under Sequential Sampling" compares statistical experiments when observations are inexpensive and acquired sequentially until a decision maker chooses to stop.
Ruth: The thesis is that they introduce two orders, small-cost decision dominance and large-budget stopping dominance, and they show that for generic pairs of experiments, these two orders actually coincide.
Warren: They characterize both comparisons by stating that there's a strict dominance of every single pairwise Kullback–Leibler divergence across all states.
Iris: The key step in proving this is a uniform exact-conversion theorem, which basically means any finite output stopping policy from one experiment can be perfectly reproduced using another.
Ruth: That conversion involves determining the first-order expected sample requirements based on those pairwise KL rates plus a square root remainder that is uniform across all policies.
Warren: This square root remainder is what accounts for the flexibility of endogenous stopping, letting you stop early on easy histories while continuing on difficult ones under an expected sample budget.
Iris: So, what they’re saying is that if those pairwise KL inequalities are strict, then this resulting linear saving eventually dominates the square root remainder term.
Ruth: This leads to the main result where one experiment yields at least as much value in every decision problem once observations are sufficiently cheap if and only if it can reproduce every terminal experiment attainable from the other under large expected sample budgets.
Warren: It ties this comparison back to things like sequential testing and experimental design, showing that with repeated sampling and endogenous stopping, the strict versions of Blackwell sufficiency generically characterize both operational orders.
Iris: They also compare these endogenous-stopping orders against fixed-sample orders, finding that for KL-generic pairs, the Moscarini–Smith order implies this stopping order.
Ruth: The main idea is that both comparisons are ultimately characterized by the same condition: the experiment has a strictly larger directed pairwise KL divergence for every ordered pair of states.
Warren: The quantities Iij E represent the rates at which a source produces evidence for state i against state j, and rho i F to G is that resulting first-order conversion factor under state i.
Iris: This gives us a strong statement about how the structure of information accumulation determines the relative quality of two experimental designs under sequential sampling.
Conclusion: Ruth: So, looking at "Ranking Experiments under Sequential Sampling," we see that the core finding is a convergence between two seemingly different ways of comparing experiments under sequential data acquisition.
Warren: The authors show that small-cost decision dominance and large-budget stopping dominance have the same generic answer for most pairs of experiments.
Iris: This convergence is driven by the strict dominance of every pairwise Kullback–Leibler divergence, which is a measure of how fast an experiment accumulates evidence against another state.
Ruth: It means that if you can afford enough data under a large expected sample budget, one experiment will be better than or equal to the other in every decision problem.
Warren: And this holds true because of the uniform exact-conversion theorem, which precisely quantifies the expected sample size needed for reproduction.
Iris: The implication is that for sequential sampling, you don't need to worry about complex stopping rules if you can afford a budget large enough to let the fundamental divergence rates do their work.
Ruth: It’s a statement about the power of information accumulation under sequential constraints, showing how flexibility in stopping rules translates directly into evidence accumulation rates.
Warren: This framework gives us a solid way to think about what survives when you have limited but flexible sampling resources, and it connects this idea to other areas like repeated experiments.
Iris: The paper does rely on the assumption that these pairwise KL inequalities are strict for the generic pairs they compare, which is something we should keep in mind for applying this result.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19