Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach

arXiv:2611.16992 · q-fin.MF · Submitted 2026-11-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Media Trading with Censored Auction Prices".

Warren: Here's the gist of the paper. Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach is a framework that models real-time bidding as a dynamic resource allocation problem under partial information where market prices are only revealed upon successful transactions, which is crucial for balancing performance, margins, and budget constraints The resulting approach combines stochastic control methods with reinforcement learning techniques to construct an approximate solution for this partially observed control problem.

Ruth: Problem Formulation The agency faces a dynamic resource-allocation problem where the decision is not simply how much to bid on an individual impression but how bids determine campaign performance, transaction margins, and the depletion of the client’s budget over a campaign horizon.

Warren: The observation constraint changes the nature of optimization because a high bid is more likely to generate an impression and reveal the clearing price but may also consume more of the available budget.

Ruth: The problem has two complementary aspects, one from modeling perspective and one from computational perspective.

Paper summary: Ruth: So to recap on "Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach," the central idea is modeling real-time bidding as a dynamic resource allocation problem under partial information > It focuses on how agencies must balance performance, margins, and budget constraints when you only see the price when you win a bid >

Warren: The main claim is that they formulate this using a continuous time stochastic control model with a latent price process to capture the missing market prices > And they show that by running exploratory subcampaigns under certain conditions, you can construct an estimator for that price signal with a mean square error of order N to the minus one >

Iris: The paper is organized in sections describing the RTB decision problem and a simple benchmark first, then it introduces the stochastic model and observation structure > Crucially, they discuss estimation and calibration in Section Four before presenting the HJB based approach and reinforcement learning methods in Sections Five and Six >

Ruth: What matters for us right now is that they’re combining these two different mathematical toolsets to solve what's fundamentally a partially observed control problem > It’s not just one neat trick, but a combination of stochastic control theory and modern deep learning techniques like physics informed neural networks >

Warren: The methodology section is dense, but it points out that the estimated price feedback they create is actually just a plug in strategy rather than solving the underlying partially observed control problem itself > That's an important caveat for how you interpret their results >

Iris: And looking at Section Seven where they discuss the numerical experiments, they admit their limitations and show where the estimation method stops working effectively > It’s a realistic look at what this framework can and cannot achieve in a real operational setting >

Ruth: So the takeaway here is that for an agency, this paper gives you a way to think about how to manage risk when you don't have perfect information about the market prices until it's too late > It shows how to build a strategy that adapts based on what’s observed so far rather than just relying on a fixed model >

Warren: It seems like they are trying to provide a bridge between the theoretical framework of stochastic control and the practical optimization power of reinforcement learning for this specific advertising context > And that’s where their work is situated in terms of its contribution to media trading research >

Iris: So moving forward, we need to keep watching how they handle those calibration steps because getting an accurate price signal from noisy auction data is the hardest part of making this whole thing work >

Conclusion: Ruth: Thinking about the title, "Media Trading with Censored Auction Prices," it really captures the essence of the challenge: you're trading media, but you only see the price after a successful trade, which is censored > The authors have put together a framework that handles that uncertainty by blending control theory and reinforcement learning >

Warren: I think what this paper means for us is that it gives us a structured way to approach this kind of resource allocation problem when you’re dealing with dynamic budgets and performance goals > It moves the conversation beyond just simple optimization under perfect information assumptions >

Iris: And the authors are essentially showing how to handle the estimation hurdle, which is often where these models fall apart in practice, by proposing methods like physics informed neural networks for calibration > That addresses a real weakness in applying control theory to empirical market data >

Ruth: So if you’re an agency head looking at this, it suggests that your strategy shouldn't just be about bidding higher or lower based on what you see right now, but about building a system that can reliably estimate the hidden price process while optimizing your long-term budget and performance goals >

Warren: And from a quantitative standpoint, it shows an alternative route to policy optimization when the dynamics aren't fully calibrated, using those actor critic methods as an option alongside the traditional Hamilton Jacobi Bellman iteration > It’s about finding a viable path when you can’t solve the whole thing perfectly >

Iris: The limitation they flag is that their estimated price feedback isn't actually the solution to the full problem, which means you still have to be careful how much you trust that signal > So, it's not a magic answer but a sophisticated tool for building a more robust control system >

Ruth: Exactly. It’s not about finding one perfect policy; it’s about creating a framework where the agency can make better decisions despite the fact that they only have partial information available at any given moment > That's what this paper is really offering us in terms of practical application >

Warren: So, to summarize, this work provides a mathematical language for understanding and tackling the trade-offs inherent in real-time media bidding where information about market prices is inherently incomplete > It’s a lot of theory put together to help solve a very messy business problem >

Iris: It’s an interesting blend of control theory and AI techniques applied directly to marketing economics, which is definitely worth paying attention to for anyone working in this space >

Yun-Ithry Gamrani, Huyên Pham

École Polytechnique · Siggmund Paris

q-fin.MF

Submitted: 2026-11-12

Updated: 2026-11-12

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach is a framework that models real-time bidding as a dynamic resource allocation problem under…

Key concepts

Stochastic Control Model
This mathematical framework is used to describe the dynamic decision-making process of the marketing agency over time. It accounts for randomness in market outcomes and budget constraints, helping to determine the best sequence of bids to maximize expected rewards.
Latent Price Process
The actual market price is not directly observable; it's hidden or 'latent.' The model treats this as a process that evolves over time, which must be inferred by the agency based on past successful bids and observed transaction outcomes.
Physics-Informed Neural Network (PINN)
A type of neural network used to help calibrate the operational model. It is trained not just on data but also incorporates physical laws or known mathematical relationships (like those governing price dynamics) to create more accurate predictions for the hidden price distributions.

Terminology

Summary

Media Trading with Censored Auction Prices: A Stochastic Control and Reinforcement Learning Approach is a framework that models real-time bidding as a dynamic resource allocation problem under partial information where market prices are only revealed upon successful transactions, which is crucial for balancing performance, margins, and budget constraints The resulting approach combines stochastic control methods with reinforcement learning techniques to construct an approximate solution for this partially observed control problem. p. 0

Where the paper says it

The framework clarifies the decisions and information trade-offs faced by the agency, while distinguishing full-information benchmarks and estimated-price policies from the genuinely partially observed control problem. Keywords: real-time bidding; stochastic control; operations research; partially observed decisions; reinforcement learning; budget allocation.

Problem Formulation

The agency faces a dynamic resource-allocation problem where the decision is not simply how much to bid on an individual impression but how bids determine campaign performance, transaction margins, and the depletion of the client’s budget over a campaign horizon. The observation constraint changes the nature of optimization because a high bid is more likely to generate an impression and reveal the clearing price but may also consume more of the available budget. p. 0

Where the paper says it

The observation constraint changes the nature of the optimization. A high bid is more likely to generate an impression and reveal the clearing price, but may also consume more of the available budget.

The problem has two complementary aspects, one from modeling perspective and one from computational perspective. p. 1
Where the paper says it

The problem has two complementary aspects. From a modeling perspective, one needs to represent the market price, campaign performance and remaining budget, together with an observation mechanism that depends on the bid.

Price Signal Construction

To construct a usable price signal, the authors consider parallel exploratory subcampaigns in which bids are randomized subject to an upper bound. Under a stylized cross-sectional sampling model, the resulting price estimator is consistent and satisfies a mean-square error bound of order N −1, where N is the number of exploratory observations. p. 0

Where the paper says it

To construct a price signal, we consider parallel exploratory subcampaigns in which bids are randomized subject to an upper bound. Under a stylized cross-sectional sampling model, the resulting price estimator is consistent and satisfies a mean-square error bound of order N −1 , where N is the number of exploratory observations.

The empirical winning fraction can be inverted to estimate the hidden price using an inverse estimator derived from Lemma 4. p. 8
Where the paper says it

3) where ϕ is the standard Gaussian density. The empirical win rate can therefore be inverted to estimate the hidden price.

1.

Model Calibration and Estimation

The operational model requires two distinct estimation questions: inferring the current market price from censored auction outcomes and fitting the dynamics of prices and campaign performance to historical data. The forward equation admits a unique strong solution, and calibration involves controlling the residual of (4. p. 6

Where the paper says it

Under (H1)–(H2), equation (3. 1) admits a unique strong solution and, for every α ∈ AF , equation (3.

6) together with the discrepancy between observed and model-implied data. p. 1
Where the paper says it

They study keyword-level bid optimization in sponsored-search advertising using limited aggregate feedback, a Bayesian model of campaign metrics, and a bandit-based online optimization method. Their study includes simulations using real advertising data.

Physics-informed neural networks are used to approximate the forward densities and their derivatives in this calibration process. p. 1
Where the paper says it

We next connect this price signal to two computational approaches. The first combines forward-equation calibration with a full-information Hamilton–Jacobi–Bellman (HJB) benchmark, approximated through policy iteration and physics-informed neural networks (PINNs) [13].

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Stochastic control formulation for dynamic bidding allows AI systems to model intertemporal objectives and constraints explicitly, balancing campaign performance, transaction margins, and budget depletion in real-time bidding scenarios.

  2. Model-based control using Hamilton-Jacobi-Bellman equations with physics informed neural networks enables the construction of an estimated price signal by combining forward equation calibration with policy iteration.

  3. Entropy regularized actor critic methods allow AI systems to learn bidding policies directly from observable information, using a relaxed measure-valued control formulation and an entropy penalty to favor exploration.

  4. The paper proposes a price estimator SbN using randomized exploratory bids, which provides a consistent estimator with a mean-square error bound of order N minus 1 for the hidden market price S.

  5. Forward equation calibration, utilizing physics informed neural networks, allows AI systems to estimate unknown dynamics like market clearing prices and campaign performance coefficients from historical data by controlling residual equations.

  6. The improved system can implement a budget pacing heuristic based on a proportional integral derivative PID controller, which maintains expenditure close to a prescribed schedule without needing access to the latent market price.

Abstract

In real-time bidding, marketing agencies must balance campaign performance, transaction margins and the use of a finite advertising budget. A key difficulty is that market-clearing prices are typically observed only when bids are successful, so the information available for subsequent decisions depends on the bidding policy. We formulate this problem using a continuous-time stochastic control model with a latent price process. To construct a price signal, we consider parallel exploratory subcampaigns in which bids are randomized subject to an upper bound. Under a stylized cross-sectional sampling model, the resulting price estimator is consistent and satisfies a mean-square error bound of order N-1, where N is the number of exploratory observations. We then describe a model-based approach combining forward-equation calibration with Hamilton-Jacobi-Bellman policy iteration and a physics-informed neural network, alongside an entropy-regularized actor-critic approach based on observable information in a model-free setting. Synthetic experiments illustrate the estimator and the computational procedures. The framework clarifies the decision and information trade-offs faced by the agency, while distinguishing full-information benchmarks and estimated-price policies from the genuinely partially observed control problem.

Sources

Related papers