A LATE-Targeting Experimental Design for Multi-Arm RCTs

summary

The gist

A choice-theoretic analysis unifies extensions of monotone compliance and motivates a two-stage design to identify local average treatment effects under imperfect compliance The Core Problem…

In short

Conventional multi-arm experiments fail to find specific treatment effects when there are three or more arms and compliance is strongly restricted. The study proposes a two-stage design: subjects report a preferred arm, and randomization occurs based on that report. This method restores one-sided noncompliance, allowing researchers to identify the effect between the reported arm and others for those who would comply.

Key concepts

Monotone Compliance Extensions
These are different ways to restrict how subjects can deviate from their initial treatment assignment. The paper examines various types, like ordered or strictly stable compliance. These restrictions limit the kinds of patterns that can be observed in experimental data, often leading to identification problems when many arms are involved.
Choice Model
This model describes how subjects make decisions by weighing the perceived benefits of an arm against a cost associated with deviating from their stated preference. This framework is used to understand the underlying economic incentives that drive compliance behavior in the proposed design.
One-Sided Noncompliance
This concept refers to a situation where subjects are only noncompliant if they deviate from the treatment they would have received under a specific alternative. In this study, by conditioning randomization on a reported preference, the design aims to isolate and identify the effect between two arms for those who would comply with one of them.
Wald Ratio
A Wald ratio is a statistical tool used in this context to calculate the effect of one treatment arm relative to every other alternative, specifically for subjects who report a certain preference. It helps quantify the margin-specific effect identified by the proposed design.

Terminology used across episodes

This episode discusses

The paper

A LATE-Targeting Experimental Design for Multi-Arm RCTs · Read on arXiv

Tetsuya Kaji

University of Chicago

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "A LATE-Targeting Experimental Design for Multi-Arm RCTs".

Warren: Here's the gist of the paper. We study the design of multi-arm experiments that identify local average treatment effects under imperfect compliance; this work is significant because conventional randomization fails to identify margin-specific effects even under strong extensions of monotone compliance The Core Problem and Proposed Solution Conventional multi-arm randomized controlled trials fail to identify margin-specific effects when there are three or more arms and strong extensions of monotone compliance are present The authors propose a two-stage design where subjects report a preferred arm, and randomization is conditional on this report, which restores one-sided noncompliance to identify the effect between the two for compliers This design leverages choice-theoretic analysis to motivate a protocol that aligns experimental design with subjects' incentives Compliance Extensions and Impossibility Results The paper reviews several extensions of monotone compliance, including ordered monotone compliance, stable and strictly stable compliance, and unordered monotone compliance The authors establish an impossibility result showing that with three or more arms and unrestricted potential outcomes, no nontrivial margin-specific group effect is identified under these combined restrictions This failure occurs because the data do not allow for the construction of two populations with the same observable distribution but different effects for that group The Choice Model and Identification Mechanism The authors develop a choice model where subjects weigh perceived benefits against a positive cost of deviation, which generates transitive compliance behavior The proposed protocol uses the survey report to tilt assignment probability toward the reported arm, ensuring subjects are incentivized to report a true favorite Under the restriction of compliance to stated preference, a Wald ratio identifies the effect of the reported arm relative to each alternative for subjects who report that arm but would accept the alternative Feasibility and Administrative Control The feasibility of compliance to stated preference depends on the administrator’s ability to mediate treatment switching, which can be achieved by reassigning a subject to their reported favorite if deviation costs are modeled appropriately The paper suggests that when subjects can obtain equivalent treatments elsewhere, the restriction must instead rest on a behavioral assumption like strict stability with the reported favorite as the common deviation destination The paper concludes that while strong compliance restrictions do not restore identification in conventional experiments, economic choice behavior suggests a design where a lottery tilted toward the reported arm elicits a favorite When treatment receipt is confined to the assigned arm or the reported favorite, conditioning on the report restores one-sided noncompliance, allowing an appropriate Wald ratio to identify the favorite’s effect relative to each alternative for subjects who would comply with the alternative The feasibility of compliance to stated preference depends on administrative control over treatment switching Appendices

Paper summary: Ruth: So, we're looking at this paper now called "A LATE-Targeting Experimental Design for Multi-Arm RCTs". It starts by saying that when you have three or more arms in a multi-arm experiment, you can't find those specific local average treatment effects if there's strong monotone compliance. Warren Right. Conventional randomization just doesn't work to isolate the effect for each individual arm when people are following the rules pretty strictly.

Iris: What it claims is that they use a choice-theoretic analysis to unify these different kinds of compliance extensions, like ordered or stable monotonicity, and this motivates a two-stage design protocol. Ruth So what does that two-stage design actually look like?

Warren: It suggests subjects first report which arm they prefer, and then the randomization happens conditional on that report. If we restrict treatment receipt to the reported arm or the assigned one, they restore one-sided noncompliance for those who are compliant with their report. Iris So you're using the reporting mechanism to identify the effect between two arms specifically for those individuals who would comply with that reported arm?

Ruth: Exactly. It says this design allows you to identify the effect between two arms for their compliers, provided they report correctly and we control treatment switching. Warren That sounds like a clever way to restore identification when the initial setup is messy because of monotone compliance issues.

Iris: The paper sets up an impossibility result showing that with three or more arms and unrestricted potential outcomes, no nontrivial margin-specific group effect can be identified even under these strong restrictions. Ruth So the core problem they're solving is that conventional methods fail to find those effects when you stack too many arms and people are highly compliant.

Warren: The impossibility result comes from the idea that the data just don't allow for constructing two groups with the same observable distribution but different effects for that group. Ruth That points to a fundamental limitation in what we can observe about these treatment effects when compliance is this rigid.

Iris: They build a choice model where subjects weigh perceived benefits against a positive cost of deviation, which creates transitive compliance behavior. Warren So the protocol uses the survey report to tilt the assignment probability toward the reported arm, incentivizing subjects to be honest about their true preference.

Ruth: That reporting mechanism is key because it aligns experimental design with what people actually want to do in terms of treatment selection under these constraints. Warren And when you restrict compliance just to stated preference, a Wald ratio then identifies the effect of that reported arm relative to each alternative for subjects who would actually accept the alternative.

Iris: The feasibility depends on the administrator being able to mediate treatment switching, which can happen if they reassign a subject to their reported favorite if deviation costs are modeled correctly. Ruth So for this design to work in the real world, we need some control over how subjects switch treatments based on that report.

Warren: The paper suggests when subjects can get equivalent treatments elsewhere, the restriction has to shift instead to a behavioral assumption like strict stability with the reported favorite as the common deviation destination. Iris That means if they can easily find another good treatment, then the focus needs to be on where they *would* deviate from their stated preference.

Ruth: So in summary, this paper shows that when you have multiple arms and strong compliance, a standard RCT fails to capture those specific effects because the data structure doesn't support it. Warren The proposed solution is this two-stage design: report your preference, then we randomize based on that report to isolate the effect for compliers.

Iris: They also discuss how this design applies in relation to choice models, which really ties the experimental design back into what we know about how people make decisions under incentive structures. Ruth That's important because it grounds the statistical identification mechanism in behavioral incentives rather than just abstract mathematical constraints.

Warren: It shows that if you can control the deviation pathway—whether by reassigning or by modeling stability—you can get identification back for those specific group effects. Iris The main limitation they point out is that this feasibility relies heavily on the administrator's ability to manage treatment switching in a way that aligns with their chosen compliance model.

Ruth: That’s the practical hurdle. So, while this design offers a way to identify those margin-specific effects under imperfect compliance, it's contingent on how well we can manage those deviation costs in practice. Warren It’s less about the math and more about the logistics of getting people to stick with what they report if they can get something better elsewhere.

Iris: The paper ends by showing that this conditional randomization allows for an appropriate Wald ratio to identify the favorite's effect relative to other alternatives for subjects who would comply with that alternative. Ruth So, the implication is a new experimental blueprint where we use self-reported preference as a structured way to perform analysis when compliance isn't perfect across many arms.

Conclusion: Ruth: So, we've been digging into this paper called "A LATE-Targeting Experimental Design for Multi-Arm RCTs." It’s about how to set up experiments with three or more treatment arms when people aren't following the rules perfectly—that monotone compliance problem.

Warren: Right. The authors are showing that conventional randomization just doesn't work anymore if you stack too many arms and compliance is strong. They propose a two-stage design where subjects report what they prefer first, and then we randomize based on that report to actually find those specific local average treatment effects for the compliers.

Iris: I want to point out that before we get into the magic of the design, there's this impossibility result they have. They show that with three or more arms and no extra assumptions, you just can't identify a real group effect because the data structure doesn't allow it.

Ruth: That’s what’s interesting. It sets up these hard limits for what we can observe when compliance is that rigid. So, how does this new design actually work to get around those limits?

Warren: The trick is using that choice model where subjects weigh the benefits against a cost of deviation, which creates this transitive compliance behavior. Then they use the survey report to tilt the assignment probability toward the reported arm, making sure people report what they would actually do if they were honest.

Iris: And when you restrict compliance just to stated preference, their Wald ratio then lets you figure out how much better that reported arm is compared to others for those specific people who would have complied with it. That’s the identification mechanism.

Ruth: So, the big implication here is that if we can control treatment switching—if the administrator can actually honor those reports by reassigning subjects—we get a new way to analyze these messy multi-arm scenarios. It moves us from just seeing what happens randomly to designing experiments based on what people actually prefer or report.

Warren: Exactly. It changes the blueprint for how we set up these complex randomized controlled trials when we're dealing with imperfect human behavior instead of perfect compliance. Next up, let’s look at how feasible this is in the real world with those administrative controls in place.

More episodes

← Home