Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials

Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.

Tom: We're looking at the paper: "Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials".

Jane: Here's the gist of the paper. Large language models (LLMs) can safely improve the precision of randomized controlled trials (RCTs) when used to generate predictions as covariates in an RCT analysis The authors develop a pipeline for best leveraging LLM predictions in this context and apply it to three different case studies Pruning redundant reasoning tokens Large language models (LLMs) are increasingly used in statistical research and applications, but they are also notorious for unreliable or biased information The paper explores whether LLMs can be used to improve the precision of randomized controlled trials (RCTs) in a safe and rigorous way They find that these predictions can safely improve precision, particularly when the RCT lacks predictive covariates or contains covariates—such as text data—that are well-suited to LLMs How the training signal is built The method involves a three-stage process for obtaining useful LLM predictions First, question formulation involves asking the LLM to predict which observation is more likely to exhibit a specific quality Second, covariate construction converts these pairwise comparisons into a count for each observation of the number of times it was chosen over its partner Third, modeling and evaluation check the significance of the LLM-generated covariate in a regression model fit to the original RCT covariates plus the new LLM-generated covariate Where the gains come from The utility of LLM predictions varies across case studies In Case Study 1, where rectangular, quantitative covariate data is present, the method does little to improve precision However, in the third example where abstracts are used as covariates, our method substantially improves precision In one case, the improvement is equivalent to increasing the sample size by nearly 60% Discussion of effectiveness The proposed pipeline is effective in reducing the standard error of the treatment effect estimate in specific settings For Case Study 2, where a pretest covariate captures each student’s algebra ability prior to treatment, the LLM predictions are still useful even though this does not translate to a meaningful reduction in effective sample size For Case Study 3, an RCT about the impact of paper accessibility on citations, we can reduce variance significantly by including LLM predictions The ideal case study for our method is small, follows similar patterns as other studies in the same context, has few covariates, and includes LLM-friendly information such as texts or images Limitations and future direction The main advantage of using LLM predictions to improve precision here is that they can be incorporated in a safe and rigorous way while preserving the unbiasedness of the estimator If the LLM predictions are not independent of treatment assignment because they incorporate post-treatment information, these guarantees may not hold The authors do not believe gpt-4o-mini used post-treatment information to make its predictions for our first two case studies Users of this method will need to evaluate the risk of post-treatment information on a case-by-case basis The paper suggests that text or image-based RCTs are a particularly strong candidate because this method can leverage texts or images while traditional covariate adjustment methods cannot A Summary Statistics Case Study 1 covariates include demographic information along with data on past arrests and convictions for 1,003 defendants arrested on drug-related charges in the District of Columbia in 2002 and 2003 Case Study 2 includes demographic covariates for 19,053 students in addition to each student’s standardized score on an algebra readiness exam administered before the experiment Case Study 3 involves four original RCT covariates: log number of authors, log number of pages, whether the paper was a review paper or not, and whether the paper was self-archived or not Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and Table 3 present the estimator standard errors by journal for Case Study 3 Table 4 shows summary statistics of covariates for Case Study 1 Table 5 shows summary statistics of covariates for Case Study 2 Table 6 and Table 7 display the regression results for Case Study 1 and Case Study 2, respectively Table 2 and

Paper summary: Jane: So wrapping up what we've heard about "Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials," the core idea is that LLMs can safely improve the precision of RCT results, especially when dealing with non-standard covariates like text.

Tom: The authors developed a pipeline for generating those LLM predictions and applying them as covariates to an RCT analysis, finding that this method works well in specific settings.

Lu: They found that the method is effective in reducing the standard error of the treatment effect estimate in certain situations, particularly when we have text or image data involved.

Meng: The major caution they give is that if those LLM predictions aren't independent of the treatment assignment, meaning they use post-treatment information, then those guarantees about unbiasedness might not hold up.

Lalam: So the final thought is that users need to carefully evaluate the risk of post-treatment information on a case-by-case basis before using this method.

Jane: It suggests that while LLMs are powerful for generating covariates, we still need rigorous checks to ensure the predictions aren't contaminated by information that should be unknown at the time of treatment assignment.

Tom: And ultimately, this paper shows a way to safely incorporate LLM predictions into an RCT analysis when the data structure is right, pushing us toward more flexible ways to handle complex covariates in causal inference.

Conclusion: Tom: So we’ve gone through how these large language models can actually help make randomized controlled trials more precise, and now Jane, what’s the actual takeaway here?

Jane: Well, at its heart this paper is about using LLMs to generate predictions that fit into the analysis of an RCT. The title itself is pretty straightforward—leveraging those LLMs to improve precision in those trials. It’s taking something messy, like text data or other non-standard information, and turning it into a way to better estimate the treatment effect.

Tom: I see. So instead of just ignoring that text or that weird covariate, you feed it into the model as a predictor? What are the authors actually saying about how this works in practice?

Jane: They’re proposing two main ways to do it: first, using LLMs to turn text into numbers—like turning a paragraph about a student's background into something the math can use. Second, they leverage the general knowledge inside the LLM to predict what the outcome might be. The big thing is that they show this can safely improve precision without messing up how we calculate things.

Tom: Safe and rigorous, right? That’s what I need to hear when we talk about these models in research settings. But what’s the catch? What are they warning us about?

Jane: They absolutely do. The authors are really careful here about this part—if the LLM predictions aren't independent of who got the treatment, meaning it uses information that comes *after* the treatment happened, those safety guarantees break down. You have to check that post-treatment information isn't creeping in.

Tom: So it’s a high-risk, high-reward situation. If you use this method, what kind of data do you actually need to be happy with? Is there a specific type of dataset where this is going to shine?

Jane: They suggest that the ideal case study is one that’s small and simple—maybe an RCT with only a few covariates, but those covariates are things like text or images. Traditional methods often struggle when you have those kinds of inputs, so this seems designed for exactly those tricky situations.

Tom: It sounds like we’re looking at a tool that handles complexity better than traditional stats tools can manage alone. So what does this mean for the future of how we run these trials?

Jane: It means we might start seeing more sophisticated ways to handle unstructured data in causal inference, moving past just clean numerical inputs. It opens up new avenues for researchers who are dealing with real-world messy data instead of just perfect datasets.