Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps".
Jane: Here's the gist of the paper. Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing <ref:2605.27153#pg2> EXATLAS is a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps <ref:2605.27153#pg2>.
Tom: The Core Framework EXATLAS is introduced as a framework for constructing an atlas from joint experimental evidence <ref:2605.27153#pg4>.
Jane: At the core of EXATLAS is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments <ref:2605.27153#pg5>.
Tom: This process yields three cases: if composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence <ref:2605.27153#pg5>.
Title and authors: Tom: Now let’s talk about who wrote this. The paper is "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps." We have Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, and Xiao Liu as the co-first authors >
Jane: It sounds like a team from different universities collaborating on this. The authors are from the University of Chicago, Rice University, and Yale University.
Lu: Yes, that’s right. They have a mix of representation learning folks and those working on literature understanding tools like LLMs >
Meng: I wonder what their specific expertise is in connecting all these different domains: experimental design, LLMs for literature understanding, and building this atlas framework >
Lalam: What they bring to the table is a way to organize the experimental archive into something that explicitly models how studies compose, conflict, or leave gaps >
Tom: So it’s not just one type of researcher doing this; it's an effort to bring together different skill sets—from the mechanics of model training to the high-level structure of scientific discovery >
Jane: And that’s what makes this kind of work so interesting for us. It shows that building these kinds of systems requires bringing together multiple specialized fields >
Lu: I think it highlights how far we’ve come with LLMs in generating research ideas and hypotheses, but the authors are pointing out that those methods often retrieve literature too shallowly, relying on keyword search or feeding titles as background >
Meng: So they’re suggesting a step up in complexity—moving from just retrieving existing ideas to explicitly modeling the relationships between them through this atlas >
Tom: That makes sense. It’s about moving from shallow retrieval to deeper structural understanding of the literature itself. We're moving toward making sure the AI understands how studies actually relate to each other, not just what words they contain >
Jane: And that structure is exactly what we need if we want to find those hidden conflicts and gaps that drive real scientific progress >
Lu: So this paper is about using these structural tools—like LLMs and graph stores—to build a more rigorous way to model the relationships among studies than just keyword searching allows >
Meng: It’s about getting a better signal from the literature, which is crucial when you’re trying to build something that needs to be accurate enough for production use >
Lalam: What they bring is this specific approach of constructing an experimental atlas that explicitly models how studies compose, conflict, or leave gaps >
The paper's summary: Tom: So let’s summarize what EXATLAS actually does in more detail. It takes a target study and searches for prior studies that are locally close in treatment and outcome space to see if their effects can be composed to predict the target effect >
Jane: So, if it finds a match, they check three things: success, disagreement, or failure of composition against the observed result >
Lu: If composition succeeds and agrees with the result, EXATLAS links the target to consistent prior evidence >
Meng: If composition succeeds but disagrees with the result, EXATLAS proposes candidate moderators or higher-level theories that could explain why they diverge despite being related >
Lalam: And if composition fails, EXATLAS suggests bridge experiments to close that gap between what we have and the target study >
Tom: It’s essentially a three-step decision process for any given target: link, reconcile, or bridge. That’s pretty concrete.
Jane: It shows how they systematically handle the three major problems in social science research: linking studies, resolving conflicts between findings, and finding missing pieces in the middle >
Lu: The paper is arguing that this systematic approach is a way to move past simply listing papers and start building something coherent >
Meng: So instead of just having a list of papers, we have a structured map where you can see exactly where the connections are strong and where they break down >
Tom: It moves us from just cataloging data to actually using that data to guide future research in a more intelligent way >
Jane: It’s about giving researchers and AI tools a way to see the latent structure hidden in the archive of experiments >
Lu: The paper is constructing an experimental atlas that explicitly models how studies compose, conflict, or leave gaps using these relations to guide scientific discovery >
The paper's improvements: Tom: So what are they saying are the improvements over existing literature? They aren't just doing a new kind of stuff; they’re focusing on how the atlas can be used to guide future work.
Jane: They focus on how this atlas can help researchers and AI tools move beyond shallow retrieval to deeper structural understanding of the literature >
Lu: The improvement is that instead of just keyword search, which only works on titles or abstracts, they’re using methods like chaining ideas across prior work to model those relationships more deeply >
Meng: So it’s about getting a better signal from the literature so that when you build something complex, the foundation is solid enough to handle real-world complexity >
Tom: And that makes sense. It suggests that if we can map out those structural relationships, we can use that map to guide future research in a more intelligent way >
Jane: So it’s about moving from finding isolated ideas to seeing how they connect into a coherent narrative of scientific development >
Conclusion: Tom: To wrap up this discussion on "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps." The main implication is that this framework gives us a way to use experimental archives as a map instead of just a pile >
Jane: It suggests that an experimental archive supports a calibrated estimate of where its own evidence reaches >
Lu: If social science is to escape the incoherency problem, it needs infrastructure for reading the archive as a map rather than just piling papers >
Meng: From my side, I’m focused on the practicalities: running this at scale and what breaks when you try to deploy it in a real setting >
Lalam: And we have to remember that these generated ideas should be treated as candidate research directions rather than validated scientific claims because they might reflect biases in the LLM training data >
Tom: So we’ve discussed how EXATLAS works—from composition success to conflict reconciliation, and how it suggests bridge experiments when things don't line up perfectly >
Jane: It’s a tool for organizing all that experimental evidence so we can see the overall structure of what we know and where the gaps are >
Lu: The paper is about using these relations to guide scientific discovery based on this systematic approach to linking, reconciling, and bridging gaps >
Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans
University of Chicago · Rice University · Yale University
cs.CY
Submitted: 2026-05-26
Updated: 2026-10-29
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts…
Key concepts
- EXATLAS Framework
- A system designed to map experimental evidence by asking if a target experiment can be reconstructed from prior studies. If it can, it links them; if not, it suggests ways to reconcile conflicts or propose new experiments.
- Composability Assessment
- The process of checking how closely a new target experiment's data matches a weighted combination of existing experimental results. This determines if the target is close enough to be predicted by prior evidence.
- Bridge Experiments
- New experiments proposed by the AI when a target cannot be composed from existing studies. These are designed to close gaps in the literature, and human raters judge them based on feasibility and plausibility.
Terminology
Summary
Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing <ref:2605. p. 0 ✉ Correspondence: liuxiao@uchicago. edu
Abstract
Social and behavioral science runs thousands of
experiments each year, yet their findings rarely
accumulate into a coherent map of what is
known, what conflicts, and what remains missing. edu
Abstract
Social and behavioral science runs thousands of
experiments each year, yet their findings rarely
accumulate into a coherent map of what is
known, what conflicts, and what remains missing. We introduce EXATLAS, a framework for
turning an archive of experiments into an atlas:
a structured map in which studies link, conflict,
or leave bridgeable gaps.Where the paper says it
Where the paper says it
The Core Framework
EXATLAS is introduced as a framework for constructing an atlas from joint experimental evidence <ref:2605.27153#pg4>. At the core of EXATLAS is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments <ref:2605. p. 0 We introduce EXATLAS, a framework for constructing such an atlas from joint experimental evidence. At the core of EXATLAS is a composition step that asks whether a target experiment can
be reconstructed from prior observed experiments. This yields
three cases. If composition succeeds and agrees
with the observed result, EXATLAS links the
target to consistent prior evidence. If composition succeeds and agrees
with the observed result, EXATLAS links the
target to consistent prior evidence. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain
it. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain
it. If the composition fails, EXATLAS proposes
bridge experiments to close the gap.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Representing Experimental Evidence
Most experiments in social and behavioral science test causal relations between treatments and outcomes; each experiment i is encoded as ei = (Ti, Oi, Zi, τi) <ref:2605.27153#pg10>. EXATLAS first uses an LLM to enrich these descriptions into context-aware descriptions <ref:2605. p. 2 Specifically, each experiment i is encoded as
ei = (Ti
, Oi
, Zi
, τi),
where Ti
is the treatment description, Oi
is the
outcome or dependent-variable description, Zi contains contextual information such as experiment
settings, population, and control conditions, and τi
is the observed treatment effect. Because treatments and outcomes are often described briefly in papers, EXATLAS first uses an
LLM (OpenAI o3 model) to enrich them into
context-aware descriptions, for instance, enriching
"creativity" to "employee creativity in the organization" by reading the content of the original paper. Let Tei and Oei denote
the enriched treatment and outcome descriptions. We define
xi =
emb(Tei) ∥ emb(Oei) ∥ emb(Tei)⊙emb(Oei)
,
where emb(·) denotes the text embedding function,
∥ denotes concatenation, and ⊙ denotes elementwise multiplication.Where the paper says it
Where the paper says it
Assessing Composability and Error Bounds
Composability is assessed by checking if the target representation xt can be closely approximated by a weighted combination of prior experiment representations. The composition error decomposes into three terms: extrapolation, local model bias, and unobserved residual. p. 1 The method is rejectable by design. Theoretically, the composition error decomposes into three
terms: extrapolation (how far the target sits from its
reconstruction), local model bias (how curved the
treatment-effect surface is in the neighborhood),
and unobserved residual (variation in factors no
embedding captures). We provide an error bound for composition under local
smoothness of the treatment-effect surface. On
held-out targets certified as locally supported,
EXATLAS recovers effect direction in 98. Given a target study,
EXATLAS searches for prior studies that are locally close in treatment and outcome space and
asks whether their observed effects can be composed to predict the target effect. This yields
three cases.Where the paper says it
Where the paper says it
Where the paper says it
Generating Theory and Experiments
When a target is composable but the composition disagrees with the observed effect, EXATLAS uses an LLM to generate a reconciliation: an explanation for why the source experiments and the target experiment may diverge despite being compositionally related. When a target cannot be composed, EXATLAS iteratively proposes bridge experiments until the target enters the composable region. p. 1 When
a target cannot be composed, the gap itself is the
finding. EXATLAS iteratively proposes bridge experiments until the target enters the composable
region.Where the paper says it
Human Evaluation of Outputs
Human raters judge EXATLAS’s bridge suggestions more plausible and more connected to the source experiments than forced composition of existing experiments. The EXATLAS Bridge proposed experiments are evaluated on Feasibility, Plausibility, and Connectedness. p. 6 Evaluation dimensions. Each experiment (contributing experiment in Forced Composition and
connecting experiment in EXATLAS Bridge) is
evaluated independently along three dimensions
using a five-point Likert scale: Feasibility, Plausibility, and Connectedness. In contrast, EXATLAS
Bridge performed better on the other two dimensions. For Plausibility, EXATLAS Bridge received
higher ratings than Forced Composition, 3.Where the paper says it
Where the paper says it
Conclusion
EXATLAS is a rejectable framework for turning fragmented experimental archives into an atlas of links, conflicts, and gaps <ref:2605.27153#pg9>. It suggests that an experimental archive, read compositionally, supports a calibrated estimate of where its own evidence reaches. p. 7 We do not claim that LLMs replace human participants or that composition based on text embeddings is universally accurate. We claim something
narrower but more useful: that an experimental
archive, read compositionally, supports a calibrated
estimate of where its own evidence reaches, where
it falls short, and where those limits point to meaningful research directions. We claim something
narrower but more useful: that an experimental
archive, read compositionally, supports a calibrated
estimate of where its own evidence reaches, where
it falls short, and where those limits point to meaningful research directions. If social science is to
escape what Watts (2017) called the incoherency
problem, it needs infrastructure for reading the
archive as a map rather than a pile.Where the paper says it
Where the paper says it
Limitations
The strongest result applies only to locally supported targets. EXATLAS cannot prove from text alone that an embedding space is mechanism-preserving. p. 8 It should not be
generalized to targets that require extrapolation beyond the archive. More fundamentally, EXATLAS
cannot prove from text alone that an embedding
space is mechanism-preserving. Local composition inherits these biases;
the atlas is dense where the literature is dense and
silent where it is silent. The meta-competing-theory reconciliation
module is a generator, not a verifier.Where the paper says it
Where the paper says it
Ethics Statement
The research ideas, bridge experiments, and theoretical reconciliations generated by LLMs may reflect biases present in their training data. They should therefore be treated as candidate research directions rather than validated scientific claims. p. 8 Ethics Statement
The research ideas, bridge experiments, and theoretical reconciliations generated by LLMs may
reflect biases present in their training data and may
unintentionally resemble existing work without
proper attribution. They should therefore be treated
as candidate research directions rather than validated scientific claims, and should not be adopted
for practical use without careful expert review, literature checking, and empirical validation. They should therefore be treated
as candidate research directions rather than validated scientific claims, and should not be adopted
for practical use without careful expert review, literature checking, and empirical validation. Because the framework operates on human-subject
experimental evidence, researchers should also consider whether proposed experiments involve sensitive populations, identity-related manipulations,
or other ethical risks before implementation.Where the paper says it
Where the paper says it
References
Gati V. Aher, Rosa I. p. 8 References
Gati V. Aher, Rosa I. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In
Proceedings of the 40th International Conference on
Machine Learning, volume 202 of Proceedings of
Machine Learning Research, pages 337–371. Abdullah Almaatouq, Thomas L Griffiths, Jordan W Suchow, Mark E Whiting, James Evans, and Duncan J
Watts. Abdullah Almaatouq, Thomas L Griffiths, Jordan W Suchow, Mark E Whiting, James Evans, and Duncan J
Watts. Beyond playing 20 questions with nature: Integrative experiment design in the social and
behavioral sciences. Behavioral and Brain Sciences,
47:e33. Social psychology of creativity: A consensual assessment technique. Journal of
Personality and Social Psychology, 43(5):997–1013. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan,
and Sung Ju Hwang. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan,
and Sung Ju Hwang. Researchagent: Iterative
research idea generation over scientific literature with
large language models. In Proceedings of the 2025
Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long Papers), pages 6709–6738. Marcel Binz, Elif Akata, Matthias Bethge, Franziska
Brändle, Fred Callaway, Julian Coda-Forno, Peter
Dayan, Can Demircan, Maria K Eckstein, Noémi
Élteto, and 1 others. Marcel Binz, Elif Akata, Matthias Bethge, Franziska
Brändle, Fred Callaway, Julian Coda-Forno, Peter
Dayan, Can Demircan, Maria K Eckstein, Noémi
Élteto, and 1 others. A foundation model ˝
to predict and capture human cognition. Nature,
644(8078):1002–1009. Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. Specializing large language models to simulate survey response distributions for global populations. In
Proceedings of the 2025 Conference of the Nations of
the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies
(Volume 1: Long Papers), pages 3141–3154. Ziyan Cui, Ning Li, and Huaikang Zhou. Ziyan Cui, Ning Li, and Huaikang Zhou. A largescale replication of scenario-based experiments in
psychology and management using large language
models. Nature Computational Science, 5:627–634. Dean, Jillian M. Dean, Jillian M. Hender, Thomas L. Hender, Thomas L. Rodgers, and Eric L. Rodgers, and Eric L. Identifying
quality, novel, and creative ideas: Constructs and
scales for idea evaluation. Journal of the Association
for Information Systems, 7(10). Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. Trends in Cognitive Sciences, 27(7):597–600. Yuwei Guo, Zihan Zhao, Xiaowei Liu, Xiangning Yu,
and Deyu Zhou. Yuwei Guo, Zihan Zhao, Xiaowei Liu, Xiangning Yu,
and Deyu Zhou. From script to stage: Automating experimental design for social simulations
with llms. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae,
and Robb Willer. Predicting results of social
science experiments using large language models. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae,
and Robb Willer. Predicting results of social
science experiments using large language models. Preprint, arXiv:2404. Horton, Apostolos Filippas, and Benjamin S. Horton, Apostolos Filippas, and Benjamin S. Large language models as simulated
economic agents: What can we learn from homo
silicus? Large language models as simulated
economic agents: What can we learn from homo
silicus? Working Paper 31122, National Bureau of
Economic Research. Working Paper 31122, National Bureau of
Economic Research. Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu
Zhang, Zixiao Wang, Paul Pu Liang, Daniel
Khashabi, Lauren M Gardner, and Tianxing He. Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu
Zhang, Zixiao Wang, Paul Pu Liang, Daniel
Khashabi, Lauren M Gardner, and Tianxing He. Can a society of generative agents simulate human
behavior and inform public health policy? Can a society of generative agents simulate human
behavior and inform public health policy? a case
study on vaccine hesitancy. In Second Conference on
Language Modeling. Howcroft, Anya Belz, Miruna-Adriana
Clinciu, Dimitra Gkatzia, Sadid A. Howcroft, Anya Belz, Miruna-Adriana
Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad
Mahamood, Simon Mille, Emiel van Miltenburg,
Sashank Santhanam, and Verena Rieser. Hasan, Saad
Mahamood, Simon Mille, Emiel van Miltenburg,
Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: NLG
needs evaluation sheets and standardised definitions. Association for Computational Linguistics. Thomas Samuel Kuhn. Thomas Samuel Kuhn. The Structure of Scientific
Revolutions. University of Chicago Press, Chicago. Imre Lakatos. Imre Lakatos. The methodology of scientific research programmes: Volume 1: Philosophical papers, volume 1. Cambridge university press. Sun Young Lee, Marko Pitesa, Stefan Thau, and
Madan M Pillutla. Sun Young Lee, Marko Pitesa, Stefan Thau, and
Madan M Pillutla. Discrimination in selection
decisions: Integrating stereotype fit and interdependence theories. Academy of Management Journal,
58(3):789–812. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao,
Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli
Zhao, Tian Feng, and Lidong Bing. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao,
Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli
Zhao, Tian Feng, and Lidong Bing. Chain
of ideas: Revolutionizing research via novel idea
development with LLM agents. Association
for Computational Linguistics. Yu Li, Lehui Li, Lin Chen, Qingmin Liao, Fengli Xu,
and Yong Li. Yu Li, Lehui Li, Lin Chen, Qingmin Liao, Fengli Xu,
and Yong Li. Agentexpt: Automating ai experiment design with llm-based resource retrieval
agent. arXiv preprint arXiv:2511. Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis
Uhlmann, Thomas Pfeiffer, Emotion Expression
Collaboration, and 1 others. Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis
Uhlmann, Thomas Pfeiffer, Emotion Expression
Collaboration, and 1 others. Can large language models help predict results from a complex
behavioural science study? Royal Society Open Science, 11(9). Xiao Liu, Haokun Liu, and Chenhao Tan. Xiao Liu, Haokun Liu, and Chenhao Tan. Heuristic-based ideation for guiding llms toward structured creativity. io/2026/blog/2026/ideationheuristics/. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and
Jeff Clune. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and
Jeff Clune. Towards end-to-end automation of
ai research. Nature, 651(8107):914–919. Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. Large
language models do not simulate human psychology. arXiv preprint arXiv:2508. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? Can llms generate novel research ideas? a large-scale
human study with 100+ nlp researchers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A Details of the Theoretical Justification
The theoretical error bound shows that composition is effective when the target experiment is close to its composed approximation, and the treatment-effect surface is smooth in the neighborhood where composition occurs. The error bound decomposes into extrapolation error, local curvature error, and residual error from unobserved factors not captured by the latent representation. p. 4 The second term is local curvature error: composition is more accurate when
the treatment-effect function is approximately linear in the local neighborhood. The third term is
residual error from unobserved factors not captured
by the latent representation and bounds the floor on
composition error.Where the paper says it
B Implementation Details of EXATLAS
The composition assessment uses sentence-transformers/all-mpnet-base-v2 for embeddings. EXATLAS solves the constrained optimization problem to find reconstruction weights. p. 3 EXATLAS instead forms a local candidate
set of up to K nearby experiments and solves the
constrained optimization problem:
α
⋆
t = arg min
α
rt(α),
subject to αtj ≥ 0,
X
j∈C(t)
αtj = 1. 3 Estimating the Treatment Effect and
Linking Consistent Experiments
If a target experiment et
is composable, EXATLAS
estimates its treatment effect by applying the same
weights to the observed effects of the source experiments: τˆt =
P
j∈C(t) α
⋆
tjτj . The threshold λ used in §4. 2 was selected to maximize prediction accuracy, so composable-with-mismatch cases,
the Reconcile module’s input, are correspondingly
scarce under this conservative setting (1/72).Where the paper says it
Where the paper says it
C Evaluation Metrics for the Composition Experiment
Four metrics are used to evaluate predicted treatment effects: sign match, MSE, MAE, and Spearman’s ρ. EXATLAS achieves the strongest overall performance on predicting effects, matching the reported effect direction for 98. p. 5 Table 1 shows that EXATLAS achieves
the strongest overall performance on predicting
effects. It matches the reported effect direction
for 98. We provide an error bound for composition under local
smoothness of the treatment-effect surface. On
held-out targets certified as locally supported,
EXATLAS recovers effect direction in 98.Where the paper says it
Where the paper says it
D Baselines for the Composition Experiment
Four baselines are compared: Direct Prediction, Nearest-Experiment RAG, Contributing-Experiment RAG, and Synthetic Participants. EXATLAS Composition achieves a SignMatch of 98. p. 5 5 Sonnet is used for the simulation baseline because it
achieves the best performance in that study. Table 1 shows that EXATLAS achieves
the strongest overall performance on predicting
effects.Where the paper says it
E Human Evaluation of Connecting Experiments
Human raters judge the proposed bridge experiments to be feasible, plausible, and well connected to the target literature. EXATLAS Bridge received higher ratings on Plausibility and Connectedness compared to Forced Composition. p. 7 In contrast, EXATLAS
Bridge performed better on the other two dimensions. For Plausibility, EXATLAS Bridge received
higher ratings than Forced Composition, 3.Where the paper says it
F Prompts
The Bridge-Experiment Generation Prompt asks the model to propose bridge experiments that logically connect a target experiment to an existing body of literature. The Conflict-Reconciliation Prompt is used to probe whether a synthetic experimental finding contradicts prior literature and elicit a unified meta/competing-theoretical account. p. 6 Taken together, the cases illustrate
Reconcile’s role across modes: surfacing a crosscell conflict in a hidden moderator structure that
neither party would likely have located unaided,
articulating the relevant prior literature more exhaustively than any single expert, and exposing
a productive disagreement over what counts as a
theoretical contribution. 3 Human Evaluation: Connecting
Experiments
We then conduct a human evaluation to assess
whether the connecting experiments proposed by
EXATLAS are useful research directions for targets
that are not locally composable from the existing
experimental archive.Where the paper says it
The assumptions state that treatment assignment is randomized, and the latent mechanism representation captures relevant features for treatment-effect variation across experiments. The local smoothness assumption requires the treatment-effect function to be locally smooth around the target and its selected source experiments. p. 4 We
assume that the treatment effect can be written as
τi = µ(mi) + ϵ
µ
i
, |ϵ
µ
i
| ≤ δunobs,
where µ is a smooth treatment-effect function, ϵ
µ
i
captures residual variation due to factors not represented in mi
, and δunobs bounds the size of this
unobserved residual. The target experiment is locally composable if
its latent representation can be approximated by a
convex combination of source representations:
mt =
X
j∈C(t)
αtjmj+ϵt
, αtj ≥ 0,
X
j∈C(t)
αtj = 1.Where the paper says it
A.2 Proof of the Error Bound
The error bound shows that composition is effective when two conditions hold: the target is locally composable from prior experiments, and the treatment-effect surface is smooth in the neighborhood where composition occurs. The residual term is bounded by 2δunobs. p. 4 Under local
smoothness of the treatment-effect function µ, the
error of composing source effects is bounded by
|τt − τ
comp
t
| ≤ L∥ϵt∥ +
H
2
X
j∈C(t)
αtj∥mj − m¯ t∥
2
+
ϵ
µ
t −
X
j∈C(t)
αtj ϵ
µ
j
,
where m¯ t =
P
j∈C(t) αtjmj . Each term is empirically diagnosable.Where the paper says it
A.1 Assumptions
The assumptions state that random assignment makes treatment and control groups comparable, and the observed outcome satisfies consistency. The latent mechanism representation is used as an empirical proxy for the latent mechanism representation. p. 4 In practice, the latent representation mi
is not directly available to EXATLAS. It uses the treatment–
outcome feature representation xi as an empirical
proxy for the latent mechanism representation.Where the paper says it
A3 Local smoothness
The local smoothness assumption requires a local Lipschitz condition, µ(m) − µ(m′) ≤ Lm − m', and bounded local curvature, ∇2µ(m)op ≤ H.
A4 Local support
The target representation lies approximately in the convex hull of nearby source representations. The residual epsilon t measures the degree to which the target lies outside local source support. p. 4 Here ϵt
is the latent reconstruction residual. Small
∥ϵt∥ means that the target lies close to the local
structure induced by the source experiments.Where the paper says it
A5 Representation validity
EXATLAS uses the text-based experiment representation xi, constructed from treatment and outcome descriptions, as an empirical proxy for the latent mechanism representation. The paper treats feature geometry as a candidate-retrieval and reconstruction device, not as a causal identification argument by itself. p. 8 More fundamentally, EXATLAS
cannot prove from text alone that an embedding
space is mechanism-preserving. The paper therefore treats feature geometry as a candidate-retrieval
and reconstruction device, not as a causal identification argument by itself.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Reconstruct treatment effects from existing experiments to predict target effects with high accuracy, achieving 98.6% directional agreement on locally supported targets. This allows AI systems to generate calibrated estimates of where existing experimental evidence reaches and falls short, guiding future theory and experimentation based on
the archive of social experiments contains more latent structure than current practice extracts.
-
Automatically identify theoretical tensions by comparing a target effect with the composed estimate from nearby experiments, proposing candidate moderators or higher-level theories that explain the divergence. This capability allows AI to
reconcile conflicts
by generatinga meta/competing-theory proposal
when composition succeeds but disagrees, transforming disagreements intocandidate theoretical explanations.
-
Generate novel and empirically grounded bridge experiments to close gaps in the literature when a target cannot be composed from existing studies. This enables AI to propose connecting experiments that are judged by human raters as
plausible and connected to the source literature,
turninggaps in the archive into actionable experiment designs.
Abstract
Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing. We introduce EXATLAS, a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps. Given a target study, EXATLAS searches for prior studies that are locally close in treatment and outcome space and asks whether their observed effects can be composed to predict the target effect. This yields three cases. If composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it. If the composition fails, EXATLAS proposes bridge experiments to close the gap. We provide an error bound for composition under local smoothness of the treatment-effect surface. On held-out targets certified as locally supported, EXATLAS recovers the direction of the reported causal effect in 98% of cases across two experimental domains, with an average Spearman correlation of rho = 0.770. Human evaluations further suggest that its proposed bridge experiments are plausible and exhibit connectedness, and that its conflict explanations are useful for theory generation. These results suggest that the archive of social experiments contains more latent structure than current practice extracts, and that making this structure explicit can guide both future theory and future experimentation.
Sources
- From Script to Stage: Automating Experimental Design for Social Simulations with LLMs
- Automated Social Science: Language Models as Scientist and Subjects
- AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent
- Exploring the Limits of Large Language Models as a Reflection of Human Cognition: An Illustration in the Context of Moral Judgement
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance