Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps

arXiv:2605.27153 · cs.CY · Submitted 2026-05-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps".

Jane: Here's the gist of the paper. Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing <ref:2605.27153#pg2> EXATLAS is a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps <ref:2605.27153#pg2>.

Tom: The Core Framework EXATLAS is introduced as a framework for constructing an atlas from joint experimental evidence <ref:2605.27153#pg4>.

Jane: At the core of EXATLAS is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments <ref:2605.27153#pg5>.

Tom: This process yields three cases: if composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence <ref:2605.27153#pg5>.

Title and authors: Tom: Now let’s talk about who wrote this. The paper is "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps." We have Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, and Xiao Liu as the co-first authors >

Jane: It sounds like a team from different universities collaborating on this. The authors are from the University of Chicago, Rice University, and Yale University.

Lu: Yes, that’s right. They have a mix of representation learning folks and those working on literature understanding tools like LLMs >

Meng: I wonder what their specific expertise is in connecting all these different domains: experimental design, LLMs for literature understanding, and building this atlas framework >

Lalam: What they bring to the table is a way to organize the experimental archive into something that explicitly models how studies compose, conflict, or leave gaps >

Tom: So it’s not just one type of researcher doing this; it's an effort to bring together different skill sets—from the mechanics of model training to the high-level structure of scientific discovery >

Jane: And that’s what makes this kind of work so interesting for us. It shows that building these kinds of systems requires bringing together multiple specialized fields >

Lu: I think it highlights how far we’ve come with LLMs in generating research ideas and hypotheses, but the authors are pointing out that those methods often retrieve literature too shallowly, relying on keyword search or feeding titles as background >

Meng: So they’re suggesting a step up in complexity—moving from just retrieving existing ideas to explicitly modeling the relationships between them through this atlas >

Tom: That makes sense. It’s about moving from shallow retrieval to deeper structural understanding of the literature itself. We're moving toward making sure the AI understands how studies actually relate to each other, not just what words they contain >

Jane: And that structure is exactly what we need if we want to find those hidden conflicts and gaps that drive real scientific progress >

Lu: So this paper is about using these structural tools—like LLMs and graph stores—to build a more rigorous way to model the relationships among studies than just keyword searching allows >

Meng: It’s about getting a better signal from the literature, which is crucial when you’re trying to build something that needs to be accurate enough for production use >

Lalam: What they bring is this specific approach of constructing an experimental atlas that explicitly models how studies compose, conflict, or leave gaps >

The paper's summary: Tom: So let’s summarize what EXATLAS actually does in more detail. It takes a target study and searches for prior studies that are locally close in treatment and outcome space to see if their effects can be composed to predict the target effect >

Jane: So, if it finds a match, they check three things: success, disagreement, or failure of composition against the observed result >

Lu: If composition succeeds and agrees with the result, EXATLAS links the target to consistent prior evidence >

Meng: If composition succeeds but disagrees with the result, EXATLAS proposes candidate moderators or higher-level theories that could explain why they diverge despite being related >

Lalam: And if composition fails, EXATLAS suggests bridge experiments to close that gap between what we have and the target study >

Tom: It’s essentially a three-step decision process for any given target: link, reconcile, or bridge. That’s pretty concrete.

Jane: It shows how they systematically handle the three major problems in social science research: linking studies, resolving conflicts between findings, and finding missing pieces in the middle >

Lu: The paper is arguing that this systematic approach is a way to move past simply listing papers and start building something coherent >

Meng: So instead of just having a list of papers, we have a structured map where you can see exactly where the connections are strong and where they break down >

Tom: It moves us from just cataloging data to actually using that data to guide future research in a more intelligent way >

Jane: It’s about giving researchers and AI tools a way to see the latent structure hidden in the archive of experiments >

Lu: The paper is constructing an experimental atlas that explicitly models how studies compose, conflict, or leave gaps using these relations to guide scientific discovery >

The paper's improvements: Tom: So what are they saying are the improvements over existing literature? They aren't just doing a new kind of stuff; they’re focusing on how the atlas can be used to guide future work.

Jane: They focus on how this atlas can help researchers and AI tools move beyond shallow retrieval to deeper structural understanding of the literature >

Lu: The improvement is that instead of just keyword search, which only works on titles or abstracts, they’re using methods like chaining ideas across prior work to model those relationships more deeply >

Meng: So it’s about getting a better signal from the literature so that when you build something complex, the foundation is solid enough to handle real-world complexity >

Tom: And that makes sense. It suggests that if we can map out those structural relationships, we can use that map to guide future research in a more intelligent way >

Jane: So it’s about moving from finding isolated ideas to seeing how they connect into a coherent narrative of scientific development >

Conclusion: Tom: To wrap up this discussion on "Building an Atlas of Social Experiments to Link Studies, Reconcile Conflicts, and Bridge Gaps." The main implication is that this framework gives us a way to use experimental archives as a map instead of just a pile >

Jane: It suggests that an experimental archive supports a calibrated estimate of where its own evidence reaches >

Lu: If social science is to escape the incoherency problem, it needs infrastructure for reading the archive as a map rather than just piling papers >

Meng: From my side, I’m focused on the practicalities: running this at scale and what breaks when you try to deploy it in a real setting >

Lalam: And we have to remember that these generated ideas should be treated as candidate research directions rather than validated scientific claims because they might reflect biases in the LLM training data >

Tom: So we’ve discussed how EXATLAS works—from composition success to conflict reconciliation, and how it suggests bridge experiments when things don't line up perfectly >

Jane: It’s a tool for organizing all that experimental evidence so we can see the overall structure of what we know and where the gaps are >

Lu: The paper is about using these relations to guide scientific discovery based on this systematic approach to linking, reconciling, and bridging gaps >

Jiawei Zhang, Honglin Bao, Pengda Wang, Alex Yan, Xiao Liu, James A. Evans

University of Chicago · Rice University · Yale University

cs.CY

Submitted: 2026-05-26

Updated: 2026-10-29

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts…

Key concepts

EXATLAS Framework
A system designed to map experimental evidence by asking if a target experiment can be reconstructed from prior studies. If it can, it links them; if not, it suggests ways to reconcile conflicts or propose new experiments.
Composability Assessment
The process of checking how closely a new target experiment's data matches a weighted combination of existing experimental results. This determines if the target is close enough to be predicted by prior evidence.
Bridge Experiments
New experiments proposed by the AI when a target cannot be composed from existing studies. These are designed to close gaps in the literature, and human raters judge them based on feasibility and plausibility.

Terminology

Summary

Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing <ref:2605. p. 0

Where the paper says it

✉ Correspondence: liuxiao@uchicago. edu Abstract Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing.

27153#pg2> EXATLAS is a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps <ref:2605. p. 0
Where the paper says it

edu Abstract Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing. We introduce EXATLAS, a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps.

27153#pg2>.

The Core Framework

EXATLAS is introduced as a framework for constructing an atlas from joint experimental evidence <ref:2605.27153#pg4>. At the core of EXATLAS is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments <ref:2605. p. 0

Where the paper says it

We introduce EXATLAS, a framework for constructing such an atlas from joint experimental evidence. At the core of EXATLAS is a composition step that asks whether a target experiment can be reconstructed from prior observed experiments.

27153#pg5>. This process yields three cases: if composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence <ref:2605. p. 0
Where the paper says it

This yields three cases. If composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence.

27153#pg5>. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it <ref:2605. p. 0
Where the paper says it

If composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it.

27153#pg5>. If the composition fails, EXATLAS proposes bridge experiments to close the gap <ref:2605. p. 0
Where the paper says it

If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it. If the composition fails, EXATLAS proposes bridge experiments to close the gap.

27153#pg5>.

Representing Experimental Evidence

Most experiments in social and behavioral science test causal relations between treatments and outcomes; each experiment i is encoded as ei = (Ti, Oi, Zi, τi) <ref:2605.27153#pg10>. EXATLAS first uses an LLM to enrich these descriptions into context-aware descriptions <ref:2605. p. 2

Where the paper says it

Specifically, each experiment i is encoded as ei = (Ti , Oi , Zi , τi), where Ti is the treatment description, Oi is the outcome or dependent-variable description, Zi contains contextual information such as experiment settings, population, and control conditions, and τi is the observed treatment effect. Because treatments and outcomes are often described briefly in papers, EXATLAS first uses an LLM (OpenAI o3 model) to enrich them into context-aware descriptions, for instance, enriching "creativity" to "employee creativity in the organization" by reading the content of the original paper.

27153#pg10>. Experiments are then represented in a shared treatment–outcome space using the formula xi = emb(Tei) emb(Oei) emb(Tei)⊙emb(Oei). p. 2
Where the paper says it

Let Tei and Oei denote the enriched treatment and outcome descriptions. We define xi = emb(Tei) ∥ emb(Oei) ∥ emb(Tei)⊙emb(Oei) , where emb(·) denotes the text embedding function, ∥ denotes concatenation, and ⊙ denotes elementwise multiplication.

Assessing Composability and Error Bounds

Composability is assessed by checking if the target representation xt can be closely approximated by a weighted combination of prior experiment representations. The composition error decomposes into three terms: extrapolation, local model bias, and unobserved residual. p. 1

Where the paper says it

The method is rejectable by design. Theoretically, the composition error decomposes into three terms: extrapolation (how far the target sits from its reconstruction), local model bias (how curved the treatment-effect surface is in the neighborhood), and unobserved residual (variation in factors no embedding captures).

On held-out targets certified as locally supported, EXATLAS recovers effect direction in 98. p. 0
Where the paper says it

We provide an error bound for composition under local smoothness of the treatment-effect surface. On held-out targets certified as locally supported, EXATLAS recovers effect direction in 98.

6% of cases. p. 0
Where the paper says it

Given a target study, EXATLAS searches for prior studies that are locally close in treatment and outcome space and asks whether their observed effects can be composed to predict the target effect. This yields three cases.

Generating Theory and Experiments

When a target is composable but the composition disagrees with the observed effect, EXATLAS uses an LLM to generate a reconciliation: an explanation for why the source experiments and the target experiment may diverge despite being compositionally related. When a target cannot be composed, EXATLAS iteratively proposes bridge experiments until the target enters the composable region. p. 1

Where the paper says it

When a target cannot be composed, the gap itself is the finding. EXATLAS iteratively proposes bridge experiments until the target enters the composable region.

Human Evaluation of Outputs

Human raters judge EXATLAS’s bridge suggestions more plausible and more connected to the source experiments than forced composition of existing experiments. The EXATLAS Bridge proposed experiments are evaluated on Feasibility, Plausibility, and Connectedness. p. 6

Where the paper says it

Evaluation dimensions. Each experiment (contributing experiment in Forced Composition and connecting experiment in EXATLAS Bridge) is evaluated independently along three dimensions using a five-point Likert scale: Feasibility, Plausibility, and Connectedness.

Results show that EXATLAS Bridge performed better on Plausibility and Connectedness compared to Forced Composition. p. 7
Where the paper says it

In contrast, EXATLAS Bridge performed better on the other two dimensions. For Plausibility, EXATLAS Bridge received higher ratings than Forced Composition, 3.

Conclusion

EXATLAS is a rejectable framework for turning fragmented experimental archives into an atlas of links, conflicts, and gaps <ref:2605.27153#pg9>. It suggests that an experimental archive, read compositionally, supports a calibrated estimate of where its own evidence reaches. p. 7

Where the paper says it

We do not claim that LLMs replace human participants or that composition based on text embeddings is universally accurate. We claim something narrower but more useful: that an experimental archive, read compositionally, supports a calibrated estimate of where its own evidence reaches, where it falls short, and where those limits point to meaningful research directions.

If social science is to escape what Watts (2017) called the incoherency problem, it needs infrastructure for reading the archive as a map rather than a pile. p. 7
Where the paper says it

We claim something narrower but more useful: that an experimental archive, read compositionally, supports a calibrated estimate of where its own evidence reaches, where it falls short, and where those limits point to meaningful research directions. If social science is to escape what Watts (2017) called the incoherency problem, it needs infrastructure for reading the archive as a map rather than a pile.

Limitations

The strongest result applies only to locally supported targets. EXATLAS cannot prove from text alone that an embedding space is mechanism-preserving. p. 8

Where the paper says it

It should not be generalized to targets that require extrapolation beyond the archive. More fundamentally, EXATLAS cannot prove from text alone that an embedding space is mechanism-preserving.

The archive is itself biased, and the meta-competing-theory reconciliation module is a generator, not a verifier. p. 8
Where the paper says it

Local composition inherits these biases; the atlas is dense where the literature is dense and silent where it is silent. The meta-competing-theory reconciliation module is a generator, not a verifier.

Ethics Statement

The research ideas, bridge experiments, and theoretical reconciliations generated by LLMs may reflect biases present in their training data. They should therefore be treated as candidate research directions rather than validated scientific claims. p. 8

Where the paper says it

Ethics Statement The research ideas, bridge experiments, and theoretical reconciliations generated by LLMs may reflect biases present in their training data and may unintentionally resemble existing work without proper attribution. They should therefore be treated as candidate research directions rather than validated scientific claims, and should not be adopted for practical use without careful expert review, literature checking, and empirical validation.

Researchers should also consider whether proposed experiments involve sensitive populations, identity-related manipulations, or other ethical risks before implementation. p. 8
Where the paper says it

They should therefore be treated as candidate research directions rather than validated scientific claims, and should not be adopted for practical use without careful expert review, literature checking, and empirical validation. Because the framework operates on human-subject experimental evidence, researchers should also consider whether proposed experiments involve sensitive populations, identity-related manipulations, or other ethical risks before implementation.

References

Gati V. Aher, Rosa I. p. 8

Where the paper says it

References Gati V. Aher, Rosa I.

Arriaga, and Adam Tauman Kalai. p. 8
Where the paper says it

Aher, Rosa I. Arriaga, and Adam Tauman Kalai.

2023. Using large language models to simulate multiple humans and replicate human subject studies. p. 8
Where the paper says it

Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies.

Abdullah Almaatouq, Thomas L Griffiths, Jordan W Suchow, Mark E Whiting, James Evans, and Duncan J Watts. p. 8
Where the paper says it

In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 337–371. Abdullah Almaatouq, Thomas L Griffiths, Jordan W Suchow, Mark E Whiting, James Evans, and Duncan J Watts.

2024. Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences. p. 8
Where the paper says it

Abdullah Almaatouq, Thomas L Griffiths, Jordan W Suchow, Mark E Whiting, James Evans, and Duncan J Watts. Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences.

Teresa M. Amabile. 1982. Social psychology of creativity: A consensual assessment technique. p. 8
Where the paper says it

Behavioral and Brain Sciences, 47:e33. Social psychology of creativity: A consensual assessment technique.

Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. p. 8
Where the paper says it

Journal of Personality and Social Psychology, 43(5):997–1013. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang.

2025. Researchagent: Iterative research idea generation over scientific literature with large language models. p. 8
Where the paper says it

Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models.

Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Élteto, and 1 others. p. 8
Where the paper says it

In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6709–6738. Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Élteto, and 1 others.

2025. A foundation model˝ to predict and capture human cognition. p. 8
Where the paper says it

Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Élteto, and 1 others. A foundation model ˝ to predict and capture human cognition.

Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. p. 8
Where the paper says it

Nature, 644(8078):1002–1009. Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich.

2025. Specializing large language models to simulate survey response distributions for global populations. p. 8
Where the paper says it

Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. Specializing large language models to simulate survey response distributions for global populations.

Ziyan Cui, Ning Li, and Huaikang Zhou. p. 8
Where the paper says it

In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3141–3154. Ziyan Cui, Ning Li, and Huaikang Zhou.

2025. A largescale replication of scenario-based experiments in psychology and management using large language models. p. 8
Where the paper says it

Ziyan Cui, Ning Li, and Huaikang Zhou. A largescale replication of scenario-based experiments in psychology and management using large language models.

Douglas L. Dean, Jillian M. p. 8
Where the paper says it

Nature Computational Science, 5:627–634. Dean, Jillian M.

Hender, Thomas L. p. 8
Where the paper says it

Dean, Jillian M. Hender, Thomas L.

Rodgers, and Eric L. p. 8
Where the paper says it

Hender, Thomas L. Rodgers, and Eric L.

Santanen. 2006. Identifying quality, novel, and creative ideas: Constructs and scales for idea evaluation. p. 8
Where the paper says it

Rodgers, and Eric L. Identifying quality, novel, and creative ideas: Constructs and scales for idea evaluation.

Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. p. 8
Where the paper says it

Journal of the Association for Information Systems, 7(10). Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray.

2023. Can ai language models replace human participants? p. 8
Where the paper says it

Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants?

Trends in Cognitive Sciences. p. 8
Where the paper says it

Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600.

Yuwei Guo, Zihan Zhao, Xiaowei Liu, Xiangning Yu, and Deyu Zhou. p. 8
Where the paper says it

Trends in Cognitive Sciences, 27(7):597–600. Yuwei Guo, Zihan Zhao, Xiaowei Liu, Xiangning Yu, and Deyu Zhou.

2026. From script to stage: Automating experimental design for social simulations with llms. p. 8
Where the paper says it

Yuwei Guo, Zihan Zhao, Xiaowei Liu, Xiangning Yu, and Deyu Zhou. From script to stage: Automating experimental design for social simulations with llms.

Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. p. 9
Where the paper says it

Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. Predicting results of social science experiments using large language models.

2024. Predicting results of social science experiments using large language models. p. 9
Where the paper says it

Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. Predicting results of social science experiments using large language models.

John J. Horton, Apostolos Filippas, and Benjamin S. p. 9
Where the paper says it

Preprint, arXiv:2404. Horton, Apostolos Filippas, and Benjamin S.

Manning. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? p. 9
Where the paper says it

Horton, Apostolos Filippas, and Benjamin S. Large language models as simulated economic agents: What can we learn from homo silicus?

Working Paper 31122, National Bureau of Economic Research. p. 9
Where the paper says it

Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research.

Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu Zhang, Zixiao Wang, Paul Pu Liang, Daniel Khashabi, Lauren M Gardner, and Tianxing He. p. 9
Where the paper says it

Working Paper 31122, National Bureau of Economic Research. Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu Zhang, Zixiao Wang, Paul Pu Liang, Daniel Khashabi, Lauren M Gardner, and Tianxing He.

2025. Can a society of generative agents simulate human behavior and inform public health policy? p. 9
Where the paper says it

Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu Zhang, Zixiao Wang, Paul Pu Liang, Daniel Khashabi, Lauren M Gardner, and Tianxing He. Can a society of generative agents simulate human behavior and inform public health policy?

a case study on vaccine hesitancy. p. 9
Where the paper says it

Can a society of generative agents simulate human behavior and inform public health policy? a case study on vaccine hesitancy.

David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. p. 9
Where the paper says it

In Second Conference on Language Modeling. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A.

Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. p. 9
Where the paper says it

Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser.

2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. p. 9
Where the paper says it

Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.

Thomas Samuel Kuhn. p. 9
Where the paper says it

Association for Computational Linguistics. Thomas Samuel Kuhn.

1962. The Structure of Scientific Revolutions. p. 9
Where the paper says it

Thomas Samuel Kuhn. The Structure of Scientific Revolutions.

Imre Lakatos. p. 9
Where the paper says it

University of Chicago Press, Chicago. Imre Lakatos.

1978. The methodology of scientific research programmes: Volume 1: Philosophical papers, volume 1. p. 9
Where the paper says it

Imre Lakatos. The methodology of scientific research programmes: Volume 1: Philosophical papers, volume 1.

Sun Young Lee, Marko Pitesa, Stefan Thau, and Madan M Pillutla. p. 9
Where the paper says it

Cambridge university press. Sun Young Lee, Marko Pitesa, Stefan Thau, and Madan M Pillutla.

2015. Discrimination in selection decisions: Integrating stereotype fit and interdependence theories. p. 9
Where the paper says it

Sun Young Lee, Marko Pitesa, Stefan Thau, and Madan M Pillutla. Discrimination in selection decisions: Integrating stereotype fit and interdependence theories.

Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli Zhao, Tian Feng, and Lidong Bing. p. 9
Where the paper says it

Academy of Management Journal, 58(3):789–812. Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli Zhao, Tian Feng, and Lidong Bing.

2025a. Chain of ideas: Revolutionizing research via novel idea development with LLM agents. p. 9
Where the paper says it

Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli Zhao, Tian Feng, and Lidong Bing. Chain of ideas: Revolutionizing research via novel idea development with LLM agents.

Yu Li, Lehui Li, Lin Chen, Qingmin Liao, Fengli Xu, and Yong Li. p. 9
Where the paper says it

Association for Computational Linguistics. Yu Li, Lehui Li, Lin Chen, Qingmin Liao, Fengli Xu, and Yong Li.

2025b. Agentexpt: Automating ai experiment design with llm-based resource retrieval agent. p. 9
Where the paper says it

Yu Li, Lehui Li, Lin Chen, Qingmin Liao, Fengli Xu, and Yong Li. Agentexpt: Automating ai experiment design with llm-based resource retrieval agent.

Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, Thomas Pfeiffer, Emotion Expression Collaboration, and 1 others. p. 9
Where the paper says it

arXiv preprint arXiv:2511. Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, Thomas Pfeiffer, Emotion Expression Collaboration, and 1 others.

2024. Can large language models help predict results from a complex behavioural science study. p. 9
Where the paper says it

Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, Thomas Pfeiffer, Emotion Expression Collaboration, and 1 others. Can large language models help predict results from a complex behavioural science study?

Xiao Liu, Haokun Liu, and Chenhao Tan. p. 9
Where the paper says it

Royal Society Open Science, 11(9). Xiao Liu, Haokun Liu, and Chenhao Tan.

2026. Heuristic-based ideation for guiding llms toward structured creativity. p. 9
Where the paper says it

Xiao Liu, Haokun Liu, and Chenhao Tan. Heuristic-based ideation for guiding llms toward structured creativity.

Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. p. 9
Where the paper says it

io/2026/blog/2026/ideationheuristics/. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune.

2026. Towards end-to-end automation of ai research. p. 9
Where the paper says it

Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of ai research.

Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. p. 9
Where the paper says it

Nature, 651(8107):914–919. Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen.

2025. Large language models do not simulate human psychology. p. 9
Where the paper says it

Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, and Benjamin Paaßen. Large language models do not simulate human psychology.

Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. p. 9
Where the paper says it

arXiv preprint arXiv:2508. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto.

2025. Can llms generate novel research ideas? p. 9
Where the paper says it

Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas?

a large-scale human study with 100+ nlp researchers. p. 9
Where the paper says it

Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.

A Details of the Theoretical Justification

The theoretical error bound shows that composition is effective when the target experiment is close to its composed approximation, and the treatment-effect surface is smooth in the neighborhood where composition occurs. The error bound decomposes into extrapolation error, local curvature error, and residual error from unobserved factors not captured by the latent representation. p. 4

Where the paper says it

The second term is local curvature error: composition is more accurate when the treatment-effect function is approximately linear in the local neighborhood. The third term is residual error from unobserved factors not captured by the latent representation and bounds the floor on composition error.

B Implementation Details of EXATLAS

The composition assessment uses sentence-transformers/all-mpnet-base-v2 for embeddings. EXATLAS solves the constrained optimization problem to find reconstruction weights. p. 3

Where the paper says it

EXATLAS instead forms a local candidate set of up to K nearby experiments and solves the constrained optimization problem: α ⋆ t = arg min α rt(α), subject to αtj ≥ 0, X j∈C(t) αtj = 1. 3 Estimating the Treatment Effect and Linking Consistent Experiments If a target experiment et is composable, EXATLAS estimates its treatment effect by applying the same weights to the observed effects of the source experiments: τˆt = P j∈C(t) α ⋆ tjτj .

The threshold λ is determined by maximizing (1−scaled MSE)×Coverage to balance accuracy and coverage. p. 5
Where the paper says it

The threshold λ used in §4. 2 was selected to maximize prediction accuracy, so composable-with-mismatch cases, the Reconcile module’s input, are correspondingly scarce under this conservative setting (1/72).

C Evaluation Metrics for the Composition Experiment

Four metrics are used to evaluate predicted treatment effects: sign match, MSE, MAE, and Spearman’s ρ. EXATLAS achieves the strongest overall performance on predicting effects, matching the reported effect direction for 98. p. 5

Where the paper says it

Table 1 shows that EXATLAS achieves the strongest overall performance on predicting effects. It matches the reported effect direction for 98.

61% of targets. p. 0
Where the paper says it

We provide an error bound for composition under local smoothness of the treatment-effect surface. On held-out targets certified as locally supported, EXATLAS recovers effect direction in 98.

D Baselines for the Composition Experiment

Four baselines are compared: Direct Prediction, Nearest-Experiment RAG, Contributing-Experiment RAG, and Synthetic Participants. EXATLAS Composition achieves a SignMatch of 98. p. 5

Where the paper says it

5 Sonnet is used for the simulation baseline because it achieves the best performance in that study. Table 1 shows that EXATLAS achieves the strongest overall performance on predicting effects.

61%.

E Human Evaluation of Connecting Experiments

Human raters judge the proposed bridge experiments to be feasible, plausible, and well connected to the target literature. EXATLAS Bridge received higher ratings on Plausibility and Connectedness compared to Forced Composition. p. 7

Where the paper says it

In contrast, EXATLAS Bridge performed better on the other two dimensions. For Plausibility, EXATLAS Bridge received higher ratings than Forced Composition, 3.

F Prompts

The Bridge-Experiment Generation Prompt asks the model to propose bridge experiments that logically connect a target experiment to an existing body of literature. The Conflict-Reconciliation Prompt is used to probe whether a synthetic experimental finding contradicts prior literature and elicit a unified meta/competing-theoretical account. p. 6

Where the paper says it

Taken together, the cases illustrate Reconcile’s role across modes: surfacing a crosscell conflict in a hidden moderator structure that neither party would likely have located unaided, articulating the relevant prior literature more exhaustively than any single expert, and exposing a productive disagreement over what counts as a theoretical contribution. 3 Human Evaluation: Connecting Experiments We then conduct a human evaluation to assess whether the connecting experiments proposed by EXATLAS are useful research directions for targets that are not locally composable from the existing experimental archive.

The assumptions state that treatment assignment is randomized, and the latent mechanism representation captures relevant features for treatment-effect variation across experiments. The local smoothness assumption requires the treatment-effect function to be locally smooth around the target and its selected source experiments. p. 4

Where the paper says it

We assume that the treatment effect can be written as τi = µ(mi) + ϵ µ i , |ϵ µ i | ≤ δunobs, where µ is a smooth treatment-effect function, ϵ µ i captures residual variation due to factors not represented in mi , and δunobs bounds the size of this unobserved residual. The target experiment is locally composable if its latent representation can be approximated by a convex combination of source representations: mt = X j∈C(t) αtjmj+ϵt , αtj ≥ 0, X j∈C(t) αtj = 1.

A.2 Proof of the Error Bound

The error bound shows that composition is effective when two conditions hold: the target is locally composable from prior experiments, and the treatment-effect surface is smooth in the neighborhood where composition occurs. The residual term is bounded by 2δunobs. p. 4

Where the paper says it

Under local smoothness of the treatment-effect function µ, the error of composing source effects is bounded by |τt − τ comp t | ≤ L∥ϵt∥ + H 2 X j∈C(t) αtj∥mj − m¯ t∥ 2 + ϵ µ t − X j∈C(t) αtj ϵ µ j , where m¯ t = P j∈C(t) αtjmj . Each term is empirically diagnosable.

A.1 Assumptions

The assumptions state that random assignment makes treatment and control groups comparable, and the observed outcome satisfies consistency. The latent mechanism representation is used as an empirical proxy for the latent mechanism representation. p. 4

Where the paper says it

In practice, the latent representation mi is not directly available to EXATLAS. It uses the treatment– outcome feature representation xi as an empirical proxy for the latent mechanism representation.

A3 Local smoothness

The local smoothness assumption requires a local Lipschitz condition, µ(m) − µ(m′) ≤ Lm − m', and bounded local curvature, ∇2µ(m)op ≤ H.

A4 Local support

The target representation lies approximately in the convex hull of nearby source representations. The residual epsilon t measures the degree to which the target lies outside local source support. p. 4

Where the paper says it

Here ϵt is the latent reconstruction residual. Small ∥ϵt∥ means that the target lies close to the local structure induced by the source experiments.

A5 Representation validity

EXATLAS uses the text-based experiment representation xi, constructed from treatment and outcome descriptions, as an empirical proxy for the latent mechanism representation. The paper treats feature geometry as a candidate-retrieval and reconstruction device, not as a causal identification argument by itself. p. 8

Where the paper says it

More fundamentally, EXATLAS cannot prove from text alone that an embedding space is mechanism-preserving. The paper therefore treats feature geometry as a candidate-retrieval and reconstruction device, not as a causal identification argument by itself.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Reconstruct treatment effects from existing experiments to predict target effects with high accuracy, achieving 98.6% directional agreement on locally supported targets. This allows AI systems to generate calibrated estimates of where existing experimental evidence reaches and falls short, guiding future theory and experimentation based on the archive of social experiments contains more latent structure than current practice extracts.

  2. Automatically identify theoretical tensions by comparing a target effect with the composed estimate from nearby experiments, proposing candidate moderators or higher-level theories that explain the divergence. This capability allows AI to reconcile conflicts by generating a meta/competing-theory proposal when composition succeeds but disagrees, transforming disagreements into candidate theoretical explanations.

  3. Generate novel and empirically grounded bridge experiments to close gaps in the literature when a target cannot be composed from existing studies. This enables AI to propose connecting experiments that are judged by human raters as plausible and connected to the source literature, turning gaps in the archive into actionable experiment designs.

Abstract

Social and behavioral science runs thousands of experiments each year, yet their findings rarely accumulate into a coherent map of what is known, what conflicts, and what remains missing. We introduce EXATLAS, a framework for turning an archive of experiments into an atlas: a structured map in which studies link, conflict, or leave bridgeable gaps. Given a target study, EXATLAS searches for prior studies that are locally close in treatment and outcome space and asks whether their observed effects can be composed to predict the target effect. This yields three cases. If composition succeeds and agrees with the observed result, EXATLAS links the target to consistent prior evidence. If composition succeeds but disagrees, EXATLAS reconciles the conflict and proposes candidate moderators or higher-level theories that could explain it. If the composition fails, EXATLAS proposes bridge experiments to close the gap. We provide an error bound for composition under local smoothness of the treatment-effect surface. On held-out targets certified as locally supported, EXATLAS recovers the direction of the reported causal effect in 98% of cases across two experimental domains, with an average Spearman correlation of rho = 0.770. Human evaluations further suggest that its proposed bridge experiments are plausible and exhibit connectedness, and that its conflict explanations are useful for theory generation. These results suggest that the archive of social experiments contains more latent structure than current practice extracts, and that making this structure explicit can guide both future theory and future experimentation.

Sources

Related papers