Buildrix: An Open Agentic AI Benchmark for Building Engineering

arXiv:2606.25139 · eess.SY, cs.SY · Submitted 2026-06-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Buildrix: An Open Agentic AI Benchmark for Building Engineering".

Dev: Here's the gist of the paper. Agentic AI offers significant potential to automate complex building-engineering workflows Buildrix presents an open, community-driven platform for developing, sharing, executing, and evaluating agentic AI skills for building engineering.

Rosa: Platform Components Buildrix integrates three main components to facilitate its function The platform includes a Python command-line package for developing, validating, publishing, installing, and managing skills and test cases, a web-based Hub for organizing open challenges, reusable skills, test cases, reviews, and benchmark results, and a local agent harness that supports skill discovery, external toolchain provisioning, progressive context loading, and multi-step workflow execution.

Dev: Buildrix skills are organized as standardized, self-contained packages containing task instructions, executable scripts, dependencies, and supporting resources.

Rosa: Skill Development Framework The study introduces a modular skill-development and sharing framework for building engineering workflows A skill is the unit of reusable engineering capability, organized as a self-contained folder centered on SKILL.md.

Title and authors: Rosa: Now let's talk about the title and who wrote this paper, "Buildrix: An Open Agentic AI Benchmark for Building Engineering". It sets a very specific scope for what they’re building.

Dev: It’s clear that the focus here is moving past just single demos and aiming for something standardized that other people can actually use as a baseline.

Rosa: They’re calling it an open platform, which means the goal isn't to build some closed system where only their team can play with it, but to create a community resource.

Dev: That opens up the idea of sharing skills and challenges, which is a big shift from how most proof-of-concept projects operate in this field.

Rosa: The authors are Zixin Jianga and Bing Donga from Syracuse University's Department of Mechanical and Aerospace Engineering. Their background seems to give them a strong grounding in the physical aspects of building engineering.

Dev: That engineering background probably explains why they are so focused on creating standardized, self-contained skill packages rather than just giving people a general-purpose LLM prompt template.

Rosa: It feels like their goal is to provide the infrastructure—the platform—so that different AI teams can build specialized knowledge modules for building engineering and then plug them into this structure.

Dev: So, when we look at the authors, it suggests they are focused on bridging the gap between theoretical AI potential and practical application in a construction or building context.

The paper's summary: Rosa: Moving on to the summary of "Buildrix: An Open Agentic AI Benchmark for Building Engineering," it boils down to what they’re trying to fix.

Dev: They identify that existing applications are too isolated, lacking reusable domain capabilities, and crucially, they lack human-verified evaluation cases and a standardized benchmarking infrastructure.

Rosa: The core of Buildrix is integrating three parts: the development package for skills and test cases; the Hub for organizing challenges and results; and the local agent harness for running those tasks.

Dev: That harness is what allows agents to discover skills, get external tools, load context progressively, and execute multi-step workflows.

Rosa: The paper emphasizes that their skills are standardized packages containing instructions, scripts, dependencies, and resources so they become truly reusable domain capabilities.

Dev: And the key finding is that this structure allows for skill composition where the output of one skill feeds into the next to solve higher-level engineering tasks.

Rosa: It moves away from a single agent trying to plan everything at once toward an autonomous ecosystem coordinating multiple specialist agents and tools, which they call agentic AI.

Dev: So, it’s about moving from static frameworks like AutoGen or LangGraph where the topology is predefined, to more open-ended systems that can dynamically route messages and tool calls among different agents based on the situation.

Rosa: The summary really paints a picture of building this infrastructure—the platform—to support complex, interdependent subtasks in engineering applications.

The paper's improvements: Dev: They suggest several ways to improve the concept, focusing on making the development process more accessible and reliable for the community.

Rosa: One big suggestion is that they want a modular skill-development framework where users can develop, package, test, and publish their own agent skills directly into the Buildrix Hub.

Dev: This is about giving people ownership over creating those building engineering modules rather than just consuming pre-made ones from some central source.

Rosa: They also mention providing a mechanism for users to upload human-verified golden test cases that can then be promoted for reproducible benchmark evaluation against shared skills.

Dev: That verification step is essential because it’s how you ensure the benchmark scores are actually meaningful and not just based on whatever the agent outputted randomly.

Rosa: Another improvement is focusing on using different levels of autonomy depending on how risky a specific engineering task is, which makes sense for risk management.

Dev: And they argue that loop engineering is central to moving from isolated tool demonstrations toward achieving true end-to-end engineering autonomy.

Rosa: So, in short, the improvements are about creating a robust ecosystem where users can contribute reliable building knowledge and test cases and then measure their agents against it.

Conclusion: Dev: To wrap up, "Buildrix: An Open Agentic AI Benchmark for Building Engineering" successfully demonstrated that multiple skills can be effectively composed into an end-to-end workflow covering things like building stock generation and energy simulation.

Rosa: The main implication is that agentic AI can genuinely support these complex building-engineering workflows when you combine domain knowledge, executable tools, persistent workspaces, and verification mechanisms.

Dev: It’s not just about the AI being smart; it's about creating this entire infrastructure—the platform—that allows the necessary coordination to happen reliably in a practical engineering setting.

Taro: I think what’s most interesting is how they addressed the failure modes during that iterative perceive–reason–act–verify loop, because that shows you can actually build something that handles when the world misbehaves gracefully.

Rosa: I agree; seeing an agent recover from those failures really proves it’s more than just a theoretical exercise; it has to be able to cope with real-world messiness.

Dev: So, the main takeaway is that for building engineering, we need this kind of open infrastructure where you can build and rigorously test these interconnected agentic systems.

Taro: Yeah, so they built the whole loop necessary for autonomy; it's about making sure the system doesn't just stop when it hits an unexpected issue.

Department of Mechanical and Aerospace Engineering, Syracuse University

eess.SY, cs.SY

Submitted: 2026-06-23

Updated: 2026-11-11

Comments: 15 pages, 8 figures, 4 tables

Code: https://github.com/Bugs-Owner/buildrix

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 93/100

The gist: Agentic AI offers significant potential to automate complex building-engineering workflows Buildrix presents an open, community-driven platform for developing, sharing, executing…

Key concepts

Buildrix Skills
These are standardized packages representing reusable engineering capabilities. Each skill contains task instructions, executable scripts, dependencies, and resources. They allow for modular development where specific engineering tasks can be built from these self-contained units.
Agent Harness Loop
This is the core execution mechanism that allows agents to perform end-to-end tasks using a 'perceive–reason–act–verify' cycle. It enables agents to interpret goals, select tools, check results, and recover from failures through an iterative diagnose–modify–retry process.
Golden Test Cases
These are high-quality test cases verified by domain experts that contain human-verified numerical data and rubrics. They serve as the standard for reproducible benchmark evaluation against a skill's output to generate reliable performance scores.
Progressive Disclosure
This design principle limits the initial context requirement by hiding detailed skill instructions until they are needed. This prevents context bottlenecks when many skills are installed, ensuring agents only load necessary information at runtime for selected tasks.

Terminology

Summary

Agentic AI offers significant potential to automate complex building-engineering workflows Buildrix presents an open, community-driven platform for developing, sharing, executing, and evaluating agentic AI skills for building engineering. p. 0

Where the paper says it

Buildrix: An Open Platform for Sharing and Benchmarking Agentic AI Skills in Building Engineering Zixin Jianga , Bing Donga aDepartment of Mechanical and Aerospace Engineering, Syracuse University, Syracuse, NY, USA Abstract Agentic AI offers significant potential to automate complex building-engineering workflows. However, most existing applications remain isolated proof-of-concept demonstrations and lack reusable domain capabilities, human-verified evaluation cases, and standardized benchmarking infrastructure.

Platform Components

Buildrix integrates three main components to facilitate its function The platform includes a Python command-line package for developing, validating, publishing, installing, and managing skills and test cases, a web-based Hub for organizing open challenges, reusable skills, test cases, reviews, and benchmark results, and a local agent harness that supports skill discovery, external toolchain provisioning, progressive context loading, and multi-step workflow execution. Buildrix skills are organized as standardized, self-contained packages containing task instructions, executable scripts, dependencies, and supporting resources. p. 0

Where the paper says it

Buildrix integrates three components: a Python command-line package for developing, validating, publishing, installing, and managing skills and test cases; a web-based Hub for organizing open challenges, reusable skills, test cases, reviews, and benchmark results; and a local agent harness that supports skill discovery, external toolchain provisioning, progressive context loading, and multi-step workflow execution. Buildrix skills are organized as standardized, self-contained packages containing task instructions, executable scripts, dependencies, and supporting resources.

Skill Development Framework

The study introduces a modular skill-development and sharing framework for building engineering workflows A skill is the unit of reusable engineering capability, organized as a self-contained folder centered on SKILL.md. This mandatory file provides both discovery metadata and task-level instructions. p. 3

Where the paper says it

Following the open Agent Skills standard [26, 32], each skill is organized as a self-contained folder centered on SKILL. This mandatory file provides both discovery metadata and task-level instructions.

Skills are designed for progressive disclosure, where only short skill descriptions are exposed at startup, loading full instructions only when a relevant skill is selected. p. 6
Where the paper says it

buildrix/skills/, and links or exposes it to the selected agent’s skill directory so that the agent can discover it at runtime. Discovery follows the progressive-disclosure mechanism described above: the agent first reads only short skill descriptions and loads the full instructions, scripts, and references only when the user request matches a relevant skill.

Skills can also be composable, allowing the output of one skill to become the input of another to solve higher-level engineering tasks. p. 4
Where the paper says it

Skills are also composable. The output of one skill can become the input of another, allowing an agent to solve a higher-level engineering task by combining multiple specialized capabilities.

Evaluation and Benchmarking

Buildrix provides a standardized benchmark evaluation mechanism for agentic AI in building engineering Quantitative test cases can be verified by domain experts and promoted to golden test cases for reproducible benchmark evaluation. The benchmark runner executes an accepted skill against a golden test case in a controlled workspace to produce criterion scores that the Hub aggregates into a leaderboard. p. 6

Where the paper says it

Whereas the harness executes a skill to complete useful work, the benchmark runner executes a skill specifically to evaluate it. It runs an accepted skill against a golden test case in a controlled workspace, compares the generated outputs with the human-expert reference, and produces percriterion scores that the Hub aggregates into a leaderboard.

Only golden test cases are eligible for benchmark scoring because they contain human-verified numerical reference data and verifiable rubrics. p. 6
Where the paper says it

It runs an accepted skill against a golden test case in a controlled workspace, compares the generated outputs with the human-expert reference, and produces percriterion scores that the Hub aggregates into a leaderboard. Only golden test cases are eligible for benchmark scoring because only they contain human-verified numerical reference data and verifiable rubrics.

Workflow Execution and Resilience

The agent harness supports end-to-end autonomous building engineering tasks through an iterative perceive–reason–act–verify loop. The case study showed that the harness enabled agents to interpret user goals, select relevant skills, execute tools, inspect intermediate results, recover from errors, and generate verifiable outputs. p. 2

Where the paper says it

This study demonstrates how an agent harness, combined with customized building-domain skills, can execute realworld engineering workflows with limited human intervention. Through representative case studies, the paper shows how Buildrix enables agents to interpret user goals, select relevant skills, execute tools, inspect intermediate results, recover from errors, and generate verifiable outputs.

The system demonstrated resilience by handling 12 failed tool calls across five main failure categories through an iterative diagnose–modify–retry loop. p. 8
Where the paper says it

The execution path was not failure-free. The agent encountered 12 failed tool calls covering five main failure categories: data-schema mismatch, incorrect function usage, software-version incompatibility, invalid model parameters, and missing input dependencies.

Key Findings on Design Principles

The research emphasizes several design principles for building applications First, small is beautiful, suggesting that agentic AI architectures should be efficient and modular rather than scaling up complexity by default. Second, the platform prioritizes high-quality, reusable skills and test cases over skill quantity alone. p. 0

Where the paper says it

This study presents Buildrix, an open, community-driven platform for developing, sharing, executing, and evaluating agentic AI skills for building engineering. Buildrix integrates three components: a Python command-line package for developing, validating, publishing, installing, and managing skills and test cases; a web-based Hub for organizing open challenges, reusable skills, test cases, reviews, and benchmark results; and a local agent harness that supports skill discovery, external toolchain provisioning, progressive context loading, and multi-step workflow execution.

Third, the authors suggest that a practical architecture should use different levels of autonomy according to task risk. Finally, the paper argues that loop engineering is central to moving from isolated tool demonstrations toward end-to-end engineering autonomy. p. 1
Where the paper says it

A modular open-source framework is needed to allow researchers and practitioners to define engineering challenges, contribute reusable skills, share human-validated golden test cases, and benchmark agent performance in a transparent and extensible way. To address these gaps, this study presents Buildrix, an open platform for tackling real-world building-engineering problems through an agentic AI harness and communitydefined engineering tasks, reusable agent skills, golden test cases, and standardized benchmark evaluation, thereby moving agentic AI for buildings from isolated demonstrations toward scalable, verifiable, and community-driven engineering autonomy.

Conclusion and Future Direction

Buildrix successfully demonstrated that multiple skills could be composed into an end-to-end workflow covering building-stock generation, weather processing, heatwave identification, EnergyPlus simulation, and retrofit analysis. The results suggest that agentic AI can support complex building-engineering workflows when domain knowledge, executable tools, persistent workspaces, and verification mechanisms are combined. p. 2

Where the paper says it

This study demonstrates how an agent harness, combined with customized building-domain skills, can execute realworld engineering workflows with limited human intervention. Through representative case studies, the paper shows how Buildrix enables agents to interpret user goals, select relevant skills, execute tools, inspect intermediate results, recover from errors, and generate verifiable outputs.

Future work is planned to expand the skill library and golden test cases, evaluate additional models and harnesses, and validate Buildrix across larger and more diverse real-world building applications. p. 1
Where the paper says it

To address these gaps, this study presents Buildrix, an open platform for tackling real-world building-engineering problems through an agentic AI harness and communitydefined engineering tasks, reusable agent skills, golden test cases, and standardized benchmark evaluation, thereby moving agentic AI for buildings from isolated demonstrations toward scalable, verifiable, and community-driven engineering autonomy. Before diving into technical details, we briefly introduce two latest agentic AI concepts used in this study: agent harnesses and agent skills.

Appendix Details

The appendix details the complete command set of the Buildrix package The appendix also shows the structure of the Buildrix Hub, which is a community web platform hosting skills and challenges. The LLM Reviewer prompts are also provided in Appendix C. p. 2

Where the paper says it

4) that define task prompts, required deliverables, reference outputs, and evaluation criteria. After submission, skills and test cases are first checked by an LLM reviewer (details in Section 2.

References

[1] Wetter, M., & Sulzer, M. (2024). A call to action for building energy system modelling in the age of decarbonization. p. 1

Where the paper says it

From relatively static agent frameworks such as AutoGen [16] and LangGraph [17], which rely on predefined agent topologies while allowing messages, decisions, and tool calls to be dynamically routed among agents, to more open-ended systems such as AutoGPT [18] and Voyager [19], which allow the reasoning trajectory and, in some cases, the execution structure to be constructed at runtime, agentic AI is evolving toward greater autonomy, flexibility, and adaptability for complex, multi-round tasks. In the building domain, recent studies have begun to explore such multi-agent agentic AI systems for building energy system modeling and control [3], EnergyPlus automation [20, 21], and related engineering workflows [22].

Journal of Building Performance Simulation, 17(3), 383–393

[2] Blum, D., Wang, Z., Weyandt, C., Kim, D., Wetter, M., Hong, T., & Piette, M. A. (2022). Field demonstration and implementation analysis of model predictive control in an office HVAC system. p. 3

Where the paper says it

Contributors create and submit challenges, skills, and test cases through the Buildrix package; the Buildrix Hub organizes, reviews, and benchmarks submitted artifacts; and end users install selected skills and execute them through an agent harness for real-world building engineering tasks. resenting a synthetic exercise with a single predetermined answer, each challenge captures an authentic engineering need, such as conducting urban-scale energy simulations, identifying extreme heat events for resilience analysis, or designing a model-predictive control sequence under a demand-response tariff.

Applied Energy, 318, 119104

[3] Jiang, Z., Xu, W., & Dong, B. (2026). An Agentic AI-Enabled Physics-Informed Machine Learning Framework for Grid-Interactive, Decarbonized Building Operations. Advances in Applied Energy, 100273

[4] White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H.,... & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. p. 0

Where the paper says it

Pre-trained ∗Corresponding author. on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6].

arXiv preprint arXiv:2302. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

11382

[5] Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R.,... & Sui, Z. (2024, November). p. 7

Where the paper says it

The heating systems included gas boilers, gas and electric furnaces, electric baseboard systems, and existing air-source heat pumps. The weather and heat-wave skills retrieved and analyzed the 2024 Syracuse hourly weather record.

A survey on in-context learning. p. 0
Where the paper says it

Pre-trained ∗Corresponding author. on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6].

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. p. 0
Where the paper says it

Pre-trained ∗Corresponding author. on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6].

1107–1128)

[6] Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y.,... & Wen, J. R. (2026). A survey of large language models. p. 0

Where the paper says it

Buildrix provides an open foundation for reusable capability development, transparent evaluation, and community-driven advancement of agentic AI in building engineering. Keywords: Agentic AI, Large Language Models, Agent Skills, Agent Harness, Building Engineering Highlights • Develop Buildrix, a CLI for creating, and sharing reusable skills.

Frontiers of Computer Science, 20(12), 2012627

[7] Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024). EPlus-LLM: A large language model-based computing platform for automated building energy modeling. p. 0

Where the paper says it

In the building domain, early applications have explored LLMs for tasks such as energy model generation [7], code compliance checking [8], and data schema alignment [9]. However, a standalone LLM has no direct access to the external environment: it cannot read project files, invoke databases, execute simulation engines, or operate domain-specific tools.

Applied Energy, 367, 123431

[8] Madireddy, S., Gao, L., Din, Z. U., Kim, K., Senouci, A., Han, Z., & Zhang, Y. (2025). Large language model-driven code compliance checking in building information modeling. p. 0

Where the paper says it

on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6]. In the building domain, early applications have explored LLMs for tasks such as energy model generation [7], code compliance checking [8], and data schema alignment [9].

Electronics, 14(11), 2146

[9] Perini, M., Antonucci, D., Giudice, R., Piscitelli, M. S., & Capozzoli, A. (2025). BrickLLM: A Python library for generating Brick-compliant RDF graphs using LLMs. SoftwareX, 30, 102121

[10] Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D.,... & Chen, E. (2024). Understanding the planning of LLM agents: A survey. p. 1

Where the paper says it

In building applications, tool-augmented agents have been explored for BIM modeling [12], building performance simulation [13], and building energy performance analysis [14], etc. However, real building applications often require multitask coordination and long-horizon planning.

arXiv preprint arXiv:2402. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

02716

[11] Ferrag, M. A., Tihanyi, N., & Debbah, M. (2025). From LLM reasoning to autonomous AI agents: A comprehensive review. p. 1

Where the paper says it

To address this challenge, the current frontier is moving toward agentic AI [15], in which an autonomous ecosystem coordinates an orchestrator, multiple specialist agents, and tools to decompose complex goals and execute interdependent subtasks. From relatively static agent frameworks such as AutoGen [16] and LangGraph [17], which rely on predefined agent topologies while allowing messages, decisions, and tool calls to be dynamically routed among agents, to more open-ended systems such as AutoGPT [18] and Voyager [19], which allow the reasoning trajectory and, in some cases, the execution structure to be constructed at runtime, agentic AI is evolving toward greater autonomy, flexibility, and adaptability for complex, multi-round tasks.

arXiv preprint arXiv:2504. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

19678

[12] Du, C., Esser, S., Nousias, S., & Borrmann, A. (2026). Text2BIM: Generating Building Models Using a Large Language Model-Based Multiagent Framework. p. 6

Where the paper says it

Example Skill: ResStock Building Generation The resstock-building-generation skill is used to demonstrate how a building-engineering workflow is packaged in Buildrix. As illustrated in Figure 2, the skill enables an agent to generate representative residential EnergyPlus models using the well documented NREL ResStock GitHub package in response to a natural-language request.

Journal of Computing in Civil Engineering, 40(2), 04025142

[13] Zhang, L., Ford, V., Chen, Z., & Chen, J. (2025). Automatic building energy model development and debugging using large language models agentic workflow. p. 0

Where the paper says it

Buildrix provides an open foundation for reusable capability development, transparent evaluation, and community-driven advancement of agentic AI in building engineering. Keywords: Agentic AI, Large Language Models, Agent Skills, Agent Harness, Building Engineering Highlights • Develop Buildrix, a CLI for creating, and sharing reusable skills.

Energy and Buildings, 327, 115116

[14] Lin, X., Prabowo, A., Razzak, I., Xue, H., Amos, M., Behrens, S., & Salim, F. D. (2024). Bitsa: Leveraging time series foundation model for building energy analytics. p. 0

Where the paper says it

on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6]. In the building domain, early applications have explored LLMs for tasks such as energy model generation [7], code compliance checking [8], and data schema alignment [9].

In 2024 IEEE International Conference on Data Mining Workshops (ICDMW) (pp. p. 7
Where the paper says it

The heating systems included gas boilers, gas and electric furnaces, electric baseboard systems, and existing air-source heat pumps. The weather and heat-wave skills retrieved and analyzed the 2024 Syracuse hourly weather record.

891–894)

[15] Acharya, D. B., Kuppan, K., & Divya, B. (2025). Agentic AI: Autonomous intelligence for complex goals—A comprehensive survey. p. 1

Where the paper says it

A single agent, particularly when based on a local model with limited capacity, may struggle to plan across many tools, coordinate multiple subtasks, and maintain reliable execution over complex engineering workflows. To address this challenge, the current frontier is moving toward agentic AI [15], in which an autonomous ecosystem coordinates an orchestrator, multiple specialist agents, and tools to decompose complex goals and execute interdependent subtasks.

IEEE Access, 13, 18912–18936

[16] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E.,... & Wang, C. (2024). AutoGen: Enabling next-gen LLM applications via multi-agent conversations. p. 1

Where the paper says it

To address this challenge, the current frontier is moving toward agentic AI [15], in which an autonomous ecosystem coordinates an orchestrator, multiple specialist agents, and tools to decompose complex goals and execute interdependent subtasks. From relatively static agent frameworks such as AutoGen [16] and LangGraph [17], which rely on predefined agent topologies while allowing messages, decisions, and tool calls to be dynamically routed among agents, to more open-ended systems such as AutoGPT [18] and Voyager [19], which allow the reasoning trajectory and, in some cases, the execution structure to be constructed at runtime, agentic AI is evolving toward greater autonomy, flexibility, and adaptability for complex, multi-round tasks.

In First Conference on Language Modeling

[17] LangChain. (n.d.). LangGraph. Retrieved May 26, 2026, from https://www. p. 0

Where the paper says it

25139v1 [eess. SY] 23 Jun 2026

langchain.com/langgraph

[18] Significant Gravitas. (2023). AutoGPT [Computer software]. p. 1

Where the paper says it

To address this challenge, the current frontier is moving toward agentic AI [15], in which an autonomous ecosystem coordinates an orchestrator, multiple specialist agents, and tools to decompose complex goals and execute interdependent subtasks. From relatively static agent frameworks such as AutoGen [16] and LangGraph [17], which rely on predefined agent topologies while allowing messages, decisions, and tool calls to be dynamically routed among agents, to more open-ended systems such as AutoGPT [18] and Voyager [19], which allow the reasoning trajectory and, in some cases, the execution structure to be constructed at runtime, agentic AI is evolving toward greater autonomy, flexibility, and adaptability for complex, multi-round tasks.

GitHub

[19] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y.,... & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. p. 0

Where the paper says it

Buildrix provides an open foundation for reusable capability development, transparent evaluation, and community-driven advancement of agentic AI in building engineering. Keywords: Agentic AI, Large Language Models, Agent Skills, Agent Harness, Building Engineering Highlights • Develop Buildrix, a CLI for creating, and sharing reusable skills.

arXiv preprint arXiv:2305. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

16291

[20] Xu, W., Wan, H., Goel, S., & Antonopoulos, C. A. (2025). Development of a dynamic multi-agent network for building energy modeling: A case study towards scalable and autonomous energy modeling. p. 1

Where the paper says it

In the building domain, recent studies have begun to explore such multi-agent agentic AI systems for building energy system modeling and control [3], EnergyPlus automation [20, 21], and related engineering workflows [22]. Despite this rapid progress, several critical gaps remain before agentic AI can become a reliable and scalable solution for real-world building engineering workflows.

Energy and Buildings, 116712

[21] Li, H., Zhang, L., Zhou, H., & Hong, T. (2026). MCPenabled agentic AI workflow for building energy modelling: framework and use cases. p. 1

Where the paper says it

A modular open-source framework is needed to allow researchers and practitioners to define engineering challenges, contribute reusable skills, share human-validated golden test cases, and benchmark agent performance in a transparent and extensible way. To address these gaps, this study presents Buildrix, an open platform for tackling real-world building-engineering problems through an agentic AI harness and communitydefined engineering tasks, reusable agent skills, golden test cases, and standardized benchmark evaluation, thereby moving agentic AI for buildings from isolated demonstrations toward scalable, verifiable, and community-driven engineering autonomy.

Journal of Building Performance Simulation, 1–27

[22] Lee, J., Song, J., Koo, J., Choi, S., Hwang, J., Saif, S. M. H.,... & Yoon, S. (2025). Agentic built environments: a review. p. 2

Where the paper says it

The Buildrix Hub is a centralized web service that organizes submitted artifacts into open challenges, a skill library, and a test-case repository. It also manages artifact review, golden test-case approval, benchmark execution, and skill ranking.

Energy and Buildings, 116159

[23] Kate, K., Pedapati, T., Basu, K., Rizk, Y., Chenthamarakshan, V., Chaudhury, S.,... & Abdelaziz, I. (2025). LongFuncEval: Measuring the effectiveness of long context models for function calling. p. 9

Where the paper says it

opened the associated script after receiving the error, and corrected the function call. The generated models also produced an EnergyPlus IDD version conflict with the locally provisioned simulation engine.

arXiv preprint arXiv:2505. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

10570

[24] Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A.,... & Liu, X. (2025). Towards a science of scaling agent systems. p. 1

Where the paper says it

A larger number of agents and tools can increase the context burden, expand the decision space, raise computational cost, and reduce execution reliability [23, 24]. Recent work in the broader AI community instead suggests that “small is beautiful” [25]: rather than scaling up system complexity by default, agentic AI architectures should be efficient, modular, scalable, and matched to the intrinsic complexity of the target workflow.

arXiv preprint arXiv:2512. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

08296

[25] Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S.,... & Molchanov, P. (2025). Small language models are the future of agentic AI. p. 0

Where the paper says it

Buildrix provides an open foundation for reusable capability development, transparent evaluation, and community-driven advancement of agentic AI in building engineering. Keywords: Agentic AI, Large Language Models, Agent Skills, Agent Harness, Building Engineering Highlights • Develop Buildrix, a CLI for creating, and sharing reusable skills.

arXiv preprint arXiv:2506. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

02153

[26] Anthropic. (2024, December 20). p. 7

Where the paper says it

The heating systems included gas boilers, gas and electric furnaces, electric baseboard systems, and existing air-source heat pumps. The weather and heat-wave skills retrieved and analyzed the 2024 Syracuse hourly weather record.

Building effective agents. p. 0
Where the paper says it

The agent observes the environment through tool outputs, reasons about the next step, and acts by issuing additional tool calls [11]. Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks.

Anthropic Engineering. p. 0
Where the paper says it

Buildrix: An Open Platform for Sharing and Benchmarking Agentic AI Skills in Building Engineering Zixin Jianga , Bing Donga aDepartment of Mechanical and Aerospace Engineering, Syracuse University, Syracuse, NY, USA Abstract Agentic AI offers significant potential to automate complex building-engineering workflows. However, most existing applications remain isolated proof-of-concept demonstrations and lack reusable domain capabilities, human-verified evaluation cases, and standardized benchmarking infrastructure.

p. 0
Where the paper says it

Buildrix: An Open Platform for Sharing and Benchmarking Agentic AI Skills in Building Engineering Zixin Jianga , Bing Donga aDepartment of Mechanical and Aerospace Engineering, Syracuse University, Syracuse, NY, USA Abstract Agentic AI offers significant potential to automate complex building-engineering workflows. However, most existing applications remain isolated proof-of-concept demonstrations and lack reusable domain capabilities, human-verified evaluation cases, and standardized benchmarking infrastructure.

p. 0
Where the paper says it

Buildrix: An Open Platform for Sharing and Benchmarking Agentic AI Skills in Building Engineering Zixin Jianga , Bing Donga aDepartment of Mechanical and Aerospace Engineering, Syracuse University, Syracuse, NY, USA Abstract Agentic AI offers significant potential to automate complex building-engineering workflows. However, most existing applications remain isolated proof-of-concept demonstrations and lack reusable domain capabilities, human-verified evaluation cases, and standardized benchmarking infrastructure.

https://www.anthropic.com/engineering/building-effective-agents

[27] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? p. 6

Where the paper says it

Example Skill: ResStock Building Generation The resstock-building-generation skill is used to demonstrate how a building-engineering workflow is packaged in Buildrix. As illustrated in Figure 2, the skill enables an agent to generate representative residential EnergyPlus models using the well documented NREL ResStock GitHub package in response to a natural-language request.

In International Conference on Learning Representations (Vol. p. 0
Where the paper says it

Pre-trained ∗Corresponding author. on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6].

p. 0
Where the paper says it

Pre-trained ∗Corresponding author. on web-scale corpora, modern LLMs, often deployed as chatbots or generative AI assistants, can interpret natural-language instructions and generate responses for a wide range of tasks through prompt engineering [4] and in-context learning [5], such as question answering, summarization, code generation, document analysis, and structured-data extraction [6].

2024, pp. 54107–54157)

[28] Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., & Scialom, T. (2024, May). p. 7

Where the paper says it

The heating systems included gas boilers, gas and electric furnaces, electric baseboard systems, and existing air-source heat pumps. The weather and heat-wave skills retrieved and analyzed the 2024 Syracuse hourly weather record.

GAIA: a benchmark for general AI assistants. p. 1
Where the paper says it

• Third, the field lacks standardized golden test cases and benchmark protocols for evaluating agentic AI in building engineering. In computer science domains, benchmark datasets and reference solutions provide a common basis for comparing algorithms and tracking progress, such as SWE-bench [27] and GAIA [28], etc.

In International Conference on Learning Representations (Vol. 2024, pp. 9025–9049)

[29] OpenAI. Introducing Codex. p. 6

Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

p. 6
Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

https://openai. p. 6
Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

p. 6
Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

com/index/introducing-codex/, May 2025a

[30] Lin, J., Liu, S., Pan, C., Lin, L., Dou, S., Huang, X.,... & Gui, T. (2026). Agentic harness engineering: Observability-driven automatic evolution of codingagent harnesses. p. 1

Where the paper says it

To address these gaps, this study presents Buildrix, an open platform for tackling real-world building-engineering problems through an agentic AI harness and communitydefined engineering tasks, reusable agent skills, golden test cases, and standardized benchmark evaluation, thereby moving agentic AI for buildings from isolated demonstrations toward scalable, verifiable, and community-driven engineering autonomy. Before diving into technical details, we briefly introduce two latest agentic AI concepts used in this study: agent harnesses and agent skills.

arXiv preprint arXiv:2604. p. 0
Where the paper says it

Memory modules and retrieval-augmented generation (RAG) extend the agent’s effective context, while fine-tuning can further specialize the underlying model for target tasks. More recently, protocols such as the Model Context Protocol (MCP) have arXiv:2606.

25850

[31] Zhang, B., Lazuka, K., & Murag, M. (2025, October 16). Equipping agents for the real world with Agent Skills. p. 2

Where the paper says it

Through representative case studies, the paper shows how Buildrix enables agents to interpret user goals, select relevant skills, execute tools, inspect intermediate results, recover from errors, and generate verifiable outputs. To the authors’ knowledge, this is among the first studies to systematically introduce and demonstrate harness-based agentic AI for real-world building engineering workflows.

Anthropic Engineering. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills

[32] Agent Skills overview. https://platform. p. 0

Where the paper says it

Buildrix: An Open Platform for Sharing and Benchmarking Agentic AI Skills in Building Engineering Zixin Jianga , Bing Donga aDepartment of Mechanical and Aerospace Engineering, Syracuse University, Syracuse, NY, USA Abstract Agentic AI offers significant potential to automate complex building-engineering workflows. However, most existing applications remain isolated proof-of-concept demonstrations and lack reusable domain capabilities, human-verified evaluation cases, and standardized benchmarking infrastructure.

claude.com/docs/en/agents-and tools/agent skills/overview

[34] Anthropic. (2025, April 18). Claude Code: Best practices for agentic coding. p. 6

Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

Anthropic Engineering. https://www.anthropic.com/engineering/claude-code-best-practices

[35] OpenAI. (2025, May). Introducing Codex. https://openai.com/index/introducing-codex

[36] Google. (2025, June 25). p. 7

Where the paper says it

Five heat-wave events containing 22 heat-wave days were identified, as shown in Figure 3(a). The most severe event lasted seven days in mid-June and reached a maximum outdoor temperature of 34.

Gemini CLI: Your opensource AI agent. p. 6
Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

Google Blog

[37] OpenClaw. (2026). OpenClaw: Open-source AI coding assistant [Computer software]. p. 1

Where the paper says it

This limits reuse, comparison, and community-driven improvement. A modular open-source framework is needed to allow researchers and practitioners to define engineering challenges, contribute reusable skills, share human-validated golden test cases, and benchmark agent performance in a transparent and extensible way.

https://openclawlab.com

[38] Nous Research. (2026). Hermes Agent: A selfimproving autonomous AI agent [Computer software]. p. 6

Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

https://hermes-agent. p. 6
Where the paper says it

is supported by the Buildrix CLI, a Python package that prepares building-engineering skills for use with commandline agent harnesses. Recent command-line harnesses, such as Claude Code [34], OpenAI Codex CLI [35], Gemini CLI [36], Open Claw [37], and Hermes Agent [38], allow language models to inspect project files, execute commands, install dependencies, revise intermediate outputs, and continue multi-step reasoning before returning control to the user.

nousresearch.com

[40] VoltAgent. (2026). Awesome Agent Skills: A curated collection of agent skills from official development teams and the community [Computer software]. p. 0

Where the paper says it

Buildrix provides an open foundation for reusable capability development, transparent evaluation, and community-driven advancement of agentic AI in building engineering. Keywords: Agentic AI, Large Language Models, Agent Skills, Agent Harness, Building Engineering Highlights • Develop Buildrix, a CLI for creating, and sharing reusable skills.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Buildrix provides an open platform for developing, sharing, executing, and evaluating agentic AI skills for building engineering by integrating a Python command-line package for development with a web-based Hub for organizing challenges and benchmarks.

  2. The agent harness supports skill discovery, external toolchain provisioning, progressive context loading, and multi-step workflow execution to enable end-to-end autonomy in real-world building engineering tasks.

  3. Buildrix skills are organized as standardized, self-contained packages containing task instructions, executable scripts, dependencies, and supporting resources to allow for reusable domain capabilities.

  4. The platform allows users to define real-world building engineering challenges and submit human-verified golden test cases that can be promoted for reproducible benchmark evaluation against shared skills.

  5. The harness supports an iterative perceive–reason–act–verify loop, allowing the agent to inspect intermediate results, recover from errors (e.g., data-schema mismatch, software-version incompatibility), and adapt workflows at runtime.

  6. Progressive disclosure limits the initial context requirement by more than 94% compared with loading all skill instructions and scripts at startup, preventing context bottlenecks as the number of installed skills increases.

  7. The system allows for skill composition where the output of one skill can become the input of another, enabling agents to solve higher-level engineering tasks by chaining specialized capabilities.

  8. The benchmark runner executes an accepted skill against a golden test case in a controlled workspace, comparing generated outputs with human-expert references to produce criterion scores aggregated into a leaderboard.

Abstract

Agentic AI has demonstrated remarkable progress across science and engineering, including mathematics, physics, chemistry, and biology. However, how well AI agents can perform real-world building engineering tasks remains largely unexplored. To address this gap, we introduce Buildrix, an open benchmark for evaluating AI agents on real building engineering tasks across eight domains. We investigated three questions: how AI agents compare with human engineers, whether expert knowledge improves agent performance, and how engineers can collaborate effectively with AI agents. The benchmark shows that AI agents achieved an average score of 78 out of 100, compared with 84 for human engineers, while completing tasks in an average of 9 minutes at a cost of 0.58 per run, compared with 8-160 reported working hours for human engineers, demonstrating promising potential. Interestingly, providing expert knowledge did not consistently improve agent performance and sometimes led to worse results. In contrast, structured task instructions substantially improved performance, increasing the average score from 27 to 75 and the success rate from 5% to 76%. These findings highlight both the potential and limitations of AI agents in building engineering, as well as the importance of structured task specifications and clear deliverable requirements for effective human-agent collaboration. More broadly, Buildrix provides an open evaluation framework for tracking AI progress, identifying capability gaps, and guiding the development of more capable and reliable AI agents to accelerate the adoption of AI in building engineering practice.

Sources

Related papers