Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy
cs.HC, cs.AI
Submitted: 2026-02-18
Updated: 2026-11-06
Comments: 40 pages, 11 figures. Re-set in the HUBS Lab paper layout (HUBS Lab Paper No. 2026-002); content unchanged
Code: https://github.com/humanbuildingsynergy/HUBS_GPT-based_SCALE_Framework
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS).
Terminology
Abstract
This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.
Sources
- Context-aware LLM-based AI Agents for Human-centered Energy Management Systems in Smart Buildings
- Leveraging Large Language Models for enhanced personalised user experience in Smart Homes
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
- Retrieval-Augmented Generation for Large Language Models: A Survey
- GPT-4o System Card
- Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems
- Explainable Iterative Data Visualization Refinement via an LLM Agent