A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems".
Jane: Here's the gist of the paper. A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess LLM-integrated HEMS across multi-turn dialogues, which is crucial because previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric.
Tom: Framework Methodology The study proposes a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency.
Jane: A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring.
Tom: GQM proceeds in three steps: (1) define evaluation goals that specify what must be measured; (2) derive operational questions that must be answered to assess progress toward each goal; and (3) identify metrics that provide the quantitative data needed to answer each question.
Paper summary: Tom: So, to wrap up this paper "A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems," the authors are basically saying that we need to stop relying on single-turn accuracy and start using a structured method like Goal Question Metric.
Jane: They introduce this framework to evaluate LLM-integrated HEMS across five categories: task performance, factual accuracy, interaction quality, control capability and system efficiency >
Lu: The main implication is that this provides a way to systematically measure the complexity of these systems in a real-world conversational setting, using twenty-three metrics derived from those goals >
Meng: For production teams like us, it suggests that we need to look at how the AI interacts with the user, not just what data it retrieves correctly >
Lalam: The study shows that interaction quality metrics, specifically personalization calibration, are actually a more discriminating factor for telling agentic systems apart from non-agentic ones >
Tom: The authors also highlight that while they have this framework to test the interaction part of things, they admit the paper doesn't address the actual energy savings or how well the underlying scheduling logic is optimized yet >
Jane: That’s a fair limitation. It’s a great way to evaluate the *conversational* aspect, but it stops short of measuring whether we actually save power or if it’s making better long-term decisions for the homeowner >
Tom: So, in simple terms, this paper gives us the blueprint for testing how good a conversational energy assistant is at talking to you, not just how smart its energy calculations are >
Conclusion: Tom: So, this paper lays out this whole evaluation framework for conversational home energy management systems that uses goal question metric to assess them across five main areas >
Jane: It takes all those complex interactions and breaks them down into things like task performance factual accuracy interaction quality control capability and system efficiency >
Tom: The authors built twenty-three specific metrics to measure all of that, which is way more detailed than just checking if the AI answered a question right or wrong >
Lu: They use an LLM as a judge pipeline for scoring these metrics automatically, which they validated against human coders to make sure the scores were reliable >
Meng: So what does this actually mean for us when we're trying to build these assistants? It’s about moving past just looking at response accuracy on a single turn >
Jane: Exactly. It suggests that we need a much broader way of testing these systems because they are constantly talking and doing things over time >
Tom: And the comparison they did across four different HEMS setups showed that interaction quality, especially how well it personalizes things, is more important than just factual accuracy when you're trying to tell if an AI is acting like a true agent >
Jane: That's a really interesting finding because it points toward a different kind of intelligence we need to build in these systems >
Tom: But the authors were pretty clear about what this framework doesn’t cover, which is that actual energy savings or how good the underlying scheduling logic is for long-term use >
Lu: They acknowledge that they’re only looking at the conversational interface and not the real-world impact on your electricity bill or whether it's making better decisions over months >
Jane: So, this gives us a solid toolkit for evaluating the conversation, but we still have to figure out if those conversations actually lead to better energy management in practice >
University of Arizona
cs.HC
Submitted: 2026-09-04
Updated: 2026-11-13
Comments: 25 pages, 7 figures, 9 tables. Re-set in the HUBS Lab paper layout (HUBS Lab Paper No. 2026-006); content unchanged
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Importance score: 84/100
The gist: A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess…
Key concepts
- Goal-Question-Metric (GQM) Methodology
- A structured approach used to design evaluations. It involves three steps: first defining what needs measuring (goals), second creating questions to assess progress toward those goals, and third identifying specific metrics that quantify the answers. This ensures every measurement is tied directly to a defined objective.
- LLM-as-Judge Pipeline
- A system where a Large Language Model (LLM) automatically scores the performance of HEMS responses based on predefined rubrics. This pipeline was validated against human coders, showing strong agreement (ICC $\geq 0.73$) for many metrics, allowing for scalable and automated evaluation of complex conversational interactions.
- Interaction Quality Dimensions
- This category measures how well the HEMS interacts with the user across four aspects: question answering completeness, personalization calibration, technical accessibility, and action transparency. The study found that 'personalization calibration' was a highly discriminating factor for distinguishing agentic systems from non-agentic ones.
- Multi-Agent Architecture
- The HEMA system uses three specialized agents—Analysis, Knowledge, and Control—coordinated by a classifier. Each agent uses the ReAct paradigm to chain reasoning and tool calls. This architecture allows the system to retrieve data, understand concepts, and execute device commands deterministically.
Terminology
Summary
A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess LLM-integrated HEMS across multi-turn dialogues, which is crucial because previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. p. 0 A Comprehensive Evaluation Framework for Conversational
Home Energy Management Systems
WOOYOUNG JUNG, University of Arizona, USA
The growing complexity in home energy management (HEM) demands advanced systems that guide occupants
toward informed energy decisions while reflecting their background, preferences, and contextual information. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies
relied on single-turn or single-task evaluations with response accuracy as the primary metric.Where the paper says it
Framework Methodology
The study proposes a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. p. 0 This study introduces a comprehensive evaluation framework of
LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories:
task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23
metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable
scalable automated scoring. This work extends
GQM into the evaluation of multi-turn conversational AI. GQM proceeds in three steps: (1) define evaluation goals that specify what must be measured;
(2) derive operational questions that must be answered to assess progress toward each goal; and
(3) identify metrics that provide the quantitative data needed to answer each question. Consequently,
whether such systems can deliver effective interactions across the extended multi-turn dialogues typical of
real-world use remains an open question. This study introduces a comprehensive evaluation framework of
LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories:
task performance, factual accuracy, interaction quality, control capability, and system efficiency.Where the paper says it
Where the paper says it
Where the paper says it
Metric Definition and Validation
Table 2 operationalizes these goals and questions into 23 measurable metrics. For example, Task Performance includes the goal achievement rate, defined as Proportion of evaluation runs in which the user’s predefined goal was satisfied before reaching the maximum turn limit. p. 4 The number of metrics per category reflects the dimensionality of each goal: interaction quality,
which spans four sub-dimensions (question answering completeness, personalization calibration,
technical accessibility, and action transparency), requires finer-grained decomposition than goals
that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS
Evaluation metric Description Scale
Task Performance
Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal
was satisfied before reaching the maximum turn limit
% (0–100%)
Turns to completion Number of conversational turns elapsed when the goal was achieved Count
Factual Accuracy
Factual accuracy rate Proportion of numerical claims that fell within 5% error of the
corresponding ground truth value
% (0–100%)
Mean error percentage Average absolute percentage error against ground truth across all
verifiable numerical claims
% (0–100%+)
Factual claims Count of verifiable quantitative claims (e. These are measures that can be objectively
extracted from human-AI conversation data; the evaluation procedure is described in Section 4. The number of metrics per category reflects the dimensionality of each goal: interaction quality,
which spans four sub-dimensions (question answering completeness, personalization calibration,
technical accessibility, and action transparency), requires finer-grained decomposition than goals
that address a single evaluative concern. The framework has three design properties. First, GQM traceability ensures every metric traces
to an explicit goal and no goal is left unmeasured.Where the paper says it
Where the paper says it
Where the paper says it
LLM-as-Judge Reliability
The reliability of the LLM-as-judge pipeline was validated against three trained human coders, after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC ≥ 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0. p. 0 Its reliability is validated against three trained human coders: after iterative rubric
calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC ≥ 0. 73), three of them
perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via
mean absolute error (0. However, before crossconfiguration results from this pipeline could be interpreted, its reliability had to first be established. Stage 1 therefore conducted a human validation study on a stratified subset of conversations to
quantify agreement between human coders and automated scores.Where the paper says it
Where the paper says it
Comparative Performance Analysis
The framework was applied to compare four HEMS configurations across diverse scenarios and user personas, revealing their respective strengths and weaknesses. The results demonstrated that interaction quality metrics – particularly over-personalization – are more discriminating than factual accuracy for differentiating agentic from non-agentic architectures. p. 4 The number of metrics per category reflects the dimensionality of each goal: interaction quality,
which spans four sub-dimensions (question answering completeness, personalization calibration,
technical accessibility, and action transparency), requires finer-grained decomposition than goals
that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS
Evaluation metric Description Scale
Task Performance
Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal
was satisfied before reaching the maximum turn limit
% (0–100%)
Turns to completion Number of conversational turns elapsed when the goal was achieved Count
Factual Accuracy
Factual accuracy rate Proportion of numerical claims that fell within 5% error of the
corresponding ground truth value
% (0–100%)
Mean error percentage Average absolute percentage error against ground truth across all
verifiable numerical claims
% (0–100%+)
Factual claims Count of verifiable quantitative claims (e. The number of metrics per category reflects the dimensionality of each goal: interaction quality,
which spans four sub-dimensions (question answering completeness, personalization calibration,
technical accessibility, and action transparency), requires finer-grained decomposition than goals
that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS
Evaluation metric Description Scale
Task Performance
Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal
was satisfied before reaching the maximum turn limit
% (0–100%)
Turns to completion Number of conversational turns elapsed when the goal was achieved Count
Factual Accuracy
Factual accuracy rate Proportion of numerical claims that fell within 5% error of the
corresponding ground truth value
% (0–100%)
Mean error percentage Average absolute percentage error against ground truth across all
verifiable numerical claims
% (0–100%+)
Factual claims Count of verifiable quantitative claims (e.Where the paper says it
Where the paper says it
Key Findings and Limitations
The results demonstrate that a single-metric or accuracy-only approach would provide an incomplete and, in some cases, misleading characterization of conversational HEMS quality. The more discriminating dimension proved to be personalization calibration: Vanilla+Energy Report exhibited 0. p. 0 73), three of them
perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via
mean absolute error (0. To demonstrate the framework’s effectiveness, a total of 970 dialogues – 16
scenarios and five personas – were generated and evaluated using four conversational HEMS configurations:
a HEMS using a vanilla LLM with raw energy data, one with a vanilla LLM with an energy analysis report,
one with a vanilla LLM with an energy analysis report and chain-of-thought prompting, and a multi-agent
HEMS. To ensure a fair comparison, the two vanilla LLM configurations with structured data (the second
and third configurations) received a pre-computed energy analysis report generated using the
analysis tools available to HEMA. This ensured that these configurations had access to the same
analytical results that HEMA would produce during its tool-mediated workflow. A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems 9
Table 5. Details of the scenarios considered in comparative analysis
Scenario Description Primary goal
Analysis-focused scenarios
Appliance analysis Identify energy-consuming appliances
Find top energy consumers and get reduction recommendations
Peak reduction strategy Reduce peak hour usage Develop strategy to shift usage to off-peak
hours
HVAC optimization Optimize heating/cooling efficiency
Learn optimal settings, scheduling, and
maintenance tips
Energy comparison Compare usage over time Compare energy usage between different
periods
Solar consideration Evaluate solar panel viability Understand costs, savings, and factors affecting solar ROI
General knowledge-focused scenarios
Understand utility rate Learn TOU rate structure Understand what TOU pricing means and
when rates are highest
Weather energy impact Understand weather effects on
energy
Learn how weather affects bills and get
weather-based advice
Utility rate details Get specific rate information Learn exact tier prices, peak hours, and seasonal differences
Rebate inquiry Find rebate programs Learn about available rebates, eligibility,
and how to apply
Control-focused scenarios
Thermostat adjustment Adjust smart thermostat Set optimal temperature based on rate structure
EV charging schedule Schedule EV charging Configure charging during off-peak hours
Device status check Review all smart devices Get overview of device status and energy
consumption
Water heater optimization
Optimize water heater schedule Align heating schedule with TOU rates
while ensuring hot water availability
Pool pump scheduling Schedule pool pump operation Set pump runtime during optimal hours for
savings
Vacation preparation Configure multiple devices for
vacation
Set up HVAC, water heater, and pool pump
to minimize energy while away
Demand response event Handle demand response notification
Adjust devices to reduce load during event
period while maintaining comfort
in its operational context constitutes a recognized evaluation approach [20]. A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems 9
Table 5. Details of the scenarios considered in comparative analysis
Scenario Description Primary goal
Analysis-focused scenarios
Appliance analysis Identify energy-consuming appliances
Find top energy consumers and get reduction recommendations
Peak reduction strategy Reduce peak hour usage Develop strategy to shift usage to off-peak
hours
HVAC optimization Optimize heating/cooling efficiency
Learn optimal settings, scheduling, and
maintenance tips
Energy comparison Compare usage over time Compare energy usage between different
periods
Solar consideration Evaluate solar panel viability Understand costs, savings, and factors affecting solar ROI
General knowledge-focused scenarios
Understand utility rate Learn TOU rate structure Understand what TOU pricing means and
when rates are highest
Weather energy impact Understand weather effects on
energy
Learn how weather affects bills and get
weather-based advice
Utility rate details Get specific rate information Learn exact tier prices, peak hours, and seasonal differences
Rebate inquiry Find rebate programs Learn about available rebates, eligibility,
and how to apply
Control-focused scenarios
Thermostat adjustment Adjust smart thermostat Set optimal temperature based on rate structure
EV charging schedule Schedule EV charging Configure charging during off-peak hours
Device status check Review all smart devices Get overview of device status and energy
consumption
Water heater optimization
Optimize water heater schedule Align heating schedule with TOU rates
while ensuring hot water availability
Pool pump scheduling Schedule pool pump operation Set pump runtime during optimal hours for
savings
Vacation preparation Configure multiple devices for
vacation
Set up HVAC, water heater, and pool pump
to minimize energy while away
Demand response event Handle demand response notification
Adjust devices to reduce load during event
period while maintaining comfort
in its operational context constitutes a recognized evaluation approach [20].Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Future Directions
Future work will replicate the framework across multiple households and utility structures, validate system rankings with real residential users, and apply the framework to emerging conversational HEMS architectures. The evaluation draws on data from a single household in Austin, Texas [40], chosen to provide a controlled context in which ground-truth values can be precisely defined. p. 7 This comparative analysis utilized two weeks of data from a single household (a 302. 4 m2
singlefamily house built in 2011 in Austin, Texas, with solar panels, an EV charger, a pool pump, an
HVAC unit, and 14 total appliances; data from Pecan Street Inc. (3) Can the proposed evaluation framework successfully distinguish different conversational
HEMS’ performance? By addressing these questions, this study makes three major contributions: (1) it proposes and
validates a comprehensive evaluation framework, grounded in the Goal-Question-Metric (GQM)
methodology, comprising 23 metrics across five categories – task performance, factual accuracy,
interaction quality, control capability, and system efficiency – for assessing conversational HEMS
(RQ1); (2) it employs an LLM-as-judge approach for scalable automated scoring validated against
human coders (RQ2); and (3) it applies this framework to compare four HEMS configurations across
diverse scenarios and user personas, providing evidence of discriminative validity (RQ3).Where the paper says it
Where the paper says it
A Home Energy Management Assistant (HEMA) employs a multi-agent architecture (Figure 7) comprising three specialized agents – Analysis, Knowledge, and Control – coordinated by a self-consistency classifier that runs four parallel Chain-of-Thought classification attempts and aggregates results through majority voting. p. 7 (3) A vanilla LLM fed with the comprehensive energy data analysis report and guided by a
chain-of-thought (CoT) prompt (Vanilla+Energy Report+CoT). (4) A multi-agent architecture of HEMS, named Home Energy Management Assistant (HEMA)
[24] (more details of HEMA are provided in Appendix A). +
RAG
OpenAI Assistants API
ReAct + few-shot Multi-agent (orchestrator + specialists)
LLM GPT-4o-mini Starling 7B,
Qwen 75B
GPT-4o Mistral 7B Llama 70B, Qwen
32B, GPT-OSS
120B
Functional scope Energy-saving
strategies
Smart home actions
Energy analysis,
device control
Parameter extraction
Appliance scheduling
Interaction type Single-turn Single-turn Single-turn Agent-driven sequential conversation
Single-task
User representation
Researcherdesigned
prompts
Researcherdesigned
prompts
Researcherdesigned
prompts
LLM-as-user
with three difficulty levels
Researcherdesigned scenarios
Evaluation metrics
Accuracy User preference
reflection in system response, latency
Response accuracy, system
performance*
Parameter retrieval accuracy
Success
rate, system
performance*
*
Including system latency, token usage
Eng. : engineering, RAG: retrieval augmented generation, ReAct: reasoning and acting
As summarized in Table 1, these systems span a range of LLM techniques from CoT prompting
[24] and RAG-augmented recommendation [42] to multi-turn ReAct interfaces [37] and hierarchical
multi-agent scheduling [34], with functional scopes ranging from energy analysis and device control
[18, 19] to structured parameter extraction and appliance coordination. (CoT)
Prompt eng. +
RAG
OpenAI Assistants API
ReAct + few-shot Multi-agent (orchestrator + specialists)
LLM GPT-4o-mini Starling 7B,
Qwen 75B
GPT-4o Mistral 7B Llama 70B, Qwen
32B, GPT-OSS
120B
Functional scope Energy-saving
strategies
Smart home actions
Energy analysis,
device control
Parameter extraction
Appliance scheduling
Interaction type Single-turn Single-turn Single-turn Agent-driven sequential conversation
Single-task
User representation
Researcherdesigned
prompts
Researcherdesigned
prompts
Researcherdesigned
prompts
LLM-as-user
with three difficulty levels
Researcherdesigned scenarios
Evaluation metrics
Accuracy User preference
reflection in system response, latency
Response accuracy, system
performance*
Parameter retrieval accuracy
Success
rate, system
performance*
*
Including system latency, token usage
Eng. 6 Wooyoung Jung
Table 2 continued from previous page
Evaluation metric Description Scale
Over-personalized
response
Count of responses referencing user-specific data to answer a general knowledge question not requiring personalization
Count
Under-personalized response
Count of responses providing a generic answer to a data-specific
question, failing to leverage available household data
Count
Appropriate general response
Count of responses correctly providing a conceptual explanation
for a general knowledge question
Count
Technical terms
explained
Count of domain-specific terms (e. , TOU, load shifting, kWh)
defined or explained when first introduced
Count
Information-beforeaction rate
Proportion of control actions preceded by the system communicating relevant device state or contextual information to the user
before executing the action
% (0–100%)
Action confirmation
rate
Proportion of control actions followed by explicit confirmation of
the completed action
% (0–100%)
Action explanation rate Proportion of control actions accompanied by an explanation of
why the setting or schedule was chosen
% (0–100%)
Control Capability
Target device accuracy Proportion of device changes affecting only the intended target
device(s) without unintended side effects
% (0–100%)
Schedule correctness Proportion of load-shifting actions scheduled during off-peak hours
per the utility’s TOU rate structure. 2 System Performance Evaluation
The system performance evaluation procedure followed a two-stage design. Of the 23 metrics,
eight were computed programmatically from system logs: turns to completion, average system
response length, response latency, and token usage from conversation metadata, and target device
accuracy, schedule correctness, mode correctness, and constraint compliance rate from deterministic
comparison of device states before and after control actions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve system evaluation by using a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. This provides a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.
-
Implement an LLM as judge pipeline for scalable automated scoring of 23 metrics using iterative rubric calibration against human coders to ensure reliability. This enables the automated assessment of multi-turn conversations across different HEMS configurations at scale while mitigating stochastic variation through a three-run median voting procedure.
-
Enhance system performance by utilizing an agentic architecture, such as Home Energy Management Assistant (HEMA), which comprises Analysis, Knowledge, and Control agents coordinated by a self-consistency classifier. This architecture enables superior personalization judgment and eliminates numerical fabrication by grounding claims in tool outputs.
-
Improve interaction quality by explicitly measuring diagnostic granularity through the separation of over-personalization and under-personalization behaviors using a response appropriateness matrix (Table 3). This allows for targeted diagnosis of where and how a system falls short, such as identifying when
inappropriately injecting user-specific data into responses to general knowledge questions
occurs. -
Enhance control capability by evaluating schedule correctness and target device accuracy deterministically from system logs, while also analyzing action transparency metrics across different scenarios. This helps identify failure modes where the agent
consistently communicated completed actions and their rationale to users, but frequently proceeded to execute device commands without first gathering or communicating relevant device state information.
Abstract
The growing complexity in home energy management (HEM) demands advanced systems that guide occupants toward informed energy decisions reflecting their background, preferences, and context. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. Whether such systems deliver effective interactions across the extended multi-turn dialogues typical of real-world use remains an open question. This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. Its reliability is validated against three trained human coders: after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC >= 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0.04-0.28). To demonstrate the framework's effectiveness, 970 dialogues -- 16 scenarios and five personas -- were generated and evaluated across four conversational HEMS configurations spanning a sophistication gradient, from a vanilla LLM with raw energy data to a multi-agent HEMS. The framework distinguished the four configurations across multiple evaluation dimensions, revealing their respective strengths and weaknesses. This study contributes to conversational HEMS by providing a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.
Sources
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- Context-aware LLM-based AI Agents for Human-centered Energy Management Systems in Smart Buildings
- Quantifying the Persona Effect in LLM Simulations
- Scaling Laws for Neural Language Models
- ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
- Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Agentic AI Home Energy Management System: A Large Language Model Framework for Residential Load Scheduling
- Evaluating Quality of Chatbots and Intelligent Conversational Agents
- Leveraging Large Language Models for enhanced personalised user experience in Smart Homes
- Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
- Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
- Agent AI with LangGraph: A Modular Framework for Enhancing Machine Translation Using Large Language Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems
- Explainable Iterative Data Visualization Refinement via an LLM Agent