A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems

arXiv:2610.00073 · cs.HC · Submitted 2026-09-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems".

Jane: Here's the gist of the paper. A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess LLM-integrated HEMS across multi-turn dialogues, which is crucial because previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric.

Tom: Framework Methodology The study proposes a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency.

Jane: A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring.

Tom: GQM proceeds in three steps: (1) define evaluation goals that specify what must be measured; (2) derive operational questions that must be answered to assess progress toward each goal; and (3) identify metrics that provide the quantitative data needed to answer each question.

Paper summary: Tom: So, to wrap up this paper "A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems," the authors are basically saying that we need to stop relying on single-turn accuracy and start using a structured method like Goal Question Metric.

Jane: They introduce this framework to evaluate LLM-integrated HEMS across five categories: task performance, factual accuracy, interaction quality, control capability and system efficiency >

Lu: The main implication is that this provides a way to systematically measure the complexity of these systems in a real-world conversational setting, using twenty-three metrics derived from those goals >

Meng: For production teams like us, it suggests that we need to look at how the AI interacts with the user, not just what data it retrieves correctly >

Lalam: The study shows that interaction quality metrics, specifically personalization calibration, are actually a more discriminating factor for telling agentic systems apart from non-agentic ones >

Tom: The authors also highlight that while they have this framework to test the interaction part of things, they admit the paper doesn't address the actual energy savings or how well the underlying scheduling logic is optimized yet >

Jane: That’s a fair limitation. It’s a great way to evaluate the *conversational* aspect, but it stops short of measuring whether we actually save power or if it’s making better long-term decisions for the homeowner >

Tom: So, in simple terms, this paper gives us the blueprint for testing how good a conversational energy assistant is at talking to you, not just how smart its energy calculations are >

Conclusion: Tom: So, this paper lays out this whole evaluation framework for conversational home energy management systems that uses goal question metric to assess them across five main areas >

Jane: It takes all those complex interactions and breaks them down into things like task performance factual accuracy interaction quality control capability and system efficiency >

Tom: The authors built twenty-three specific metrics to measure all of that, which is way more detailed than just checking if the AI answered a question right or wrong >

Lu: They use an LLM as a judge pipeline for scoring these metrics automatically, which they validated against human coders to make sure the scores were reliable >

Meng: So what does this actually mean for us when we're trying to build these assistants? It’s about moving past just looking at response accuracy on a single turn >

Jane: Exactly. It suggests that we need a much broader way of testing these systems because they are constantly talking and doing things over time >

Tom: And the comparison they did across four different HEMS setups showed that interaction quality, especially how well it personalizes things, is more important than just factual accuracy when you're trying to tell if an AI is acting like a true agent >

Jane: That's a really interesting finding because it points toward a different kind of intelligence we need to build in these systems >

Tom: But the authors were pretty clear about what this framework doesn’t cover, which is that actual energy savings or how good the underlying scheduling logic is for long-term use >

Lu: They acknowledge that they’re only looking at the conversational interface and not the real-world impact on your electricity bill or whether it's making better decisions over months >

Jane: So, this gives us a solid toolkit for evaluating the conversation, but we still have to figure out if those conversations actually lead to better energy management in practice >

University of Arizona

cs.HC

Submitted: 2026-09-04

Updated: 2026-11-13

Comments: 25 pages, 7 figures, 9 tables. Re-set in the HUBS Lab paper layout (HUBS Lab Paper No. 2026-006); content unchanged

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 84/100

The gist: A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess…

Key concepts

Goal-Question-Metric (GQM) Methodology
A structured approach used to design evaluations. It involves three steps: first defining what needs measuring (goals), second creating questions to assess progress toward those goals, and third identifying specific metrics that quantify the answers. This ensures every measurement is tied directly to a defined objective.
LLM-as-Judge Pipeline
A system where a Large Language Model (LLM) automatically scores the performance of HEMS responses based on predefined rubrics. This pipeline was validated against human coders, showing strong agreement (ICC $\geq 0.73$) for many metrics, allowing for scalable and automated evaluation of complex conversational interactions.
Interaction Quality Dimensions
This category measures how well the HEMS interacts with the user across four aspects: question answering completeness, personalization calibration, technical accessibility, and action transparency. The study found that 'personalization calibration' was a highly discriminating factor for distinguishing agentic systems from non-agentic ones.
Multi-Agent Architecture
The HEMA system uses three specialized agents—Analysis, Knowledge, and Control—coordinated by a classifier. Each agent uses the ReAct paradigm to chain reasoning and tool calls. This architecture allows the system to retrieve data, understand concepts, and execute device commands deterministically.

Terminology

Summary

A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems introduces a comprehensive evaluation framework derived from the Goal-Question-Metric methodology to assess LLM-integrated HEMS across multi-turn dialogues, which is crucial because previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. p. 0

Where the paper says it

A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems WOOYOUNG JUNG, University of Arizona, USA The growing complexity in home energy management (HEM) demands advanced systems that guide occupants toward informed energy decisions while reflecting their background, preferences, and contextual information. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric.

Framework Methodology

The study proposes a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. p. 0

Where the paper says it

This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring.

GQM proceeds in three steps: (1) define evaluation goals that specify what must be measured; (2) derive operational questions that must be answered to assess progress toward each goal; and (3) identify metrics that provide the quantitative data needed to answer each question. p. 3
Where the paper says it

This work extends GQM into the evaluation of multi-turn conversational AI. GQM proceeds in three steps: (1) define evaluation goals that specify what must be measured; (2) derive operational questions that must be answered to assess progress toward each goal; and (3) identify metrics that provide the quantitative data needed to answer each question.

The five domain-specific evaluation goals are Task Performance, Factual Accuracy, Interaction Quality, Control Capability, and System Efficiency. p. 0
Where the paper says it

Consequently, whether such systems can deliver effective interactions across the extended multi-turn dialogues typical of real-world use remains an open question. This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency.

Metric Definition and Validation

Table 2 operationalizes these goals and questions into 23 measurable metrics. For example, Task Performance includes the goal achievement rate, defined as Proportion of evaluation runs in which the user’s predefined goal was satisfied before reaching the maximum turn limit. p. 4

Where the paper says it

The number of metrics per category reflects the dimensionality of each goal: interaction quality, which spans four sub-dimensions (question answering completeness, personalization calibration, technical accessibility, and action transparency), requires finer-grained decomposition than goals that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS Evaluation metric Description Scale Task Performance Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal was satisfied before reaching the maximum turn limit % (0–100%) Turns to completion Number of conversational turns elapsed when the goal was achieved Count Factual Accuracy Factual accuracy rate Proportion of numerical claims that fell within 5% error of the corresponding ground truth value % (0–100%) Mean error percentage Average absolute percentage error against ground truth across all verifiable numerical claims % (0–100%+) Factual claims Count of verifiable quantitative claims (e.

Interaction Quality spans four sub-dimensions: question answering completeness, personalization calibration, technical accessibility, and action transparency. p. 4
Where the paper says it

These are measures that can be objectively extracted from human-AI conversation data; the evaluation procedure is described in Section 4. The number of metrics per category reflects the dimensionality of each goal: interaction quality, which spans four sub-dimensions (question answering completeness, personalization calibration, technical accessibility, and action transparency), requires finer-grained decomposition than goals that address a single evaluative concern.

The framework has three design properties: GQM traceability ensures every metric traces to an explicit goal and no goal is left unmeasured. p. 5
Where the paper says it

The framework has three design properties. First, GQM traceability ensures every metric traces to an explicit goal and no goal is left unmeasured.

LLM-as-Judge Reliability

The reliability of the LLM-as-judge pipeline was validated against three trained human coders, after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC ≥ 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0. p. 0

Where the paper says it

Its reliability is validated against three trained human coders: after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC ≥ 0. 73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0.

04–0.28). The human validation study served a dual purpose: establishing measurement reliability and calibrating the automated pipeline. p. 9
Where the paper says it

However, before crossconfiguration results from this pipeline could be interpreted, its reliability had to first be established. Stage 1 therefore conducted a human validation study on a stratified subset of conversations to quantify agreement between human coders and automated scores.

Comparative Performance Analysis

The framework was applied to compare four HEMS configurations across diverse scenarios and user personas, revealing their respective strengths and weaknesses. The results demonstrated that interaction quality metrics – particularly over-personalization – are more discriminating than factual accuracy for differentiating agentic from non-agentic architectures. p. 4

Where the paper says it

The number of metrics per category reflects the dimensionality of each goal: interaction quality, which spans four sub-dimensions (question answering completeness, personalization calibration, technical accessibility, and action transparency), requires finer-grained decomposition than goals that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS Evaluation metric Description Scale Task Performance Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal was satisfied before reaching the maximum turn limit % (0–100%) Turns to completion Number of conversational turns elapsed when the goal was achieved Count Factual Accuracy Factual accuracy rate Proportion of numerical claims that fell within 5% error of the corresponding ground truth value % (0–100%) Mean error percentage Average absolute percentage error against ground truth across all verifiable numerical claims % (0–100%+) Factual claims Count of verifiable quantitative claims (e.

HEMA maintained 80%+ goal achievement across nearly all persona-scenario combinations, while non-agentic configurations showed high variance from 0% to 100%. p. 4
Where the paper says it

The number of metrics per category reflects the dimensionality of each goal: interaction quality, which spans four sub-dimensions (question answering completeness, personalization calibration, technical accessibility, and action transparency), requires finer-grained decomposition than goals that address a single evaluative concern. Performance evaluation metrics for LLM-integrated HEMS Evaluation metric Description Scale Task Performance Goal achievement rate Proportion of evaluation runs in which the user’s predefined goal was satisfied before reaching the maximum turn limit % (0–100%) Turns to completion Number of conversational turns elapsed when the goal was achieved Count Factual Accuracy Factual accuracy rate Proportion of numerical claims that fell within 5% error of the corresponding ground truth value % (0–100%) Mean error percentage Average absolute percentage error against ground truth across all verifiable numerical claims % (0–100%+) Factual claims Count of verifiable quantitative claims (e.

Key Findings and Limitations

The results demonstrate that a single-metric or accuracy-only approach would provide an incomplete and, in some cases, misleading characterization of conversational HEMS quality. The more discriminating dimension proved to be personalization calibration: Vanilla+Energy Report exhibited 0. p. 0

Where the paper says it

73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0. To demonstrate the framework’s effectiveness, a total of 970 dialogues – 16 scenarios and five personas – were generated and evaluated using four conversational HEMS configurations: a HEMS using a vanilla LLM with raw energy data, one with a vanilla LLM with an energy analysis report, one with a vanilla LLM with an energy analysis report and chain-of-thought prompting, and a multi-agent HEMS.

64 ± 0.65 over-personalized responses per conversation despite having access to correctly structured, pre-computed analysis data, while HEMA achieved near-zero over-personalization (0. p. 9
Where the paper says it

To ensure a fair comparison, the two vanilla LLM configurations with structured data (the second and third configurations) received a pre-computed energy analysis report generated using the analysis tools available to HEMA. This ensured that these configurations had access to the same analytical results that HEMA would produce during its tool-mediated workflow.

08 ± 0.29). The framework does not address actual energy savings, long-term user satisfaction, or the optimality of underlying scheduling logic. p. 8
Where the paper says it

A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems 9 Table 5. Details of the scenarios considered in comparative analysis Scenario Description Primary goal Analysis-focused scenarios Appliance analysis Identify energy-consuming appliances Find top energy consumers and get reduction recommendations Peak reduction strategy Reduce peak hour usage Develop strategy to shift usage to off-peak hours HVAC optimization Optimize heating/cooling efficiency Learn optimal settings, scheduling, and maintenance tips Energy comparison Compare usage over time Compare energy usage between different periods Solar consideration Evaluate solar panel viability Understand costs, savings, and factors affecting solar ROI General knowledge-focused scenarios Understand utility rate Learn TOU rate structure Understand what TOU pricing means and when rates are highest Weather energy impact Understand weather effects on energy Learn how weather affects bills and get weather-based advice Utility rate details Get specific rate information Learn exact tier prices, peak hours, and seasonal differences Rebate inquiry Find rebate programs Learn about available rebates, eligibility, and how to apply Control-focused scenarios Thermostat adjustment Adjust smart thermostat Set optimal temperature based on rate structure EV charging schedule Schedule EV charging Configure charging during off-peak hours Device status check Review all smart devices Get overview of device status and energy consumption Water heater optimization Optimize water heater schedule Align heating schedule with TOU rates while ensuring hot water availability Pool pump scheduling Schedule pool pump operation Set pump runtime during optimal hours for savings Vacation preparation Configure multiple devices for vacation Set up HVAC, water heater, and pool pump to minimize energy while away Demand response event Handle demand response notification Adjust devices to reduce load during event period while maintaining comfort in its operational context constitutes a recognized evaluation approach [20].

p. 8
Where the paper says it

A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems 9 Table 5. Details of the scenarios considered in comparative analysis Scenario Description Primary goal Analysis-focused scenarios Appliance analysis Identify energy-consuming appliances Find top energy consumers and get reduction recommendations Peak reduction strategy Reduce peak hour usage Develop strategy to shift usage to off-peak hours HVAC optimization Optimize heating/cooling efficiency Learn optimal settings, scheduling, and maintenance tips Energy comparison Compare usage over time Compare energy usage between different periods Solar consideration Evaluate solar panel viability Understand costs, savings, and factors affecting solar ROI General knowledge-focused scenarios Understand utility rate Learn TOU rate structure Understand what TOU pricing means and when rates are highest Weather energy impact Understand weather effects on energy Learn how weather affects bills and get weather-based advice Utility rate details Get specific rate information Learn exact tier prices, peak hours, and seasonal differences Rebate inquiry Find rebate programs Learn about available rebates, eligibility, and how to apply Control-focused scenarios Thermostat adjustment Adjust smart thermostat Set optimal temperature based on rate structure EV charging schedule Schedule EV charging Configure charging during off-peak hours Device status check Review all smart devices Get overview of device status and energy consumption Water heater optimization Optimize water heater schedule Align heating schedule with TOU rates while ensuring hot water availability Pool pump scheduling Schedule pool pump operation Set pump runtime during optimal hours for savings Vacation preparation Configure multiple devices for vacation Set up HVAC, water heater, and pool pump to minimize energy while away Demand response event Handle demand response notification Adjust devices to reduce load during event period while maintaining comfort in its operational context constitutes a recognized evaluation approach [20].

Future Directions

Future work will replicate the framework across multiple households and utility structures, validate system rankings with real residential users, and apply the framework to emerging conversational HEMS architectures. The evaluation draws on data from a single household in Austin, Texas [40], chosen to provide a controlled context in which ground-truth values can be precisely defined. p. 7

Where the paper says it

This comparative analysis utilized two weeks of data from a single household (a 302. 4 m2 singlefamily house built in 2011 in Austin, Texas, with solar panels, an EV charger, a pool pump, an HVAC unit, and 14 total appliances; data from Pecan Street Inc.

Replication across diverse utility structures, climates, and device configurations is a priority. p. 1
Where the paper says it

(3) Can the proposed evaluation framework successfully distinguish different conversational HEMS’ performance? By addressing these questions, this study makes three major contributions: (1) it proposes and validates a comprehensive evaluation framework, grounded in the Goal-Question-Metric (GQM) methodology, comprising 23 metrics across five categories – task performance, factual accuracy, interaction quality, control capability, and system efficiency – for assessing conversational HEMS (RQ1); (2) it employs an LLM-as-judge approach for scalable automated scoring validated against human coders (RQ2); and (3) it applies this framework to compare four HEMS configurations across diverse scenarios and user personas, providing evidence of discriminative validity (RQ3).

The framework does not address actual energy savings, long-term user satisfaction, or the optimality of underlying scheduling logic.

A Home Energy Management Assistant (HEMA) employs a multi-agent architecture (Figure 7) comprising three specialized agents – Analysis, Knowledge, and Control – coordinated by a self-consistency classifier that runs four parallel Chain-of-Thought classification attempts and aggregates results through majority voting. p. 7

Where the paper says it

(3) A vanilla LLM fed with the comprehensive energy data analysis report and guided by a chain-of-thought (CoT) prompt (Vanilla+Energy Report+CoT). (4) A multi-agent architecture of HEMS, named Home Energy Management Assistant (HEMA) [24] (more details of HEMA are provided in Appendix A).

Each agent follows the ReAct paradigm [55], chaining tool calls with intermediate reasoning before synthesizing a response. p. 2
Where the paper says it

+ RAG OpenAI Assistants API ReAct + few-shot Multi-agent (orchestrator + specialists) LLM GPT-4o-mini Starling 7B, Qwen 75B GPT-4o Mistral 7B Llama 70B, Qwen 32B, GPT-OSS 120B Functional scope Energy-saving strategies Smart home actions Energy analysis, device control Parameter extraction Appliance scheduling Interaction type Single-turn Single-turn Single-turn Agent-driven sequential conversation Single-task User representation Researcherdesigned prompts Researcherdesigned prompts Researcherdesigned prompts LLM-as-user with three difficulty levels Researcherdesigned scenarios Evaluation metrics Accuracy User preference reflection in system response, latency Response accuracy, system performance* Parameter retrieval accuracy Success rate, system performance* * Including system latency, token usage Eng. : engineering, RAG: retrieval augmented generation, ReAct: reasoning and acting As summarized in Table 1, these systems span a range of LLM techniques from CoT prompting [24] and RAG-augmented recommendation [42] to multi-turn ReAct interfaces [37] and hierarchical multi-agent scheduling [34], with functional scopes ranging from energy analysis and device control [18, 19] to structured parameter extraction and appliance coordination.

The Analysis agent provides 15 tools for energy data retrieval and statistical analysis; the Knowledge agent provides 7 tools for RAG over energy documents, weather data, and built-in concept explanations. p. 2
Where the paper says it

(CoT) Prompt eng. + RAG OpenAI Assistants API ReAct + few-shot Multi-agent (orchestrator + specialists) LLM GPT-4o-mini Starling 7B, Qwen 75B GPT-4o Mistral 7B Llama 70B, Qwen 32B, GPT-OSS 120B Functional scope Energy-saving strategies Smart home actions Energy analysis, device control Parameter extraction Appliance scheduling Interaction type Single-turn Single-turn Single-turn Agent-driven sequential conversation Single-task User representation Researcherdesigned prompts Researcherdesigned prompts Researcherdesigned prompts LLM-as-user with three difficulty levels Researcherdesigned scenarios Evaluation metrics Accuracy User preference reflection in system response, latency Response accuracy, system performance* Parameter retrieval accuracy Success rate, system performance* * Including system latency, token usage Eng.

The Control agent provides 9 tools for device discovery, state monitoring, and command execution following a discovery-first validation approach. p. 5
Where the paper says it

6 Wooyoung Jung Table 2 continued from previous page Evaluation metric Description Scale Over-personalized response Count of responses referencing user-specific data to answer a general knowledge question not requiring personalization Count Under-personalized response Count of responses providing a generic answer to a data-specific question, failing to leverage available household data Count Appropriate general response Count of responses correctly providing a conceptual explanation for a general knowledge question Count Technical terms explained Count of domain-specific terms (e. , TOU, load shifting, kWh) defined or explained when first introduced Count Information-beforeaction rate Proportion of control actions preceded by the system communicating relevant device state or contextual information to the user before executing the action % (0–100%) Action confirmation rate Proportion of control actions followed by explicit confirmation of the completed action % (0–100%) Action explanation rate Proportion of control actions accompanied by an explanation of why the setting or schedule was chosen % (0–100%) Control Capability Target device accuracy Proportion of device changes affecting only the intended target device(s) without unintended side effects % (0–100%) Schedule correctness Proportion of load-shifting actions scheduled during off-peak hours per the utility’s TOU rate structure.

The system is implemented using LangGraph [51] for deterministic state-graph routing. p. 9
Where the paper says it

2 System Performance Evaluation The system performance evaluation procedure followed a two-stage design. Of the 23 metrics, eight were computed programmatically from system logs: turns to completion, average system response length, response latency, and token usage from conversation metadata, and target device accuracy, schedule correctness, mode correctness, and constraint compliance rate from deterministic comparison of device states before and after control actions.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve system evaluation by using a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. This provides a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.

  2. Implement an LLM as judge pipeline for scalable automated scoring of 23 metrics using iterative rubric calibration against human coders to ensure reliability. This enables the automated assessment of multi-turn conversations across different HEMS configurations at scale while mitigating stochastic variation through a three-run median voting procedure.

  3. Enhance system performance by utilizing an agentic architecture, such as Home Energy Management Assistant (HEMA), which comprises Analysis, Knowledge, and Control agents coordinated by a self-consistency classifier. This architecture enables superior personalization judgment and eliminates numerical fabrication by grounding claims in tool outputs.

  4. Improve interaction quality by explicitly measuring diagnostic granularity through the separation of over-personalization and under-personalization behaviors using a response appropriateness matrix (Table 3). This allows for targeted diagnosis of where and how a system falls short, such as identifying when inappropriately injecting user-specific data into responses to general knowledge questions occurs.

  5. Enhance control capability by evaluating schedule correctness and target device accuracy deterministically from system logs, while also analyzing action transparency metrics across different scenarios. This helps identify failure modes where the agent consistently communicated completed actions and their rationale to users, but frequently proceeded to execute device commands without first gathering or communicating relevant device state information.

Abstract

The growing complexity in home energy management (HEM) demands advanced systems that guide occupants toward informed energy decisions reflecting their background, preferences, and context. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. Whether such systems deliver effective interactions across the extended multi-turn dialogues typical of real-world use remains an open question. This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. Its reliability is validated against three trained human coders: after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC >= 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0.04-0.28). To demonstrate the framework's effectiveness, 970 dialogues -- 16 scenarios and five personas -- were generated and evaluated across four conversational HEMS configurations spanning a sophistication gradient, from a vanilla LLM with raw energy data to a multi-agent HEMS. The framework distinguished the four configurations across multiple evaluation dimensions, revealing their respective strengths and weaknesses. This study contributes to conversational HEMS by providing a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.

Sources

Related papers