Forecasting Daily Household Electricity Demand in a Hot-Arid City: A Comparative Benchmark of Local, Global, Deep and Foundation Models on ESHED-38
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Statistics. The day in statistics: methods, theory and the inference under them.
Tom: We're looking at the paper: "Forecasting Daily Household Electricity Demand in a Hot-Arid City".
Jane: Here's the gist of the paper. This study benchmarks 24 forecasting configurations on ESHED-38 to determine which model best predicts daily household electricity demand in a hot-arid city and reveals that zero-shot foundation models are among the most accurate options.
Tom: Benchmark Setup and Evaluation Protocol The study benchmarks 24 forecasting configurations on ESHED-38, a public record of daily smart-meter consumption for 38 homes in Erbil, Iraq.
Jane: Every configuration is evaluated on the same 54,720 forecasts, produced from 48 weekly origins across a held-out period of 359 days at lead times of up to 30 days.
Tom: The configurations span five base families, plus two cross-family ensembles: naive baselines, per-house statistical models, global gradient-boosted trees, global deep networks, and pre-trained time-series foundation models applied zero-shot.
Paper summary: Tom: So we’ve talked about what the paper is testing—benchmarking twenty four configurations on ESHED-thirty-eight to find the best daily demand predictor for that hot city <ref:2611.14535#pg1>. Now let's look at what they actually claim about their findings.
Jane: They conclude that the best methods reduce the error of a seasonal naive rule by twenty to twenty-five percent and those leading models beat that baseline in every house, which is a strong statement given the context of the data.
Lu: They also noted something about how consumption behaves here, saying daily demand is dominated by a slowly drifting level with a weak weekly cycle and only a seasonal swing on top, which explains why certain model types are performing better than others.
Meng: That sounds like it points toward the importance of recent data in these models. If the skill comes mostly from that recent mean, then models that heavily rely on historical averages might be more useful than complex architectures if they don't capture that drift well.
Lalam: The paper suggests why those pre trained models work so well: because the consumption pattern here is driven by this slowly drifting level, and the survey attributes only account for a tiny fraction of the total skill gain, about zero point one percent across all houses.
Tom: That’s a huge piece of information for anyone building a forecasting system. It tells us that trying to add more complex features from surveys or deep network architectures might not be paying off much compared to just getting the recent trend right.
Jane: And they also found something about the weather data itself, which is important because we need good inputs. They discovered that most of the weather record behaves like a repeated annual sequence rather than a measured daily series, with between sixty-four and seventy-four percent of dates flagged for each variable due to this.
Lu: That means the weather information used in their test period probably isn't actually the weather that occurred, which they call an unresolved issue, but it’s a major caveat for anyone trying to use ESHED-thirty-eight to study actual weather effects <ref:2611.14535#pg1>.
Meng: So if we were deploying this in a real system, we need to be careful how much weight we put on the weather variables when those variables are mostly just repeating annual sequences. What's the next thing they found regarding model performance?
Tom: They found that Chronos-two which is used without any training on this data, is the most accurate model one day ahead and it's significantly better than all twenty-three other configurations tested <ref:2611.14535#pg3>.
Jane: And at one week out, Chronos-two is statistically tied with TimesFM-two point five, Chronos Bolt and a cross family ensemble that was fixed before testing, and those ensembles lead at thirty days out <ref:2611.14535#pg3>. That shows the model's performance varies depending on the forecast horizon.
Lu: The researchers also looked into how much skill comes from different inputs for the global gradient boosted trees, finding that almost all of their skill comes from just the recent mean, which aligns with what we saw in their analysis of those tree models.
Lalam: It’s clear they found a strong performance win for zero shot foundation models in this specific benchmark, which is a big signal if you're looking at new AI techniques.
Conclusion: Tom: So to wrap up the paper "Forecasting Daily Household Electricity Demand in a Hot-Arid City: A Comparative Benchmark of Local, Global, Deep and Foundation Models on ESHED-thirty-eight", the main message is that while many complex models exist, they don't all offer massive gains here <ref:2611.14535#pg1,Forecasting Daily Household Electricity Demand in a Hot-Arid City: A Comparative>.
Jane: They found that the best methods reduce error by twenty to twenty-five percent compared to a seasonal naive rule in every house across this benchmark. The leading models beat that baseline in every single house tested.
Lu: The authors point out that for this specific demand, the deep networks trained from scratch don't outperform exponential smoothing on average, and the survey attributes only contribute about zero point one percent to the total gain.
Meng: For us who are thinking about deployment, it means that we shouldn't overcomplicate our model architecture if we can rely on getting a very accurate recent mean rather than chasing those deep architectures that don't consistently show better results.
Lalam: The conclusion is that zero shot foundation models are the most computationally efficient of the high performing options in this study, and Chronos-two is the best model one day ahead and needs only seconds of computation <ref:2611.14535#pg3>.
Tom: They also found a major limitation they have to mention, which is that they couldn't get firm conclusions about the value of weather information because it appears to be largely reconstructed from repeated annual sequences.
Jane: So for anyone looking at ESHED-thirty-eight to study weather effects, the paper strongly suggests that you need verified weather data for those specific periods if you want any reliable results <ref:2611.14535#pg1>.
Lu: That’s a very practical takeaway: don't trust the weather inputs in this dataset unless you can verify them yourself because they might just be repeating patterns.
Meng: Exactly. The work shows us what works and what doesn't, and for production, that often means sticking to simpler, more robust methods that rely on recent history rather than chasing the most complex AI architecture.
Lalam: Ultimately, the study helps narrow down which specific tool you should use based on the forecast horizon you need to cover.
stat.AP
Submitted: 2026-10-12
Updated: 2026-10-12
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 82/100
The gist: This study benchmarks 24 forecasting configurations on ESHED-38 to determine which model best predicts daily household electricity demand in a hot-arid city and reveals that zero-shot foundation…
Key concepts
- Zero-shot foundation models
- These are large AI models trained on vast amounts of general data without being specifically trained for electricity demand forecasting. The study found these were among the most accurate methods, performing well even without specific training on the ESHED-38 dataset.
- MASE (Mean Absolute Scaled Error)
- This is a metric used to evaluate forecast accuracy, scaling the error relative to a simple naive forecast. Lower MASE values indicate better performance in predicting daily household electricity demand over time.
- Weather Data Quality Screen
- The study found that most weather records behaved like repeated annual sequences rather than true daily measurements. A quality screen flagged 64-74% of dates for each variable, suggesting much of the recorded weather data was reused annually.
Terminology
Summary
This study benchmarks 24 forecasting configurations on ESHED-38 to determine which model best predicts daily household electricity demand in a hot-arid city and reveals that zero-shot foundation models are among the most accurate options. p. 0
Benchmark Setup and Evaluation Protocol
The study benchmarks 24 forecasting configurations on ESHED-38, a public record of daily smart-meter consumption for 38 homes in Erbil, Iraq. Every configuration is evaluated on the same 54,720 forecasts, produced from 48 weekly origins across a held-out period of 359 days at lead times of up to 30 days. p. 0 The configurations span five base families, plus two cross-family ensembles: naive baselines, per-house statistical models, global gradient-boosted trees, global deep networks, and pre-trained time-series foundation models applied zero-shot. p. 0 Every configuration is evaluated on the same 54,720 forecasts. p. 0
Model Performance Findings
The best configurations cut the geometric-mean scaled error of a seasonal-naive forecast by 20 to 25 percent. Chronos-2, used without any training on these data, is the most accurate model one day ahead and is significantly better there than all 23 alternatives. p. 0 At one week it is statistically tied with TimesFM-2. p. 05, Chronos-Bolt and a cross-family ensemble fixed before testing, and the two ensembles lead at 30 days. p. 0 Deep networks trained from scratch do not outperform exponential smoothing on average. p. 0 The household survey adds negligible skill. p. 0
Data Quality and Weather Analysis
A data-quality audit found that most of the weather record behaves like a repeated annual sequence rather than a measured daily series. The result is striking, as between 64 and 74 percent of dates are flagged for each variable. p. 4 These flags form three contiguous blocks, suggesting that much of the weather record was filled by reusing one annual sequence. p. 0 The value of weather information for these homes is therefore treated as unresolved, and the protocol is released as a baseline for future work. p. 0
Key Insights from Model Analysis
The results suggest why pre-trained models do so well, as daily consumption in these homes is dominated by a slowly drifting level, with a weak weekly cycle and a seasonal swing on top. The tree models’ importances and SHAP values say the same: almost all of their skill comes from the recent mean. The survey attributes help in 47, 47 and 66 percent of houses respectively, but all 22 survey attributes together account for 0. p. 31 percent of the total gain. p. 0
Conclusion
The best methods reduce the error of a seasonal-naive rule by 20 to 25 percent and beat it in every house. Zero-shot foundation models are the most computationally efficient of the high-performing options in this benchmark: Chronos-2 is significantly the best model one day ahead and needs seconds of computation. p. 0 A cross-family ensemble fixed before testing is the most robust option at longer horizons, statistically tied with the best foundation models. p. 0 Deep networks trained from scratch and global tree ensembles do not consistently or significantly outperform per-house exponential smoothing on a panel of this size. p. 0 The household survey provides negligible average benefit to LightGBM. p. 0 Finally, metered demand is driven by winter heating, and the weather record appears to be largely reconstructed from repeated annual sequences, which prevents any firm conclusion about the value of weather information. p. 8
Data and Code Availability
ESHED-38 is publicly available from Mendeley Data (doi:10.17632/dkts3x2yh5.2) under a CC BY 4.0 licence. The notebook that implements the complete pipeline, including data cleaning, the weather screen, all 24 configurations, evaluation and every figure, is available from the author on request. p. 1
Acknowledgements
The author thanks Truska Jaf and Ibrahim Ismael Hamarash for making the ESHED-38 dataset openly available.
References
[1] T. Jaf and I. I. Hamarash. Erbil Smart Home Energy Dataset – 38 houses [ESHED-38 (2020–2025)], Version 2. p. 2
[2] P. Montero-Manso and R. p. 2 J. Hyndman. Principles and algorithms for forecasting groups of time series: Locality and globality. p. 0
[3] A. F. Ansari, L. Stella, C. Turkmen, et al. Chronos: Learning the language of time series. p. 2
[4] A. Das, W. Kong, R. Sen, and Y. Zhou. A decoder-only foundation model for time-series forecasting. p. 2
[5] R. J. Hyndman and G. p. 2 Athanasopoulos. Forecasting: Principles and Practice. p. 2
[6] T. Hong and S. p. 1 Fan. Probabilistic electric load forecasting: A tutorial review. p. 0
[7] W. Kong, Z. Y. Dong, Y. Jia, D. J. Hill, Y. Xu, and Y. Zhang.
[8] S. Makridakis, E. Spiliotis, and V. Assimakopoulos.
[9] S. Makridakis, E. Spiliotis, and V. Assimakopoulos.
[10] R. J. Hyndman, A. p. 2 B. Koehler, R. p. 2 D. Snyder, and S. Grose.
[11] R. J. Hyndman and Y Khandakar. p. 2
[12] V Assimakopoulos and K Nikolopoulos.
[13] G Ke, Q Meng, T Finley, et al. LightGBM: A highly efficient gradient boosting decision tree. p. 2
[14] T Chen and C Guestrin. XGBoost: A scalable tree boosting system.
[15] C Challu, K G Olivares, B N Oreshkin, F Garza Ramirez, M Mergenthaler Canseco, and A Dubrawski.
[16] Y Nie, N H Nguyen, P Sinthong, and J Kalagnanam.
[17] B Lim, S O Arık, N Loeff, and T Pfister.
[18] D Salinas, V Flunkert, J Gasthaus, and T Januschowski.
[19] R. J Hyndman and A B Koehler. p. 2 Another look at measures of forecast accuracy. p. 0
[20] T Gneiting and A E Raftery. p. 2 Strictly proper scoring rules, prediction, and estimation. p. 2
[21] T Gneiting, F Balabdaoui, and A E Raftery. p. 2 Probabilistic forecasts, calibration and sharpness. p. 2
[22] J Demˇsar. p. 2 Statistical comparisons of classifiers over multiple data sets. p. 0
[23] M Friedman. p. 0 The use of ranks to avoid the assumption of normality implicit in the analysis of variance. p. 7
[24] P B Nemenyi. p. 0 Distribution-free multiple comparisons. p. 1
[25] F Wilcoxon. p. 0 Individual comparisons by ranking methods. p. 2
[26] S Holm. p. 0 A simple sequentially rejective multiple test procedure. p. 1
[27] L J Tashman. p. 2 Out-of-sample tests of forecasting accuracy: An analysis and review. p. 1
[28] C Bergmeir and J M Ben´ıtez. p. 2 On the use of cross-validation for time series predictor evaluation. p. 0
[29] V Vovk, A Gammerman, and G Shafer. Algorithmic Learning in a Random World. p. 8
[30] T Akiba, S Sano, T Yanase, T Ohta, and M Koyama. Optuna: A next-generation hyperparameter optimization framework. p. 0
[31] C Genest. Vincentization revisited.
[32] Nixtla. StatsForecast, MLForecast and NeuralForecast: Open-source forecasting libraries. p. 0
[33] Amazon Science. Chronos-2 model card. p. 0
[34] Google Research. p. 1 TimesFM 2.5 (200M, PyTorch) model card. p. 0
--- Page 6 ---
Figure 1: Daily consumption of every house (rows, ordered by median use) over the full record, on a log(1 + kWh) colour scale.
Figure 2: Daily station weather after the solar corrections: air temperature (mean, minimum, maximum), relative humidity, pressure, mean and gust wind speed, irradiance and daily solar energy. p. 5
Figure 3: Weather quality screen. p. 6
Figure 4: Distribution of daily consumption over all 70,718 house-days: histogram with kernel density (left), normal Q–Q plot (centre) and distribution of log(1 + kWh) (right). p. 7
Figure 5: Daily consumption per house, ordered by median. p. 7
Figure 6: Seven-day mean consumption across houses (dark line) with the band between the 10th and 90th percentile houses, and the seven-day mean air temperature (red, right axis). p. 8
Figure 7: Left: daily consumption against daily mean temperature for a random sample of house-days, with a LOWESS curve of the cross-house mean. p. 8
Figure 8: Day-of-week profile (left) and monthly profile (centre), each relative to the house mean with 95 percent confidence intervals, and the distribution of annual mean consumption per house for the complete years 2021–2024. p. 9
Figure 9: Autocorrelation (top, 60 lags) and partial autocorrelation (bottom, 30 lags) for a low-consumption (H17), a median (H29) and a high-consumption house (H32). p. 9
Figure 10: Pearson correlation between the cross-house mean and the daily weather variables, after the solar corrections. p. 5 p. 5
Figure 11: Share of houses that own or have each binary survey attribute (left), and the difference in mean daily consumption between houses with and without it (right). p. 7
Figure 12: Mean daily consumption per house against the numeric survey attributes, with Pearson correlations. p. 3
How the training signal is built
The features for global gradient-boosted trees include the house’s consumption lagged by 1, 2, 3, 7, 14, 21, 28 and 364 days; rolling means and a day of week. Each house’s series is standardised by its own mean and standard deviation before training. p. 7 For the weather track, it additionally includes degree days relative to comfort thresholds and calendar inputs like Friday indicators. p. 0 The most important weather feature is pressure (0. p. 37 percent), which hardly reflects a physical driver of consumption. p. 4
Where the gains come from
The leading configurations bring MASE down from 0.861 for the seasonal-naive forecast to 0. p. 0673–0.674, cut MAE from 7. p. 005. Chronos-2 is the best model one day ahead. p. 0 The seasonal-naive forecast has its worst origin a week later (MASE 1. p. 032 on 2 March), when consumption fell back as the weather warmed. p. 3 The finding that does matter is the one in Section 3. p. 13: the weather supplied to the models for the test period is probably not the weather that occurred. p. 0 The results support five conclusions. p. 1 The most important open item for anyone using ESHED-38 to study weather effects is verified weather data for the same period. p. 1
Limitations of the Study
Several limitations qualify these findings, including that all houses are in one city and share a single weather record, so microclimate effects cannot be studied. The survey was carried out after the metering period, so attributes may be wrong for earlier years, and it contains little variation. p. 3 The meter may not capture all of a household’s energy use, particularly power from generators. p. 8 The weather screen used here is heuristic. p. 0 With 38 houses many differences among the leading configurations are not statistically significant, and they are reported as ties. p. 0 The test covers a single year, and the December 2024 shock shows how strongly one period can affect averages. p. 3
--- Page 8 ---
Figure 9: Autocorrelation (top, 60 lags) and partial autocorrelation (bottom, 30 lags) for a low-consumption (H17), a median (H29) and a high-consumption house (H32).
Figure 10: Pearson correlation between the cross-house mean and the daily weather variables, after the solar corrections.
Figure 18: Pairwise win rate at h7: the share of the 38 houses in which the row configuration achieves a lower sCRPS than the column configuration. p. 0
Figure 20: MASE (h7) by calendar month of the target day for every configuration. p. 0
Figure 21: MASE (h7) by mean temperature of the target day, for the best configuration of each family and the seasonal-naive baseline. p. 0
Figure 22: MASE (h7) averaged over houses for each of the 48 forecast origins in the test period. p. 0
Figure 23: Left: reliability diagram for the best configuration of each family, plotting the observed share of outcomes below each quantile against its nominal level; Right: mean 80 percent interval width (relative to mean consumption) against observed coverage at h7. p. 8
Figure 24: Critical-difference diagram for sCRPS at h1 (Nemenyi, α = 0. p. 005, CD = 5.90).
Figure 25: Critical-difference diagram for sCRPS at h7. p. 4
Figure 26: Critical-difference diagram for sCRPS at h30. p. 3
Figure 27: Effect of adding the recorded weather (red) compared with the univariate version (blue) on MASE (left) and sCRPS (right) at h7. p. 8
Figure 28: Percentage change in MASE (h7) from adding weather, scored separately on dates whose temperature passed the quality screen and on flagged dates. p. 5
--- Page 30 ---
Figure 29: Survey-attribute ablation for LightGBM [exog]: per-house difference in sCRPS between the model with and without survey attributes, sorted, at h1, h7 and h30.
Figure 30: Normalised split-gain importance of the 35 most important LightGBM [exog] features. p. 3
Figure 31: SHAP summary for LightGBM [exog]. p. 2
Figure 32: Optuna tuning of LightGBM: best validation MASE found after each trial, for both tracks.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Foundation models can produce forecasts of nearbest accuracy for new datasets
with well-calibrated intervals, for a new set of homes in seconds
because they learn to recognize thatdaily consumption in these homes is dominated by a slowly drifting level, with a weak weekly cycle and a seasonal swing on top.
-
The system can be optimized by focusing on recent consumption levels, as the analysis showed
almost all of their skill comes from the recent mean,
which suggests thata robust generalist is hard to beat
when the signal is mostly level and noise. -
When deployed for new locations, the system can generate forecasts without needing
tuning, no training history and no choice of features
by leveraging pre-trained models like Chronos-2 or TimesFM-2.5 directly on the new data. -
The system can provide a more robust forecast for longer horizons by using a cross-family ensemble fixed before testing, which is
statistically tied with the best foundation models.
Related papers
- Unraveling time-varying causal effects of multiple exposures: integrating Functional Data Analysis with Multivariable Mendelian Randomization
- Strategic Causal Policy Learning with Manipulable Eligibility
- BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
- Automating structural reliability analysis with a multi-agent large language model framework
- Improving Minority Population Sampling with BISG Probabilities: Evidence from a Survey of Jewish Americans
- When Do Sports Leagues Stop Being Competitive? Bayesian Detection of Competitive-Balance Eras from Match-Level Data