LEAF: Navigating Concept Drift in Cellular Networks

arXiv:2109.03011 · cs.NI, cs.LG, cs.PF · Submitted 2021-09-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LEAF: Navigating Concept Drift in Cellular Networks".

Jane: The gist: LEAF is a new methodology for concept drift mitigation that detects, explains,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now that we know the context, let's look at what LEAF actually does in the paper. The paper introduces LEAF as this explainable machine learning approach for networking models that works with any supervised regressors.

Jane: It sounds like they are building a pipeline with three main parts: first, a drift detector; second, tools to explain drift for features; and finally, a drift mitigator.

Lu: That's the structure they propose; it’s meant to be transparent about how the system finds and fixes that concept drift. They use Kolmogorov-Smirnov Windowing on time-series of estimated errors to detect when a model drifts, which tells us when something is happening.

Meng: So it starts by flagging the change in the distribution of those output errors, which is a pretty standard way to spot a shift in performance metrics. But how do they actually figure out *why* it's drifting?

Lalam: That's where the explainer module comes in; they want to characterize model errors simply based on the model input and output. They rank features by permutation-based feature importance and then group features by their correlations to find what’s driving the drift.

The paper's summary: Tom: It seems like the most important part of this paper is how they explain that drift using Local Error Approximation, or LEA. They develop LEAplot and LEAgram to visualize local error components.

Jane: So these tools help operators see for which features, where, and how much concept drift is occurring by mapping it to the under-trained region of a model. That should give network engineers some real guidance instead of just a generic alarm.

Lu: And this isn't just about spotting the change; they are also focused on informed mitigation. They use forgetting and over-sampling strategies based on information from the drift explainer, combined with dispersion, which is like the coefficient of variance or standard deviation for those KPIs.

Meng: So if the dispersion of a KPI is high, they switch to a different strategy—they use non-linear cubic weights of LEA to focus only on those regions of high error in the newest drifting instances. That's an adaptive way to retrain.

The paper's improvements: Tom: So what makes this approach better than just periodic or triggered retraining? The paper shows that LEAF consistently outperforms existing mitigation approaches, and it reduces costly retraining operations by as much as seventy-six point nine percent compared to periodic retraining.

Jane: That reduction in operation cost is huge for anyone running these kinds of large-scale models constantly, because retraining can be very expensive. And they show it beats baseline schemes across all models and KPIs except for CDR, which they flag as an exception.

Lu: For example, for the LSTM model, LEAF achieves a seven point seven one percent to fifty point one three percent less NRMSE compared to triggered retraining when forecasting demand, which is a much better number than what triggered retraining gets them.

Meng: I’m interested in that NRMSE comparison because it's a direct measure of forecast error; if LEAF can bring that down so drastically, it means the model is actually getting much smarter faster.

Conclusion: Tom: So to wrap up, the paper "LEAF: Navigating Concept Drift in Cellular Networks" uses explainable AI and informed mitigation to handle concept drift in cellular demand forecasting. It consistently outperforms periodic and triggered retraining while cutting down on retraining costs significantly.

Jane: The implication is that this framework isn't just a theoretical exercise; it offers a practical way to deploy ML models in dynamic networking environments by giving operators actionable explanations for *why* the model is struggling.

Lu: From my view, the real power here is the ability to map spatio-temporal changes of error with plausible reasons, like hardware installations or firmware updates, because they use LEAgram for that decomposition.

Meng: I think from an engineering standpoint, being able to use that information to select the optimal data subset for retraining based on dispersion levels is what makes it useful in practice.

Lalam: It shows how we can build more reliable and interpretable AI systems by making the drift detection process itself explainable, which helps build trust in these complex forecasting tools.

Shinan Liu, Francesco Bronzino†, Paul Schmitt‡

University of Chicago · École Normale Supérieure de Lyon · University of Hawaii

cs.NI, cs.LG, cs.PF

Submitted: 2021-09-07

Updated: 2023-02-02

Journal ref: Proc. ACM Netw., Vol. 1, No. CoNEXT2, Article 7. Publication date: September 2023

DOI: 10.1145/3609422

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 82/100

The gist: The gist: LEAF is a new methodology for concept drift mitigation that detects, explains, and mitigates drift in regression-based machine learning models applied to cellular network Key Performance

Key concepts

Concept Drift: This occurs when the underlying relationship between input data (features) and the desired output (target KPI) changes over time. In networking, this means a model trained on old traffic patterns becomes inaccurate when new traffic patterns emerge, requiring adaptation.
Local Error Approximation of Features (LEAF): LEAF is an explainable framework that detects drift using a distribution change detector. It then identifies which specific features and time intervals are causing the most error by approximating the model's error locally. This allows for targeted mitigation rather than retraining the entire model.
Informed Mitigation: This strategy uses LEAF's explanation to decide how to retrain. It combines forgetting and over-sampling with dispersion metrics (like variance) of KPIs. If a KPI is highly variable, the system focuses on high-error regions using non-linear weights, ensuring retraining targets the most relevant data for adaptation.

Terminology

Concept Drift — 7× in this paper · explained in 5 episodes
Accuracy — 2× in this paper · explained in 1 episode
Local Error Approximation of Features — 1× in this paper

Summary

The gist: LEAF is a new methodology for concept drift mitigation that detects, explains, and mitigates drift in regression-based machine learning models applied to cellular network Key Performance Indicators (KPIs) by leveraging local error approximation.

Characterization of Concept Drift

The paper first characterizes concept drift in a large cellular network for a major metropolitan area in the United States using more than four years of KPI data The authors find that concept drift occurs across many important key performance indicators (KPIs), independently of the model, training set size, and time interval They demonstrate concept drift in a large cellular network comparing different KPIs, model families, training set sizes, and periods across many regression models and tasks The diverse, longitudinal nature of the dataset used presents a challenging concept drift problem

LEAF Framework Overview

LEAF is introduced as an explainable ML approach for networking models that works with any supervised regressors and provides explanations for black-box models The framework implements a pipeline of three sequential components: (1) a drift detector, (2) a set of tools to explain drift for features, and (3) a drift mitigator The detector applies the well-known Kolmogorov-Smirnov Windowing (KSWIN) method on the time-series to identify a change in the distribution of the output error, providing an indicator of whether drift is occurring

Explaining Drift

The explainer module is designed around the goal of characterizing errors of a regression model simply based on the model input and output It determines the most representative features that contribute to drift by first ranking features by permutation-based feature importance and then grouping features by their correlations The Local Error Approximation (LEA) is used to decompose the error of a model on any corresponding dataset using selected representative features as input LEAplot and LEAgram are developed to visualize local error components and provide guidance for model compensation

Informed Mitigation

The mitigation module is based on the idea of informed adaptation, where the key is to find the optimal data subset to retrain the model and mitigate drift This involves forgetting and over-sampling strategies using information from LEAF’s drift explainer, combined with dispersion (coefficient of variance or Std/Mean) of the target KPIs For instance, when dispersion is high, the strategy uses non-linear (cubic) weights of LEA to focus on regions of high error in the latest drifting instances The framework also allows for multi-group LEAF, which iteratively optimizes which retrain data to use based on the input of the drift explanation component

Evaluation and Performance

LEAF consistently outperforms existing mitigation approaches, such as periodic and triggered retraining, while reducing costly retraining operations by as much as 76.9% (compared to periodic retraining) The evaluation shows that LEAF consistently outperforms all baseline schemes across models and KPIs For instance, for LSTM, LEAF is drastically better at reducing NRMSE compared to triggered retraining, achieving 7.71% to 50.13% less NRMSE than triggered retraining Furthermore, LEAF consistently improves model performance compared to the static model when applied to DVol forecasting, with the NRMSE going as low as 0.02 The results demonstrate that LEAF outperforms other methods across KPIs, except for CDR

Conclusion

The LEAF framework employs explainable AI and informed mitigation to address the critical problem of concept drift in cellular demand forecasting The results based on more than four years of KPI data show that LEAF consistently outperforms both periodic and triggered retraining while reducing the cost of retraining The authors believe that the LEAF framework can be applied beyond cellular networks, to other network management problems that use black-box models for regression-based prediction However, the evaluation of LEAF to date has been conducted on fully labeled datasets; thus, a promising direction could be to improve LEAF to cope with semi-supervised or unsupervised models with partial or no labels.

References

[1] AutoGluon AI. accessed July, 2021. AutoGluon: AutoML for Text, Image, and Tabular Data. https://auto.gluon.ai/stable/index.html

[2] David Alvarez-Melis and Tommi S Jaakkola. 2018. On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049 (2018).

[3] Daniel W Apley and Jingyu Zhu. 2020. Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82, 4 (2020), 1059–1086.

[4] Behnaz Arzani, Kevin Hsieh, and Haoxian Chen. 2021. Interpretable Feedback for AutoML and a Proposal for Domain-customized AutoML for Networking. In Proceedings of the Twentieth ACM Workshop on Hot Topics in Networks. 53–60.

[5] Sara Ayoubi, Noura Limam, Mohammad A Salahuddin, Nashid Shahriar, Raouf Boutaba, Felipe Estrada-Solano, and Oscar M Caicedo. 2018. Machine learning for cognitive network management. IEEE Communications Magazine 56, 1 (2018), 158–165.

[6] Stephen H Bach and Marcus A Maloof. 2008. Paired learners for concept drift. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, 23–32.

[7] Federico Barbero, Feargus Pendlebury, Fabio Pierazzi, and Lorenzo Cavallaro. 2020. Transcending transcend: Revisiting malware classification with conformal evaluation. arXiv preprint arXiv:2010.03856 (2020).

[8] Roberto Souto Maior Barros and Silas Garrido T Carvalho Santos. 2018. A large-scale comparison of concept drift detectors. Information Sciences 451 (2018), 348–370.

[9] Albert Bifet and Ricard Gavalda. 2007. Learning from time-changing data with adaptive windowing. In Proceedings of the 2007 SIAM international conference on data mining. SIAM, 443–448.

[10] Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32.

[16] Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363 (2018).

[27] Alex Kantchelian, Sadia Afroz, Ling Huang, Aylin Caliskan Islam, Brad Miller, Michael Carl Tschantz, Rachel Greenstadt, Anthony D Joseph, and JD Tygar. 2013. Approaches to adversarial drift. In Proceedings of the 2013 ACM workshop on Artificial intelligence and security. 99–110.

[28] Bartosz Krawczyk, Leandro L Minku, Joao Gama, Jerzy Stefanowski, and Michał Woźniak. 2017. Ensemble learning for data stream analysis:

Improvements for AI systems

Bold Header: Concept Drift Detection via KSWIN

The system can detect drift by applying Kolmogorov-Smirnov Windowing (KSWIN) [41, 49] on the time-series to identify a change in the distribution of the output error, providing an indicator of whether drift is occurring. This allows for prompt mitigation when sudden changes occur, such as during COVID-19 lockdowns.

Bold Header: Explainable Drift Attribution via LEAgram

The system can provide an end-to-end interpretation of a model or a mitigation scheme by using LEAgram to show the decomposition of CatBoost NRMSE time-series, allowing operators to associate spatio-temporal changes of error with plausible reasons (e.g., hardware installations, firmware updates, etc.) based on the location and duration of events.

Bold Header: Targeted Mitigation via Informed Adaptation

The system can apply forgetting and over-sampling strategies using the information derived from LEAF to compensate for concept drift, specifically by using non-linear (cubic) weights [29] of EL to focus on the regions of high error in the latest drifting instances when KPI dispersion is high.

Abstract

Operational networks commonly rely on machine learning models for many tasks, including detecting anomalies, inferring application performance, and forecasting demand. Yet, model accuracy can degrade due to concept drift, whereby the relationship between the features and the target to be predicted changes. Mitigating concept drift is an essential part of operationalizing machine learning models in general, but is of particular importance in networking's highly dynamic deployment environments. In this paper, we first characterize concept drift in a large cellular network for a major metropolitan area in the United States. We find that concept drift occurs across many important key performance indicators (KPIs), independently of the model, training set size, and time interval -- thus necessitating practical approaches to detect, explain, and mitigate it. We then show that frequent model retraining with newly available data is not sufficient to mitigate concept drift, and can even degrade model accuracy further. Finally, we develop a new methodology for concept drift mitigation, Local Error Approximation of Features (LEAF). LEAF works by detecting drift; explaining the features and time intervals that contribute the most to drift; and mitigates it using forgetting and over-sampling. We evaluate LEAF against industry-standard mitigation approaches (notably, periodic retraining) with more than four years of cellular KPI data. Our initial tests with a major cellular provider in the US show that LEAF consistently outperforms periodic and triggered retraining on complex, real-world data while reducing costly retraining operations.