Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data".
Dev: Here's the gist of the paper. Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques This paper presents a methodology for the automated analysis and visualization of correlation structures in large PQ datasets The approach adapts an existing framework for shorter observation periods and enhances it with aggregation and distance-based visualization techniques Daily Spearman correlation coefficients are averaged via Fisher’s z-transformation and aggregated across phases, parameters, and sites The resulting correlation structures are visualized using hierarchical clustering and multidimensional scaling to reveal consistent and recurring relationships Correlation Analysis Techniques The methodology distinguishes between two types of correlation: autocorrelation and cross-correlation The Spearman rank correlation coefficient is a nonparametric measure for assessing the association between two time series This coefficient captures monotonic dependencies, i.e., variables that consistently increase or decrease together Mean correlation coefficients are calculated using the Fisher z-transformation to appropriately average daily coefficients and phase combinations Aggregation and Distance Representation Correlation matrices are constructed between PQ parameters for multiple measurement sites, which results in a large number of correlation matrices The aggregation matrix A is defined as A = Σ c ij with c ij = Σ 1(r̅ ij ≥ τ where 1(⋅) denotes the indicator function Distance is calculated using D ij = Σ (1 - A ij, 0 for i ≠ j and D ii = 0 This distance matrix is suitable for clustering and projection-based visualization Visualization Results Hierarchical clustering visualizes the share of significant correlations between voltage and current PQ parameters derived from the aggregation matrix Multidimensional scaling (MDS) embeds the distance matrix into a low-dimensional space while preserving pairwise distances as accurately as possible The results reveal that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized Site Correlation Analysis The methodology was demonstrated using correlations between different measurement sites, identifying three exemplary groups of measurement sites Group 1 comprises sites that exhibit no or only very few correlations with other sites Group 2 includes measurement sites that show a higher mutual share of significant correlations, between 20 % and 60 % of the PQ parameters Group 3 consists of measurement sites from the high-voltage level (110 kV), which also exhibit pronounced mutual correlations The highest shares occur for voltage quality parameters U03 (47 %) and Upst (33 %), indicating that variations of these parameters tend to propagate over large parts of the network The paper introduced a methodology for analyzing largescale power quality correlation structures using robust averaging and aggregation techniques By using hierarchical clustering and multidimensional scaling, complex dependencies become interpretable, revealing that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized The approach enables systematic comparison across parameters and measurement sites and supports the identification of robust, recurring correlation patterns Future research will focus on correlation-based segmentation to identify temporal changes, operational regimes, and periods of similar power quality behavior ACKNOWLEDGMENT The authors would like to thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for funding this project (project number 521923789) REFERENCES
1: M.
Rosa: Domagk, J.
Dev: Meyer, T.
Rosa: Wang, D.
Paper summary: Rosa: So looking at the "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data" paper by Domagk and Meyer, it seems like the main point is that they created a robust methodology for handling those massive power quality datasets.
Dev: They tackled how to find systematic dependencies that are usually invisible with conventional analysis techniques by using aggregation and distance based visualization methods.
Rosa: The authors adapt an existing framework for shorter observation periods and enhance it with ways to aggregate daily Spearman correlation coefficients across phases, parameters, and sites.
Taro: It matters because they use hierarchical clustering and multidimensional scaling to visualize these structures, showing that low-order harmonics and flicker have widespread spatial correlations while higher-order harmonics are more localized.
Dev: They also found that the measurement sites naturally fall into groups based on their correlation patterns with other sites, identifying groups one through three based on mutual significant correlations.
Rosa: The implication is that this method helps us systematically compare parameters across measurement sites and pin down those robust, recurring patterns in power quality behavior.
Taro: It gives us a way to see where the most common problems are spreading versus where the rare ones are stuck locally in the network structure.
Conclusion: Rosa: So, to wrap up this part, we’ve seen how they used these correlation structures—those patterns of how different power quality things link together across many sites and measurements—to find some very clear groupings in the data.
Dev: Yeah, they took all that massive dataset and used aggregation and distance representation to make sense of it.
Rosa: The title itself, "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data," it just tells you what they did: they found patterns across a huge amount of power quality information.
Dev: It sounds like they were trying to take something that’s usually just a giant pile of numbers and turn it into something actually understandable for engineers.
Rosa: Exactly. They showed that by using tools like hierarchical clustering and multidimensional scaling, those complex dependencies become visible, not just as random noise.
Dev: And what they found is interesting because it splits the power quality issues into two main types of things: low-order harmonics and flicker versus higher-order harmonics.
Rosa: Low-order stuff, like the basic voltage or current signals, those are showing up with pretty widespread correlations across different parts of the system.
Dev: But those higher-order harmonics? They tend to stay very localized. That means you don't see them everywhere; they stick around specific places in the network.
Rosa: And when they look at different measurement sites, they found these three distinct groups based on how much they correlate with each other.
Dev: Group one has almost no connections to anything else, Group two has a decent mix of correlations, and then Group three is made up of those high voltage sites that show the strongest mutual links.
Rosa: That suggests that if you want to understand power quality across a whole network, you don't have to look at every single measurement in isolation.
Dev: It points toward a way for operators to systematically compare parameters and identify these robust, recurring patterns instead of just looking at isolated events.
TUD Dresden University of Technology · TransnetBW
eess.SP
Submitted: 2026-03-13
Updated: 2026-03-13
Comments: 5 pages, 10 figures, submitted to IEEE conferences
Journal ref: 2026 22nd International Conference on Harmonics & Quality of Power (ICHQP), Dresden, Germany, 2026, pp. 1-5
DOI: 10.1109/ICHQP67927.2026.11756843
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 76/100
The gist: Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques This…
Key concepts
- Spearman Rank Correlation Coefficient
- This is a statistical tool used to measure the relationship between two time series without assuming they follow a normal distribution. It specifically identifies monotonic dependencies, meaning variables that consistently increase or decrease together, which is crucial for understanding how power quality components interact over time.
- Aggregation Matrix (A)
- This matrix summarizes the correlations across many measurement sites and parameters. It aggregates individual correlation coefficients based on a threshold ($ au$), creating a single representation of the overall correlation structure that is useful for subsequent distance calculations and clustering.
- Multidimensional Scaling (MDS)
- MDS is a visualization technique that takes high-dimensional distance data, like the calculated distance matrix between sites, and projects it into a lower-dimensional space (e.g., 2D or 3D). This allows researchers to visually map out how different measurement locations are related based on their correlation patterns.
- Hierarchical Clustering
- This method groups data points (in this case, measurement sites) based on the similarity of their relationships. It builds a tree-like structure showing which sites are most closely correlated with each other, helping to identify distinct groups of network behavior.
Terminology
Summary
Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques This paper presents a methodology for the automated analysis and visualization of correlation structures in large PQ datasets
Methodology Overview
The approach adapts an existing framework for shorter observation periods and enhances it with aggregation and distance-based visualization techniques Daily Spearman correlation coefficients are averaged via Fisher’s z-transformation and aggregated across phases, parameters, and sites The resulting correlation structures are visualized using hierarchical clustering and multidimensional scaling to reveal consistent and recurring relationships
Correlation Analysis Techniques
The methodology distinguishes between two types of correlation: autocorrelation and cross-correlation The Spearman rank correlation coefficient is a nonparametric measure for assessing the association between two time series This coefficient captures monotonic dependencies, i.e., variables that consistently increase or decrease together Mean correlation coefficients are calculated using the Fisher z-transformation to appropriately average daily coefficients and phase combinations
Aggregation and Distance Representation
Correlation matrices are constructed between PQ parameters for multiple measurement sites, which results in a large number of correlation matrices The aggregation matrix A is defined as A = Σ c ij with c ij = Σ 1(r̅ ij ≥ τ where 1(⋅) denotes the indicator function Distance is calculated using D ij = Σ (1 - A ij, 0 for i ≠ j and D ii = 0 This distance matrix is suitable for clustering and projection-based visualization
Visualization Results
Hierarchical clustering visualizes the share of significant correlations between voltage and current PQ parameters derived from the aggregation matrix Multidimensional scaling (MDS) embeds the distance matrix into a low-dimensional space while preserving pairwise distances as accurately as possible The results reveal that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized
Site Correlation Analysis
The methodology was demonstrated using correlations between different measurement sites, identifying three exemplary groups of measurement sites Group 1 comprises sites that exhibit no or only very few correlations with other sites Group 2 includes measurement sites that show a higher mutual share of significant correlations, between 20 % and 60 % of the PQ parameters Group 3 consists of measurement sites from the high-voltage level (110 kV), which also exhibit pronounced mutual correlations The highest shares occur for voltage quality parameters U03 (47 %) and Upst (33 %), indicating that variations of these parameters tend to propagate over large parts of the network
Conclusions
The paper introduced a methodology for analyzing largescale power quality correlation structures using robust averaging and aggregation techniques By using hierarchical clustering and multidimensional scaling, complex dependencies become interpretable, revealing that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized The approach enables systematic comparison across parameters and measurement sites and supports the identification of robust, recurring correlation patterns Future research will focus on correlation-based segmentation to identify temporal changes, operational regimes, and periods of similar power quality behavior
ACKNOWLEDGMENT
The authors would like to thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for funding this project (project number 521923789)
REFERENCES
[1] M. Domagk, J. Meyer, T. Wang, D. Feng, and W. p. 4 REFERENCES
[1] M. Feng, and W. Feng, and W. Huang, ‘Automatic
Identification of Correlations in Large Amounts of Power Quality Data
from Long-Term Measurement Campaigns’, in 2021 26th International
Conference and Exhibition on Electricity Distribution (CIRED), Online
Conference,: Institution of Engineering and Technology, Nov.Where the paper says it
Where the paper says it
[2] R. A. Fisher, ‘Frequency Distribution of the Values of the Correlation Coefficient in Samples from an Indefinitely Large Population’, Biometrika, vol. p. 4 Huang, ‘Automatic
Identification of Correlations in Large Amounts of Power Quality Data
from Long-Term Measurement Campaigns’, in 2021 26th International
Conference and Exhibition on Electricity Distribution (CIRED), Online
Conference,: Institution of Engineering and Technology, Nov. Fisher, ‘Frequency Distribution of the Values of the Correlation
Coefficient in Samples from an Indefinitely Large Population’,
Biometrika, vol. Fisher, ‘Frequency Distribution of the Values of the Correlation
Coefficient in Samples from an Indefinitely Large Population’,
Biometrika, vol. 4, 1915, doi: 10.Where the paper says it
Where the paper says it
[3] F. Znidi, H. Davarikia, and H. p. 4 2307/2331838. Davarikia, and H. Davarikia, and H. Rathore, ‘Power Systems Transient
Stability Indices: Hierarchical Clustering Based Detection of Coherent
Groups Of Generators’, in 2024 IEEE Texas Power and Energy
Conference (TPEC), Feb. Rathore, ‘Power Systems Transient
Stability Indices: Hierarchical Clustering Based Detection of Coherent
Groups Of Generators’, in 2024 IEEE Texas Power and Energy
Conference (TPEC), Feb. 1109/TPEC60005.Where the paper says it
Where the paper says it
Where the paper says it
[4] W. S. Torgerson, ‘Multidimensional scaling: I. p. 4 1109/TPEC60005. Torgerson, ‘Multidimensional scaling: I. Torgerson, ‘Multidimensional scaling: I. Theory and method’,
Psychometrika, vol. Theory and method’,
Psychometrika, vol. 401–419, Dec. 401–419, Dec. 1952, doi:
10.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[5] F. Belmudes, D. p. 4 1007/BF02288916. Belmudes, D. Belmudes, D. Ernst, and L. Ernst, and L. Wehenkel, ‘Pseudo-Geographical
Representations of Power System Buses by Multidimensional Scaling’,
in 2009 15th International Conference on Intelligent System
Applications to Power Systems, Nov.Where the paper says it
Where the paper says it
Where the paper says it
[6] V. Wei, N. Ivkin, V. Braverman, and A. p. 4 Wehenkel, ‘Pseudo-Geographical
Representations of Power System Buses by Multidimensional Scaling’,
in 2009 15th International Conference on Intelligent System
Applications to Power Systems, Nov. Braverman, and A. Braverman, and A. Szalay, ‘Sketch and Scale
Geo-distributed tSNE and UMAP’, in 2020 IEEE International
Conference on Big Data (Big Data), Dec. Szalay, ‘Sketch and Scale
Geo-distributed tSNE and UMAP’, in 2020 IEEE International
Conference on Big Data (Big Data), Dec. 996–1003.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated identification of systematic dependencies in power quality data can be achieved by applying a methodology that adapts long-term correlation analysis to shorter observation periods, as described in Section I. This allows AI systems to detect recurring relationships that might otherwise be missed due to data volume or observation length constraints.
-
The system can perform robust estimation of parameter relationships by calculating mean correlation coefficients using Fisher's z-transformation instead of relying on single correlation values over the entire period, as shown in Section III C. This compensates for the non-linearity of the distribution, providing a more reliable measure of dependency strength.
-
AI systems can efficiently summarize large sets of pairwise correlations across multiple measurement sites by constructing an aggregation matrix where c i j is defined as a sum over significant correlations (Section IV). This provides a compact representation highlighting robust correlation patterns rather than analyzing every individual site-to-site correlation matrix directly.
-
A system can visualize complex correlation structures through hierarchical clustering, which preserves pairwise relationships when using aggregation-based distances (Section V B). This allows for the interpretation of shared significant correlations between voltage and current parameters across different measurement sites by observing early merges in the dendrogram (Figure 4).
-
The AI system can project PQ parameter relationships into a lower-dimensional space using multidimensional scaling to visualize how parameters are positioned based on their aggregation-based distances (Section V C). This allows for the identification of clusters, such as Group 1 sites exhibiting no or very few correlations, and Group 2 sites with high mutual correlation shares.
-
The system can automatically distinguish between different site types by analyzing the resulting clustering and MDS visualizations (Section VI). It can identify groups like Group 3 sites from the high-voltage level (110 kV) that exhibit pronounced mutual correlations, which is crucial for understanding network propagation behavior.
Abstract
Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques. This paper presents a methodology for the automated analysis and visualization of correlation structures in large PQ datasets. Building on an existing framework, the approach is adapted for shorter observation periods and enhanced with aggregation and distance-based visualization techniques. Daily Spearman correlation coefficients are averaged via Fishers z-transformation and aggregated across phases, parameters, and sites. The resulting correlation structures are visualized using hierarchical clustering and multidimensional scaling to reveal consistent and recurring relationships. The methodology is demonstrated using data from 85 measurement sites within the German transmission system.
Related papers
- Open-H-RF: A Large-Scale Ultrasound Raw Channel Dataset Initiative
- Scalable Attention for 5G NR Channel Estimation
- Low-Altitude Wireless Networks: A Comprehensive Survey
- Sensing-Assisted OFDM with Index Modulation: A Waveform for Optical Wireless Integrated Communication and Localisation
- Iteration-Free Candidate Pruning and an Index-Domain VLSI Architecture for Sort-Free MIMO Box Decoding up to 1024-QAM
- A Foundation Model for Large-Scale Wireless Network Planning, Operation and Optimization