Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data
summary
The gist
Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques This…
In short
The methodology analyzes massive power quality datasets to find hidden dependencies between different parameters and measurement locations. It uses statistical averaging and distance metrics to visualize these relationships, revealing that low-order harmonics show broad network influence while higher-order ones are localized.
Key concepts
- Spearman Rank Correlation Coefficient
- This is a statistical tool used to measure the relationship between two time series without assuming they follow a normal distribution. It specifically identifies monotonic dependencies, meaning variables that consistently increase or decrease together, which is crucial for understanding how power quality components interact over time.
- Aggregation Matrix (A)
- This matrix summarizes the correlations across many measurement sites and parameters. It aggregates individual correlation coefficients based on a threshold ($ au$), creating a single representation of the overall correlation structure that is useful for subsequent distance calculations and clustering.
- Multidimensional Scaling (MDS)
- MDS is a visualization technique that takes high-dimensional distance data, like the calculated distance matrix between sites, and projects it into a lower-dimensional space (e.g., 2D or 3D). This allows researchers to visually map out how different measurement locations are related based on their correlation patterns.
- Hierarchical Clustering
- This method groups data points (in this case, measurement sites) based on the similarity of their relationships. It builds a tree-like structure showing which sites are most closely correlated with each other, helping to identify distinct groups of network behavior.
Terminology used across episodes
This episode discusses
- Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data · Paper Radio
The paper
Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data · Read on arXiv
TUD Dresden University of Technology · TransnetBW
Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques. This paper presents a methodology for the automated analysis and visualization of correlation structures in large PQ datasets. Building on an existing framework, the approach is adapted for shorter observation periods and enhanced with aggregation and distance-based visualization techniques. Daily Spearman correlation coefficients are averaged via Fishers z-transformation and aggregated across phases, parameters, and sites. The resulting correlation structures are visualized using hierarchical clustering and multidimensional scaling to reveal consistent and recurring relationships. The methodology is demonstrated using data from 85 measurement sites within the German transmission system.
DOI: 10.1109/ICHQP67927.2026.11756843
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data".
Dev: Here's the gist of the paper. Large-scale power quality (PQ) measurement campaigns generate vast amounts of multivariate data, in which systematic dependencies are difficult to identify using conventional analysis techniques This paper presents a methodology for the automated analysis and visualization of correlation structures in large PQ datasets The approach adapts an existing framework for shorter observation periods and enhances it with aggregation and distance-based visualization techniques Daily Spearman correlation coefficients are averaged via Fisher’s z-transformation and aggregated across phases, parameters, and sites The resulting correlation structures are visualized using hierarchical clustering and multidimensional scaling to reveal consistent and recurring relationships Correlation Analysis Techniques The methodology distinguishes between two types of correlation: autocorrelation and cross-correlation The Spearman rank correlation coefficient is a nonparametric measure for assessing the association between two time series This coefficient captures monotonic dependencies, i.e., variables that consistently increase or decrease together Mean correlation coefficients are calculated using the Fisher z-transformation to appropriately average daily coefficients and phase combinations Aggregation and Distance Representation Correlation matrices are constructed between PQ parameters for multiple measurement sites, which results in a large number of correlation matrices The aggregation matrix A is defined as A = Σ c ij with c ij = Σ 1(r̅ ij ≥ τ where 1(⋅) denotes the indicator function Distance is calculated using D ij = Σ (1 - A ij, 0 for i ≠ j and D ii = 0 This distance matrix is suitable for clustering and projection-based visualization Visualization Results Hierarchical clustering visualizes the share of significant correlations between voltage and current PQ parameters derived from the aggregation matrix Multidimensional scaling (MDS) embeds the distance matrix into a low-dimensional space while preserving pairwise distances as accurately as possible The results reveal that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized Site Correlation Analysis The methodology was demonstrated using correlations between different measurement sites, identifying three exemplary groups of measurement sites Group 1 comprises sites that exhibit no or only very few correlations with other sites Group 2 includes measurement sites that show a higher mutual share of significant correlations, between 20 % and 60 % of the PQ parameters Group 3 consists of measurement sites from the high-voltage level (110 kV), which also exhibit pronounced mutual correlations The highest shares occur for voltage quality parameters U03 (47 %) and Upst (33 %), indicating that variations of these parameters tend to propagate over large parts of the network The paper introduced a methodology for analyzing largescale power quality correlation structures using robust averaging and aggregation techniques By using hierarchical clustering and multidimensional scaling, complex dependencies become interpretable, revealing that low-order harmonics and flicker exhibit widespread spatial correlations, whereas higher-order harmonics remain largely localized The approach enables systematic comparison across parameters and measurement sites and supports the identification of robust, recurring correlation patterns Future research will focus on correlation-based segmentation to identify temporal changes, operational regimes, and periods of similar power quality behavior ACKNOWLEDGMENT The authors would like to thank the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) for funding this project (project number 521923789) REFERENCES
1: M.
Rosa: Domagk, J.
Dev: Meyer, T.
Rosa: Wang, D.
Paper summary: Rosa: So looking at the "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data" paper by Domagk and Meyer, it seems like the main point is that they created a robust methodology for handling those massive power quality datasets.
Dev: They tackled how to find systematic dependencies that are usually invisible with conventional analysis techniques by using aggregation and distance based visualization methods.
Rosa: The authors adapt an existing framework for shorter observation periods and enhance it with ways to aggregate daily Spearman correlation coefficients across phases, parameters, and sites.
Taro: It matters because they use hierarchical clustering and multidimensional scaling to visualize these structures, showing that low-order harmonics and flicker have widespread spatial correlations while higher-order harmonics are more localized.
Dev: They also found that the measurement sites naturally fall into groups based on their correlation patterns with other sites, identifying groups one through three based on mutual significant correlations.
Rosa: The implication is that this method helps us systematically compare parameters across measurement sites and pin down those robust, recurring patterns in power quality behavior.
Taro: It gives us a way to see where the most common problems are spreading versus where the rare ones are stuck locally in the network structure.
Conclusion: Rosa: So, to wrap up this part, we’ve seen how they used these correlation structures—those patterns of how different power quality things link together across many sites and measurements—to find some very clear groupings in the data.
Dev: Yeah, they took all that massive dataset and used aggregation and distance representation to make sense of it.
Rosa: The title itself, "Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data," it just tells you what they did: they found patterns across a huge amount of power quality information.
Dev: It sounds like they were trying to take something that’s usually just a giant pile of numbers and turn it into something actually understandable for engineers.
Rosa: Exactly. They showed that by using tools like hierarchical clustering and multidimensional scaling, those complex dependencies become visible, not just as random noise.
Dev: And what they found is interesting because it splits the power quality issues into two main types of things: low-order harmonics and flicker versus higher-order harmonics.
Rosa: Low-order stuff, like the basic voltage or current signals, those are showing up with pretty widespread correlations across different parts of the system.
Dev: But those higher-order harmonics? They tend to stay very localized. That means you don't see them everywhere; they stick around specific places in the network.
Rosa: And when they look at different measurement sites, they found these three distinct groups based on how much they correlate with each other.
Dev: Group one has almost no connections to anything else, Group two has a decent mix of correlations, and then Group three is made up of those high voltage sites that show the strongest mutual links.
Rosa: That suggests that if you want to understand power quality across a whole network, you don't have to look at every single measurement in isolation.
Dev: It points toward a way for operators to systematically compare parameters and identify these robust, recurring patterns instead of just looking at isolated events.
More episodes
- Daily Summary for 2026-11-19
- 2111.07910-Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
- 2203.02149-HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
- 2205.10102-Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
- 2311.10959-Structure-Aware Sparse-View X-ray 3D Reconstruction
- 2403.04116-Radiative Gaussian Splatting for Efficient X-ray Novel View Synthesis
- 2410.02572-Combining Pre- and Post-Demosaicking Noise Removal for RAW Video
- 2604.17453-Learned Nonlocal Feature Matching and Filtering for RAW Image Denoising
- 2603.20387-End-to-End Multi-Task Learning for Adjustable Joint Noise Reduction and Hearing Loss Compensation
- 2507.20825-Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G