Correlation of divergency: c delta. Being different in a similar way or not

summary

Video file (mp4)

The gist

The correlation-of-divergency coefficient, cδ, is a custom statistical measure designed to quantify the similarity of internal divergence patterns between two groups of paired values.

In short

The correlation-of-divergency coefficient (cδ) is a custom statistic designed to measure if the patterns of internal differences between two sets of values mirror each other. It compares how scores stand apart within each set, rather than looking at raw score correlations. A high positive cδ means outliers in both sets behave similarly relative to their own groups.

Key concepts

Correlation-of-Divergency Coefficient (cδ)
This is a custom statistical measure that quantifies the similarity between two groups of paired values by comparing their internal divergence patterns. It checks whether the way scores differ within one group is similar to how they differ in another group, focusing on pattern matching rather than direct value association.
Divergence Score Gaps
For every score in a set, this measures how far that specific score lies from every other score in its own set. These individual gaps are then combined into a single divergence score using a power mean, which helps create the final comparison metric.
Scale-Invariance
cδ is designed to be scale-invariant, meaning it remains unchanged even if the overall level or units of the scores are shifted. This allows it to compare patterns across different contexts, such as comparing bacterial growth under two different drugs or protein binding fingerprints.

Terminology used across episodes

This episode discusses

The paper

Correlation of divergency: c delta. Being different in a similar way or not · Read on arXiv

School of Design, The Hong Kong Polytechnic University · Department of Computing, The Hong Kong Polytechnic University · Research Institute for Quantum Technology, The Hong Kong Polytechnic University · Department of Communication Science, Vrije University

Matched sets may have unlike mean scores and spreads yet share the same way in which some scores stand apart. Mean gaps and raw-score correlations do not answer this question. Correlation of divergency, c delta, is put forward to fill that gap. For every score in each of two equally sized, matched sets, it finds how far that score lies from every other score in its own set. It then draws these gaps into one divergence score through a power mean. The Pearson correlation between the two lists of divergence scores is c delta, which lies between-1 and 1. A high positive value means that the same people or things stand out in both sets, no matter which way their raw scores point. A negative value means that a score far from the rest in one set tends to be matched with a score near the rest in the other, whilst a value near zero shows little linear link. As c delta stems from within-set gaps, it is unchanged by shifts in overall level or units. It can show whether the same people feel sharply shut out in matched neighbourhood sets, whether the same bacterial strains grow oddly under two drugs, or whether the same proteins stand out in two drug-binding fingerprints. It should sit alongside means and raw-score correlations rather than take their place: c delta does not show whether an outlier is high or low, nor why it arose. The power p sets the weight given to wide gaps. With p=1 it uses Gini distance, and with p=2 root-mean-square distance. Higher p gives more weight to the furthest gaps but cuts Kish's effective sample size. The raw scores and the choice of p must therefore be read with care.

Transcript

Introduction to the show: ident: Quantum Radio. Generated commentary on the latest quantum physics and condensed matter papers.

Kai: We're looking at the paper: "Correlation of divergency".

Mira: Here's the gist of the paper. The correlation-of-divergency coefficient, cδ, is a custom statistical measure designed to quantify the similarity of internal divergence patterns between two groups of paired values.

Kai: The gist: cδ compares whether the way values differ within one group mirrors the way values differ within another.

Mira: The concept and formulation of cδ cδ is a custom equation developed to measure the association between patterns of divergence within two groups of values (x and y) It compares how each value in x differs from all other values in x, and does the same for y, then assesses whether these patterns of divergence are related To calculate the numerator (signal), for each xi, sum the squared differences between xi and all other xj (j ≠ i), and do the same for yi Take the square root of each sum (for magnitude), multiply the results for xi and yi, and sum over all i The denominator is the product of the average internal divergences of both groups This ensures that the coefficient is scale-invariant.

Kai: Interpretation and behavior of cδ A high value of the cδ coefficient indicates that the patterns of divergence within x and y are similar, such that when a value in x is distant from others in x, the corresponding value in y is likewise distant from others in y (being different in the same way).

Paper summary: Kai: So looking at the whole thing again, "Correlation of divergency: c delta. Being different in a similar way or not" really tries to address that gap we talked about by focusing purely on comparing internal divergence patterns between two groups of paired values >

Mira: That’s the gist—c delta compares whether the way values differ within one group mirrors the way values differ within another, which is what it's designed to do >

Lev: It seems like a very specific tool for quantifying similarity in dispersion structures, which is useful when you're comparing things that aren't directly paired up in a simple way >

Kai: The authors say it’s original because it focuses on comparing divergence patterns rather than just direct associations, which is a big thing given how often people default to Pearson or Spearman >

Mira: And they acknowledge its limitations, like the standard version using squared differences amplifies outliers quadratically, and the influence function grows quadratically with distance from the data bulk >

Lev: They also mention that for truly robust measures, like Gini’s mean difference which uses absolute differences instead of squared ones, you get a linear influence function >

Kai: So they suggest that for real inference with c delta, permutation testing is the most natural path forward because there isn't a closed-form null distribution yet >

Mira: And they advise analysts to report the normalized c delta over c delta max and use permutation-based p-values when comparing across different datasets >

Lev: So, in short, it’s a bespoke measure for checking if variability structures match up between two groups of values where you can't rely on simple linear associations >

Conclusion: Kai: So we’re wrapping up on "Correlation of divergency: c delta," which is basically this new way to measure how similar two sets of data are in their internal patterns of spread, not just if the average numbers line up >

Mira: Right, it’s about that correlation coefficient between divergence patterns. The authors are trying to fill a gap where mean scores and raw score correlations just don't tell you if the *way* things stand apart is the same across two different groups >

Lev: From my side, I’m thinking about how this would translate to error correction. If we have two different noise profiles, this measure could tell us if those noise profiles are structured similarly in their deviation patterns >

Kai: Exactly. The title itself says it’s about "being different in a similar way or not." It sounds like they’re moving beyond just looking at the mean difference and asking a deeper question about the structure of the spread itself >

Mira: I agree. It shifts the focus from whether high values in set A correspond to high values in set B, to whether an outlier in A is matched by an outlier in B, even if that outlier is low or high depending on which way you look at it >

Lev: And for real hardware, that’s important. If we’re dealing with systems where the structure of the error matters more than the absolute level of error, this could be a useful diagnostic tool to compare two different experimental setups >

Kai: So, essentially, they've created a metric that checks if two things are 'different' in the same structural way, regardless of how high or low those differences are measured on a scale >

Mira: That’s the core idea. It’s about comparing the shapes of the distributions internally rather than just their centers or raw pairings >

Lev: And they have to be careful about how they calculate it. The paper mentions using a power mean, where you can weigh those bigger gaps differently with a parameter p >

Kai: Yeah, that p parameter is important. It lets you decide if you care more about the really wide deviations or the smaller ones when calculating this correlation >

Mira: And they also flag that standard calculations are sensitive to outliers because of how they use squared differences, so you’ve gotta read those limitations closely before you trust the numbers >

Lev: That makes sense. For us in error correction, we need to know if these structural similarities hold up when we introduce noise that might create weird outliers >

Kai: So for now, it looks like this is a new way to compare the internal variability of two datasets, and it’s definitely something worth watching as people try to apply it where simple correlations fall short >

More episodes

← Home