Implementation Guidelines for Data Quality Metrics

arXiv:2610.10919 · cs.DB, cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Implementation Guidelines for Data Quality Metrics".

Tom: The gist The authors make ISO DQ metrics executable by classifying them into generalizable, parameterized, and non-generalizable metrics and proposing implementation guidelines realized in an open-source library called dqmeasure.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’ve covered the structure of the guidelines, but now let’s talk about what this "Implementation Guidelines for Data Quality Metrics" actually says about solving that standardization problem.

Jane: Essentially, the paper argues that because those ISO standards leave it unclear how to compute certain things, practitioners end up making their own ad hoc checks >

Lu: That’s right. They show that tools often implement checks in ways that don't align with the actual dimensions of data quality defined in ISO/IEC twenty-five thousand twelve > <ref:2610.10919#pg1>

Tom: So the main contribution is this systematic way of implementing those metrics, which is independent of any specific tool you might be using >

Jane: They introduce a model for executable Data Quality Metrics, which is this tuple (s, U, θ, φ), where s is the subject, U are the units counted >

Meng: That tuple structure sounds like a formal way to describe what you’re measuring and what parameters you need to calculate it >

Tom: Right. And then they have this score calculation formula: X equals A over B equals summed over u in U phi theta of u multiplied by U size >

Jane: That formula is how the final score is calculated, where phi theta of u specifies for each unit, to what degree it fulfills the requirement >

Lu: It’s a very clear mathematical way to define how you aggregate those individual unit scores into one overall metric >

Tom: The whole point is that this model allows you to calculate a score based on those specific parameters and units defined in the standard.

Jane: This moves it away from just being vague textual definitions toward something measurable and computable across different systems >

The paper's summary: Tom: Now let’s look at what they suggest improves things beyond just having a classification scheme; they are proposing these five implementation guidelines as the actual solution to that mess.

Jane: They detail how you go from the abstract idea in a standard to an executable metric, starting with specifying the four components of Definition one > <ref:2610.10919#pg2>

Lu: Then there’s categorizing the DQM into Generalizable, Parameterized, or Non-generalizable based on its reference >

Tom: And then parameterizing those category K metrics by either learning parameters from clean reference data or setting them by an expert user >

Meng: That part about learning parameters from reference data sounds really practical for real-world use because it cuts out a lot of manual setup work >

Jane: Plus, there's harmonizing the results so that scores are comparable and missing values are handled consistently across different metrics >

Tom: And then the validation step, which is testing if the DQM actually meets its intended purpose by injecting errors and checking if the score decreases monotonically with those errors >

Lu: That last part, making sure the score drops as the error rate goes up, that ties it directly to how well it tracks data deterioration or ML performance >

Tom: So these guidelines are designed to resolve all those ambiguities that plague how we try to implement standardized data quality checks across different systems.

The paper's improvements: Jane: So wrapping up this discussion on the "Implementation Guidelines for Data Quality Metrics," it’s clear they are focused on making abstract standards operational >

Tom: They provide a concrete path, through these five guidelines, to turn those textual DQ definitions into something that can be computed systematically >

Lu: The big picture here is moving away from tools that just do low-level checks and aligning them directly with the high-level dimensions of data quality >

Meng: For engineers, this means they have a blueprint for building reliable monitoring systems that don't rely on guesswork when implementing DQ checks >

Jane: It suggests that by adopting this structure, we can finally compare different data quality assessments across different sources and tools fairly.

Tom: The paper introduces dqmeasure as the reference implementation following these guidelines, which is a library of twenty DQM definitions >

Lu: And they show that when you use this approach, the metrics detect injected errors and their scores decline with diminishing data quality or downstream ML performance >

Meng: They also showed that for most metrics, they scale linearly with the number of rows you feed them, which is important for large datasets >

Jane: So ultimately, this work gives practitioners a systematic way to build robust, comparable data quality monitoring systems based on established standards.

Tom: That’s it for today’s paper on the Implementation Guidelines for Data Quality Metrics. We'll be back next time with something else interesting from arXiv.

Conclusion: Tom: So we’ve seen how they take those fuzzy definitions from ISO standards and give us a real blueprint for actually building data quality metrics that work >

Jane: Exactly, it’s about taking the abstract idea of data quality dimensions and turning them into something you can compute using this new model with the subject, units, parameters, and then the fulfillment scores >

Lu: The classification into Generalizable Parameterized and Non-generalizable DQM is really smart because it tells you exactly how much manual effort you're going to need for any given metric >

Meng: From an engineering standpoint, I like that they propose those five implementation guidelines, it gives a clear step-by-step process instead of just throwing a bunch of math at you >

Lalam: The idea of learning parameters from clean reference data instead of relying on expert input is really powerful for scaling up monitoring systems across massive datasets >

Tom: And the results are pretty solid too, showing that these DQM models actually detect errors and their scores drop when the quality or ML performance goes down >

Jane: It means you can track how data quality deteriorates without needing a human to constantly input new rules for every single check >

Lu: They also showed that these metrics scale linearly with the size of your data, which is something we really needed to see for practical use on big-scale systems >

Meng: Scaling linearly with the number of rows is crucial; it means you can run these checks on huge datasets without your monitoring pipeline slowing down significantly >

Lalam: And Lalam thinks this approach could fundamentally improve how we build trust in AI outputs by providing a more rigorous, automated way to check the input data quality >

Tom: So, that’s the gist of "Implementation Guidelines for Data Quality Metrics" — it’s a solid framework for making standardized data quality checks executable and scalable >

Jane: It gives us a clear path forward from those high-level standards to actual, working tools that are measurable and comparable across different systems >

Lu: This opens up so much potential for integrating DQ directly into the machine learning pipeline in a way that respects the established quality dimensions >

Meng: I’m looking forward to seeing how this translates into actual production code, because right now, it sounds like a really well-thought-out theoretical framework >

Lalam: The potential for AI systems to self-audit their input quality based on these executable metrics is really something to think about for improving culture in the industry >

Tom: It is definitely a solid piece of work, and we’ve got some big ideas coming up soon that will connect this DQ framework with other deep learning challenges.

Philipp Jung, Katinka Becker, Felix Biessmann, Valerie Restat, Martin Seyferth, Daniel Schwabe, Lisa Ehrlinger

Berliner Hochschule für Technik Berlin, Germany · Physikalisch-Technische Bundesanstalt Berlin, Germany · Einstein Center Digital Future Berlin, Germany · FernUniversität in Hagen Hagen, Germany

cs.DB, cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Comments: 14 pages, 7 figures, 5 tables. Submitted to EDBT 2027

Code: https://github.com/calgolab/dqmeasure

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The authors make ISO DQ metrics executable by classifying them into generalizable, parameterized, and non-generalizable metrics and proposing implementation guidelines realized in an

Key concepts

Generalizable Metrics (G): These are fixed assertions that require no extra input beyond the data itself to evaluate. They are automatically executable because they don't need any external settings or parameters to function correctly.
Parameterized Metrics (K): These metrics require specific inputs, such as the number of outliers, which can be learned from clean reference data or set by a human expert. The authors provide guidelines for how to determine these necessary parameters.
Non-generalizable Metrics (N): These metrics demand qualitative judgment and cannot be automated using fixed rules. They require human input to assess their quality, meaning they fall outside the scope of fully automated data quality checks.
Executable DQMs Model: A mathematical model defining a metric as (s, U, $\theta$, $\phi_{\theta}$). This structure allows for scoring by measuring how well each unit in a set fulfills its requirement to calculate an overall quality score.

Terminology

Accuracy — 56× in this paper · explained in 1 episode
Consistency — 41× in this paper · explained in 3 episodes
AUC — 11× in this paper · explained in 2 episodes
Generalizability — 7× in this paper · explained in 6 episodes
Aggregation — 6× in this paper
Core — 5× in this paper · explained in 2 episodes
Benchmark — 2× in this paper · explained in 2 episodes
International Conference on Information Quality — 2× in this paper
Parameterization — 2× in this paper · explained in 2 episodes
Abundance — 1× in this paper · explained in 1 episode
Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalization — 1× in this paper · explained in 13 episodes
Synthetic Data Generation — 1× in this paper · explained in 5 episodes
Trace — 1× in this paper · explained in 2 episodes
Transfer Learning — 1× in this paper · explained in 13 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

The gist The authors make ISO DQ metrics executable by classifying them into generalizable, parameterized, and non-generalizable metrics and proposing implementation guidelines realized in an open-source library called dqmeasure.

Bridging the Gap Between Standards and Implementation

The paper addresses a gap between data quality (DQ) dimensions defined in textual form by standards like ISO/IEC 25024 and DQ tools that implement low-level checks which are not aligned with these dimensions. The authors make the ISO DQ metrics executable by classifying all data-level metrics of both standards into (i) generalizable metrics that need no input beyond the data, (ii) parameterized metrics whose parameters can be learned from clean reference data or set by an expert, and (iii) non-generalizable metrics that need qualitative judgment and cannot be automated. For the metrics that can be automated, i.e., categories (i) and (ii), we propose implementation guidelines that resolve what the standards leave open.

Model of Executable DQMs and Classification Scheme

The paper introduces a model of executable DQMs defined as a tuple (s, U, θ, φ) where s is the subject, U are the units counted, θ comprises all parameters required to evaluate the DQM, and φtheta: U → [0, 1] specifies for each unit to which degree it fulfills the requirement. This model allows for a score calculation as X = A/B = Í u∈U φtheta (u) U. The authors propose a new classification scheme for DQMs based on their generalizability, identifying three categories: Generalizable (G), Parameterized (K), and Non-generalizable (N). Category G DQMs are fixed canonical assertions that do not require a parameter for evaluation, while Category K DQMs require one or more parameters to evaluate, such as the number of outliers. Category N DQMs require qualitative input to evaluate and cannot be automated.

Implementation Guidelines

The authors propose five implementation guidelines (IG1–IG5) that describe the steps required to turn a textual definition of a DQM into an executable DQM. These guidelines are:

IG1: Specify, which requires specifying the four components of Definition 1.
IG2: Categorize, which assigns the DQM to one of the three categories (G, K, or N) based on its reference.
IG3: Parameterize, which sets parameters for category K DQMs by learning them from reference data or setting them by an expert user.
IG4: Harmonize, which ensures results are comparable through uniform orientation of scores and consistent handling of missing values.
IG5: Validate, which checks if a DQM fulfills its intended purpose by testing it with injected errors to ensure the score decreases monotonically with errors belonging to its DQ dimension.

Evaluation Results

Experiments on real-world and synthetic datasets show that the DQMs (i) detect injected errors, (ii) decline in score together with downstream ML performance, and (iii) scale linearly with the number of rows. In one experiment on a real-world dataset, the DQMs track deteriorating DQ without manual input. The results show that DQMs react strongly to error scenarios, and as the error rate increases, their scores fall monotonically with diminishing DQ and ML performance. Furthermore, the experiments demonstrate that the DQMs scale linearly with the size of the data.

Conclusion

The work introduces dqmeasure, a library of 20 DQMs that learns required parameters from clean reference data instead of relying on expert knowledge and manual input. The authors demonstrate that the DQMs from dqmeasure detect injected errors, that their scores decrease monotonically with diminishing DQ and ML performance, and that they scale linearly with the number of rows. In future work, they plan to extend dqmeasure to DQ dimensions and DQMs that are not covered so far. The source code is available in the public GitHub repository https://github.com/calgolab/dqmeasure.

How it works

Evaluation Results

Improvements for AI systems

Bold header: Executable Data Quality Metrics (DQMs)

This system can automate continuous DQ monitoring by realizing ISO DQ metrics as executable DQM tuples (s, U, θ, φ), allowing for automated scoring based on clean reference data rather than manual rule definition.

Bold header: Automated Parameter Learning

The system can learn the parameters of a DQM from clean reference data instead of relying on manually defined rules, enabling DQMs of category K to be implemented by fitting or profiling parameters from reference data, such as determining the accepted domain values for Syntactic data accuracy (Acc-I-1).

Bold header: Automated Error Detection and Impact Assessment

The system can detect injected errors and report the score decreases monotonically with an increasing number of injected errors while simultaneously showing how the DQMs detect data errors that harm an application, such as how the classifier's ROC-AUC falls when error rates increase.

Bold header: Contextual Error Analysis

The system can provide a detailed breakdown of error impact, showing which DQMs react to the errors and illustrating that some metrics, like Syntactic data accuracy and Data format consistency detect the typos in diagnosis.

Bold header: Scalable Runtime Monitoring

The system can scale linearly with data size by running DQMs on massive datasets (e.g., 41 million rows) in under two minutes, ensuring that DQMs scale linearly with the number of rows for most metrics.

Abstract

Despite decades of data quality (DQ) research, a gap remains between DQ dimensions, such as accuracy or completeness, which the literature defines in textual form, and DQ tools, which typically implement low-level checks that are not aligned with these dimensions. ISO/IEC 25024 and ISO/IEC 5259 attempt to bridge this gap by defining DQ metrics for each dimension. However, these DQ metrics are hardly used, because the standards leave open how to implement them: for example, the metric for syntactic accuracy counts syntactically accurate values, but does not state how to decide that a value is syntactically accurate. This simply moves the problem to another level without solving it. As a result, DQ assessment currently cannot build on the standards. In this paper, we make the ISO DQ metrics executable. We classify all data-level metrics of both standards into (i) generalizable metrics that need no input beyond the data, (ii) parameterized metrics whose parameters can be learned from clean reference data or set by an expert, and (iii) non-generalizable metrics that need qualitative judgment and cannot be automated. For the metrics that can be automated, i.e., categories (i) and (ii), we propose implementation guidelines that resolve what the standards leave open. We realize the guidelines in dqmeasure, an open-source library of 20 metrics that learns these parameters from reference data instead of relying on manually defined rules. Our experiments on real-world and synthetic datasets show that the metric scores decrease monotonically with an increasing number of injected errors, decline together with downstream ML performance, and scale linearly with the number of rows, which enables automated DQ monitoring based on the standards.