Data Quality

Quality Is Something You Verify for the Path¶

Data readiness is a use-case property, and quality is no exception. A use case that drafts a newsletter from a one-line brief needs almost no quality work; a use case that reads a customer's account and acts on it needs the dimensions below verified on that specific path. In traditional analytics, poor data quality leads to bad reports. In a use case that routes through a system of record, it leads to a system that is confidently and systematically wrong, repeating that error across every decision it touches. The failure mode is not a wrong number in a cell; it is a biased hiring algorithm, a fraud model that misses an emerging attack pattern, or a clinical decision tool that underperforms for minority populations.

More than 80% of AI projects fail — about twice the failure rate of IT projects that do not involve AI (RAND, 2024)

The core problem: enterprise data infrastructure was designed for reporting and transactions, not for feeding AI systems. Most data lakes were built with volume as the primary goal; quality, organization, and context were afterthoughts. AI requires all three, on whichever path a given use case actually depends on.

The Six Quality Dimensions (and What Each Means for AI)¶

The DAMA-DMBOK framework defines six primary data quality dimensions. Each has specific implications when applied to AI workloads that do not exist in traditional Business Intelligence (BI) contexts.

1. Accuracy¶

Values correctly reflect the real-world entity or event they represent.

In traditional analytics: An inaccurate customer address means a mis-delivered mailer.

In AI: Systematic inaccuracies in training labels teach the model the wrong ground truth. A fraud detection model trained on miscoded transactions will encode those errors as signal.

Accuracy failures in AI often come from:

Automated data ingestion pipelines with no validation step
Source system field repurposing (a notes column being used for structured values)
Historical data that was accurate when written but reflects a superseded business definition

2. Completeness¶

All required values are present and populated.

In AI: Missing values are not just gaps — they are a pattern the model will learn from. If null values in a demographic field correlate with a protected class, a model can learn to use missingness as a proxy feature. It ends up encoding bias without a single explicit sensitive field in the training set.

Completeness assessment for AI must include:

Null rate by column, by time range, by data source segment
Whether missing data is Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR). The three carry very different implications for model fairness.
Impact of imputation strategies on downstream model behavior

3. Consistency¶

The same entity or concept is represented the same way across systems and over time.

Inconsistency is the most common cause of silent model degradation. Examples:

A status field coded as 1/0 in one system and Y/N in another, joined in a training pipeline
A product category taxonomy that changed mid-dataset, creating a structural break in the time series
Customer IDs that were re-used after account deletion

For AI, consistency must be enforced at the schema level (standard field types, controlled vocabularies) and validated during every pipeline run — not assumed from documentation.

4. Timeliness¶

Data reflects the current state of the real-world entity, and is available when needed.

Timeliness has two dimensions for AI:

Inference latency — for a use case that reads a system of record at decision time (fraud, personalization, dynamic pricing), the feature pipeline must deliver data within the decision window. Batch features inserted into a real-time path create a temporal mismatch. This applies whether or not you trained the model making the decision.
Training data recency (if you train your own model) — stale training data encodes outdated patterns. A credit risk model trained on pre-pandemic behavior fails as spending patterns shift. A customer churn model built before a major product change is working against itself.

5. Validity¶

Values conform to defined domain rules, formats, and reference data.

Validity failures are often invisible in BI but catastrophic in AI:

A phone number field containing free-text notes
Date fields with values from 1900 or 2099 due to default handling in legacy systems
Numeric fields with sentinel values (-1, 999) used to indicate missing data, interpreted as real values by a model

Validity checks are the simplest to automate (schema validation, range checks, regex) and among the highest-value quality investments for AI readiness.

6. Uniqueness¶

Each real-world entity is represented exactly once.

Duplicate records are a training amplifier: they increase the effective weight of the duplicated example without increasing its information content. In datasets assembled from multiple sources, duplicates are extremely common and often invisible without cross-system entity resolution.

For AI specifically:

Customer-level deduplication requires probabilistic entity matching, not just exact ID matching — this applies to any use case that looks up an entity through a system of record, whether or not a model was trained on that data.
(If you train your own model) duplicates in the training set can cause overfitting to specific examples, and duplicates that leak across train/test splits inflate evaluation metrics.

If You Train or Fine-Tune Your Own Models¶

Everything above applies to any use case that reads structured data through a system of record. The four failure modes below are different: they assume you are training or fine-tuning a model on your own labeled dataset, which is a minority of enterprise AI work today. Most teams deploy a model someone else trained and consume data at inference time, not training time — for that work, the dimensions above are the whole story. Read this section only if your use case includes a training or fine-tuning step.

96% of organizations encounter data quality problems when training AI models (Dimensional Research, 2019) 80% of ML project effort is consumed by label noise, inconsistent tagging, and missing context (Cognilytica, 2020)

Beyond the standard dimensions, training your own model introduces failure modes that have no direct analog in traditional data quality work.

1. Label Noise¶

Supervised learning depends on accurately labeled training data, but humans make labeling mistakes. They disagree on subjective categories, miss subtle distinctions, and introduce their own biases. Even benchmark datasets like ImageNet contain a significant number of mislabeled samples.

Label noise substantially degrades model performance when more than 20% of training data is mislabeled (Budach et al., 2022)

Sources of label noise:

Crowdsourced annotation without inter-annotator agreement checks
Automated labeling using heuristics treated as ground truth
Annotation drift (guidelines changed mid-project; early labels do not reflect later standards)
Domain ambiguity (a computer vision defect model labeled by rotating shifts of workers who apply criteria inconsistently)

Mitigation requires inter-annotator agreement scoring, confidence-weighted training, and human-in-the-loop QA throughout the annotation process — not just at the end.

2. Representation Bias¶

Training datasets often underrepresent certain populations, scenarios, or edge cases. Models perform well on majority groups but fail for minorities or rare events.

Bias manifests through:

Historical bias — training on past decisions that encoded human prejudice (hiring data, loan approvals)
Sampling bias — data collected in a way that systematically excludes certain populations (clinical trials, product usage analytics)
Proxy features — variables correlated with protected attributes that the model learns to use even when the protected attribute is excluded

Mitigation: demographic balance audits, fairness metric tracking during training, reweighting or synthetic minority oversampling (SMOTE), and governance requirements embedded in model development workflows.

3. Data Drift¶

Production data distributions shift over time, causing model performance to degrade silently. This is one of the most underestimated quality risks in deployed AI systems.

Drift manifests in three forms:

Covariate shift — feature distributions change (e.g., customer demographics of the user base evolve)
Prior shift — label distributions change (e.g., fraud rate increases; the model was not trained on the new base rate)
Concept drift — the relationship between features and labels changes (e.g., what predicts churn shifts after a product redesign)

The harder problem is not detecting drift but distinguishing temporary fluctuations from permanent shifts that require retraining. Continuous monitoring with automated retraining triggers addresses both.

4. Data Poisoning¶

A targeted attack in which adversaries introduce malicious or misleading data into the training pipeline. The poisoned data distorts training, producing models with deliberately engineered failure modes — often invisible in standard evaluation.

This is an emerging security concern as AI is deployed in high-stakes contexts. Mitigation requires data provenance controls, anomaly detection on incoming training data, and adversarial robustness testing.

Building a Data Quality Framework for AI¶

A data quality framework operationalizes governance policies through rules, metrics, and remediation workflows. For AI, it must be continuous — not a one-time pre-project audit.

The Five-Step Framework¶

Step 1: Profile and baseline Run automated profiling on all candidate datasets. Capture null rates, value distributions, cardinality, schema, and row counts. Establish the baseline that monitoring will compare against.

Step 2: Define expectations as code Translate business rules and quality requirements into testable assertions. Tools like Great Expectations allow teams to express these declaratively and version them alongside data pipeline code.

Step 3: Integrate checks into the pipeline Quality is preserved — or lost — in the data pipeline itself. Checks must run at ingestion, transformation, and feature engineering stages. A failing check should block the pipeline, not produce a warning that gets ignored.

Step 4: Monitor for drift in production Deploy statistical process control monitors that compare production data distributions against training baselines. Alert when distributions diverge beyond defined thresholds. For AI, monitoring the data is as important as monitoring the model.

Step 5: Close the loop with remediation workflows Quality failures must route to named owners for remediation — not to a generic inbox. Track time-to-remediation as a governance metric. Connect quality scores to downstream model performance metrics so business impact is visible.

Tooling Landscape (2025–2026)¶

Tool	Category	Best for
Great Expectations	Open-source testing	Defining data expectations as code; CI/CD integration; developer-first teams
Monte Carlo	Observability platform	End-to-end data reliability; freshness SLAs; anomaly detection; enterprise scale
Soda Core	Open-source testing	SQL-native quality checks; lightweight alternative to GX
dbt tests	Transformation-layer	Quality checks embedded in dbt transformation models
Atlan	Data catalog + DQ	In-warehouse DQ checks (Snowflake, Databricks); centralized quality trust signals
Bigeye	ML-driven observability	Rule-based + ML-driven checks tightly integrated with Snowflake / BigQuery
Databand (IBM)	Pipeline observability	Airflow/Spark pipeline health; upstream/downstream impact tracking
Deequ (AWS)	Open-source (Spark)	Large-scale Spark-based dataset unit testing

Key trend: Data quality testing and data observability are converging. Teams now expect pipelines built the way modern software is built — with automated testing, version control, and continuous monitoring (DataKitchen, 2026).

Data Quality Scoring¶

A data quality score provides a single composite measure of a dataset's fitness for AI use. It should be:

Dimension-weighted — not all dimensions matter equally for every use case. A real-time fraud model weights timeliness heavily; a batch churn model weights completeness and consistency.
Tied to a business KPI — otherwise the scorecard becomes a vanity report.
Computed continuously — a score produced once at project start decays as data changes.

Sample scoring approach:

Dimension	Weight	Score	Weighted Score
Accuracy	25%	82	20.5
Completeness	20%	91	18.2
Consistency	20%	74	14.8
Timeliness	15%	88	13.2
Validity	10%	95	9.5
Uniqueness	10%	78	7.8
Total	100%		84.0

A score below 70 on any single dimension should block a dataset from production AI use until remediated.

Practical Readiness Checklist¶

[ ] Data profiling completed on the datasets this use case's path actually touches — null rates, distributions, cardinality documented
[ ] Quality expectations defined as code and committed to version control
[ ] Quality checks integrated into the data pipeline (blocking, not advisory)
[ ] Drift monitoring deployed in production — alert thresholds defined
[ ] Data quality scores tied to named owners with remediation SLAs
[ ] (If you train your own model) Label quality reviewed — inter-annotator agreement scored
[ ] (If you train your own model) Demographic balance audit completed — underrepresented groups identified and addressed
[ ] (If you train your own model) Training/test split validated for data leakage and duplicate records across splits
[ ] (If you train your own model) Data poisoning risk assessed for training data sourced externally
[ ] (If you train your own model) Quality score baseline established before the first training run

Sources¶

RAND — The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, 2024. By some estimates more than 80 percent of AI projects fail, twice the rate of failure for information technology projects that do not involve AI. View source · verified 2026-06-20 · primary
Dimensional Research (commissioned by Alegion) — Artificial Intelligence and Machine Learning Projects Are Obstructed by Data Issues, 2019. 96% of enterprises run into problems with data quality, data labeling required to train AI, and building model confidence. View source · verified 2026-06-22 · primary
Cognilytica — Data Preparation & Labeling for AI, 2020. as much as 80% of machine learning project time is spent on aggregating, cleaning, labeling, and augmenting machine learning model data. View source · verified 2026-06-20 · ⚠ secondary mirror
Budach, Feuerpfeil, Ihde, Nathansen, Noack, Patzlaff, Naumann & Harmouch — The Effects of Data Quality on Machine Learning Performance on Tabular Data, 2022. up to 20% of training labels could be flipped without performance losses of no more than 10% in F1-score. View source · verified 2026-06-20 · primary
DataKitchen — The 2026 Open-Source Data Quality and Data Observability Landscape, 2026. the convergence of data quality testing and data observability is creating a new expectation: reliable, transparent data pipelines built the same way modern software systems ensure reliability. View source · verified 2026-06-20 · primary