Data quality checks every AI project needs before launch

Before an AI project goes live, run automated checks on the exact data it will train on, retrieve from and receive in production: schema and types, nulls and duplicates on key columns, allowed values and ranges, freshness of every source, overlap and leakage between training and test data, a sample audit of labels, and a comparison between training data and live inputs. Write each check as a query that returns the bad rows, decide in advance which failures block the launch, and keep the checks running after launch day.

The rest of this article covers what each check catches, and which ones I would refuse to launch without.

A launch that passed every demo

Here is an illustration. A distributor builds a model to flag orders likely to be returned. Offline, it scores well. The demo goes well. Two weeks after launch, the operations team stops looking at the flags because they seem random.

The post-mortem finds three things, none of them in the model code. The training table had a column, return_reason, that is only filled in after a return happens, so the model had partly learned to read the answer. The test set was split after duplicate orders had been copied across from a migration, so some test rows were near-copies of training rows. And in production, the region field arrived as free text from a new ordering app, while in training it had been a clean code from the ERP.

Each would have been caught by a check a few lines long. Nobody wrote them because the data looked fine in a notebook.

This is a narrower question than general data readiness. Readiness asks if you have the right data at all. Quality checks assume you do and ask whether it is still correct today, in the form the AI will actually see it.

A check is a query that returns bad rows

The most useful framing I know comes from dbt. Its data tests are select statements that look for failing records: "If the data test returns zero failing rows, it passes." You get the offending rows rather than a red light, so someone can fix them, and the same check runs in development, in the launch pipeline and every day after.

dbt ships four generic tests, and they cover a surprising share of real problems: unique, not_null, accepted_values (a status column must be one of a fixed set) and relationships (every customer_id in orders exists in customers, which is referential integrity). Each test can be configured to warn or to error, with thresholds through warn_if and error_if, and failing rows can be stored in an audit table for later inspection.

You do not need dbt for this. A folder of SQL files that return violations, plus a script that fails the pipeline when an error-level query returns rows, is enough. What matters is that checks are versioned with the pipeline and have a severity decided before anyone is under deadline pressure.

Structure: does the data look like you promised?

Google's ML Test Score paper (Breck et al., 2017) makes this its first data test: feature expectations are captured in a schema so they can be checked automatically. Its example is pleasingly blunt: an adult human is surely between one and ten feet tall. The paper suggests building the schema from statistics on the training data, then adjusting it with domain knowledge, or writing expectations down first and comparing them to the data so the numbers you happen to have do not anchor you.

TensorFlow Data Validation does the first half for you. It computes statistics, infers a draft schema (which features are present, their types, how often they appear, their allowed domains) and reports anomalies when a new dataset does not match. Its documentation is explicit that you should review the inferred schema and refine it, because the inference is only a heuristic.

For a typical business dataset, the structural checks I would write first are these:

• every column the model or prompt depends on exists and has the expected type
• primary keys are unique and not null
• categorical fields only contain allowed values, with a list someone has signed off
• numbers sit in a plausible range (no negative quantities, no 400-year-old customers)
• foreign keys resolve, so an order never points to a customer that does not exist
• null rate per key column stays under a written limit

The hard part is knowing which columns matter. A KPI or a model usually depends on fewer columns than people think, and those are the ones that deserve strict checks. CloudOEngine's AI reads your real data schemas to propose KPIs on the data layer, and the columns behind the KPIs you keep are a sensible first list for strict checks.

Freshness has to be a number

"Recent data" is not a check. "The orders table must have a row loaded in the last 6 hours, warn after 6 and fail after 12" is. dbt expresses this with warn_after and error_after against a loaded_at_field, set per source with stricter values on the tables that matter most. Any scheduler can do the same with a max(loaded_at) query.

This belongs on a launch list because stale data does not look broken. A feed that stopped on Tuesday still returns rows on Friday, and every structural check passes. The AI answers confidently from Tuesday's world. Set the limit from the decision the AI supports: a stock-availability assistant needs minutes, a monthly churn model can live with a day.

Leakage and overlap: the checks that make your score honest

These two checks protect the number you will use to decide if you launch at all, so I would add them first to an existing project.

scikit-learn's documentation defines leakage as using information that would not be available at prediction time when building the model, which "results in overly optimistic performance estimates". Its own worked example is sobering. On a dataset with random targets, where chance accuracy is about 0.5, selecting features on all the data before splitting produced an accuracy of 0.76. Doing the selection on the training split only brought it back to 0.5. The same trap applies to scalers, imputers and anything else that is fitted.

Leakage also hides in columns. The return_reason field in the illustration above is a classic case: a value that is only known after the outcome. The check is a review rather than a query. For every feature, someone writes down when its value becomes known relative to the moment of prediction, and anything known later is removed.

Overlap is the simpler cousin. If identical or near-identical records sit in both training and test sets, the test set measures memory. Lee et al. (2021) found train-test overlap affecting over 4% of the validation set of standard language modelling datasets. For tabular data, the check is a join on a normalised key across the split. For text, use hashing for exact matches and a near-duplicate method for the rest. If your records have a natural grouping (one customer, one store, one patient), split by group so that the same entity cannot appear on both sides.

Labels are data too

Teams test inputs and then trust labels because a person made them. Northcutt, Athalye and Mueller (2021) estimated an average of at least 3.3% label errors across 10 widely used benchmark test sets, and at least 6% in the ImageNet validation set. Those are curated public datasets. Labels exported from a ticketing system, where agents pick the first category in a dropdown under time pressure, are unlikely to be cleaner.

You cannot query your way out of this, but you can measure it. Before launch, pull a random sample of a few hundred labelled records from the test set, have two people label them independently without seeing the original label, and count disagreements. If a meaningful share of the sample is wrong, your accuracy figure has an error bar nobody has written down, and you should fix the test set before tuning the model against it. The same idea applies to the reference answers in an LLM evaluation set.

Training and serving must compute the same thing

The ML Test Score paper calls the check for training/serving skew perhaps "the most important and least implemented test", and says this kind of error causes production issues across a wide range of teams. Skew happens because features are often computed twice: once in a batch job for training and once in an application at prediction time. A new feature gets backfilled for training with one piece of code and computed live with another. A timezone, a rounding rule or a default for missing values differs, and the model quietly receives inputs it has never seen.

There are two ways to check it. The direct way is to log what the live system actually sends to the model for a sample of requests, recompute the same features through the training path for the same records, and compare value by value. The statistical way is to compare distributions. TFDV supports skew comparators between training and serving statistics: L-infinity distance for categorical features, and Jensen-Shannon divergence for both numeric and categorical ones. Its own example flags a payment type feature where cash appears noticeably more often in serving than in training.

Before launch you have no production traffic, but you usually have a pilot. Run the live path on real inputs for a week without acting on the output, and compare.

Documents need checks as well

For a RAG assistant, the document set is the training data, and the checks are just as mechanical.

Look for exact and near-duplicate documents, because three copies of an old policy outvote one copy of the new one at retrieval time. Find superseded versions and either remove them or mark them, so "leave policy v3 FINAL" and "leave policy v4" are not both answering questions. Flag files whose extracted text is empty, very short or full of replacement characters, which usually means a scanned PDF without OCR or a broken parser. Require a date and an owner on every document. And check that every document has an access label before it is indexed, because a file with no label tends to end up visible to everyone. How documents are split also affects quality, which we covered in chunking strategies for RAG.

That last check is easier when sources are not scattered across separate tools. In CloudOEngine, connected databases and files sit under one permission model and one audit trail, so an access rule is defined once rather than re-created inside each index.

Which checks block a launch

Too many blocking checks and people learn to override them. Too few and the gate is decoration. These are the defaults I would start with.

Thresholds are judgment calls, and the ML Test Score paper is honest that alert thresholds need careful tuning to balance false positives against false negatives. Start strict on a few checks, loosen with evidence, and write down every change.

A failing check also needs somewhere to go. A red row in a dashboard nobody opens is the same as no check. Route each failure to a named owner with the failing rows attached. In CloudOEngine, agents run real tools, such as filing a ticket, and every step and tool call is traced. With Autopilot's approval rules, a person approves the action (in the app or through an email approve link) before it runs. See the AI runtime.

Adding checks to a project that is already late

If launch is close and none of this exists, do it in this order:

1. Run the leakage review and the train-test overlap check, because they decide if your headline accuracy is real.
2. Add unique, not-null and type checks on the ten or so columns the model or prompt actually uses.
3. Put a freshness limit on every source with a warn and an error level.
4. Audit a few hundred labels from the test set with two independent reviewers.
5. Run a shadow week and compare live inputs with training data.
6. Decide severities, name owners, and wire failures to tickets.

The first two steps take a day and are the most likely to change the launch decision. The rest turns a one-off check into something that keeps running, because data that was clean in October will not stay clean on its own.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting data (databases, files and camera streams), running pipelines, and running AI, including an LLM gateway, agents with tool access, RAG, Vision AI and live video. Everything sits under one permission model and one audit trail, so the data your checks validate and the AI that uses it are governed by the same access rules.

Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets. That matters for the document checks above, where a missing access label is a launch blocker rather than a cleanup task.

CloudOEngine's AI reads your real data schemas to propose KPIs. The columns behind the KPIs you adopt are the ones that deserve the strictest checks.

When a check fails, someone has to act. Agents in CloudOEngine run real tools, for example filing a ticket, and every step and tool call is traced and inspectable. With Autopilot, agents are triggered and governed by approval rules: a person approves the action, in the app or through an email approve link, before it runs.

CloudOEngine is built in Bengaluru, India.

Book a demo or see the Data layer.

Frequently asked questions

What data quality checks should run before an AI launch?

At minimum: schema and type checks, null and uniqueness checks on key columns, allowed-value and range checks, referential integrity between tables, a freshness threshold on every source, a train-test overlap and leakage check, a sample audit of labels, and a comparison of training data against the data the live system will see.

Which data quality failures should block a launch?

Block on failures that make your evaluation untrustworthy or your inputs wrong: missing or mistyped key columns, duplicate IDs, stale sources past their limit, test data that overlaps training data, and large skew between training and serving. Distribution shifts and small null-rate changes are usually warnings that a named owner reviews.

What is data leakage in machine learning?

Leakage is when information that would not be available at prediction time is used to build the model. scikit-learn's documentation notes it produces overly optimistic performance estimates and poorer results in production, and a common cause is fitting preprocessing steps on all the data before splitting it.

What is training-serving skew?

It is a difference between the data a model was trained on and the data it receives in production, often because features are computed by different code in each place. Google's ML Test Score paper calls the check for it perhaps the most important and least implemented test.

How common are label errors in test data?

More common than most teams assume. A 2021 study by Northcutt, Athalye and Mueller estimated an average of at least 3.3% label errors across 10 widely used benchmark test sets, and at least 6% in the ImageNet validation set.

Do RAG and LLM assistants need data quality checks too?

Yes. The document set is the data. Check for duplicate and superseded documents, files whose text extraction came back empty or garbled, missing dates and owners, and documents without access labels, before they are indexed.

Sources

1. Add data tests to your DAG (dbt documentation)
2. freshness (dbt documentation)
3. Get started with TensorFlow Data Validation (TensorFlow)
4. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (Breck et al., Google, IEEE Big Data 2017)
5. Common pitfalls and recommended practices: data leakage (scikit-learn)
6. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (Northcutt, Athalye, Mueller, 2021)
7. Deduplicating Training Data Makes Language Models Better (Lee et al., 2021)

#data quality #data validation #data leakage #training-serving skew #mlops