When I bring a bag of shopping home, I check it on the kitchen table before anything goes into the fridge. I do not think of it as checking. It just happens.

I count the eggs, because the box should hold six. I look for a cracked one. I read the date on the milk. Each check takes a second, and each one looks for one kind of problem. Counting the eggs will never find a crack. Reading the date will never find a missing egg.

How careful should I be? If I throw away every egg with a speck of dirt on it, I will throw away good food every week. If I only glance at the bag, one day a bad egg will end up in the pan. There is a cost on both sides, and I cannot make both of them zero.

There is also a problem no check on the table can find. Milk that sat warm in the shop for an hour looks exactly like good milk. The date is right, the bottle is sealed, and it is the right size. Only a thermometer in the shop could have told me.

A machine learning model is the cook in that kitchen. It uses whatever reaches it, and it cannot tell a good number from a bad one. Nothing crashes when bad data arrives. The model gives an answer, the answer is wrong, and nobody is told.

This lesson is about the checks that stand between new data and the model. The chapter's second lesson, data ingestion pipelines, already built six checks on the table itself: row counts, copies, missing keys, key order, column names and empty columns. Its lab measured them, and I will not repeat that here.

Lesson 2 ended by saying a pipeline also needs checks on the range and spread of each column. That is where this lesson starts. A file can have the right number of rows, the right columns and no copies, and still carry a pressure written in the wrong unit.

Here is the plan. First, the words, and six ways data can be wrong. Then the gate, which decides what happens to a batch that fails. Then two real tools, run on real data. Then the lab: five years of real weather readings, six faults I put in on purpose, and eight checks. I counted both kinds of mistake a check can make, and what each fault did to a model. After that come drift, real teams, and when a check earns its place.

A batch is the data that arrives together. Here one batch is one day of weather: 24 rows, one for each hour. A check is a rule the batch must pass. Great Expectations, one of the tools in this lesson, calls a check an expectation.

A check can be wrong in two ways. A false alarm is when it fires on a batch that had nothing wrong with it. A miss is when it stays quiet on a batch that did have a fault. A strict check has more false alarms. A loose check has more misses.

A range check asks whether each value lies between two limits. A category check asks whether each value is one of a known list. A consistency check is a rule between two columns. A stuck check asks whether a sensor's readings change at all during the day. A mean check asks whether the day's average stays near the usual average.

A gate is the point a batch cannot pass until its checks pass. To quarantine a batch is to set it aside, with the reason it failed, instead of deleting it. MAE, mean absolute error, is the average size of a model's mistake, in the units of what it predicts.

"Data quality" sounds vague until you split it into questions you can answer. A common list has six, and a batch can pass five of them and still be ruined by the sixth.

Completeness asks whether the values are there. The lab's data is real, and it has gaps: the fine-dust reading is missing in 2,067 of 43,824 hours, 669 of them in 2010. PM2.5 is what the model predicts, so the lab simply leaves those hours out: the model learns from 24,418 reference hours and is scored on 17,339 serving hours. When an input column has gaps instead, a model can learn that "missing" means something, and a sudden rise in gaps changes its answers.

Validity asks whether each value fits its type and range. A temperature of -999 is not a temperature. It is a code some feeds use for "no reading". A value like that, which stands in for "no value", is called a sentinel. Consistency asks whether related values agree. Dew point is the temperature at which the air would start to form dew. The US National Weather Service says it "can never be greater than the air temperature". The real data broke that rule once, on 8 October 2010 at 07:00.

Uniqueness asks whether each key appears once. Timeliness asks whether the data came on time. Lessons 1 and 2 of this chapter measured those two, so this lab does not.

Accuracy asks whether a value matches the world. It is the hardest, because a value can be present, valid, consistent and unique, and still be wrong. No rule can see that. You need a second source to compare against. In the lab, a thermometer that read 3 degrees high set off no check more often than a good day did.

The most common setup I see is a check that writes a report after the data has already landed. By the time anyone reads it, the bad rows are in the table, and a model may have trained on them. A check you do not act on is only a comment.

A gate works the other way round. New data lands somewhere private first. The checks run there. Only a batch that passes is published to the table that models and dashboards read.

This pattern has a name: write-audit-publish. Michelle Ufford's 2017 talk on data quality at Netflix listed it as an "ETL pattern for high-quality big data jobs". The Apache Iceberg table format builds it in. Its documentation says "writes are performed on a separate audit-branch independent from the main table history". After validation, "the main branch can be fastForward to the head of audit-branch". A branch here is a separate line of versions of the table, like a branch in Git: writes go to it without touching the main one. Fast-forward just means main now points at the checked data.

Why put the gate at the data layer, and not inside each model? Because one gate protects every reader of the table: this model, the next model, and every dashboard. Checks written inside each model drift apart over time. And the day a new reader appears, it has no checks at all.

Checks are code, and several tools exist to write them. I ran two of them, on this lesson's data. For the others I only quote their own documentation.

Great Expectations keeps checks in a suite, a named list of expectations. You run the suite on a batch and get back a result for each one. In version 1.x each expectation is a class, such as ExpectColumnValuesToBeBetween. One setting matters a lot for this lesson: mostly. Its documentation says an expectation is "successful if at least mostly fraction of values match". The default is 1, every row. I tried it on one real day from the lab, with three hours of temperature set to -999. With the default, the range check failed. With mostly=0.85 it passed, because 21 of 24 rows is more than 85%.

Pandera describes a pandas table as a schema: each column, its type, and its checks. With lazy=True it collects every failure before it stops, into a table called failure_cases. Pandera can also write a first schema from data, with infer_schema. Its docs warn that "these inferred schemas are rough drafts that shouldn't be used for validation without modification". That warning matters here, as you will see.

I wrote the lab's design into the docstring of its script, validation_demo.py, on 30 September 2026, before it ever ran. The script is the same file you can copy from this lesson and run yourself.

The data is the Beijing PM2.5 dataset, public on OpenML and in the UCI archive. It holds 43,824 hours from 2010 to 2014. Each hour has the weather at Beijing's airport and the PM2.5 reading at the US Embassy. PM2.5 is the amount of very fine dust in the air, in micrograms per cubic metre. The weather columns are dew point (DEWP) and temperature (TEMP), both in degrees C. There is also air pressure (PRES) in hectopascals (hPa), and a wind direction code (cbwd).

The split. The first three years, 1,096 days, are the reference. The checks take their limits from them, and the model learns from them. The last two years, 730 days, are serving: new data arriving one day at a time, as a daily file from a weather feed would.

The faults. I made six kinds of silent fault. For each one, I took a copy of every serving day and put that fault into it, one fault at a time. None of them changes a column's type or leaves a value empty, so none would trip lesson 2's table checks.

The model is scikit-learn's HistGradientBoostingRegressor: many small decision trees, built one after another. It predicts PM2.5 from nine columns: the hour, the month, the three sensors and the wind code. The last three are running totals of wind speed, hours of snow and hours of rain. I turned early stopping off. Lesson 2 found that it holds back a random tenth of the rows, so results moved with the random seed. One run.

The first question is the one I most often see skipped. On the 730 clean serving days, with no fault at all, how often did each check fire?

Every loose check stayed quiet on all 730 days. The strict ones did not. The strict range check fired on 19 days, the strict stuck check on 12, and the strict mean check on 12. Together, the strict gate would have held 37 good days, 5.1% of them. That is about one day in twenty, with nothing wrong.

Look at how the row rate and the day rate relate. The strict range check failed 117 of the 17,520 clean serving rows, 0.7%, and one failing row is enough to hold the whole day. If those 117 rows had been scattered at random, about 14.9% of days would have fired. That is 1 minus (1 minus 117 / 17,520) to the power 24. The check fired on 2.6%. I added that arithmetic to the report after a review.

The difference is that the failing rows came in groups, about 6 on each day that fired. That makes sense for weather: a cold, dry spell lasts hours, not one hour. So you cannot guess a check's day rate from its row rate alone. The safe way is to run it on clean history and count.

Google's data validation team found the same kind of problem at a much larger scale. Their 2019 paper, by Breck and others, tried a standard statistical test on 100 million points, with 0.01% of them replaced. A person would call the two samples the same. The test, they wrote, "would needlessly fire an alert 7 out of 10 times and would most likely be considered a flaky detection method".

Now the faults. For each one, I counted the days on which each check fired, out of the 730 faulty copies.

Be careful how you read this table. I built several checks with one fault in mind: the stuck check for the stuck sensor, the category check for the renamed code. Many other cells were certain by arithmetic. A value of -999 or a pressure of 101 hPa lies outside any sensible range, so every range check had to catch them. A swap puts the dew point above the air temperature in every hour where the air was warmer, so the dew point rule had to catch it. Those full columns show the checks were written correctly, not that checks in general catch faults.

The useful cells are the ones the arithmetic did not settle in advance. The stuck sensor set off the dew point rule on 72 days, when the air cooled below the frozen dew point. The mean checks caught the pressure in kilopascals on every day, which the band made certain too, but they barely noticed the stuck sensor.

The swap is the interesting one. The loose range check never saw it, because both columns hold values that are possible temperatures. The strict range check caught it on 301 days, and the strict mean check on 172. The one rule between the two columns caught it on all 730 days.

The renamed code was caught on 691 days. The other 39 days had no calm-wind hour at all, so there was nothing to rename. And the offset column matches the clean column exactly. No check fired on the thermometer error more often than it fired on clean days.

A check that fires is only half the story. The other half is what the fault costs if nothing stops it. So I scored the clean model on each faulty copy of the 730 serving days. The model's error on clean serving days was an MAE of 46.39 micrograms per cubic metre.

The swap was by far the most costly. It raised the MAE from 46.39 to 129.92, almost three times as bad. The -999 hours and the stuck sensor each added about 8.5. The pressure in kilopascals, the fault every range check caught, added only 2.02.

That surprised me. One possible reason is how trees work: they split on "pressure below some value", and a pressure of 101 is simply below every split. So the model treats every hour as the lowest pressure it knows, and pressure is only one of nine inputs. I did not test that reason. The lesson is that how easy a fault is to catch is a different question from what it costs.

A fault can also reach a model through its training data. So I also trained a model on each faulty copy of the reference years, then fed it clean serving days. This chart shows both, with the swap left off because it would squash the scale. After a review, I added a third case to the report: the fault on both sides, in training and in serving.

The first two are the same kind of mistake seen from two sides. A served fault means a model trained on clean history is fed faulty data. A trained fault here means a model trained on faulty history is fed clean data. Both are a mismatch between what the model learned and what it is fed: training-serving skew. To a model that learned pressures near 101, a clean pressure of 1016 is far outside anything it saw.

The offset is the fault this lesson is really about. The loose gate let a thermometer reading 3 degrees high through on all 730 days. The strict gate held 37 of them, but those were exactly the 37 days it holds with no fault at all. No check fired on the offset more often than on clean days. It cost the model 0.75 MAE on these two years, a gap the block bootstrap cannot tell apart from zero.

Why did no check react to it? Every value it produced was still possible. A 3 degree error sits well inside the range of a normal day and the spread of a normal year. It cannot break the dew point rule either, because it only makes the air warmer. It is the accuracy problem from the six questions: a value that is well-formed, and wrong.

After the results, I asked a follow-up question: how big must the error be before any check notices? I added 0 to 20 degrees to every serving day's temperature and ran the gates again.

Up to +3, the strict gate held exactly its 37 false-alarm days and nothing more. From +4 it started to hold extra days, slowly: 22.2% of days at +10, still with most days passing. The loose gate stayed quiet until +16. Meanwhile the model's error climbed the whole way, to 51.42 at +20.

What would a second source look like here? Suppose there were a second weather station nearby. A job could compare this airport's hourly temperature with that station's, and watch the difference between them. The two will never agree exactly, so the check is on the difference over weeks, not on one hour. A difference that moves by 3 degrees and stays there is the offset, found. I did not build this, because the dataset holds one station.

So the checks and the harm move on different scales. A small error costs the model something from the first degree, and the checks see nothing until the error is large. A tighter band would only add false alarms. What would help is a second source: a nearby weather station, a second sensor, or a person checking a sample now and then.

The renamed wind code shows a different kind of silence. The category check caught it on 691 of the 730 days. But when a check does not run, what does the pipeline itself do with a value it has never seen?

The lab's feature job turns the wind code into a pandas category with a fixed list of codes. The pandas docs say "values not in categories will be replaced with NaN", a missing value. pandas 3.0.6 also prints a deprecation warning, saying a future version will raise an error. The job did not stop. The demo counts that warning, in a line I added after the first run: it came twice, once for each run that met the new code.

Then the model met a missing wind code. scikit-learn's guide says that when a feature had no missing values in training, "samples with missing values are mapped to whichever child has the most samples". So at each split on the wind code, those hours went down the busier branch.

The cost was tiny on this model. Calm wind came in 4,006 of the 17,520 serving hours, 22.9%. On those hours the MAE went from 57.21 to 57.43. One possible reason is that the model has another wind column to lean on, the running total of wind speed. A different model, or a dashboard counting calm hours, could be hurt far more. The check fired even though this model barely cared, and that is fine: a check protects every reader, not just one model.

Put the checks together into the two gates, and the trade shows up clearly.

Both gates caught every day of four faults: the kilopascals, the -999 hours, the stuck sensor and the swap. Both let through the renamed days that had no calm-wind hour, which was right, because nothing on those days had changed. I checked that in the report, day by day. Several of those catches were certain by arithmetic, as the table showed.

On these six faults, every faulty day the strict gate held and the loose gate let through was one of the strict gate's own 37 false-alarm days. The report checks that day by day. With the offset it held those same 37 days, and let the other 693 through. So is the loose gate simply better? Not quite. The offset sweep shows what the strict gate buys. At +10 degrees it held 22.2% of days, 125 more than on clean days, while the loose gate held nothing until +16. The strict gate costs good days and buys earlier warning of a large error.

A gate also needs a rule for what happens when it fires. Decide it before the failure, so nobody has to make it up at 3 in the morning.

Some rules must always hold: a known unit, a known category, dew point not above air temperature. When one of those fails, hold the whole file. The table keeps serving yesterday's data while a person looks. My view is that a model on day-old data is usually safer than a model on wrong data, though this lab did not measure that.

Most checks in this lesson look at one value at a time. The mean checks are different. They look at a profile: numbers that describe a whole batch, such as its average, its spread, or how often each category appears. A profile check can see a batch whose values are each possible but whose shape has moved.

That shape is called a distribution: how the values of a column are spread out, how many are low and how many high. When it moves over time, that is drift. When the data a model learned from and the data it now serves differ at the same moment, that is training-serving skew.

These are the mistakes I would look for first when I review a validation setup. They are general practice, not measurements, except where I say the lab measured something.

A gate that only warns. If a failed check writes a log line and the job carries on, you have a report, not a gate. Great Expectations returns success as False and raises nothing. Something in your pipeline has to act on it. Feed the gate a broken batch on purpose, and watch it stop.

Validating after the write. If the data lands in the shared table first, readers can see bad rows before any check runs. Stage it, check it, then publish it.

Limits taken from the data's own minimum and maximum. That is what infer_schema writes, and its docs call the result a rough draft. Here it fired on 19 good days in two years.

A baseline that follows the drift. If you rebuild the profile from last week every week, a slow drift moves the baseline with it, and the alarm never fires. Freeze it from data you trust, and change it on purpose when you retrain.

Only per-row checks. A batch can pass every row rule and still be wrong in shape. Add a profile check on the columns a model reads.

Dropping failed rows. A dropped row cannot be replayed or explained later.

When not to add a check. Not every column needs a rule. A free-text note that no model reads does not need a range. Every check costs run time and attention, and a check that fires often trains people to ignore it.

The old version of this lesson told four company stories. I checked each one against the company's own words, and none held up as written. Uber was said to have one feature feeding "hundreds of models", Netflix to protect "millions of users", and Airbnb to have acted because "trust in a metric collapses". I could not find any of those claims in the companies' own posts, so I have cut them.

Here is what five teams do say, in their own posts and papers.

Google. Breck and others wrote in 2019 that "hundreds of product teams use our system to validate trillions of training and serving examples per day". Their Play store case is on the previous slide.

Uber. In February 2023 Uber wrote that "data regressions are hard to catch because the most impactful ones are generally silent". In one incident, a fare component "was missing in the critical fares dataset for 10% of the sessions". It "was detected after 45 days manually by one of the data scientists". Uber built D3 to detect such drifts automatically. Its post names the same trade as this lab: "Base limits can sometimes be more aggressive and generate many false positives". So, it says, "we define conservative alerting limits on top of the base ones".

Netflix. Netflix wrote about Axion in 2022. In its words, Axion is "our fact store that is leveraged to compute ML features offline". The post says: "Corruption in data can significantly impact production model performance and A/B test results".

Airbnb. A 2019 survey found its data scientists "had trouble identifying which data sources met the high quality bar required for their work". For its Midas certification, checks "are required for certified data, and cover basic sanity checks, definitional testing, and anomaly detection".

This script is the lab. It downloads the weather data, puts in the six faults, and runs the eight checks on clean and faulty days. It builds the two gates and trains the models. It prints the false alarms, the catches, the gates and the MAE for every fault. It needs no GPU; when I ran it, it finished in about 5 seconds once the data was downloaded.

Before you run this lab. You need Python 3 and two libraries: pip install scikit-learn pandas. scikit-learn holds the model and the download, and brings NumPy with it; pandas holds the tables. The first run downloads Beijing PM2.5 from OpenML, about 1 MB, so it needs an internet connection once. After that, scikit-learn keeps a copy in a folder in your home directory (scikit_learn_data).

I ran it with scikit-learn 1.9.1 and pandas 3.0.6 on a Mac. Both run the same way on Windows and Linux, but I have not checked the numbers there. Another version may give different decimals, so the first line printed names both versions. Give it a file name, python validation_demo.py out.json, and it also saves every number at full precision; that is how results/dq-demo.json was made.

"""Data validation: which check catches which silent fault, at what cost.

Lesson 5 of 'Data Engineering for ML', made small. It needs Python 3 with
scikit-learn and pandas (pip install scikit-learn pandas). The first run
downloads Beijing PM2.5 from OpenML (about 1 MB) and keeps a copy.
    python validation_demo.py            # print the results
    python validation_demo.py out.json   # and save every number

Design, written 2026-09-30 before the first run:
  Data: hourly weather at Beijing's airport and PM2.5 at the US Embassy,
  2010-01-01 to 2014-12-31. One batch is one day, 24 rows, as a daily
  file from a weather feed would arrive. Reference: 2010 to 2012 (the
  model learns here; the checks take their profile from here). Serving:
  the 730 days of 2013 and 2014.
  Six silent faults, each put into a copy of every serving day, one at a
  time. None changes a type or leaves a value missing:
    kpa       PRES arrives in kilopascals: every value divided by 10
    sentinel  3 random hours of the day carry TEMP = -999
    stuck     DEWP stays at its first value of the day, all day
    swap      TEMP and DEWP arrive in each other's columns
    renamed   the wind direction "cv" arrives as "calm"
    offset    TEMP reads 3 degrees C too high
  Eight checks. Each fires on a day or not; a row check fires on a day
  when any row of that day fails it.
    range_loose   DEWP, TEMP in -60..60 and PRES in 900..1100 (by hand)
    range_strict  DEWP, TEMP, PRES inside the reference's min..max
    category      cbwd is one of the values the reference has
    consistency   DEWP <= TEMP (dew point cannot pass air temperature)
    stuck_loose   DEWP, TEMP or PRES has 1 distinct value in the day
    stuck_strict  the same, fewer than 3 distinct values
    mean_loose    a day's mean of DEWP, TEMP or PRES is more than 3
                  standard deviations from the reference days' mean
    mean_strict   the same, more than 2
  Reported: the share of the clean serving days each check fires on
  (false alarms) and of the faulty days (caught; the rest are missed);
  for row checks, the share of clean serving rows that fail. Two gates,
  loose (range_loose, category, consistency, stuck_loose, mean_loose)
  and strict (range_strict, category, consistency, stuck_strict,
  mean_strict): clean days held, faulty days let through.
  The model: HistGradientBoostingRegressor, random_state 0, early
  stopping off, on hour, month, DEWP, TEMP, PRES, cbwd, Iws, Is, Ir.
  It learns pm2.5 from the reference rows that have it. MAE, the mean
  absolute error in micrograms per cubic metre, on the serving rows that
  have it: clean; each fault on every serving day; each fault on every
  reference day, tested on clean serving days. One run.
  The report adds a bootstrap of the serving days (1,000 draws, seed 0)
  for each serving fault's MAE gap, and a count of the raw data's own
  problems (missing pm2.5 by year, rows with DEWP above TEMP, repeated
  hours), and nothing else.
  After the first run, two changes, neither to a number: the first line
  printed also names the pandas version, and the warning pandas gives
  for the renamed wind code is counted and printed as one short line, so
  the printed run holds everything the terminal shows.

Author: Roni Das
Created: 2026-09-30
"""
import json
import sys
import warnings

import numpy as np
import pandas as pd
import sklearn
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingRegressor

S = ["DEWP", "TEMP", "PRES"]  # the three sensor columns the checks watch
X_COLS = ["hour", "month", "DEWP", "TEMP", "PRES", "cbwd", "Iws", "Is", "Ir"]
LOOSE = {"DEWP": (-60, 60), "TEMP": (-60, 60), "PRES": (900, 1100)}
FAULTS = ["kpa", "sentinel", "stuck", "swap", "renamed", "offset"]
CHECKS = ["range_loose", "range_strict", "category", "consistency",
          "stuck_loose", "stuck_strict", "mean_loose", "mean_strict"]
GATES = {"loose": ["range_loose", "category", "consistency",
                   "stuck_loose", "mean_loose"],
         "strict": ["range_strict", "category", "consistency",
                    "stuck_strict", "mean_strict"]}

raw = fetch_openml(data_id=42891, as_frame=True, parser="auto").frame
raw["cbwd"] = raw["cbwd"].astype(str)
raw["day"] = pd.to_datetime(raw[["year", "month", "day"]])
ref, srv = raw[raw["year"] <= 2012], raw[raw["year"] >= 2013]

# The frozen profile: everything the checks know, taken from the reference.
CATS = sorted(ref["cbwd"].unique())
LO, HI = ref[S].min(), ref[S].max()
day_means = ref.groupby("day")[S].mean()
MU, SD = day_means.mean(), day_means.std()


def fault(d, kind):
    """A copy of the rows d with one silent fault in every day."""
    d = d.copy()
    if kind == "kpa":
        d["PRES"] = d["PRES"] / 10
    elif kind == "sentinel":  # the 3 hours with the smallest random key
        key = pd.Series(np.random.default_rng(0).random(len(d)), d.index)
        d.loc[key.groupby(d["day"]).rank(method="first") <= 3, "TEMP"] = -999
    elif kind == "stuck":
        d["DEWP"] = d.groupby("day")["DEWP"].transform("first")
    elif kind == "swap":
        d["TEMP"], d["DEWP"] = d["DEWP"].astype(float), d["TEMP"]
    elif kind == "renamed":
        d["cbwd"] = d["cbwd"].replace("cv", "calm")
    elif kind == "offset":
        d["TEMP"] = d["TEMP"] + 3
    return d


def run_checks(d):
    """Which checks fire on each day of d, and which rows fail."""
    rows = pd.DataFrame({
        "range_loose": ~pd.concat([d[c].between(*LOOSE[c]) for c in S],
                                  axis=1).all(axis=1),
        "range_strict": ~pd.concat([d[c].between(LO[c], HI[c]) for c in S],
                                   axis=1).all(axis=1),
        "category": ~d["cbwd"].isin(CATS),
        "consistency": d["DEWP"] > d["TEMP"],
    })
    days = rows.groupby(d["day"]).any()
    g = d.groupby("day")[S]
    distinct = g.nunique().min(axis=1)
    days["stuck_loose"], days["stuck_strict"] = distinct < 2, distinct < 3
    z = ((g.mean() - MU) / SD).abs().max(axis=1)
    days["mean_loose"], days["mean_strict"] = z > 3, z > 2
    return days[CHECKS], rows


WARNED = []  # what pandas warns about while the job builds its table


def table(d):
    """The feature job: the model's columns, cbwd as a fixed category."""
    X = d[X_COLS].copy()
    with warnings.catch_warnings(record=True) as caught_now:
        warnings.simplefilter("always")
        X["cbwd"] = pd.Categorical(X["cbwd"], categories=CATS)
    WARNED.extend(str(w.message) for w in caught_now)
    return X


def fit(d):
    d = d[d["pm2.5"].notna()]
    model = HistGradientBoostingRegressor(
        random_state=0, early_stopping=False, categorical_features="from_dtype")
    return model.fit(table(d), d["pm2.5"])


def mae(model, d):
    d = d[d["pm2.5"].notna()]
    return float((model.predict(table(d)) - d["pm2.5"]).abs().mean())


n_days = srv["day"].nunique()
out = {"scikit_learn": sklearn.__version__, "pandas": pd.__version__,
       "rows": len(raw),
       "ref_days": ref["day"].nunique(), "serving_days": n_days,
       "categories": CATS, "lo": LO.to_dict(), "hi": HI.to_dict(),
       "mu": MU.to_dict(), "sd": SD.to_dict()}
print(f"scikit-learn {sklearn.__version__}, pandas {pd.__version__}")
print(f"rows {len(raw):,}; days: reference {out['ref_days']:,}, "
      f"serving {n_days}")

clean_days, clean_rows = run_checks(srv)
out["false_alarm"] = clean_days.mean().to_dict()
out["clean_rows_failing"] = clean_rows.mean().to_dict()
print(f"false alarms, of {n_days} clean days:")
for c in CHECKS:
    print(f"  {c:<13} {clean_days[c].mean():6.1%}")

caught = {k: run_checks(fault(srv, k))[0] for k in FAULTS}
out["caught"] = {k: caught[k].mean().to_dict() for k in FAULTS}
out["caught_new"] = {k: (caught[k] & ~clean_days).mean().to_dict()
                     for k in FAULTS}
print(f"faulty days caught, % of {n_days}:")
print("  " + " " * 13 + "".join(f"{k[:5]:>6}" for k in FAULTS))
for c in CHECKS:
    print(f"  {c:<13}" + "".join(f"{caught[k][c].mean():6.0%}"
                                   for k in FAULTS))

out["gates"] = {}
print("gates: clean days held, faulty let through")
for gname, cs in GATES.items():
    held = clean_days[cs].any(axis=1).mean()
    through = {k: float(1 - caught[k][cs].any(axis=1).mean())
               for k in FAULTS}
    out["gates"][gname] = {"clean_held": float(held), "through": through}
    print(f"  {gname:<7}{held:5.1%} held;" + "".join(
        f" {through[k]:.0%}" for k in FAULTS))

model = fit(ref)
out["mae_clean"] = mae(model, srv)
out["mae_served"] = {k: mae(model, fault(srv, k)) for k in FAULTS}
out["mae_trained"] = {k: mae(fit(fault(ref, k)), srv) for k in FAULTS}
print(f"MAE, micrograms per m3; clean {out['mae_clean']:.1f}")
print(f"  {'fault':<9} {'served':>7} {'trained':>8}")
for k in FAULTS:
    print(f"  {k:<9} {out['mae_served'][k]:7.1f} "
          f"{out['mae_trained'][k]:8.1f}")

out["pandas_warnings"] = WARNED
print(f"pandas warnings: {len(WARNED)} (a category it does not know)")

if len(sys.argv) > 1:  # a file name was given: save every number too
    json.dump(out, open(sys.argv[1], "w"), indent=1)

The report lives in scripts/labs/dataeng/dq_report.py. It reads the demo's stored files, results/dq-demo.json and the printed run, and the weather data from scikit-learn's local copy. It does not trust the demo's arithmetic. The demo puts in the faults and runs the checks with pandas. The report holds the 43,824 hours as plain Python dictionaries and does it all again with loops.

Then it fits every model again on tables it builds itself. It stops unless every stored rate comes back exactly and every MAE to within a billionth. It also reads the tool script's stored results. It stops unless both tools fail exactly the rules the lab's loose checks fail on that day. And it stops unless Pandera's inferred limits equal the strict range limits. All 173 checks agreed.

What came before the run is in the demo's docstring: the data, the split, the six faults, the eight checks, the two gates, the model and the bootstrap. What came after I saw the results is in the report's docstring, written before the report first ran. So is a later part, added after a review of this lesson. It covers the faults on both sides, the block bootstrap, the scattered-rows arithmetic and the 1-degree dew point rule. That covers the swap check, why the strict checks fired, the rename details, the offset sweep and the one day. Every figure that shows one of those says "after the results".

This box has no model in it. It holds every hour of 2014, the second serving year: 8,760 hours of dew point, temperature, pressure and wind code. It also holds the limits the checks learned from 2010 to 2012, and the three hours of each day the lab's random draw gave the -999 fault. It puts in each fault, runs all eight checks, and builds the two gates, in your browser. Each number is stored as characters in base 64, which just means counting with 64 digits instead of 10, so the box stays small.

The report checked the box against the lab's own checks on 2014, count for count. Because it holds one year, not two, its counts are for 365 days. As it is, it prints that the strict range check fired on 10 clean days, the strict stuck check on 4 and the strict mean check on 3. The strict gate held 15 clean days, and the loose gate held none.

Then try your own limits. Change z > 2 in checks to z > 2.5, and count the false alarms again. Change row[1] + 3 in fault to row[1] + 10, and see which checks start to fire on the offset. Or change the loose TEMP limit in LOOSE from (-60, 60) to (-20, 45), and see what that range check now catches, and what it now holds by mistake.

Before you trust a check, I think you should be able to work it out with a pencil. So here is one real day. I picked it by a rule fixed before looking at any day: the earliest clean serving day on which a check of the strict gate fired. That was Wednesday 2 January 2013.

Start with the range check. The day's 24 dew points run from -29 to -26. The lowest dew point in the three reference years was -28. So at noon, the dew point went one degree below anything the reference had seen, and the strict range check fired. The day's pressure rose from 1033 at midnight to 1046 at 23:00. The reference's highest was 1045, so pressure broke it too.

Now the mean check. The 24 dew points add up to -659. Divided by 24, that is a mean of -27.46. The reference days' mean dew point was 1.90, with a standard deviation of 14.32. The z-score says how many standard deviations a value sits from the mean: -27.46 minus 1.90, divided by 14.32, is -2.05.

Pressure works the same way. The 24 pressures add up to 24,947, a mean of 1039.46. Against a reference mean of 1016.60 and a standard deviation of 10.15, the z-score is +2.25. Temperature's z-score was -1.91.

Map the pencil work onto the demo. The range test is d[c].between(LO[c], HI[c]) for each row. Adding up and dividing by 24 is g.mean(). Taking away the mean and dividing by the spread is (g.mean() - MU) / SD. The script does the same arithmetic for every day at once.

Two z-scores are beyond 2, and none is beyond 3. So the strict mean check fired and the loose one did not. Every loose check passed this day. This was a very cold, very dry winter day: real weather, not a fault.

Loading. fetch_openml(data_id=42891) downloads the data once and reads the local copy after that. The wind code comes back as pandas text, so the script turns it into plain strings. day is the date of each hour, built from the year, month and day columns. ref is 2010 to 2012 and srv is 2013 and 2014.

The frozen profile. CATS is the list of wind codes the reference has. LO and HI are each sensor's lowest and highest value in the reference. day_means is each reference day's average of each sensor; MU and SD are the mean and standard deviation of those averages. Everything the checks know comes from these five names.

fault(d, kind). Takes a copy of the rows and puts in one fault. For sentinel, it gives every row a random key from numpy.random.default_rng(0) and marks the three smallest keys of each day. For stuck, transform("first") gives every row its day's first dew point.

Land every file in staging first. Publish it to the shared table only after its checks pass. In Iceberg that is an audit branch; in a plain folder it can be a staging path.

Write rules you can defend. A known unit, a known list of categories, and a rule between columns, such as dew point not above air temperature. Together with the loose stuck check, these caught every fault here except the offset, and none of them fired on a clean day. Several of those catches were certain by arithmetic, so treat that as a floor on what such rules do, not a ceiling.

Run every new check on clean history first. Count how often it fires on days you know were fine. Here that would have shown the strict gate holding 37 good days before it ever went live.

Set each limit from at least a year of clean history, and choose a false-alarm rate you can live with. A month misses the seasons: here, a January would have set very different limits from a July. And zero false alarms is not the goal, because the quietest setting also misses the most. Write down the rate you chose, and why. A limit with a written reason can be reviewed. A limit copied from last year's minimum cannot.

Keep what fails, with the reason. Put it in a quarantine table that you add to on every run, with the batch id and a timestamp. dbt's store_failures table is replaced on each run, so copy from it rather than keep it.

Run the checks in CI too. CI, continuous integration, runs tests on every code change. If someone edits a transform or a check, run the suite on a sample of real data before the change merges.

It is the only step aimed at a value that is wrong but normal, like the thermometer 3 degrees high.

Add the check when it names a fault that cannot be normal. A unit, a known category, or dew point above air temperature. Here those rules, with the loose stuck check, caught every fault except the offset and never fired on a clean day. Some of those catches were certain by arithmetic.

Add it when a model reads the column. The swap cost 83.53 MAE here, and one rule between two columns caught it every day.

Add it when you can say what the gate does when it fires. Hold the file, set rows aside, or tell a person. A check with no action is a comment.

Add it once you have measured its false alarms on clean history. Then you know what it costs before it pages anyone.

Think twice when its limits are the data's own minimum and maximum. Here that fired on 19 good days in two years.

Think twice when nobody reads the column, or nobody will act on the alarm. Think twice when a normal season crosses the limit, as the cold snaps did here.

Think twice when the only fault you fear is a small wrong reading. No band will see it. Compare against a second source instead.

One city's weather, 2010 to 2014. Everything here is one dataset with three sensors and a wind code. Other data, with other columns and other seasons, will give other rates.

Six faults I chose and put in. Real faults are messier. I built several checks with one of these faults in mind, so their catches were expected. The false alarms, the partial catches and the offset were not.

Each fault on every day, alone. Real faults come and go, arrive together, and touch part of a day. This lab did not test any of that.

One model type, one run. The costs are for one kind of model. A linear model, which reads a pressure of 101 very differently from a tree, could suffer far more.

The bootstraps check luck, and nothing else. They ask whether another draw of days from the same two years would agree. The day-by-day one treats each day as unrelated to the next, and weather runs in spells. With 7-day and 30-day blocks, the three smallest gaps could not be told apart from zero. Neither says anything about another city or another year.

Some analyses came after the results. The swap check, the reasons for the false alarms, the rename details, the offset sweep and the one day were designed after I saw the first results. I wrote them into the report's docstring before it first ran, and every figure that shows them says so. The faults on both sides, the block bootstrap, the scattered-rows arithmetic and the 1-degree dew point rule came later still, after a review of this lesson.

Take one pipeline your team runs and ask the five questions on the card. The quickest are the first and the third. Does a batch wait anywhere before readers can see it? And which fault would every one of your checks miss? If you cannot name one, you have not looked yet.

Then do the measurement. Take at least a year of history you trust, and run your checks on it. Count how often each one fires. Then put one fault of your own into a copy, and count again. Here that showed a strict gate holding 37 good days, and no check firing more often on a 3 degree error than on a good day.

The card keeps the lesson's numbers. The strict gate held 37 good days in two years, and the loose gate held none. Both caught every day of four faults. With the thermometer reading 3 degrees high, no check fired more often than on a good day. The costliest fault, the swap, was caught every day by one rule between two columns. And the strict gate's false alarms bought something: at 10 degrees off it held 125 more days, while the loose gate held none until 16.

The next lesson in this chapter is about data versioning. It covers how to pin the exact data a model was trained on, with tools such as DVC, LakeFS and table time travel. Then a batch you fixed and loaded again can be traced.