Picture the data scientist who built your fraud model. Every couple of weeks, fraud patterns shift, so the model needs fresh training. Here is how that actually happens without a platform: she opens a notebook, downloads a new export of last month's transactions, runs the cells top to bottom, eyeballs the accuracy, exports a new pickle file, messages it to a backend engineer on Slack, and he swaps it into the service by hand.

Now multiply that. Ten models. Thirty models. Each one needs retraining on its own cadence. Each retrain is a person doing the same manual dance, remembering which export to use, which notebook cells to skip, which file is the good one. People go on vacation. People forget a step. Someone trains on a broken data export at 2 AM and quietly ships a worse model, and nobody notices for a week.

Retraining by hand does not scale, and it is not just about effort. It is about trust. When a human runs the steps, you cannot answer the questions that matter: which data trained this exact model, which version of the code, can we reproduce it, and did anyone actually check it was better before it went live? A training pipeline exists to answer all four, automatically, every single time.

The mental shift is worth stating plainly, because everything else in this lesson follows from it. In classical software, the artifact you ship is code, and the thing that keeps it honest is a test suite that either passes or fails. In machine learning, the artifact you ship is a model, and a model is a function of data as much as of code. Two engineers can run the identical script a week apart and get two different models, because the data underneath moved. That single fact is why the tooling here looks different from a normal CI pipeline: it has to pin data, not just code, and it has to judge an output that is graded on a curve rather than a boolean.

A training pipeline turns "a person carefully runs some steps" into "the platform runs the same steps the same way, on a trigger, with a receipt."

By the end of this lesson you will be able to draw the pipeline as a graph, name what runs it, decide when it should fire, and explain why a fresh model is not allowed to ship just because it is fresh.

A training pipeline is a directed acyclic graph, a DAG, of the steps that turn raw data into a trained, blessed model. Directed because the steps run in an order. Acyclic because you never loop back inside a single run. Each box does one job and hands its output to the next.

The classic shape is six steps: ingest the data for this run, validate it so a broken feed fails fast, transform it into the exact features the model expects, train the model, evaluate it against the model already in production, and register it, but only if it won.

The reason to draw it as a graph and not a single script is the arrows. They make the dependencies explicit. Validate cannot start until ingest finishes. Train cannot start until the features exist. And the branch at evaluate is the whole point of the pipeline: a fresh model only earns a spot in the registry if it beats the current champion on data it has never seen. Lose that comparison and the run stops, so production keeps the model it already trusts.

Two properties of the graph carry more weight than they first appear to. First, because dependencies are explicit, the engine can run independent steps in parallel and know exactly what to re-run when one step changes. A single script has none of that structure; it is just lines that happen to run in order. Second, because each box is isolated, a box can be given its own compute and its own dependencies. The train box can demand a GPU and pin a specific PyTorch version while the validate box stays a tiny CPU container. You do not get that from a monolithic notebook, and it is the difference between a pipeline that scales and one that does not.

The word blessed is doing real work in that first sentence. A trained model is not automatically a good model. Blessed means it passed the gate, and the gate is the last box for a reason: nothing upstream of it is allowed to touch production. Ingest, validate, feature, and train can all succeed, produce a model, and still be thrown away at the gate. That is not waste. That is the pipeline doing its most important job.

Drawing a DAG is easy. Running it reliably, on a schedule, retrying the flaky step, placing the training step on a GPU while the light steps run on cheap CPUs, is not. That job belongs to an orchestration engine, and understanding what it does and does not do is the single most clarifying idea in this whole topic.

An orchestrator owns the DAG. It knows the step order, launches each step when its inputs are ready, retries failures, and records whether the run passed. The important thing to understand is what it does not do: it does not run your training code itself. It is a conductor, not a worker. It asks the compute cluster to execute each step and waits for the result.

Read that picture top to bottom. The control plane decides what runs and when. Kubernetes decides where, placing each step on a node with the right hardware. And object storage carries data between steps, because separate containers on separate machines share no memory, so the only way for the train step to receive the features is to read a file the feature step wrote. This is why a training pipeline is many containers passing files, not one process passing variables, and that fact shapes almost every other decision on this page.

The engines you will hear named most often differ less in what they do than in what they treat as the unit of work. Pick the one that matches how your team already ships, not the one with the longest feature list.

The row that decides most real architectures is the first one: what a step is. In Airflow a step is a Python task, which is why teams that already run Airflow for data engineering reach for it first, and also why it can feel awkward for ML artifacts it was never designed to track. In Kubeflow a step is a container on Kubernetes, which is powerful and also means you must run Kubernetes. In Metaflow a step is a plain Python function that scales from a laptop to the cloud without the scientist rewriting anything, which is exactly the ergonomics Netflix optimized for. Underneath, the pattern is identical everywhere: the orchestrator decides what and when, the cluster decides where, and containers do the work by passing files. The names change; the shape does not.

A run should be triggered, never done out of boredom. Deciding what fires a run is a real design choice, because the trigger sets how fresh the model stays and how much compute you burn, and those two pull against each other.

A schedule is the baseline, a clock that retrains nightly or weekly so the model never drifts far from reality. It is simple and predictable, and its weakness is precisely its predictability: a sudden shift in the world between two runs goes unseen until the next tick. New data is a stronger signal, firing a run when a fresh batch of labeled examples has actually landed, so you spend compute only when there is something new to learn from.

The strongest trigger is drift, an alert forwarded from the monitoring layer saying that live inputs have moved away from the training distribution. Drift matters more than the calendar because it means the model is now being asked about a world it was never trained on, which is the exact condition under which its predictions quietly go wrong. A drift-triggered retrain spends compute at the moment the model has demonstrably become stale, not on a fixed timer that is either too eager or too late. Continuous retraining sits at the far end, always refitting on the newest data, and it earns its cost only where staleness is measured in minutes, like ads ranking or live fraud. For most systems the pragmatic answer is a blend: a schedule as a floor so nothing ever goes truly stale, drift as the real workhorse, and continuous reserved for the rare model where the bill is worth it.

Here is the question people forget. A run can finish, produce a perfectly valid model, and still not ship. That is by design, and it is the second half of the pipeline's job.

The evaluation gate compares the fresh model, the challenger, against the model already in production, the champion, on a held-out set that neither model trained on. Only if the challenger wins by a margin the team set in advance does it get promoted. Lose, and the challenger is discarded and the champion keeps serving.

Three details in that figure carry the whole idea. First, the comparison is on held-out data, always. A model looks brilliant on the data it trained on, so that number is worthless for deciding whether to ship; the held-out score is the only honest one, because it estimates how the model behaves on examples it will actually meet in production. Second, the champion and the challenger are scored on the same frozen test set, so the comparison is apples to apples rather than two numbers measured on two different slices of reality. Third, the promotion margin is set in advance and is larger than zero. Requiring the challenger to win by 0.005 rather than any positive amount protects you from promoting a model that is not really better, only luckier on this particular test set, which is a real risk when the improvement is within the noise of your evaluation.

The cultural point is as important as the mechanical one. A retrain that produces a losing model is not a failed run. It is a successful run that returned a useful answer: the current model is still the best you have, so do not touch production. Once a team internalizes that, the gate stops feeling like a gate that sometimes blocks progress and starts feeling like the thing that lets them retrain aggressively without fear, because a bad retrain simply cannot reach users.

It is tempting to think of a pipeline run as pass or fail. In reality the orchestrator tracks each run through a small state machine, and the interesting states are the ones between the two ends, because that is where reliability actually lives.

Retrying is the state that makes long pipelines survivable. A run that touches many machines over many minutes will meet a transient failure as a matter of course: a spot instance reclaimed, a rate limit, a 503 from an upstream service, a node that briefly went unhealthy. When a single step hits one of these, the orchestrator backs off and retries that step alone, not the whole run. That distinction is the difference between a twenty-minute train step that recovers from a blip and a pipeline that throws away an hour of ingest and feature work because the last step flickered.

Backfill is the other state worth knowing. Sometimes you discover that a past window was skipped, or that a bug in the feature code produced wrong results for last Tuesday. Backfill re-queues exactly that window, reprocessing the missed or broken slice without disturbing the runs that came after it. Both retrying and backfill are only possible because the orchestrator records each run's status and each step's output. Because it knows precisely what completed, it always knows what to resume and what to replay, which is the same reason a monolithic script gives you neither: a script that dies is just dead, with no record of how far it got.

A pipeline is only trustworthy if you can reproduce a run. Six weeks from now a model misbehaves in production, and you need to replay the exact run that produced it, reproduce the bug, and fix it. That only works if three inputs were frozen when the run happened, and the pipeline recorded which three.

Think of every run as a small provenance graph: pinned inputs flow in, and the run stamps them onto the model it produces, so the model always knows what made it.

The three inputs that must be frozen are data, code, and environment. Data is pinned with an immutable snapshot: a versioned object-storage path, a Delta or Iceberg table version, or a DVC hash, so that even if the warehouse tables are later overwritten, the exact bytes this run trained on can still be read. Code is pinned with the git SHA of the pipeline and feature logic, recorded on every run, so that two runs on the same data that produce different models cannot leave you guessing whether the code changed between them. Environment is pinned with the container image digest plus a lockfile that names every library version, so that a new NumPy or scikit-learn release cannot silently shift your results.

Miss any one of the three and a rerun can quietly produce a different model, which means you never really understood what you shipped. The receipt is the whole idea in one object: this data snapshot, this git SHA, this image digest, stapled to every model in the registry. And notice how cheaply the third one comes. Because each step already runs in its own container, the image digest is captured for free by the pipeline; the environment leg of the receipt costs you nothing extra, which is one more reason the per-step container design is not an accident.

It would be simpler to write one big script that ingests, trains, and evaluates in a single process. Real platforms deliberately do not, and one of the payoffs of splitting the work into isolated, addressable steps is that you can skip the ones whose inputs did not change.

The mechanism is content-addressed caching. Each step is fingerprinted by a hash of everything that feeds it: its input data, its code, and its config. The step's output is cached under that fingerprint. On the next run, a step whose fingerprint matches the cache is skipped entirely, and its cached output is handed straight to the next step. Only steps whose inputs actually changed are recomputed.

The rule is mechanical and worth memorizing: a step runs only if the hash of its inputs is not already in the cache. Follow the figure. Yesterday's data snapshot has not changed, so ingest and validate are cache hits and return in zero seconds. But the feature code was edited, so its input hash is new, so it recomputes. And here is the key consequence: the moment the feature step recomputes, everything downstream of it must recompute too, because train's input, the features, is now different. The cache does not save the train step, because train genuinely has a new input. What the cache saves is the untouched prefix, ingest and validate, that would otherwise have been redone for nothing.

This is why a small code change costs one retrain and not five, and it is also why the DAG structure earns its keep. The engine can compute these fingerprints and reason about what to skip precisely because the dependencies are explicit. The cost of splitting steps into separate containers, writing intermediate data to object storage instead of keeping it in memory, buys exactly this: addressable, cacheable, resumable boxes. For a pipeline that runs for minutes or hours, that trade is easily worth it. For a tight inner loop measured in milliseconds, it would not be.

The train step is the one that eventually outgrows a single machine, and it does so for two distinct reasons that call for two different answers. Getting this distinction right is a common interview separator, so it is worth being precise.

Data parallelism answers the throughput problem. The model fits comfortably on one GPU, but you have far more training data than one GPU can churn through in an acceptable time. So you copy the full model onto every GPU, hand each replica a different shard of each batch, and after every step the replicas synchronize by averaging their gradients in an all-reduce, which keeps all copies identical. Add more GPUs and you process more data per unit time. This is the common case, and it is what most teams mean when they say distributed training.

Model parallelism answers a different problem: the model does not fit in a single GPU's memory at all. The largest language models have more parameters than any one accelerator can hold, so their layers are sliced across several GPUs. A batch flows through GPU 0's layers, whose output becomes the input to GPU 1's layers, and so on down the line, with activations passed GPU to GPU. The GPUs run in sequence rather than in parallel replicas, which is a fundamentally different communication pattern from the gradient all-reduce of data parallelism.

The question is never which is better in the abstract; it is which resource ran out first. Out of throughput on a model that still fits: go data parallel. Out of memory before the model even loads: go model parallel. The very biggest training jobs shard both at once, splitting the model across GPUs and then replicating that whole sharded setup across groups of machines. For a foundation-level mental model, holding the two axes apart, data versus memory, is what matters.

Distributed or not, a long train step introduces a failure mode the light steps never face: it can run for hours, and the machine under it can vanish partway through. Spot instances get reclaimed with little warning, and a node can simply die. Without protection, that crash throws away every epoch of progress and forces a restart from zero.

Checkpointing is the answer. Every few epochs, the worker writes the model's state, its weights and its optimizer state, to durable storage. If the worker then dies, a fresh worker reloads the last checkpoint and resumes from there instead of from the beginning.

Trace the sequence. The worker trains through epoch 5 and writes a checkpoint, then trains epochs 6 and 7, and during epoch 7 the spot node is reclaimed and the worker process is gone. The orchestrator notices the step failed and reschedules it on a new worker, which loads the latest checkpoint, restoring the exact model and optimizer state as of epoch 5, and resumes from epoch 6. The crash cost two epochs of redone work, not the whole run.

Checkpoint frequency is the dial you tune. Checkpoint often and a crash is cheap, but you pay more write overhead and slow the training loop slightly with frequent durable writes. Checkpoint rarely and the writes are nearly free, but a crash costs more redone work. For hours-long training on cheap spot instances, where crashes are expected rather than exceptional, frequent checkpoints almost always win, because the expected cost of a crash dominates the small overhead of writing state. This is the training-step analogue of the run-level retrying you saw earlier: in both cases, durable state is what turns a fatal failure into a survivable one.

There is a layer that ties the reproducibility, the gate, and the registry together, and without it a training platform is oddly blind. It is experiment tracking, and its job is to record, for every run, what the run did.

Every run of the DAG emits three kinds of record as it executes. Params are the inputs you chose: the learning rate, the model depth, the data snapshot. Metrics are the outputs you measured: held-out AUC, precision at a threshold, training loss over epochs. Artifacts are the things the run produced: the model file itself, plus plots and evaluation reports. A tracking store like MLflow or Weights and Biases captures all three into one queryable row per run.

That store is what powers two things the rest of the lesson quietly depends on. The compare UI lets a scientist sort every run by a metric, diff the configs of the best and worst, and plot loss curves side by side, which is how anyone actually finds the run that worked. And when a challenger wins the gate, its artifact is promoted into the model registry with its full lineage attached, ready for the serving layer to pull. Without this layer, a question as basic as which hyperparameters gave us 0.926 is unanswerable a month later, because the knowledge lived in someone's terminal scrollback and is gone. With it, every run is a durable row, the gate is just a comparison over that table, and promotion is picking the top of it.

You do not draw the DAG by hand. You write it as code, and the engine builds the graph from the dependencies between steps. Here is the same six-step pipeline expressed in a Kubeflow-style Python definition. Read it as a description of boxes and arrows, not as a script that runs top to bottom.

from kfp import dsl

# Each @dsl.component becomes ONE step, and each step runs in its
# own container image with its own pinned dependencies.
@dsl.component(base_image="registry/ingest:2.4.1")
def ingest(data_snapshot: str) -> str:
    # Pull a FROZEN snapshot, not "today's live table", so the run
    # is reproducible. Returns the S3 path of the raw dataset.
    ...

@dsl.component(base_image="registry/validate:2.4.1")
def validate(raw_path: str) -> str:
    # Schema, null rates, value ranges. Raise to FAIL the whole run
    # if the upstream feed is broken.
    ...

@dsl.component(base_image="registry/features:2.4.1")
def transform(raw_path: str) -> str:
    # Same feature logic that runs at request time, or you get skew.
    ...

@dsl.component(base_image="registry/train:2.4.1")
def train(features_path: str) -> str:
    # The heavy step. Asks for a GPU node (see below).
    ...

@dsl.component(base_image="registry/train:2.4.1")
def evaluate_and_register(model_path: str, champion_auc: float) -> None:
    # THE GATE. Score on the held-out set, compare to the champion,
    # and register ONLY if the challenger wins.
    ...

@dsl.pipeline(name="fraud-retrain")
def fraud_pipeline(data_snapshot: str, champion_auc: float):
    raw = ingest(data_snapshot=data_snapshot)
    checked = validate(raw_path=raw.output)
    feats = transform(raw_path=checked.output)
    model = train(features_path=feats.output)
    model.set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1)
    evaluate_and_register(model_path=model.output, champion_auc=champion_auc)

Two things to notice. First, the arrows are implicit: validate takes the output of ingest, so the engine knows validate depends on ingest and orders them for you. Second, every step names its own base_image, so the train step can pin PyTorch and demand a GPU while the validate step stays a tiny CPU container. That per-step container choice is what makes caching, per-step retries, and the free environment receipt all possible at once.

To make the two ideas at the heart of the pipeline concrete, here is a tiny runnable model of them: content-addressed caching that decides which steps to skip, and the evaluation gate that decides whether the new model ships. Nothing here calls a real framework; it is the decision logic on its own so you can watch it run.

None of this is theoretical. The big ML shops all built exactly this machinery, usually after the by-hand approach burned them.

Netflix built Metaflow and later open sourced it. Its whole pitch is that a data scientist writes normal Python, decorates functions as steps, and the framework handles the rest: it snapshots the code and data for every run so any past run can be reproduced, it moves heavy steps to the cloud transparently, and it lets you resume a failed run from the step that broke instead of the beginning. The scientist thinks about the model. The platform handles the DAG, the containers, and the reproducibility receipt. Notice how directly that maps onto what you just learned: the snapshot is the frozen data input, the resume-from-failure is run-level retrying, and the transparent cloud scaling is the orchestrator asking the cluster for the right compute.

Uber built Michelangelo, its internal ML platform, and standardized the training pipeline across hundreds of teams. Instead of every team reinventing ingest, train, and evaluate, they plug into shared pipeline steps with a common feature layer, versioned models, and an evaluation gate baked in. That standardization is why Uber could ship models for arrival-time estimates, fraud, and pricing without each team rebuilding the plumbing, and why a model at Uber comes with its lineage attached rather than as an anonymous file someone trained once.

The lesson from both is the same as the rest of this track. The model was never the bottleneck. The repeatable, reproducible, self-running path from fresh data to a blessed production model is the bottleneck, and a training pipeline is what removes it. Once it exists, retraining stops being a person's 2 AM chore and becomes something the platform just does: triggered when the world changes, reproducible by construction, resilient to the crashes and evictions that are ordinary at scale, and honest enough to keep the old model whenever the new one fails to earn its place.