Picture the data scientist who built your fraud model. Every couple of weeks, fraud patterns shift, so the model needs fresh training. Here is how that actually happens without a platform: she opens a notebook, downloads a new export of last month's transactions, runs the cells top to bottom, eyeballs the accuracy, exports a new pickle file, messages it to a backend engineer on Slack, and he swaps it into the service by hand.

Now multiply that. Ten models. Thirty models. Each one needs retraining on its own cadence. Each retrain is a person doing the same manual dance, remembering which export to use, which notebook cells to skip, which file is the good one. People go on vacation. People forget a step. Someone trains on a broken data export at 2 AM and quietly ships a worse model, and nobody notices for a week.

Retraining by hand does not scale, and it is not just about effort. It is about trust. When a human runs the steps, you cannot answer the questions that matter: which data trained this exact model, which version of the code, can we reproduce it, and did anyone actually check it was better before it went live? A training pipeline exists to answer all four, automatically, every single time.

A training pipeline turns "a person carefully runs some steps" into "the platform runs the same steps the same way, on a trigger, with a receipt."

A training pipeline is a directed acyclic graph, a DAG, of the steps that turn raw data into a trained, blessed model. Directed because the steps run in an order. Acyclic because you never loop back inside a single run. Each box does one job and hands its output to the next.

The classic shape is six steps:

1. Ingest the data for this run.
2. Validate it, so a broken feed fails fast.
3. Feature transform it into the exact inputs the model expects.
4. Train the model.
5. Evaluate it against the model already in production.
6. Register it, but only if it won.

The reason to draw it as a graph and not a single script is the arrows. They make the dependencies explicit. Validate cannot start until ingest finishes. Train cannot start until the features exist. And the branch at evaluate is the whole point of the pipeline: a fresh model only earns a spot in the registry if it beats the current champion on data it has never seen. Lose that comparison and the run stops, so production keeps the model it already trusts.

Drawing a DAG is easy. Running it reliably, on a schedule, retrying the flaky step, placing the training step on a GPU while the light steps run on cheap CPUs, is not. That job belongs to an orchestration engine.

An orchestrator owns the DAG. It knows the step order, launches each step when its inputs are ready, retries failures, and records whether the run passed. The important thing to understand is what it does not do: it does not run your training code itself. It is a conductor, not a worker. It asks the compute cluster to execute each step and waits for the result.

The three names you will hear most, and what they schedule:

Underneath, the pattern is the same everywhere. The orchestrator decides what runs and when. Kubernetes decides where, placing each step on a node with the right hardware. And containers do the actual work, passing data to each other as files in object storage because separate containers on separate machines share no memory.

You do not draw the DAG by hand. You write it as code, and the engine builds the graph from the dependencies. Here is the same six-step pipeline expressed in a Kubeflow-style Python definition. Read it as a description of boxes and arrows, not as a script that runs top to bottom.

from kfp import dsl

# Each @dsl.component becomes ONE step, and each step runs in its
# own container image with its own pinned dependencies.
@dsl.component(base_image="registry/ingest:2.4.1")
def ingest(data_snapshot: str) -> str:
    # Pull a FROZEN snapshot, not "today's live table", so the run
    # is reproducible. Returns the S3 path of the raw dataset.
    ...

@dsl.component(base_image="registry/validate:2.4.1")
def validate(raw_path: str) -> str:
    # Schema, null rates, value ranges. Raise to FAIL the whole run
    # if the upstream feed is broken.
    ...

@dsl.component(base_image="registry/features:2.4.1")
def transform(raw_path: str) -> str:
    # Same feature logic that runs at request time, or you get skew.
    ...

@dsl.component(base_image="registry/train:2.4.1")
def train(features_path: str) -> str:
    # The heavy step. Asks for a GPU node (see below).
    ...

@dsl.component(base_image="registry/train:2.4.1")
def evaluate_and_register(model_path: str, champion_auc: float) -> None:
    # THE GATE. Score on the held-out set, compare to the champion,
    # and register ONLY if the challenger wins.
    ...

@dsl.pipeline(name="fraud-retrain")
def fraud_pipeline(data_snapshot: str, champion_auc: float):
    raw = ingest(data_snapshot=data_snapshot)
    checked = validate(raw_path=raw.output)
    feats = transform(raw_path=checked.output)
    model = train(features_path=feats.output)
    model.set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1)
    evaluate_and_register(model_path=model.output, champion_auc=champion_auc)

Two things to notice. First, the arrows are implicit: validate takes the output of ingest, so the engine knows validate depends on ingest and orders them for you. Second, every step names its own base_image. The train step can pin PyTorch and a GPU while the validate step stays a tiny CPU container. That per-step container choice is not an accident, and the next slides explain why it matters.

Two questions decide the entire behavior of a pipeline, and they are easy to mix up.

The first: should the pipeline run at all? A run should be triggered, never done out of boredom. There are three common triggers:

• Schedule. A clock. Retrain nightly or weekly as a baseline so the model never goes stale.
• New data. A signal that a fresh batch of labeled data has landed, worth learning from.
• Drift. An alert forwarded from the monitoring layer saying live inputs have drifted away from the training data. This is the strongest trigger, because it means the model is now seeing a world it was not trained on.

The second question is the one people forget: even after a run finishes, does the new model actually ship? This is the evaluation gate, and the answer is no by default.

The gate compares the fresh model, the challenger, against the model already in production, the champion, on a held-out set that neither trained on. Only if the challenger wins by a margin the team set in advance does it get promoted. Lose, and the challenger is discarded and the champion keeps serving. This is what stops the platform from ever shipping a worse model. A retrain that produces a losing model is not a failure. It is proof your current model is still the best you have.

Here is the gate as a lifecycle, with real numbers:

Notice the comparison is always on the held-out set. A model always looks brilliant on the data it trained on, so that number is worthless for deciding whether to ship. The held-out score is the only honest one.

A pipeline is only trustworthy if you can reproduce a run. Six weeks from now a model misbehaves in production, and you need to replay the exact run that produced it, reproduce the bug, and fix it. That only works if three inputs were frozen when the run happened.

Think of it as a receipt stapled to every model in the registry: this data snapshot, this git SHA, this image digest. Miss any one of the three and a rerun can quietly produce a different model, which means you never really understood what you shipped. This is exactly why each step runs in its own container. The image digest is half the receipt, captured for free by the pipeline.

It would be simpler to write one big script that ingests, trains, and evaluates in a single process. Real platforms deliberately do not. Each step runs in its own container, and there are concrete reasons.

• Different steps want different hardware. Ingest is network bound and happy on a cheap CPU. Train wants a GPU for twenty minutes. Separate containers let the scheduler put each step on the right box instead of paying for one oversized machine that sits idle most of the run.
• Different steps want different dependencies. The validate step needs a data-quality library. The train step needs PyTorch and CUDA. In one process those dependency trees fight. In separate images they never meet.
• Failures stay contained and steps are retryable. If train crashes on a flaky node, the orchestrator retries just that step. It does not re-run the expensive ingest, and a crash in one step cannot corrupt another.
• The environment is captured for free. Each step's image digest is recorded on the run. That digest is a third of your reproducibility receipt, with no extra work.

The cost is real too: passing data between containers means writing it to object storage instead of keeping it in memory, which adds latency and some plumbing. For a training pipeline that runs for minutes or hours, that trade is easily worth it. For a tight inner loop measured in milliseconds, it would not be.

None of this is theoretical. The big ML shops all built exactly this machinery, usually after the by-hand approach burned them.

Netflix built Metaflow and later open sourced it. Its whole pitch is that a data scientist writes normal Python, decorates functions as steps, and the framework handles the rest: it snapshots the code and data for every run so any past run can be reproduced, it moves heavy steps to the cloud transparently, and it lets you resume a failed run from the step that broke instead of the beginning. The scientist thinks about the model. The platform handles the DAG, the containers, and the reproducibility receipt.

Uber built Michelangelo, its internal ML platform, and standardized the training pipeline across hundreds of teams. Instead of every team reinventing ingest, train, and evaluate, they plug into shared pipeline steps with a common feature layer, versioned models, and an evaluation gate baked in. That standardization is why Uber could ship models for arrival-time estimates, fraud, and pricing without each team rebuilding the plumbing.

The lesson from both is the same as the rest of this track. The model was never the bottleneck. The repeatable, reproducible, self-running path from fresh data to a blessed production model is the bottleneck, and a training pipeline is what removes it. Once it exists, retraining stops being a person's 2 AM chore and becomes something the platform just does.