Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · Core
Attaching the correct answer to each example, such as a category or a yes or no, so a model can learn from it or be tested against it.
In short
AI data labeling is the work of attaching the right answer to each example, such as "spam" or "not spam" on an email, so a machine learning model can learn from it. Labels are often the most expensive part of a project, because a person usually has to make them.
In short
AI data labeling is the work of attaching the right answer to each example, such as "spam" or "not spam" on an email, so a machine learning model can learn from it. Labels are often the most expensive part of a project, because a person usually has to make them.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Data Labeling across a run of lessons, not one page. These 4 alone are about 292 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
What a labelling budget buys, measured on 4,601 real emails over 30 random splits. Buying the labels the model was least sure of got within 0.01 of a model trained on all 3,450 pool labels with 188 labels on average, in all 30 repeats; buying labels at random got there by 500 labels in only 12. At 50 labels the aimed labels were behind random in 21 of 30 repeats, and the emails they picked were so hard that a model scored 0.698 on them against 0.922 on a random test set. Ten labelling rules and no hand labels scored 0.891, about what 100 random labels bought. Then who labels, what Mechanical Turk charges, consensus and hidden test items, weak supervision with Snorkel, model pre-labelling, and relabelling when a label's meaning changes.
Every labelled dataset has some wrong labels. On 1,797 real handwritten digits, a lab makes up to 40 percent of the training labels wrong, randomly or by swapping look-alike digits, and trains four kinds of model. Nearest neighbour copied mistakes straight through, while a forest and 15 nearest neighbours barely noticed random ones. From 20 percent up, look-alike mistakes did far more damage. Flagging every row where an out-of-fold prediction disagreed with its label found almost all the random mistakes, but was fooled by repeated confusions. A random forest asked about the rows it trained on flagged none of them, and wrong labels in the test set lowered an unchanged model's score by about the same share.
A model can only learn from answers that have arrived, and a team can only score it on them. I retrained the same model every week on 22,656 half-hours of electricity prices while the right answers arrived half an hour, a day, a week, four weeks or twelve weeks late. Accuracy fell from 0.7636 to 0.6954, though not smoothly: four weeks scored 0.7315 against a week's 0.7299, a gap that cannot be told apart from chance. Against a fair baseline, the model the same team would have had by never retraining, weekly retraining clearly helped only when the labels came within a day; twelve weeks late it scored 0.6954 against 0.7077. And with labels twelve weeks late, a dashboard showed a fine week on 65 days when the real week was bad. What a late label costs, how late the mistakes show, what you can watch meanwhile, and what to do.
I tested twelve changes to the same 0.5B ticket model, one setting at a time: the seed, the number of training examples, the number of passes, the learning rate, and whether the loss counts the prompt. First I measured the wobble, what the seed alone does: the totals were 93, 92 and 93 whole tickets of 131, but 15 to 21 tickets flipped between right and wrong between any two seeds. Against that, two changes clearly mattered: more examples (26, 58, 78, 93 for 50, 100, 200 and 500 rows) and a learning rate far too small (58). A higher learning rate (96) and masking the prompt (97) beat the one-ticket spread of the totals, but not the flips, and the sign test cannot tell either from luck.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Many machine learning models learn from examples where the right answer is already known. This is called supervised learning. The model looks at many pairs of input and answer, and finds the pattern that links them.
The input is the raw data: an email, a photo, a sound clip. The answer attached to it is the label. Data labeling, also called annotation, is the work of attaching those answers. The person who does it is called a labeller or an annotator.
Without labels, the raw data is an unsolved problem. With them, it becomes something a model can learn from, and something you can test a model against. That is why labelling comes before almost every supervised project.
The shape of a label depends on the task. Here are common examples, from simple to hard:
The first examples take one click. The later ones can take minutes or hours for a single item, because every box or pixel is its own small decision. Published papers show how wide the range is. In 2017, researchers drew a box in 7 seconds by clicking an object's four outermost points, which they said was 5 times faster than the usual way. The Cityscapes team, who labelled 5,000 street photos pixel by pixel, wrote that one photo took more than 1.5 hours on average.
When a team says "we need labels", it chooses among three kinds of people.
Large AI labs use the same mix. OpenAI's InstructGPT paper says it hired about 40 contractors on Upwork and through Scale AI to rank chat answers. Anthropic's 2022 paper says it used US-based Mechanical Turk workers with the Masters qualification, and crowdworkers hired on Upwork. Masters are workers that Amazon's systems picked as high performers.
The work itself happens in a labelling tool, not a spreadsheet. A tool such as Label Studio shows one item at a time, records who labelled it and when, and even how many seconds it took. A labeller follows a written rulebook, called the annotation guidelines, that says what each label means and how to handle hard cases. Some of the items they see are hidden test items with a known answer, used to check their work.
On pay, the crowd platforms let the buyer set the reward per task. Mechanical Turk adds a 20 percent fee on what you pay workers, and another 20 percent on tasks given to 10 or more workers. The lesson does not give a market rate. It does quote Anthropic's paper, which says the team aimed to pay its crowdworkers "significantly above the minimum wage in California".
Labelling is also hard because people disagree, even trained ones. How often depends on the task. In the LIDC/IDRI lung scan collection, four radiologists all agreed on only 34.7 percent of the lesions that at least one of them called a nodule. In Cityscapes, two people gave the same label to 96 percent of pixels. In the InstructGPT work, labellers ranking chat answers agreed 72.6 percent of the time.
Those numbers are not a ranking of people. Where a shadow on a scan stops being a nodule is a judgement call. A pixel of road is not. So the first thing to measure on any new labelling job is how often your own people agree. The inter-annotator agreement lesson shows how to correct that number for agreement by luck.
Wrong labels then hurt the model, but not always in the way people expect. The wrong labels lesson found that random mistakes did far less harm than mistakes with a pattern.
A crowd is not reliable on its own. The checks you build around it can be. Three checks do most of the work.
More opinions cost more than in proportion. On Mechanical Turk, asking 10 workers per item crosses the 10-assignment line, so Amazon's fee doubles from 20 to 40 percent.
Because labels cost so much, teams try to buy fewer of them. The lesson tested two ideas on 4,601 real emails from the UCI Spambase dataset. It hid the labels of a pool of 3,450 emails and "bought" them back in different ways, 30 times over with different random splits.
There is a catch with aimed labels. They are the hard cases on purpose. A model scored 0.922 on a random test set, but only 0.698 on the emails active learning had picked. So never use those labels as your test set. Keep a random, carefully labelled test set, sometimes called a golden dataset, that no strategy chooses.
Yes, but the work is changing. A newer approach lets a large language model make the first pass. It suggests a label for each item, and a person confirms or corrects it. Checking a suggestion is usually faster than labelling from nothing. One 2023 study in PNAS found ChatGPT beat crowd workers by about 25 points on tweets and news articles, at under $0.003 a label. That is one study, on text, so treat it as one data point.
The risk is that a person only clicks accept. Then the labels are the model's labels with a person's name on them, and the model's blind spots go straight into your training data. Keep hidden test items under the checking step too.
The demand for human labels has not gone away. Labels also need refreshing when the meaning of the right answer changes, for example when a new kind of fraud appears. In June 2025, Scale AI announced an investment from Meta that it said values the company at over $29 billion.
The Data Labeling and Annotation lesson has the full lab, its code, a rule playground in the browser and a step-by-step plan for spending a labelling budget. It is the best next step if you have to decide how your own data gets labelled.
Amazon Mechanical Turk
A crowd platform where the buyer sets the pay per task. Amazon takes a 20 percent fee, plus 20 percent more on tasks with 10 or more workers.
OpenAI InstructGPT
About 40 contractors, hired on Upwork and through Scale AI, ranked model answers. They agreed with each other 72.6 percent of the time.
Snorkel
A Stanford tool for weak supervision. Its LabelModel combines noisy labelling rules into one label, with no hand labels.
Google Snorkel DryBell
Built labelling rules from knowledge across the company, and got classifiers of comparable quality to ones trained on tens of thousands of hand labels.
Label Studio
An open-source labelling tool. Its export records who made each label, when, and how many seconds it took.
It is work attaching the right answers to examples that a model will learn from: marking emails as spam, drawing boxes around cars, transcribing audio or ranking chatbot answers. The work is done in a labelling tool, following a written rulebook, and is checked with hidden test items and by comparing several people's answers. It is offered by crowd platforms such as Mechanical Turk, by labelling companies such as Scale AI and Appen, and directly by AI labs.
A simple one: a person reads an email and marks it spam or not spam. After a few thousand of these, a model can learn to sort new emails. A harder one: a person colours every pixel of a street photo by what it shows, road, car, person or sky. The Cityscapes team reported that this took more than 1.5 hours per photo.
Yes. The largest language models still rely on labels made and checked by people, for example people ranking chat answers. What is changing is how the work is done. Rules and language models now often make a first pass, and people check and correct it. Labels also have to be redone when the meaning of an answer changes.
In practice they mean the same thing: attaching the right answer to each example. The person who does it is called a labeller or an annotator, and both words are common. What matters more is the shape of the label, from one click for a class to more than an hour for a finely labelled photo.
Checked against these on .
Golden Dataset
A curated set of inputs with known-correct expected outputs, used to measure quality and catch regressions when you change a prompt, a model, or the retrieval stack.
LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
Feature Engineering
Turning raw data into the signals a model actually learns from. Often the highest-leverage work in a classical ML system, and where a feature store keeps definitions consistent.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Train-Serve Skew
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
This page explains the idea. The "Data Labeling and Annotation: The Expensive Bottleneck of Supervised ML" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Labeling and Annotation: The Expensive Bottleneck of Supervised ML lessoncourse 2
Take models from notebook to production, with labs on real models.