Two Ways to Catch the Rare Bag
Picture an airport scanner. Almost every bag on the belt is ordinary. Once in a long while, one needs a closer look. The officer has two ways to catch more of the rare ones.
The first is to retrain the officer: show them hundreds of extra pictures of suspicious bags until those feel common. The second is simpler: turn the sensitivity dial on the machine, so it flags a bag at a lower level of suspicion.
Machine learning has the same two choices when one class is rare, like fraud, disease or defects. You can rebalance the training data, or you can move the threshold, the dial. The imbalanced-data lesson describes both. In this lesson I measure them against each other, and the answer turns on how you set the dial.
The Words You Need First
Rare class. The thing you want to catch, when it makes up only a small share of the data. Here, 2, 5 or 10 percent.
Probability. The model's output for each row: a number from 0 to 1 saying how likely it thinks the row is rare.
Threshold. The cut-off. Rows with a probability at or above it are called rare. The usual default is 0.5.
Precision. Of the rows the model called rare, the share that really were. Low precision means many false alarms.
Recall. Of the rows that really were rare, the share the model caught. Low recall means many misses.
F1. One number that combines precision and recall. It is high only when both are high.
Rebalancing. Changing the training data or the training rules so the rare class counts for more.
Training, validation and test sets. The model learns from the training set. The validation set is kept aside to make choices on, such as where to set the threshold. The test set is kept aside until the very end, to give a fair final score.
Seed. A starting number for the random shuffles, so a run can be repeated exactly. Each seed thins and splits the data differently.