Name: System Design Masterclass
Availability: InStock

Lazy loading is the default way almost every cache in the world is filled. A request arrives, the cache is checked, and if the value is not there the application fetches it, stores it, and returns it. The next request for that key is a hit. Nothing has to be published, nothing has to be predicted, and nothing has to be maintained.

Every sentence in that description is true. The one it leaves out is who paid for it.

Two panels. The left, how it is described, runs from real traffic through the cache fills itself to no maintenance needed, with keys reading who does the work nobody, what is stored only what is wanted, cost sounds like none. The right, who actually pays, runs from one unlucky user through waits for the slow path to so everybody after does not, with keys reading who does the work a user, how many users one per distinct key, cost 13,259 of them. — ComparisonFigure 1. Every entry in a lazily filled cache was paid for by one user who waited
The person who pays is chosen at random and gets no benefit from their own wait. They took the slow path so that everybody who asks for the same thing afterwards does not.

That is not an argument against lazy loading. It is the right default nearly everywhere, and the rest of this lesson will say so more than once. It is an argument for knowing the size of the bill, because the number is much larger than the phrase self-filling suggests.

The way to find out is to count. A hundred thousand requests were drawn from a Zipf distribution over twenty thousand possible keys, which is the standard model for how real traffic distributes itself across a catalogue: a small number of very popular items and a long tail of things asked for once or twice. Then the same stream was replayed five times, warming a different number of the most popular keys before each run.

A terminal capture of the cold start lab. A cold cache, 100,000 requests, 20,000 possible keys, 13,259 distinct keys touched, a miss costing 200 ms and a hit 1 ms. A table with rows for 0, 20, 200, 2,000 and 13,000 keys warmed in advance, showing users who paid at 13,259, 13,239, 13,059, 11,263 and 3,138, the share of requests at 13.3 percent falling to 3.1, and the mean latency at 27.4 ms falling to 23.7 ms. Below it a warm-up curve showing hit rate per 5,000 requests at 59, 75, 77, 80, 83, 86, 86 and 87 percent. — terminal-figureFigure 2. The measurement. Five runs of the same request stream over a cache that starts empty
With nothing warmed, 13,259 separate users paid the full slow path, which is one for every distinct key the traffic touched. That is 13.3 percent of all requests, and the mean across the whole run was 27.4 milliseconds.

The mean is the first thing to throw away. Look at the column and it barely moves: 27.4 milliseconds at the top, 23.7 at the bottom, while thousands of users stop waiting two hundred milliseconds each. A four millisecond difference in the summary statistic corresponds to ten thousand people no longer waiting.

A hand-drawn figure. A narrow bar labelled mean 27.4 ms with a note beneath reading looks completely healthy. Below it a much wider bar labelled 13.3 percent of requests took 200 ms with a note reading which is one user in eight. A marker annotation reads the same 100,000 requests, reported two ways. — SketchFigure 3. A mean of 27 milliseconds and one user in eight waiting two hundred are the same data
The latency here has exactly two values and nothing in between. Their average is a point where no measurement landed. No request took 27 milliseconds. They took 1 millisecond or they took 200.

produces bimodal distributions by construction, which makes the mean the least informative number you can quote about one. The statistic that describes what happened is the share of requests that took the slow path.

Lazy Loading

The rest of this lesson

Continue this lesson

A Cache That Fills Itself Out Of People

Counting The People Who Paid