Name: System Design Masterclass
Availability: InStock

Moving a cache out of the process is presented as a capacity and consistency upgrade: one copy instead of one per instance. Both of those are real. What the move also does is put a fixed price under every single hit.

Two panels. A local cache lookup is a hash lookup, a pointer in your process, with no network at all, and keys reading measured p50 0.00004 ms and what can fail nothing. A remote cache lookup is a socket write, a round trip and a parse, on loopback at best, with keys reading measured p50 0.056 ms and what can fail the network. — ComparisonFigure 1. Moving a cache out of the process multiplies the cost of a hit by more than a thousand
Measured against a real on loopback, which is the most favourable network any cache will ever have. That 0.056 milliseconds is a floor, and it does not depend at all on what you put in the cache.

A hand-drawn figure. A wide bar labelled one remote cache hit on loopback, 0.056 ms. Below it a bar six pixels wide labelled one local cache hit. A note reads that the second bar is 0.00004 ms and is drawn at six pixels because it is 1,326 times smaller. — SketchFigure 2. The same operation on the same value, in and out of the process
There is no way to draw these two on one axis honestly. This is measured with no network between them at all, so on a real deployment the first bar is several times longer than drawn.

A terminal capture of the remote cache lab. A real redis-server on loopback with 400 samples per measurement. A first table shows a GET of a 200 byte value at a p50 of 0.056 ms and a p99 of 0.084, against a local dict at 0.00004 ms. A crossover table shows sha256 rounds against compute time and the GET cost, with the verdict switching from compute it to cache it between 50 and 200 rounds. A size table shows GET p50 and json.loads p50 at 1, 2, 22 and 232 kilobytes, with the deserialise share rising from 2 percent to 94. — terminal-figureFigure 3. The measurement. A real Redis, over a raw socket, with no client library in the way
A remote cache hit costs 0.056 ms and a local dictionary lookup of the same value costs 0.00004, which is a factor of 1,326.

A line chart with two series against sha256 rounds. Computing the value rises from 0.0004 ms at one round to 0.286 at a thousand. Fetching it from the remote cache is a flat line at 0.056 ms. The lines cross between fifty and two hundred rounds. — ChartFigure 4. Work cheaper than 0.056 milliseconds is made slower by caching it remotely
Fifty rounds of sha256 takes 0.016 ms, which is a third of the fetch. Two hundred takes 0.061, which is just past it. Below that line the cache reports an excellent hit rate and makes the system slower.

A great deal of what ends up in is below this line: a formatted string, a small derived object, a parsed configuration value, a permission check over data already in memory. Every one of those is faster to redo than to fetch.

Remote Cache

The rest of this lesson

Continue this lesson

The Price Of Admission

What The Floor Makes Uneconomic