Cache TTL (Time To Live): How to Choose the Value

Name: System Design Masterclass
Availability: InStock

What a TTL Is Actually Betting On

Every cache in this chapter has had a in it somewhere, and every team that has one has had the same argument: is five minutes too long, or not long enough?

Two panels. The left, how it is described, shows picking a number, feeding a freshness dial, ending in longer means more hits. Its key reads: decided by how brave you feel, argued about in a meeting, depends on nothing measurable. The right, what it actually is, shows your request stream feeding a bet on the next request, ending in pays only if it arrives in time. Its key reads: decided by key popularity, argued about with a histogram, depends on the request interval. — ComparisonFigure 1. A cache entry pays off only if the same key is asked for again before it expires
That single sentence turns the argument from a matter of taste into a question about your own traffic, which is countable. For a key requested every ten minutes, a five minute TTL never hits at all, however careful the reasoning behind it was.

So the right number depends on how often keys repeat. The next thing to establish is what that actually looks like.

Real Traffic Is Not Uniform

A line chart of the cumulative share of all requests against keys ordered by popularity as a percentage. The Zipf line rises steeply, reaching 56.2 percent at the first 1 percent of keys, 78 percent at 10 percent, and 94 percent at half. A straight diagonal line shows what equal popularity would look like. — ChartFigure 2. The busiest one percent of keys took 56.2 percent of all requests
The straight line is what people implicitly assume when reasoning about caches. The curved line is what request popularity actually looks like, and every intuition about changes once you accept it.

A very small head takes most of the traffic and is easy to cache well. A very long tail is asked for once or twice, and no helps it.

Sweeping the TTL

The script is at scripts/labs/ttl/sweep.py. It generates 400,000 requests across 20,000 keys over an hour, Zipf distributed, and reports hit rate, origin load and staleness at seven different TTLs.

A recorded terminal session. The busiest 1 percent of keys take 56.2 percent of requests, and 2,051 keys were requested exactly once. A table shows TTL against hit rate, origin requests and mean staleness: 10 seconds gives 50.1 percent, 199,640 origin requests and 5 seconds staleness; 60 seconds gives 66.8 percent, 132,694 and 30 seconds; 900 seconds gives 88.5 percent, 45,989 and 442 seconds; one hour gives 95.2 percent, 19,126 and 1,751 seconds. A note explains that the last two rows match because the window is one hour. — RecordedFigure 3. Seven TTLs against 400,000 requests of realistic traffic
The benefit is a curve that flattens. The cost is a line that does not, because mean staleness is always about half the . Those two columns moving differently is the whole argument.

A line chart with TTL in seconds on the x axis. The cache hit rate line rises from 50.1 percent at 10 seconds to 60.4 at 30, 66.8 at 60, 80.8 at 300 and 88.5 at 900, clearly flattening. A second line showing mean staleness as a percentage of the TTL stays flat at almost exactly 50 percent throughout. — ChartFigure 4. Hit rate flattens. Staleness stays at half the TTL, so it grows with it
Half of 900 seconds is 442 seconds of average wrongness, bought for 7.7 more points of hit rate than 300 seconds gave. Somewhere on every one of these graphs the two lines cross over in value.

The Ceiling Nobody Expects

Two hand-drawn bars. A long one labelled requests that can be cached, annotated as 95.2 percent being the ceiling at any TTL. A short highlighted one labelled keys asked for exactly once, annotated as 2,051 of 20,000 keys and compulsory misses. A handwritten note reads: a key requested once can never be a hit, no configuration reaches these. — SketchFigure 6. A cache that never expired would still miss on nearly five percent of requests
2,051 of the 20,000 keys appeared exactly once in 400,000 requests. Each one is guaranteed to be a miss, because there is no second request. Chasing the last few points is chasing something that does not exist.

Work out your compulsory miss rate before setting a hit rate target. A team aiming for 99 percent on traffic whose ceiling is 95 will keep raising the , keep adding staleness, and never arrive.

A line chart of requests reaching the origin out of 400,000, against TTL in seconds. It falls from 199,640 at 10 seconds to 158,455 at 30, 132,694 at 60, 76,893 at 300 and 45,989 at 900. — ChartFigure 7. Origin load is the number your database actually feels
Alert on this line, not on hit rate. Hit rate is a proportion that can be improved by adding cacheable traffic. Origin requests per second is the absolute number that decides whether the thing behind the cache survives a busy hour.

One Number for Very Different Keys

A sequence diagram with three lifelines: a popular key, the cache with a 60 second TTL, and a rare key. The popular key is asked 40 times a second, so it misses once every 60 seconds, giving 2,399 hits for every miss. The rare key is asked once every ten minutes and is always a miss, because the entry expired nine minutes ago, noted as it occupied memory for the whole minute. — SequenceFigure 8. The same sixty second TTL is excellent for one key and pure overhead for the other
A single global is applied to keys whose request intervals differ by orders of magnitude, and it can only be right for a narrow band of them.

A line chart of the mean age of the data served in seconds, against TTL in seconds. It rises from 5 seconds at a 10 second TTL to 15 at 30, 30 at 60, 149 at 300 and 442 at 900. — ChartFigure 9. The average thing you serve is half a TTL old
A request arrives at a uniformly random point in an entry's life, so the mean age is half. At a 15 minute TTL the average user is looking at seven minute old data and the worst case is fifteen, routinely.

Say the TTL out loud as an age instead of a duration. If that sentence is uncomfortable, the number is wrong, and it is a much better test than intuition.