Caching Strategies

Application Cache

0 of 7 complete

0%

Contents

Back|Caching StrategiesApplication Cache
1/7
21 min left
  1. Home
  2. System Design: Foundation
  3. Caching Strategies
  4. Application Cache
Prerequisites
Local Cacherequired
Related Topics
Lazy LoadingWeb & Content DeliveryCachingCore FundamentalsContent Delivery Network (CDN)Load Balancing & ProxiesData Locality OptimizationData Replication & DistributionIn-Memory DatabasesDatabase Types & Storage
Previous lessonLocal CacheNext lessonBrowser Caching

System Design

  • Foundation
1 of 7
  • Intermediate
  • Advanced
  • Capstone

    • AI Engineering

    • Foundation
    • Data, RAG and Agents
    • Evaluation, LLM Ops and Security

    systemdesign.academy

    • Home
    • Glossary
    • Interview prep
    • Reviews
    • About
    • Privacy
    • Terms

    this free lesson is part of

    System Design Masterclass

    • 770 lessons, beginner to principal
    • 90 real interview systems taken apart step by step
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the System Design course

    Save your progress and get the interview checklist PDF

    System Design Interview Cheat Sheet: the 7 interview stages, the 10 points a scorecard grades, 12 classic systems. 18 pages. Free with an account.

    you just read a free lesson from

    System Design Masterclass

    • 770 lessons, beginner to principal
    • 90 real interview systems taken apart step by step
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the System Design course

    The Cache That Knows What the Data Means

    A caches whatever your headers permit. A proxy caches responses. Neither of them has any idea what a user, an order or a permission is.

    Two panels. The left, infrastructure cache, shows a request passing through something in front, then reaching your code which is unaware. Its key reads: decides what config, your code knows nothing, bugs live in the config. The right, application cache, shows a request reaching your own code, then the database on a miss. Its key reads: decides what your code, your code knows everything, bugs live in your code.
    ComparisonFigure 1. The cache your code manages knows what the data means, and owns every mistake

    Only your code knows that a leaderboard may be five seconds old, or that a permission check must not be. No layer above you has that information, and that is exactly why an application cache can do things a never could.

    The price is that every correctness question is now yours, and this lesson is about the first one.

    Cache Aside, and the Race Inside It

    A sequence diagram with three lifelines: your code, the cache and the database. Your code asks the cache for user:42 and gets nothing. It queries the database and receives the row. It sets user:42 in the cache and answers the request, noted as four lines of code and the third is where the trouble is.
    SequenceFigure 2. Four lines, and every application cache begins as these four lines

    Look in the cache. If it is not there, ask the database, store it, return it. It is easy to write and easy to read, which is most of why it is everywhere. It is also correct, as long as nothing writes while a read is in flight.

    A sequence diagram with four lifelines: a reader, the cache, the database and a writer. The reader misses on user:42 and reads name equals Ann. The writer writes name equals Anna and deletes user:42, which is not there yet, noted as the delete removes nothing because the fill has not happened. The reader then sets user:42 to Ann, noted as the old value written after the delete.
    SequenceFigure 3. The writer deleted a key that did not exist yet, and the reader put the old value back

    Both threads did exactly what the pattern says. The delete arrived before the fill, so it deleted nothing, and the fill then stored a value that was already out of date.

    This entry does not go stale, it goes wrong, and it stays wrong. There is no expiry involved. Every later read is a cache hit on a value the database disagrees with.

    after “Cache Aside, and the Race Inside It”

    Could you draw this on a whiteboard tomorrow?

    Cache Invalidation in the System Design course takes it from here.

    Each step has its own diagram, so you can follow it at your own pace.

    Open the System Design course page

    How Often, Measured

    The usual response is that the interleaving is unlikely. It is worth measuring instead of assuming, and the script is at scripts/labs/appcache/aside_race.py.

    A recorded terminal session. A table with database read times of 0.5, 1, 2, 4 and 8 milliseconds across the top. Plain cache aside stores the wrong value in 6.0, 9.0, 17.3, 39.0 and 84.0 percent of trials. The same with a lock measures 5.3, 9.0, 15.7, 42.0 and 82.0 percent. A fill refused if the version moved measures 0.0 percent at every read duration.
    RecordedFigure 4. The race, made to happen and counted, 300 trials per cell

    The writer arrives at a uniformly random point in a ten millisecond window. The failure rate is simply the chance it lands inside the read, so it tracks the read time almost exactly.

    A line chart of the percentage of fills that stored the wrong value, against how long the database read takes in milliseconds. Plain cache aside rises from 6 percent at 0.5 ms to 9 at 1 ms, 17.3 at 2 ms, 39 at 4 ms and 84 at 8 ms. A second line for the same pattern with a lock sits on top of it throughout.
    ChartFigure 5. A two millisecond read on a key written every ten milliseconds loses one fill in six

    At half a millisecond the rate is 6 percent, which sounds ignorable and is not. At eight milliseconds, an ordinary uncached join, 84 percent of fills store the wrong value. This is not an unlucky scheduler. It is one ordinary event overlapping another.

    The Fix That Is Not a Fix

    Five rows. The instinct: two threads are racing, so add a lock. What the lock does: serialises the two writes to the cache. What the problem was: the reader is holding data that is already old. So the lock: changes nothing, measured at the same rate. What does work: refuse the fill if anything was written during the read. A closing line reads that mutual exclusion cannot help when the problem is that one thread's data went out of date before it took the lock.
    FigureFigure 6. Adding a lock measured exactly the same failure rate

    This is in the lab deliberately, because it is the first thing everybody reaches for and it looks like it should work. A lock guarantees the two cache writes do not interleave. It does nothing about the reader holding a row that was read before the write happened.

    When a lock does not help, it usually means the race is not between two operations but between an operation and time.

    A sequence diagram with four lifelines: a reader, the cache, the database and a writer. The reader misses and notes the version as 7, then reads name equals Ann. The writer writes name equals Anna and the version becomes 8, then deletes user:42. The reader finds the version is 8 and not 7, so the fill is refused, noted as the next reader misses and reads the correct value.
    SequenceFigure 7. The reader checks whether the world moved, and gives up if it did

    A miss is cheap. A poisoned entry is not. So when the two are in tension, throwing the fill away is always the right choice. Measured at zero failures across every read duration tested.

    Patterns, Keys and What to Cache

    Five rows. Cache aside: read check miss fill, write update the database and drop the key, tagged the default. Read through: the cache library fetches for you on a miss, tagged same race hidden. Write through: write to cache and database together synchronously, tagged no race slower writes. Write behind: write to cache and flush to the database later, tagged fast can lose data. Refresh ahead: reload a hot key just before it expires, tagged an addition not a pattern.
    FigureFigure 9. Five names, and only two change what happens during a concurrent write

    Cache aside and read through differ in who writes the code, not in the ordering, so they have the same race. When somebody proposes switching between them to fix a consistency problem, the switch will not fix it.

    Two panels. The left, update the entry, shows writer A and writer B both writing the cache, ending in whichever landed last wins. Its key reads: cache holds a guess, order matters yes, can disagree with the db. The right, delete the entry, shows both writers deleting the key, ending in the next read fetching the truth. Its key reads: cache holds nothing, order matters no, can disagree no.
    ComparisonFigure 10. Two concurrent writers updating can leave the wrong value. Two deleting cannot

    Deleting is and order independent. Updating is neither. The value you would write is a guess about what the database now contains, and the cost of being wrong is a poisoned entry while the cost of deleting is a single miss.

    Herds, Misses and Where Things Live

    A bar chart of the percentage of requests that wait for the database, with 100 concurrent readers. With nothing it is 100 percent. With single flight it is 1 percent. With stale while revalidate it is 0.
    ChartFigure 14. Serving stale while refreshing means nobody waits at all

    Stale while revalidate is the strongest tool an application cache has and the one people forget they can use. If the data is already allowed to be a minute old, allowing a minute and two seconds costs nothing and removes every refill stall.

    Two hand-drawn bars. The first, without negative caching, is labelled every request for a missing id hits the database. The second, with negative caching, is highlighted and labelled the miss itself is cached briefly. A handwritten note reads: a scanner asking for a million missing ids is a database problem until the misses are cached.
    SketchFigure 15. A cache that only stores hits leaves every miss going to the database

    If not-found is a legitimate and repeatable answer, cache it, with a shorter than a hit. If you do not, your cache has a hole in it exactly where hostile or buggy traffic will find it.

    Two panels. The left, in the process, shows config and schemas going through an immutable only bar, living until restart. Its key reads: good for things that cannot change, never for user data. The right, in the shared cache, shows sessions and aggregates going through one copy for all instances, invalidated on write. Its key reads: good for anything mutable, never for the only copy.

    Quick Check

    Knowledge Check

    4 questions - Score 80% to pass

    Q1

    Your endpoint's database read takes 8 ms and the key is written roughly every 10 ms. What fraction of cache-aside fills store a stale value?

    Q2

    You add a mutex around both the cache write in the writer and the fill in the reader. What happens to the stale-fill rate?

    Q3

    Two writers update the same row at nearly the same time. Should each write the new value into the cache, or delete the key?

    Q4

    Your cache hit rate is 98 percent and there are no errors or latency spikes. Can you conclude the cache is correct?

    The two lines sitting on top of each other is the next finding.

    Every real implementation is a version of this. calls it WATCH, databases call it compare-and-set, and some caches use a delete marker that outlives the read. The shape is always: prove nothing changed, or do not write.

    Two hand-drawn bars. The top, labelled no TTL, is a full-width bar reading cache says Ann, database says Anna, described as wrong from here forever. The bottom, labelled with a 60 second TTL, is split: the first part reads still wrong still a HIT and the remainder reads correct again. A handwritten note reads: a TTL bounds how long you are wrong, it does not stop you being wrong.
    SketchFigure 8. A TTL turns a permanent disagreement into a temporary one

    Put a on everything, and do not mistake it for correctness. During that window every read is a hit, so there is no error and no latency signal. The only way to notice is to compare the cache against the database on purpose.

    Five key patterns. user:42 is fine, one entity one key. user:42:v3 is better, the version makes a stale read impossible. orders:user:42:page:1 is fine, and now you have to invalidate all pages. search with free text produces unbounded keys and will never hit. report:2026-08-27 is excellent, immutable once the day is over.
    FigureFigure 11. The key is where you decide how invalidation will work

    Putting a version in the key turns invalidation into writing a new key, which cannot race. Free text produces one key per query, so the hit rate approaches zero. Both decisions are made when you pick the name.

    A flowchart. Starting from a value your code could cache, it asks whether it is derived from data other people can change. If no, cache it freely and the lesson does not apply. If yes, it asks whether serving a stale version would be visible or harmful. Harmful means do not cache it, or cache with a version check on every fill. Merely visible means cache with a TTL and be able to say how long a wrong value survives. Neither means cache it, and this is where almost all the real wins are.
    FlowchartFigure 12. Three questions, and most of the value is in the bottom right branch

    Teams usually cache the wrong things: the cheap lookups that were never slow, and the permission checks that must not be stale. The expensive aggregate nobody minds being five seconds old is the one sitting uncached, because it feels riskier and is not.

    A bar chart of milliseconds spent in a typical uncached endpoint. Auth is 3, the first query 8, the second query 9, join and business logic 21, serialisation 6, and the total 47.
    ChartFigure 13. The slowest single thing is usually not a query

    either query saves less than caching the finished result. Cache the highest thing you are allowed to cache: caching the assembled response removes the queries, the logic and the serialisation together.

    after “Patterns, Keys and What to Cache”

    Stuck on how this part actually works?

    The System Design course walks through it step by step in Database Query Caching.

    One of 770 lessons in the System Design course, written in the same plain style.

    See the System Design course
    ComparisonFigure 16. Immutable in the process, anything a write can change in the shared cache

    Never make a shared cache the only place a value exists. Redis will evict under memory pressure, restart, or fail over and lose the tail of its , and every one of those is normal behaviour. If losing a value would be an incident, it belongs in the database.

    Five metrics. Hit rate: necessary, and the one everybody already has. Eviction rate: high means the cache is too small to help at all. Fill latency: how long a miss costs, which sets your tail latency. Stale reads detected: sample keys and compare against the database. Keys created per second: a sudden rise means the key space just exploded.
    FigureFigure 17. Five metrics, and the fourth is the only one that can see this lesson's bug

    A poisoned entry produces no errors, no , and a perfect hit rate. Sampling a few cached keys on a schedule and comparing them against the database costs almost nothing and is the only instrument that would have caught the race measured here.

    Five rows with real product logos. Redis: WATCH and Lua give you the version check, tagged shared. Memcached: CAS tokens do the same job, older and simpler, tagged shared. PostgreSQL: sometimes the answer is an index and not a cache, tagged source. Python: cachetools for local and redis-py for shared, tagged client. Go: singleflight is in the standard extended library, tagged client.
    ArchitectureFigure 18. Both major shared caches already give you the version check

    Nothing here requires a new dependency. Before adding a cache to fix a slow endpoint, run the query plan: a cache in front of a query that needed an index is a permanent operational cost paid to avoid a one line migration.

    Four pairs of sentences. The race is theoretical becomes measured at 17 percent with a 2 ms read and 84 percent with an 8 ms read. We added a lock so it is fixed becomes measured at the same rate as no lock. The TTL keeps it fresh becomes the TTL bounds how long it stays wrong. Cache hit rate is 98 percent so we are fine becomes hit rate cannot see a wrong value, sample and compare instead.
    FigureFigure 19. Four sentences, and the first two were measured to be false

    The last one deserves sitting with. A poisoned entry has a one hundred percent hit rate on that key, so the metric everyone watches moves in the reassuring direction exactly when the cache has gone wrong.

    Six rows summarising the lesson. What it is: a cache your own code manages, key by key. What it buys: you can cache what only your code understands. The default pattern: cache aside, four lines, and a real race. How often the race fires: 6 percent at a 0.5 ms read, 84 percent at 8 ms, measured. What does not fix it: a lock, measured at the same rate. What does: refuse the fill if the version moved. A closing line reads that on a write you should delete the key rather than updating it, and refuse any fill that cannot prove the world stood still.
    FigureFigure 20. The most powerful cache you have, and the one whose bugs are entirely yours

    Two rules carry most of the value. On a write, delete rather than update, because deleting is order independent and updating is a guess. And on a fill, prove that nothing changed while you were reading, because a wasted read costs one miss and a wrong entry costs you every read that follows.

    after “Herds, Misses and Where Things Live”

    An interviewer will ask why, not only what.

    Pick it up in Redis Cache, from the System Design course.

    It ends with a short quiz, so you know it stuck.

    Look inside the System Design course $49 once, lifetime access