Guides

System design

Caching: a visual guide

A cache keeps a copy of an answer, so you do not make it twice. The caches on this page run live, so you can send requests and see where each answer comes from.

Piotr Jura,

A cache keeps a copy of an answer close to the place that asks for it. The next request gets the copy, and the database does no work.

A cachethe copy expires after 6 s
clientdatabase
cache: empty
0 hits, 0 misses
Press Clear the cache. The next requests go all the way to the database, and the first answer that comes back fills the cache again.
  • A solid dot is a hit, so the answer came from the cache.
  • An orange ring is a miss. That request went to the database.
  • A grey dash is an old answer. You see it later in the guide, when the data changes and the cache still has the copy from before.

The layers

A request passes several caches on its way to the database, and the first one that has a copy answers.

Four caches in a rowthe first one with a copy answers
Last answer came from ...
browseremptyCDNemptyappemptyRedisemptydatabasesource
0 hits, 0 misses
A layer that says copy can answer. Turn a layer off and the request goes one step further.
  • The browser cache belongs to one visitor. Press New visitor and the request goes on to the CDN, which keeps one copy for all visitors.
  • Cloudflare caches files like images, CSS and JavaScript by default, and it does not cache HTML or JSON. You have to turn that on yourself.
  • App memory is the fastest layer, because a read from memory takes about 100 nanoseconds and a round trip inside one data center takes about 0.5 milliseconds. You pay with consistency: each server has its own copy, and a restart empties it.
  • Redis is one cache for all your servers, so you pay that round trip on every read.

Expiry

Every copy gets a time to live. After that time the cache drops the copy, and the next read goes to the database.

A copy with a time to livetime to live: 4 s
Database: v1Cache: empty
no copy
hitmissold
now
0 hits, 0 misses, 0 old
4 s
The bar is the time that the copy has left. The two version numbers show what the database has and what the cache has.
  • You get 2 reads per second here, so with a time to live of 4 seconds 1 read in 8 goes to the database.
  • Press Change the data. The cache gives the old version until the copy expires, and each of those reads is a dash.
  • Set the slider to 1 second and the dashes almost stop, but now every second read is a miss.
  • In HTTP the time to live is max-age in the Cache-Control header. In Redis it is EX on the SET command.

Delete on write

You can also delete the copy when the data changes, so the next read loads the new version.

Delete the copy on a writetime to live: 8 s
Database: v1Cache: empty
no copy
hitmissold
now
0 hits, 0 misses, 0 old
8 s
The same cache as before, and now each change of the data deletes the copy. Turn Delete on write off to compare.
  • The dashes are gone, and you pay with one miss after each change.
  • Every piece of code that writes the data must also delete the copy. If one place forgets, the cache gives old answers until the copy expires, so keep the time to live as a safety net.
  • The order matters. Microsoft's cache-aside guide says to write to the database first and to delete the copy after that.

When a hot key expires

A popular key gets many reads at the same moment, and when its copy expires they all miss together.

A hot key expires3 reads per second, a query takes 2 s
Database queries now: 0
6.0 s left
hitmissold
now
0 hits, 0 misses, 0 old
Each filled box is a database query that runs now. All of them compute the same answer.
  • You get 3 reads per second and the query takes 2 seconds, so one expiry starts 6 queries.
  • Press Serve the old copy. Now one read loads the new version and the others get the old one, so the database runs 1 query and not 6.
  • HTTP calls this stale-while-revalidate. The singleflight package in Go does it another way: the other callers wait for the one query and share its result.

Which one to use

Pick what you need. If you are not sure, set cache headers on your static files and put Redis in front of your slow queries.

Which layer fitspick what you need
SharedPrivateDeleteNo trip
RedisQuery results, sessions, counters
CDNImages, CSS, JavaScript, public pages
Browser cacheFiles that one visitor loads again
App memoryConfig and small lists that rarely change
A filled square means the layer gives you that. Pick one or more needs. The rows that fit stay lit.

Sources

Share

Newsletter

Email from me

Occasional notes on agentic development — new posts, videos, and tools I'm actually using.

By submitting, you agree to receive emails from Piotr Jura at The Agentic Developer. Unsubscribe anytime. Articles and videos when I publish.