CDNs
Read firstCaching patternsLoad balancing
Someone in Chennai opens your site, and your servers are in Virginia. That is about 13,800 km in a straight line, and light in fibre covers about 200 km every millisecond, so one round trip can never be faster than about 140 ms. Before the first byte of the page arrives, the browser needs a round trip to open the connection, one for TLS and one for the request itself. That is over 400 ms of waiting that no amount of fast code will remove.
A CDN (content delivery network) fixes this by keeping copies of your files in hundreds of cities. The user in Chennai talks to a server in Chennai, a round trip of a few milliseconds, and that server only goes to Virginia for something it does not have.
Your CDN serves 99% of requests from its own copies. Someone changes a setting and it drops to 90%. Roughly how much more traffic reaches your own servers?
What a CDN actually is#
Three kinds of box, and each one has a job.
- An edge is a CDN server in a city near users, also called a point of presence. It keeps copies of popular files and answers from them.
- The origin is your own server, where the real files live. The edge goes there on a miss.
- A shield is a middle layer between the edges and the origin, usually one per region. A miss at any edge in that region asks the shield first, so the origin sees one request where it would have seen dozens.
A CDN is the outermost layer of the cache stack in the caching patterns chapter, and everything from there applies: a hit ratio, an expiry, and the days a cache goes empty. What is new is that there are hundreds of caches instead of one, run by somebody else, in cities you will never visit.
- CloudflareCDN plus security and code that runs at the edge
- AkamaiThe oldest of the large CDNs
- FastlyCDN known for fast purges and edge code
- Amazon CloudFrontThe CDN inside AWS
Why the shield exists#
A new video segment goes live and is popular at once. Without a shield, every edge that gets asked for it misses and goes to the origin, so one new file becomes hundreds of origin requests within a second. With a shield, those edges all ask the shield, and the shield asks the origin once.
The shield does nothing for files that are already warm everywhere. It earns its place on cold misses, which come in bursts: a new release, a viral post, or every cache emptying at once. Those are exactly the moments your origin is least able to take 25 times the load.
The cache key decides the hit ratio#
The edge stores each response under a cache key, and two requests share a copy only if their keys match. By default the key is usually the host, the path and the whole query string. That sounds harmless until you see what ends up in a query string.
Three rules come out of that widget.
- Strip what does not change the response. Tracking parameters like
utm_sourcechange nothing about an image, so most CDNs let you drop them from the key or sort and filter the query string. - Never key shared content on cookies. A cookie is per visitor, so a key that includes it
is a cache of one. If a response depends on who is asking, it should not be in a shared cache
at all, which is what
Cache-Control: privatesays. - Vary on a header only when the response truly varies, and normalise it first. Five
languages is five copies. The raw
Accept-Languageheader has thousands of spellings.
A page with someone's name on it, cached at the edge with a shared key, is served to the
next person who asks. This has leaked account pages on real sites. Anything personal is
Cache-Control: private or no-store, and the default for a new route should be "not
cached" until someone decides otherwise.
Keeping copies fresh#
A copy at the edge is out of date the moment you change the original. There are three ways to deal with that, and a real site uses all three for different files.
More than a cache#
A CDN sits between every user and your servers, so it ends up doing more than caching:
- TLS ends at the edge. The slow handshake happens a few milliseconds from the user, and the edge keeps warm connections open to the origin, so even a page that cannot be cached arrives faster through a CDN. (What happens when you type a URL walks through those round trips.)
- Attacks land on the edge first. A flood of traffic is spread over hundreds of cities instead of arriving at your one data centre. The security chapter comes back to this.
- Private files can still be cached. A paid video is served through a signed URL: the link carries an expiry time and a signature made with a secret only you and the CDN know. The edge checks the signature and serves its cached copy, and a link copied to a forum stops working when it expires.
- Small pieces of code can run at the edge, for redirects, A/B tests and checking a token before a request ever reaches you.
Break it#
Read every step as text
- normal · Healthy. A 99% hit ratio at the edge. The origin sees about one request in a hundred, mostly the long tail of files nobody has asked for lately.
- new release · Degraded. A popular release goes live. Every edge misses on it at once, and the shield turns those hundreds of misses into one fetch per file. The origin barely notices, which is the shield doing its one job.
- purge all · Down. Someone purges everything to fix one wrong image. Every edge and the shield are empty, the hit ratio drops towards zero, and the origin takes something like a hundred times its usual traffic until the caches refill. It may not survive long enough for them to.
- repaired · Healthy. Versioned file names for everything that can have one, so most changes need no purge at all. Purges by exact URL or by tag, never "everything". And an origin sized for a cold start of its hottest files, not only for the 1% it sees on a good day.
On 8 June 2021, one Fastly customer pushed a valid configuration change that triggered a bug Fastly had shipped a few weeks earlier, and 85% of Fastly's network started returning errors. Large news sites and shops went down together. Fastly saw it within a minute and had 95% of the network back within 49 minutes. The lesson for a design round: a CDN is a dependency like any other, and for a site that must stay up, the answer is a second CDN you can move traffic to, or an origin that can serve users directly for a while.
Trade-offs#
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Long expiry with versioned file names | Almost every request is a hit, and nothing is ever stale. | A build step that renames files, and a short expiry on the page that points at them. | Every script, style, image and video segment. |
| A regional shield | Cold misses from hundreds of edges become one origin fetch per region. | One more hop on a miss, and one more thing billed. | Any origin that would struggle with a burst of cold misses, which is most of them. |
| A short expiry instead of purging | No purge system to build or to misuse. | Changes take up to the expiry to appear, and the origin is asked again even when nothing changed. | HTML and API responses where a minute of staleness is fine. |
| A second CDN | One provider's bad day is not your outage. | Two configurations to keep in step, and a lower discount from each. | When an hour of downtime costs more than running two. |
Interview replay#
Checkpoint#
1. The hit ratio drops from 99% to 90%. Roughly what happens to the traffic reaching the origin?
2. Your script files are named app.js and cached for a year. You ship a fix. What is the clean way to make users get it?
3. Same CDN, but the site now shows each user their own dashboard. What changes first?
- India to the US is at least 140 ms a round trip, and a first request needs about three.
- 99% to 90% hit ratio is ten times the origin load. The origin sees misses, not requests.
- Versioned names, cached for a year, for every file that can have one. Short expiry only for the page that points at them.
- A shield turns hundreds of cold misses into one. It matters on the worst minute, not the average one.
A CDN keeps copies of your files in servers near users, so a request that would cross an ocean is answered from the same city: India to the US is at least 140 milliseconds a round trip, and a first request needs about three. Edges answer hits, a regional shield collapses cold misses from hundreds of edges into one fetch, and the origin sees what is left. The hit ratio is the number that matters, because the origin sees misses, and going from 99 to 90 percent is ten times the load. The cache key decides the hit ratio, so I strip tracking parameters, never key shared content on cookies, and keep anything personal out of the shared cache entirely. For freshness, files get content hashes in their names and a long expiry, the HTML that points at them gets a short one, and purges are by exact URL, never everything. Even uncached pages go through the CDN, because TLS ends near the user. And the CDN is itself a dependency, so a site that must stay up keeps a way to move traffic to a second one.