29 min read

Design a web crawler

Senior Search Queues and streams ScalingUpdated 28 October 2026

Read firstMessage queues Back of the envelope math

Publish a page today and Google often knows about it within days, before you tell anyone. Something visited it. That something is a web crawler: a program that downloads a page, pulls out every link on it, and then visits those links. Then the links on those pages, and so on, for as long as it runs.

The idea fits in one sentence. The hard part is doing it a billion times a month without knocking on any one website's door too often, without downloading the same thing twice, and without getting lost in a site that never ends. Watch one URL go round.

Pick. The frontier hands a fetcher one URL, say a page on indgeek.com, because that site is allowed a visit right now.

Two words from that loop come up all through this page:

The frontier is the to-do list: every URL we know about and have not fetched yet.
Politeness means never hitting one website too often. You are a guest on servers you do not own, and a crawler that forgets this gets blocked.

Step 1: Understand the problem#

"Crawl the web" is not a requirement yet. These six questions turn it into one. Notice that two of them are about manners, not scale. That is why interviewers like this question.

You ask	They say	What it settles
How many pages, and in how long?	A billion pages a month, refreshed all the time.	About 400 downloads a second. That sounds small, until you see in the napkin math below that it has to be spread across hundreds of thousands of websites at once.
Do we obey robots.txt and go slowly on each site?	Always. Getting blocked is worse than being slow.	robots.txt is a small file every site can publish saying which pages crawlers may visit and how often. Obeying it, plus a speed limit per site, decides the shape of the frontier. More on that in the first deep dive.
Plain HTML, or pages built by JavaScript too?	HTML for now.	A fetcher stays cheap. Running JavaScript means a full headless browser per page, roughly a hundred times the CPU.
How fresh do pages we already have need to be?	News in minutes, most pages in weeks.	We need to remember when each page was last fetched and whether it changed, so we can check again at the right time.
What happens to the pages we download?	Store them and hand them to the team that builds search.	The crawler stops at storage. Say this out loud, or you will end up designing a search engine by accident.
How do we know we have seen a page before?	By its URL, and by its content, because one page often lives at many URLs.	Two separate checks: one for URLs, one for page text. The second deep dive builds both.

What you are building#

Functional requirements

What the system does.

Start from a list of seed URLs and follow links. Seeds are the first URLs you type in by hand. After that the crawl feeds itself.
Never overload a single site. Politeness is a hard rule, not a nice to have.
Do not download the same thing twice. Neither the same URL, nor the same text under a different URL.
Fetch pages again as they change. A crawl that runs once is a snapshot, not a crawler.

Non-functional requirements

How well it has to do it. The numbers you commit to.

One billion pages a month.
Obey robots.txt and a per site speed limit, always.
If a machine dies, its work is picked up, not lost and not repeated.
Storage grows for years, so nothing can assume it fits in memory.

Out of scope

What you cut, and say so out loud.

Ranking and the search index. That is its own question: see Design Google Search.
Running JavaScript. Worth one sentence: it is about a hundred times dearer per page, so it runs as a separate, small tier.
Cleaning page text, removing menus, detecting language.

Back of the envelope#

Three inputs, and the last row is the one that matters. If this kind of estimate is new, the back of the envelope chapter walks through the method.

Fetch budget

Pages per month1000 million

Average page size120 KB

Downloads per site per second1

Pages per second1000M / 2.6M seconds = 386

Bandwidth386 x 120KB = 0.4 Gbps

Raw storage per month123 TB

Different sites being crawled at once386 / 1 = 386

386 sites at once

the number that shapes the whole design

Each site gets at most 1 download a second, so reaching 386 pages a second means about 386 different sites must be in progress at the same moment. The limit is per site, so the to-do list has to be organised per site too.

Everyone first pictures one long queue of URLs. The per site limit makes that picture wrong before you write any code.

Step 2: Propose the high level design#

The core entities

URLCleaned up and hashed to 64 bits. One row per URL, with when it was last fetched.
SiteEverything under one host name. One worker owns a site, with a timetable for when it may be visited next.
Page fingerprintA hash of the page text, so the same content under a new URL, or unchanged next month, is not stored again.
Seen filterA Bloom filter over every URL seen. A false positive skips a page, and it never misses a URL it has seen.

Never breaks: No site is fetched by two workers at once, or faster than it allows.

The data model#

One row per URL we know about. It lives in a key value store split by site, for the reason the deep dives explain. (Design a key value store shows what sits under a table like this.)

url_statekey value store, partitioned by host

url_hash	bigint	PK	A 64 bit hash of the cleaned up URL. Smaller than the URL itself, which adds up over a billion rows.
host	varchar(255)	IDX	The site, like indgeek.com. Everything for one site is stored together, because one worker owns that site.
last_crawled	timestamp		When we last fetched it. Decides when to fetch it again.
content_hash	bigint		A fingerprint of the page text last time. Same fingerprint means the page did not change.
etag	varchar(128)		A version tag the site gave us. We send it back next time and the site can answer "not changed" without sending the page.
fail_count	int		Failures in a row. Past a limit, stop trying.

Sample row

-4471903... | indgeek.com | 2026-08-20 04:11 | 88a13f... | "W/9f3c" | 0

The etag and the content hash are how a crawler saves effort. A page that never changes gets checked rarely and cheaply, so most of the download budget goes to pages that do change.

The whole system on one whiteboard#

Figure 1 is the same loop as the picture at the top, with the supporting boxes added.

Figure 1. It is a loop, not a pipeline. Two boxes do the hard work: the frontier decides what to fetch next and when, and the seen filter stops the loop from running forever.

Walking Figure 1:

Seeds start the frontier. Seeds are a hand picked list of URLs, plus sitemaps (files where sites list their own pages). After the first hour they hardly matter, because the crawl finds its own URLs.
A fetcher asks the frontier for work and gets a URL whose site is due a visit. It never picks its own work. That one rule is what lets 500 machines stay polite together.
Before downloading, the fetcher needs the site's robots.txt and its IP address. Both come from caches. Looking up an address is DNS, the internet's phone book, explained in what happens when you type a URL. At 400 lookups a second, an uncached resolver throttles you before any website notices you.
The page's text gets a fingerprint. If we already have that text, stop. If not, the raw page is written once to object storage (a huge, cheap file store, see design object storage), and a message goes on a queue for the search team. Why a queue: they read at their own pace, and a slow indexer never slows the crawl. Queue vs pub/sub vs event stream covers the choice.
The parser pulls every link out of the HTML.
Each link is cleaned up and checked against the seen filter. Links we know are dropped. New ones get a row in the URL state table.
The new URLs go back into the frontier, and the loop closes.

Your answer

Step 2 says a fetcher never picks its own work. What goes wrong if all 500 machines just take the next URL off one shared queue?

Step 3: Design deep dive#

The frontier#

The frontier has two jobs that pull against each other. Priority: important pages should be fetched sooner. Politeness: never hit one site too fast. Think of a courier with 400 parcels for one apartment block. Ringing the same bell 400 times in a minute gets the door shut. The fix is to mix in other blocks and come back later. (The general idea of slowing yourself down for someone else's sake is in rate limiting vs throttling vs backpressure.)

How does a fetcher decide which URL to download next?

First try
One big first in, first out queue
Every new URL joins the back. Every fetcher takes from the front. It is a breadth first walk of the web, and it takes five minutes to build.
Where it breaks: One page on a big site links to 300 more pages on the same site. They sit next to each other in the queue, so 300 fetchers take them together and hit that site 300 times in one second. To that site you are an attack.
Better
One queue per site, one worker per queue
Route every URL to the queue for its site. A site queue belongs to exactly one worker, and that worker waits between requests to the same site. Two machines can no longer hit one site together, because only one machine ever holds its queue.
Where it breaks: Polite, but blind. A worker takes URLs from its queues in arrival order, so a news homepage that changes every minute waits behind 10,000 old pages from a forum.
What you ship
Priority first, then site queues, then a visit timetable
Score each new URL and put it in a priority band. From there it moves into its site queue. A timetable (a min heap: a list that always hands you the smallest time first) holds the earliest moment each site may be visited again. A worker takes the site whose time has come, fetches one URL, and puts the site back with a new time.

Here is the third rung one stage at a time.

1/4 A new URL gets a score and lands in a priority band. The score can be as simple as "how many clicks from a seed" or as clever as an estimate of the page's importance. This stage decides what is crawled sooner.

The frontier is too big for memory. A billion URLs at a couple of hundred bytes each is hundreds of gigabytes, so site queues live on disk and only the front of each queue is kept in memory. That is a normal, boring thing to say out loud in the interview.

Politeness is not one number

robots.txt can set a crawl delay, so fetch and cache it for a site before anything else on that site. Sites also differ hugely: a big one is fine with ten requests a second, a small one on shared hosting struggles with one. Good crawlers watch response times and errors and slow down on their own. That is the difference between a crawler people put up with and one they block.

Not fetching the same thing twice#

"Twice" hides three different problems, and each needs its own tool:

The same URL reached from two different pages.
The same text at different URLs: print versions, tracking tags, the same article on five country domains.
The same page next month, when it probably has not changed.

Start with the first. It runs for every single link, billions of times, so it has to be cheap.

Have we seen this URL before?

First try
A set in memory
Keep every URL in a hash set. Checking is instant.
Where it breaks: A billion URLs at around 100 bytes each is about 100 GB of memory, on one machine, and it is gone if that machine restarts.
Better
Ask the URL state table
The table already has a row per URL. Look the URL up there.
Where it breaks: Every page has dozens of links. That is tens of thousands of database reads a second just to say "yes, seen it", and nearly every answer is yes.
What you ship
A Bloom filter in front of the table
A Bloom filter is a long row of bits, all 0 at the start. To add a URL, hash it a few different ways and switch on the bit each hash points to. To check a URL, look at those same bits. If any is 0, the URL is definitely new. If all are 1, it is probably seen. A billion URLs fit in about 1.2 GB with a 1% error rate.

Try it. Check the links in order and watch the last one.

A Bloom filter with 16 bits

010

011

012

013

014

015

Every link is hashed twice. Each hash picks one of the 16 bits. If both bits are already on, the filter says "seen before". If either is off, the link is definitely new.

0 of 16 bits are on. The fuller the filter, the more often a new link lands only on bits that are already on. A real crawler uses billions of bits, sized so that this happens about once in a hundred.

The last link was never added, yet the filter said "seen". Its two bits had been switched on by other links. That is a false positive, and it is the only mistake a Bloom filter makes. The opposite mistake cannot happen: a URL that went in always finds its bits on. For a crawler this is the right way round. A false positive skips one page we have not crawled, which costs little. Crawling a page twice, over and over, would cost a lot. (A hash is a fingerprint function: same input, same output, every time. The hashing article shows one live.)

Before a URL is hashed, clean it up, so small spelling differences do not count as new URLs. This is called normalising: lowercase the host, drop the #section part, sort the query parameters, and strip known tracking parameters like utm_source. It removes a large share of duplicates for free.

The same text at different URLs is the content check in Figure 1. Hash the page text and compare. An exact hash only catches perfect copies, though, and two copies of an article often differ by a date or an ad. SimHash is a fingerprint built so that similar text gets a similar fingerprint, so near copies match too.

The same page next month is handled by asking before downloading. The crawler sends the etag it saved last time: "has this changed since version W/9f3c?" If not, the site answers 304 Not Modified with no body. That costs almost nothing. Pages that keep answering 304 get checked less and less often, and pages that change every visit get checked more. No single interval works for every page, because some change every minute and some never do.

Crawler traps

A calendar with a "next month" link, for ever. A shop where every filter combination is a new URL. Nobody built these to trap you, they are just endless. Cap the depth (clicks from a seed), cap the URLs per site per pass, and watch for sites producing lots of near identical pages.

Sharding the work#

One machine cannot hold the whole frontier, so it is split across many. This is sharding: each machine owns one slice of the data (the sharding chapter covers it from the start). The question is how to slice. Split by site, not by URL. It follows from politeness: all URLs for one site must sit with one worker, or two workers will hit that site at the same time and no rate limit will save you. Try the three options.

Consumer group

part 0

lag 4consumer 0

part 1

lag 3consumer 1

part 2

lag 5consumer 2

part 3

lag 4consumer 0

part 4

lag 3consumer 1

part 5

lag 4consumer 2

Each site hashes to one slice, and each slice has one owner. Only one worker ever fetches a given site, so politeness is guaranteed by the layout. Work is roughly even while sites are roughly similar in size.

The last option has the best numbers, and it is the wrong design. That is worth saying in the interview: the metric you would naturally watch picks the wrong answer. When a machine joins or leaves, consistent hashing moves only a small share of sites to a new owner, instead of reshuffling all of them.

Break it#

Push each failure and watch which box goes red first.

Crawl health

polite

politeno per site limittrapfilter fullbounded

Healthy. One download per site per second, robots.txt cached and obeyed, 400 pages a second across hundreds of thousands of sites. Nobody notices you, which is the goal.

Read every step as text

polite · Healthy. One download per site per second, robots.txt cached and obeyed, 400 pages a second across hundreds of thousands of sites. Nobody notices you, which is the goal.
no per site limit · Down. One shared queue, and 500 fetchers land on one site together. You get rate limited, then blocked by IP, then blocked by name, and finally added to block lists other sites share.
trap · Down. A calendar with an endless next month. The frontier fills with millions of URLs from one site that differ by one parameter and hold nothing. The crawl now spends its budget feeding itself.
filter full · Degraded. The Bloom filter was sized for a billion URLs and now holds three. Most bits are 1, so most new URLs look "seen" and are silently skipped. Nothing errors. Size it with room to grow and watch how full it is.
bounded · Healthy. A timetable per site, a depth cap, a cap per site, and a filter with room to grow. Slower than it could be in theory, and it can run for years without being blocked or eating itself.

Trade-offs#

Choice	What you gain	What you pay	Pick it when
Frontier split by site	Politeness comes from the layout: only one worker can ever fetch a given site.	A very large site becomes one slow slice that more workers cannot speed up.	Always. Any other split turns politeness into a race you eventually lose.
Bloom filter for seen URLs	A billion URLs in about 1.2 GB, with a check that never touches a database.	A small share of new pages are skipped by mistake, and more as it fills.	Any set too big for memory where missing a few items is fine.
Asking "changed since?" with the etag	Checking an unchanged page costs a 304 with no body, not a full download.	Storing an etag per URL, and trusting sites to send correct ones.	Any recrawl. On a mature crawl this is where most of the saving comes from.
Skipping JavaScript	About a hundred times cheaper per page, so far more of the web for the same money.	Sites built entirely in JavaScript look empty.	The default, with a separate rendering tier for a chosen few sites.

The level bar#

One moment from this round, answered three ways. Pick the one you think is staff level before the labels appear. (What changes at senior and staff explains the gap.)

Pick the staff level answer

InterviewerYou are asked to double the crawl rate. What changes?

Show which is which without guessing

Answer A: Mid level. The fetchers were never the limit. Each site has a polite rate, so more machines just wait longer on the same sites.
Answer B: Senior. Finds the real limit and the capacity checks. Treats every page as equally worth fetching.
Answer C: Staff. Asks what doubling is for, pulls the cheapest lever first, and orders the steps by what actually limits throughput.

Then run the question cold, with a 45 minute timer and no notes, and score the run on the five things this question is really testing.

Score your last run

0 / 10

Politeness
Seen before
Same content
Traps
Splitting the work

Pick 0, 1 or 2 on every row. Be the strict interviewer, not the kind friend.

Interview replay#

Interviewer

You have 500 fetcher machines and a queue of URLs. What is wrong with each machine taking the next URL?

The opening, and it tests exactly one thing.

You

Politeness. A big site shows up many times in that queue, so several fetchers hit it at the same moment, and to that site we look like an attack. The limit that matters is per site, so the frontier has to be per site: a queue per site, one owner per queue, and a timetable of when each site may next be visited.

From the symptom to the data structure in one move. "The limit that matters is per site" is the whole answer.

Interviewer

How do you know whether you have seen a URL before, at a billion URLs?

A scale question with a specific tool in mind.

You

Normalise the URL, hash it, and check a Bloom filter. It never says "new" for a URL it has seen, so I never re-crawl in a loop. Sometimes it says "seen" for a new URL, so I skip a page now and then, which is cheap for a crawler. Normalising first, sorting query parameters and dropping tracking ones, removes a lot of duplicates before hashing.

Names which mistake is acceptable and why. That shows the tool is understood, not recalled.

Interviewer

What stops the crawl running forever on one site?

Traps. They want to hear several limits, not one clever detector.

You

Several caps. Maximum depth from a seed, maximum URLs per site per pass, and an alarm for sites producing lots of near identical pages, which is what an endless calendar looks like from outside. The site does not have to be malicious, only endless, and plenty are.

Several independent limits. The right instinct for input you do not control.

Interviewer

How would you decide when to crawl a page again?

Open ended, and it separates a one time crawl from a real crawler.

You

From how often it actually changes. I keep an etag and a content hash per URL and send a conditional request, so an unchanged page costs a 304. Pages that come back unchanged get checked less often, pages that change every time get checked more. I do not know the exact back-off curve big crawlers use, so I would start simple, double the gap after each 304, and tune it from data.

Adaptive, not a fixed interval, and an honest "I do not know, here is how I would find out", which interviewers score well.

Checkpoint#

Checkpoint

1. Why split the frontier by site and not by URL, when splitting by URL balances better?

2. Your Bloom filter says a URL is "seen", but it was never crawled. What happens?

3. A site answers 304 Not Modified to most of your recrawls. Is that a problem?

Worth memorising

A billion pages a month is about 400 downloads a second, spread politely across hundreds of thousands of sites at once.
Politeness is per site. That one fact is why the frontier is many queues, not one.
A Bloom filter holds a billion URLs in about 1.2 GB at a 1% false positive rate, and it never misses a URL it has seen.
A 304 Not Modified is the cheapest recrawl there is. Spend downloads on pages that change.

Say this in 60 seconds

A crawler is a loop: pick a URL, download it, store it, pull out the links, keep the new ones. The rule that shapes everything is politeness, so the frontier is not one queue. It is a queue per site with one owner each, plus a timetable of when each site may be visited next, which makes it impossible for two workers to hit one site at once. Avoiding duplicate work is three problems. The same URL is caught by normalising it and checking a Bloom filter, where a rare false positive just skips a page. The same text at different URLs is caught by a content fingerprint like SimHash. The same page next month is checked with its etag, so an unchanged page costs a 304. I would cap depth and URLs per site to survive endless calendars and filter pages, and split the work across machines by site, never by URL.