Walkthrough: Web Crawler

A web crawler downloads the web: start from seed URLs, fetch pages, extract links, fetch those, repeat. It is the classic pipeline-style question — unlike a product system, there are no users making requests; there is data flowing through stages. That changes how you run the framework: after requirements, spend two minutes narrating the data flow before drawing any boxes.

Step 1: Requirements

Functional:

1. Crawl the web from seed URLs, storing page text for later processing (say, search indexing or LLM training data).
2. Extract links from each page and crawl them too.
3. Respect site owners' rules (robots.txt).

Non-functional:

• Scale: 10 billion pages in under 5 days.
• Politeness: never hammer a single site — roughly 1 request/second per domain.
• Fault-tolerant: a crashed worker must not lose pages or re-crawl the world.

Step 2: Data flow (the pipeline step)

Say it as a sentence before drawing: take a URL from the frontier (the queue of URLs waiting to be crawled) → fetch the page → parse it: extract text and links → store the text → push new, unseen links back onto the frontier → repeat until empty. Every box in the design exists to serve one verb in that sentence.

Step 3: Estimation — the math that sizes the fleet

10B pages ÷ (5 days × ~100k sec/day) ≈ ~23,000 pages/sec fleet-wide. A network-optimized instance sustains roughly 3,750 pages/sec (crawling is I/O-bound — you're mostly waiting on other people's servers). 23,000 ÷ 3,750 ≈ 6.2, so ~8 machines with headroom. Storage: 10B pages × ~2MB raw HTML is ~20PB, but extracted text at ~100KB is ~1PB — say which you're keeping. Notice the surprise worth voicing: the fetch fleet is tiny; the hard parts are coordination and politeness, not compute.

Step 4: High-level design

• Frontier queue: SQS (a managed queue) holding URLs to crawl.
• Fetcher fleet: workers that take a URL, download the page, and drop raw HTML into S3.
• Parser workers: a separate stage (fed by its own queue of "page stored" events) that extracts text and links. Separating fetch from parse lets each scale independently and means a parser bug never re-downloads the web.
• Metadata DB: tracks every URL's state (seen, fetched, failed) and per-domain rules.

New links flow: parser → dedupe check → frontier. The loop closes and the crawl sustains itself.

Deep dive 1: Politeness without stalling

Two rules: honor robots.txt (the file where sites declare what may be crawled and how fast — cache it per domain, re-fetch daily) and cap at ~1 request/sec per domain. The trap: with one global queue, ten million queued URLs from one huge site would serialize behind a per-domain rate limit while the whole fleet idles. Fix: per-domain scheduling — a domain-aware layer (per-domain sub-queues, or a rate-limiter check with re-enqueue and delay) so each domain trickles at its polite rate while the fleet stays busy across millions of other domains. One slow domain must never block the pipeline.

A cousin optimization interviewers like: DNS caching. Every fetch needs a DNS lookup, and historically DNS could eat most of a crawler's elapsed time — cache aggressively and spread across multiple resolvers.

Deep dive 2: Dedupe at scale

Two layers. URL-level: normalize URLs (lowercase host, strip fragments and tracking params) and check the metadata DB before enqueueing — otherwise every popular page gets crawled thousands of times via different links. Content-level: different URLs often serve identical pages (mirrors, print views); hash the fetched content and skip processing if seen. At 10B scale, a Bloom filter (a compact probabilistic set that answers "definitely new" or "probably seen" in tiny memory) screens candidates before the authoritative DB check.

Also guard against crawler traps — calendar pages and session-ID links that generate infinite URLs. Cap crawl depth at ~15–20 hops from a seed and cap URLs per domain.

Deep dive 3: What happens when a fetcher crashes mid-page?

This exact probe tests whether you understand queue semantics. SQS gives a consumer a message with a visibility timeout: the message is hidden, not deleted, while the worker holds it. The worker deletes it only after the page is safely in S3 and metadata is updated. If the worker crashes, the timeout lapses and the URL simply reappears for another fetcher — at-least-once delivery, so a page might occasionally be fetched twice, and the dedupe layer absorbs that harmlessly. Transient failures (site down) retry with exponential backoff; URLs that keep failing land in a dead-letter queue so they can't clog the frontier forever.

Wrap-up

Recap: the pipeline is frontier → fetch → S3 → parse → dedupe → frontier; ~8 fetch machines meet 10B-in-5-days; politeness is enforced per-domain without stalling the fleet; dedupe runs at both URL and content level with a Bloom filter screen; and crash recovery falls out of visibility timeouts plus idempotent re-fetches. With more time: re-crawl scheduling by change frequency (freshness) and prioritizing important pages first.