Design a Web Crawler: System Design Interview Guide

Web crawler design shows up constantly in system design interviews at companies that touch search, data pipelines, or price-comparison products, because it forces a candidate to reason about scale, politeness, and correctness simultaneously. Unlike “design Twitter,” there’s no obvious social-graph shortcut — you have to build a mental model of the open web and defend every design choice against edge cases like infinite redirect loops, spider traps, and rate-limited hosts. This guide walks through the requirements-gathering, high-level architecture, and the four hardest sub-problems interviewers push on: politeness, the URL frontier, duplicate detection, and distributed crawling.

Clarifying Requirements First

Before drawing a single box, nail down scope. A web crawler interview can mean wildly different systems depending on the goal:

• Purpose: general search index, price monitoring, plagiarism detection, or academic archiving?
• Scale: crawl 1 billion pages, or 1 million pages of a specific domain set?
• Freshness: how often must pages be re-crawled — daily for news, monthly for static content?
• Content types: HTML only, or also PDFs, JS-rendered SPAs, images?

Most interviewers expect you to converge on: a general-purpose crawler targeting billions of pages, refreshed on a priority basis, HTML-first with pluggable content extractors. State this assumption out loud and move on — spending ten minutes here is a common failure mode.

High-Level Architecture

The canonical web crawler pipeline has five components:

1. Seed URLs — a starting set (e.g., top domains, sitemap.xml submissions)
2. URL Frontier — a prioritized, politeness-aware queue of URLs to fetch
3. Fetcher workers — pull a URL, resolve DNS, issue HTTP GET, handle redirects
4. Parser / extractor — parse HTML, extract new links, extract content for storage
5. Duplicate detector + storage — dedupe by URL and by content, persist to a document store

Draw this as a loop: the parser feeds newly discovered URLs back into the frontier, closing the crawl cycle. State clearly that this is a breadth-first, priority-queue-driven system, not a naive recursive crawl — that single sentence signals seniority.

Politeness: The Interview’s Favorite Trap

Politeness is the constraint interviewers use to separate candidates who’ve built real crawlers from those who haven’t. A crawler that hits example.com with 10,000 concurrent requests will get IP-banned within seconds and, worse, can constitute a denial-of-service attack on a small site.

Concrete politeness mechanisms to name:

• Per-host rate limiting: maintain a host → last-crawl-timestamp map; enforce a minimum delay (commonly derived from robots.txt’s Crawl-delay directive, or a default of 1-2 seconds).
• robots.txt compliance: fetch and cache it per host before crawling any page; honor Disallow rules and per-user-agent overrides.
• Queue partitioning by host: route all URLs for a given host to the same worker or worker shard so rate limits are enforceable without cross-worker coordination.
• Exponential backoff on errors: 429/503 responses should push the host’s next-crawl time further out, not retry immediately.

A strong answer explains why host-based partitioning matters: without it, politeness becomes a distributed rate-limiting problem across N independent workers, which is dramatically harder than a single queue per host.

The URL Frontier in Depth

The URL frontier is the single most-discussed data structure in this interview. It has two competing goals: priority (crawl important pages first) and politeness (don’t hammer one host).

A widely-cited design (Mercator-style) splits the frontier into two tiers:

• Front queues: F queues, each assigned a priority level (computed from PageRank estimate, update frequency, or business signal). A prioritizer routes new URLs into one of these based on score.
• Back queues: B queues, each mapped to exactly one host at a time via a routing table. A back queue selector picks a front queue (biased toward higher priority), pulls a URL, and pushes it to the back queue for that URL’s host, creating the host-worker binding needed for politeness.
• A heap keyed by earliest allowed fetch time decides which back queue is served next, guaranteeing minimum-delay enforcement per host without an explicit sleep.

Mention that the frontier must be persisted (not purely in-memory) since a billion-URL crawl runs for days and must survive worker restarts — this typically means backing the queues with a distributed queue or an embedded LSM-based store.

Duplicate Detection

Two distinct duplicate problems exist and interviewers expect you to separate them:

1. URL-level dedup: has this exact URL already been queued or fetched? Solved with a Bloom filter in front of a persistent hash set — the Bloom filter absorbs the vast majority of “have I seen this” checks in memory, falling back to disk lookup only on a probable hit.
2. Content-level dedup: different URLs (mirrors, tracking-parameter variants, syndicated content) serving near-identical content. Solved with content fingerprinting — hash the normalized page body (e.g., SimHash or MinHash for near-duplicate detection, not just exact MD5) and skip re-storing or re-extracting-links from pages whose fingerprint already exists within a similarity threshold.

The distinction matters because URL dedup is a set-membership problem (Bloom filter territory) while content dedup is a similarity-search problem (locality-sensitive hashing territory) — conflating the two is a common candidate mistake.

Distributed Crawling Trade-offs

Walk through the failure modes explicitly: a single crawl trap (a site generating infinite unique URLs via calendar pages or session IDs) can starve the frontier. Mitigate with max-depth per host and URL pattern heuristics (e.g., cap parameters, detect repeating path segments).

Storage and Freshness

Store two things per page: the raw/extracted content (blob store, e.g., object storage) and metadata (URL, fetch timestamp, content hash, HTTP status, discovered links) in a queryable database. Freshness is handled by a re-crawl scheduler that re-inserts URLs into the frontier based on observed change frequency — pages that historically change often get shorter re-crawl intervals, a classic exponential-backoff-in-reverse pattern.

Wrapping Up the Interview

A candidate who names politeness, the two-tier frontier, Bloom-filter URL dedup, fingerprint-based content dedup, and host-based partitioning for distributed workers has covered the core signal this question is designed to test. If you want a structured walkthrough of this and dozens of other recurring interview archetypes with worked examples and trade-off tables, The 0-to-1 SWE Interview Playbook (Amazon: https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20) dedicates a full chapter to crawler and pipeline-style questions, updated for 2026 interview loops.

Common Follow-Up Questions

Interviewers frequently extend this into: “How would you crawl JavaScript-rendered pages?” (headless rendering workers, more expensive, reserved for high-priority URLs), “How do you avoid re-crawling identical content across mirrors?” (canonical URL detection via <link rel="canonical"> plus content fingerprinting), and “How do you scale to 10x traffic?” (add worker shards, keep the frontier’s host-partitioning invariant, and horizontally scale the dedup store with consistent hashing). Prepare a one-sentence answer for each rather than improvising live.