Design a web crawler: seed URLs in, text out
"A crawler is a data pipeline, not a user-facing service: seed URLs in, text out. I split it into stages joined by queues so any stage can fail and retry alone, I stay polite to every site (obey robots.txt, one request per second per domain), and I let the bandwidth math size the fleet for 10B pages in 5 days."
Understanding the problem
5 min- Ask what the crawl is for. A search engine indexes and ranks pages; an LLM lab wants raw text. This design extracts text from the web to train an LLM.
- Say out loud that no crawler reaches every page. The goal is the vast majority.
- Ask whether seed URLs are given (almost always yes). If not, start from popular search engines, news sites, social media and web directories.
Functional Requirements
- Crawl the web starting from given seed URLs.
- Extract the text from each page and store it for later processing.
Below the line (out of scope):
- Processing the text (the training itself).
- Non-text content: images and video.
- Pages rendered by JavaScript, and pages behind a login.
- Security against malicious actors.
- Cost.
- Legal and privacy compliance.
Non-Functional Requirements
- Average 2 MB transferred per page (HTML plus inline resources), a worst case for bandwidth planning. HTML alone is ~30 KB.
- 10B × 2 MB = 20 PB transferred: a bandwidth number, not a storage one.
- Stored raw HTML: 10B × ~30 KB = 300 TB. Extracted text at 5 to 10 KB a page: 50 to 100 TB.
- Politeness caps one domain at 86,400 × 5 = 432,000 pages in the 5 days.
- A domain with 1M URLs needs 11.6 days at 1 request/s, which is why the goal is the vast majority of pages.
- Fault tolerant: after a failure the crawl resumes without losing progress.
- Polite: obey robots.txt and never overload a site.
- 1 request/s per domain by default.
- Efficient: the crawl finishes in under 5 days.
- Scalable: 10B pages.
- Ask about scale first. It shapes the database and the architecture.
Entities + system interface
5 minDefining the Core Entities
- URL record (Metadata DB): url, links to the raw HTML and the extracted text in S3, content hash, depth, status, attempts, next_attempt_at.
- ~300 B a row × 10B = ~3 TB of URL rows.
- Status:
discovered,pending,fetched,duplicate,blocked,skippedordead.
- Domain record (Metadata DB): parsed robots.txt rules, crawl delay,
lastCrawlTime. - Queue message: a URL reference.
- The URL and Domain records live in a Metadata DB that the fault-tolerance dive introduces; the high-level design in step 3 does not need it yet.
API or System Interface
Input: seed URLs Output: extracted text, stored for later processing // no user-facing API: a system interface and a data flow 1. Take a URL from the frontier; resolve its IP through DNS 2. Fetch the HTML 3. Extract the text 4. Store the text 5. Extract the links; add them to the frontier 6. Repeat until the frontier is empty
- This question has no API: the interface is seed URLs in and extracted text out.
- The data flow is six steps: take a URL and resolve DNS, fetch, extract text, store it, add the links to the frontier, repeat.
- Write the data flow on the board before any boxes, because the high-level design follows it step by step.
High-level design
10 min, end to end, no dives yet1) Crawl from seed URLs
- Seed URLs go into the Frontier Queue, the list of URLs to crawl.
- Kafka, Redis or SQS; the fault-tolerance dive picks one.
- The Crawler takes a URL from the Frontier Queue and resolves its IP through DNS.
- DNS caching comes up in the scale dive.
- The Crawler fetches the HTML from the Webpage, an external server outside the system boundary.
- The Crawler extracts the page's links and pushes them back onto the Frontier Queue.
- The loop repeats until the Frontier Queue is empty.
2) Extract and store text
- The Crawler extracts the text from the fetched HTML.
- The text goes to S3 Text Data, blob storage that is cheap, durable and scales.
- GCS or Azure Blob are fine too.
- One Crawler does DNS, fetch, text and links, so one failure loses all of it.
- The fault-tolerance dive handles it.
Potential deep dives
~20 min, interviewer steers1) Fault tolerance: how does one failure not lose progress?
- The one Crawler does DNS, fetch, text extraction and link extraction, so a failure anywhere loses all of it.
- Fetching fails most: sites are down, slow, or serve huge pages.
- Split the Crawler into a fetch stage and a parse stage, joined by Kafka topics, so each stage retries alone and scales alone.
- Still wrong: Kafka has no per-message delay, so retry backoff is a service you build yourself.
- Split the Crawler into pipelined stages joined by queues (SQS): the URL Fetcher and the Parser Worker.
- URL Fetcher: fetch the page, store the raw HTML in S3, send the url id to the Parsing Queue.
- Parser Worker: read the HTML, write the text to S3, push new URLs to the Frontier Queue.
- Text and URL extraction share one stage. Both are simple.
- Each stage retries alone and scales alone.
- A Metadata DB (DynamoDB, or PostgreSQL/MySQL) holds a URL table with the links to the blobs in S3.
- Queue messages carry the url id (the URL record's id), never the HTML.
- Queues are bad at large payloads.
- Changing text extraction (say, adding image alt text) re-runs only the Parser Workers, not the fetches.
Retrying failed fetches: how do you back off, and when do you stop?
Sites go down, time out, or return errors. The retry has to survive the URL Fetcher that tried.
- Retry the fetch after a short in-memory timer.
- The retry is lost if the URL Fetcher dies.
- A few seconds is rarely enough for a site that is down.
- Write failed URLs to a retry topic with the next attempt time in the message, and run a retry service that re-enqueues each URL when it is due.
- It works, but you build the backoff yourself: Kafka has no per-message delay.
- Offsets commit per partition, so out-of-order fetches either stall a partition behind one slow URL or, committed ahead, lose URLs on a crash.
- On a failed fetch, add 1 to
attemptson the URL record and hide the message for base × 2attempts plus jitter.- ChangeMessageVisibility does the hiding: SQS holds the delay, so there is no retry service.
- Visibility timeout: default 30 s, max 12 hours, and the 12 hours run from the receive.
- After 5 real fetch failures, set the row
deadand delete the message.- Count failures in
attempts, not in ApproximateReceiveCount: a politeness deferral is also a receive.
- Count failures in
- Set the Frontier Queue's retention to 14 days, because the 4-day default is shorter than the 5-day crawl.
- A dead-letter queue with maxReceiveCount 50 catches only messages that fail to process at all.
A URL Fetcher dies mid-fetch: why is the URL not lost?
- SQS only hides a received message, so a URL Fetcher that crashes before the delete leaves it to reappear for another URL Fetcher.
- Parser Workers work the same way.
- Write order: HTML to S3, url id to the Parsing Queue, URL record set
fetched(the commit), then delete the message.- Marking it first lets a crash before the send leave a fetched page nothing parses.
- Set the visibility timeout above the fetch timeout plus the writes: 60 s over a 30 s fetch timeout.
- Otherwise a slow fetch reappears and is fetched twice.
- Key every write by url id and make it idempotent, so a repeated fetch is harmless.
- A crash between the send and the delete repeats the fetch; SQS standard is at-least-once anyway.
- SQS wins over Kafka here because of per-message visibility, the built-in dead-letter queue and managed scaling.
2) Politeness: how do you obey robots.txt and never overload a site?
- The Crawler fetches whatever URL comes off the Frontier Queue, with no robots.txt check.
- It fetches Disallow paths, and nothing spaces its requests to one site.
- Fetch a domain's robots.txt the first time the domain comes up, parse it, and store the rules in the Domain table.
- Still wrong: the rules go stale over a 5-day crawl, and a URL may wait days before its fetch.
- It says nothing about a robots.txt that errors, or about thousands of domains on one shared-hosting IP.
- Store each domain's parsed robots.txt rules in the Domain table and cache them for up to 24 hours (RFC 9309).
- A 5-day crawl refetches each robots.txt about 5 times.
- Re-check the cached rules at fetch time; a URL may have waited days.
- robots.txt lists a User-agent, the Disallow paths and a Crawl-delay in seconds.
- Crawl-delay is not in the official standard and Googlebot ignores it. We honor it, and default to 1 s.
- A 4xx robots.txt allows everything; a 5xx, 429 or network failure means push the domain back and wait.
- Its URLs keep waiting. Only a parsed Disallow marks a URL
blocked. - Follow up to 5 redirects for robots.txt itself, one hop per claim.
- Its URLs keep waiting. Only a parsed Disallow marks a URL
- Cap concurrent requests per resolved IP, because thousands of domains can share one shared-hosting IP.
- The DNS cache already has the IP.
- Follow a page's redirect only within the same host, as the domain's next request; a cross-host target is enqueued as a new URL.
- Inline, it would hit another domain without its crawl delay or robots.txt check.
Crawl delay: how do two URL Fetchers not hit one domain inside its delay?
- A URL Fetcher waits out the crawl delay only for its own requests.
- Any URL Fetcher can pick up any domain's URL, so several hit one site at once.
- Take
SET domain:{d} <fetcherId> NX PX <crawlDelayMs>before the fetch. The winner fetches.- The key is a spacing token, never deleted: deleting it early allows a second request inside the delay.
- Redis TTLs run on the server's clock, so crawler clock skew does not matter. Redis unreachable: defer, never fetch unguarded.
- The loser defers the message with ChangeMessageVisibility by the key's remaining TTL (
PTTL) plus jitter.- Without jitter, every URL Fetcher waiting on one domain retries at the same instant and all but one fail again.
- The cost: a domain with n queued URLs is received about n(n+1)/2 times to fetch n pages.
- Each round one URL fetches and the rest re-hide. 10,000 URLs: ~50M receives.
- Deferred messages stay in flight; big domains hit the ~120,000 in-flight cap, and past it ReceiveMessage returns OverLimit for the whole fleet.
- Schedule domains, not URLs: the Domain Scheduler keeps each domain in a Redis sorted set scored by the earliest time it may be hit.
- URLs wait as
pendingrows in the Metadata DB; domains wait in the sorted setready, one member per domain with waiting URLs, scored in Redis server time. - It is the front-queue/back-queue design of Mercator (Heydon and Najork's crawler), with Redis as the heap.
- URLs wait as
- A URL Fetcher claims a due domain, not a URL, and sends one request per claim.
- After the fetch the Domain Scheduler rescores the domain to now + crawl delay + jitter.
- Each URL is received from SQS once and no URL Fetcher holds a message, so there is no re-receive churn and no ~120,000 in-flight cap to hit.
- The score replaces a per-domain lock: no domain gets two requests in one crawl delay.
The Domain Scheduler: how do URLs wait in the database and domains wait in Redis?
- A Parser Worker's conditional insert writes the URL row
discovered, then sends the url id to the Frontier Queue.- The Frontier Queue (SQS) stays as the durable intake, read by the Domain Scheduler instead of the URL Fetchers.
- A sweeper re-enqueues
discoveredrows older than an hour, so a crash between the insert and the send loses nothing.
- The Domain Scheduler checks robots rules, INCRs the domain's pending-URL counter, ZADDs the domain NX, flips the row to
pending, then deletes the message.- A row no longer
discoveredis a duplicate delivery: delete the message and stop. - A parsed Disallow marks the row
blocked; the depth limit or a trap cap marks itskipped. - One Lua script:
INCR pending:{domain}andZADD ready NX <now> <domain>. NX never pulls a domain's time earlier, and a domain leavesreadyonly inside a due claim, so re-adding it at now is safe. - Redis first, row second, delete last: a crash re-runs the step and over-counts, which is harmless.
- A row no longer
- A URL Fetcher claims a due domain: its score moves to now + lease and it gets a claim token.
- Lua:
ZRANGEBYSCORE ready -inf <now> LIMIT 0 k; ZREM a member whose counter is 0; move each other to now + lease (ZADD ready XX) and setclaim:{domain}= the fetcher's id, same expiry. - Lease = max(60 s, crawlDelay + fetch timeout). No other URL Fetcher sees the domain until it ends.
- Fence: no send once the claim is older than lease minus max(crawlDelay, fetch timeout), 30 s with a 30 s fetch timeout, timed from the claim call. A request that is sent starts at least one crawl delay before the lease ends and finishes before it.
- Lua:
- One request per claim: the domain's lowest-depth due
pendingrow, or its robots.txt.- A cached Disallow marks the row
blockedwith no request made. - Rules missing or older than 24 h: the request is the robots.txt fetch. A 5xx, 429 or network failure pushes the domain back; its URLs stay
pending. - Read Content-Length from the GET's response and abort past ~10 MB (
skipped); a HEAD would be a second request. - Only not-yet-due rows: rescore to their earliest
next_attempt_at. No row at all: reset the over-counted counter from a DB count.
- A cached Disallow marks the row
- Store before commit: HTML to S3 keyed by url id, url id to the Parsing Queue, then the row set
fetchedonly if stillpending.- A crash before the status write leaves the row
pending; the refetch overwrites the same keys. - A same-host redirect target's conditional insert goes straight to
pendingin this domain (with an INCR).
- A crash before the status write leaves the row
- Fetch failure:
attempts+ 1 andnext_attempt_at= now + base × 2n with jitter; the row stayspending.- After 5 real failures the row is
deadand the counter drops. - A whole domain failing (timeouts, 5xx) pushes the domain's score back exponentially instead.
- After 5 real failures the row is
- After each fetch the Domain Scheduler writes
lastCrawlTimeand rescores the domain, only if the claim token is still this fetcher's.- Lua: check
claim:{domain},DECRBY pending:{domain}by the rows that leftpending, thenZADD ready XX <now + crawlDelay + jitter>, even at 0. The next claim removes a domain at 0.
- Lua: check
- URL Fetcher crash or pause: the domain comes due again when the lease ends, and the URL is still
pending.- A fetcher paused past its fence sends nothing. A late zombie's claim token has expired, so it cannot rescore the domain.
- One gap stays open: a pause between the fence check and the socket write. A remote site cannot check a fencing token.
- A reconcile sweep every few minutes re-adds any domain with
pendingrows but noreadymember (ZADD NX), so a race can delay a URL, never strand it. - Redis loss: rebuild
readyand the counters from the Metadata DB, each score max(now, lastCrawlTime + crawlDelay), never plain now.- After a rebuild or a failover (an async replica may drop the newest claims), hold new claims for one max crawl delay plus 30 s: a fetch in flight at the loss has not written
lastCrawlTimeyet. - A replica plus AOF makes a rebuild rare. Millions of domains × ~100 B = hundreds of MB; 30,000 claims/s fits one Redis (or shard
readyby domain hash).
- After a rebuild or a failover (an async replica may drop the newest claims), hold new claims for one max crawl delay plus 30 s: a fetch in flight at the loss has not written
3) Scale: how do you crawl 10B pages in under 5 days?
- Do scaling last, once you know every stage.
- Run the crawl on one machine: even at full line rate, 12,500 pages/s, 10B pages take 9.3 days.
- About twice the 5-day target.
- Size the fleet at full line rate, 12,500 pages/s per 200 Gbps machine, so 2 machines finish in 4.6 days.
- Still wrong: latency, DNS, politeness and retries leave about 30% of line rate usable, so 2 machines really take about 15 days.
200 Gbps / 8 bits per byte = 25 GB/s per instance 25 GB/s / 2 MB per page = 12,500 pages/s at full line rate 12,500 × 30% usable = 3,750 pages/s // latency, DNS, politeness, retries 10B / 3,750 = 2,666,667 s = 30.9 days on one machine 30.9 days / 8 machines = 3.9 days, under 5 8 × 3,750 = 30,000 pages/s for the fleet
- URL Fetchers are I/O bound: one 200 Gbps instance at 30% usable fetches 3,750 pages/s.
- A network-optimized instance, c6in.32xlarge or c7gn.16xlarge class.
- 10B pages take 30.9 days on one machine, so 8 machines finish in 3.9 days.
- Parser Workers auto-scale on the depth of the Parsing Queue (Lambda, or ECS on Fargate).
- 30,000 pages/s at 1 to 2 s per fetch is 30,000 to 60,000 open fetches, 3,750 to 7,500 per machine (Little's law).
- At 1 request/s per domain, 30,000 pages/s needs 30,000+ domains every second: throughput comes from crawling millions of domains at once.
DNS: how do you keep lookups from becoming the bottleneck?
- DNS is a real bottleneck at thousands of lookups per second over millions of domains.
- Mercator measured 87% of each thread's time in DNS with the stock resolver, and 25% after a custom multi-threaded resolver.
- Cache lookups in the URL Fetchers for the record's TTL, with a floor of a few minutes.
- Cache NXDOMAIN failures too, or every link to a dead domain costs a lookup.
- Spread lookups across several resolvers and DNS providers, round-robin.
- Raise a provider's limits with money.
URL dedup: how do you avoid fetching one URL twice?
- Normalize the URL first: lowercase scheme and host, drop the default port and the fragment, sort the query parameters, strip tracking parameters.
- Also resolve relative links and
..and normalize percent-encoding. Tracking parameters:utm_*,gclid. - Keep the path's case, http vs https and
www: they can differ, and the content hash catches the ones that do not.
- Also resolve relative links and
- The insert decides: a Parser Worker enqueues a URL only if its conditional insert won.
- Postgres:
INSERT ... ON CONFLICT DO NOTHING. DynamoDB: a put withattribute_not_exists(url_hash), keyed by a hash of the normalized URL. - Check-then-enqueue would let two Parser Workers both enqueue a new URL.
- The row starts
discovered; a sweeper re-enqueuesdiscoveredrows older than an hour.
- Postgres:
- A duplicate delivery hits a row that is no longer
discoveredand is dropped without a fetch.- SQS standard can deliver a message twice.
- At ~50 outlinks a page, Parser Workers check 30,000 × 50 = 1.5M URLs/s against the URL table.
- Each is a write-sized conditional insert, so a parser-local cache or Bloom filter in front pays off.
Duplicate content: how do you skip a page you already have?
- Different URLs can serve identical content: http and https, www, mirror sites.
- Parse every page whose URL is new.
- URL dedup cannot see identical content, because the URLs differ.
- Hash the fetched content and check RedisBloom (
BF.ADD/BF.EXISTS) before parsing.- Fast, no false negatives, but a false positive skips a unique page for good, and entries cannot be deleted.
- 10B entries at 1% false positives is ~9.6 bits each = ~12 GB, one key on one shard.
- Hash the fetched content and parse only if the hash's conditional insert won.
- The key must be unique, or two Parser Workers with the same hash both see "absent" and both parse.
- Postgres: a UNIQUE index, with
ON CONFLICT DO NOTHINGon the insert. - DynamoDB: a separate ContentHash table keyed by hash with an
attribute_not_exists(hash)conditional put.- A GSI can neither enforce uniqueness nor read consistently.
- A content hash catches only byte-identical pages; near-duplicates need SimHash or MinHash, usually offline.
- Pages that differ by an ad or a timestamp: SimHash on shingles (Hamming distance) or MinHash with LSH, run on the text.
Crawler traps: how do you stop a site that never ends?
- A crawler trap is a site built to keep a crawler busy forever.
- Store a depth on the URL record (link hops from a seed: seed 0, its links 1) and stop past 15 to 20.
- A URL keeps the depth of its first discovery; dedup drops later, shorter paths.
- Depth stops deep traps (calendars,
/a/a/a/...) but not wide ones, where session ids or sort orders mint unlimited URLs at one depth. - Add a URL length cap (~2,000 characters), a repeated-path-segment check and a per-domain URL budget.
- Politeness caps a trap's fetches at 432,000 per domain in 5 days; the budget caps its Frontier Queue and URL table growth.
Other dives, one line each:
- JavaScript pages: a headless browser such as Puppeteer.
- Monitoring: Datadog or New Relic.
- Large files: read Content-Length from the GET's headers and abort past ~10 MB.
- Continual recrawls: a URL Scheduler owns recrawl timing and Parser Workers write URLs to the DB.
- Priority crawling: several SQS queues by priority.
- Nothing is deleted until the work is stored: the SQS message reappears after its visibility timeout, and under the Domain Scheduler the row is still
pendingwhen the lease ends. - Write order: HTML to S3, url id to the Parsing Queue, URL record set
fetched, then any message delete. - Visibility timeout (60 s) above the fetch timeout (30 s), and every write keyed by url id, so the repeat is harmless.
attempts+ 1 on the URL record; next try at now + base × 2n with jitter.- After 5 real failures the row is
dead. Receives do not count, since a politeness deferral is also a receive. - Frontier retention 14 days, over the 4-day default. A dead-letter queue at maxReceiveCount 50 is only a backstop.
- Fetchers claim domains, not URLs. A domain sits once in the Redis
readyset, and a claim moves its score to now + lease with a claim token. - After the fetch the Domain Scheduler rescores the domain to now + crawl delay + jitter, only if the claim token is still this fetcher's.
- 200 Gbps / 8 = 25 GB/s; / 2 MB = 12,500 pages/s; 30% usable = 3,750 pages/s.
- 10B / 3,750 = 2.67M s = 30.9 days on one machine.
- 8 machines = 3.9 days, under 5.
- Parser Workers scale separately on Parsing Queue depth.
- Each round one URL fetches and the rest re-hide, so ~n(n+1)/2 = 50M receives.
- Every deferred message counts toward the ~120,000 in-flight cap, and each deferral is a receive.
- The Domain Scheduler replaces it: URLs wait in the Metadata DB, a Redis sorted set holds each domain's next allowed time, and a URL Fetcher claims a due domain, so each URL is received once.
Final design + what is expected at each level
wrapFinal Design
- Seed URLs → Frontier Queue (SQS) → Domain Scheduler → URL Fetchers → S3 and Parsing Queue → Parser Workers → S3 text and new URLs.
- Write order: HTML to S3, url id to the Parsing Queue, URL record set
fetched, then any message delete. - 8 URL Fetcher machines at 3,750 pages/s each, caching DNS for the record's TTL across several providers.
- Write order: HTML to S3, url id to the Parsing Queue, URL record set
- The Domain Scheduler keeps one Redis score per domain, the next time it may be hit; URL Fetchers claim a due domain and send one request per claim.
- robots.txt rules and crawl delay in the Domain table, cached up to 24 h. Requests also capped per IP.
- A failed fetch bumps
attemptsand backs off with jitter; dead after 5 real failures; a dead-letter queue is a backstop only.- Frontier retention 14 days.
- Auto-scaled Parser Workers skip seen content hashes, write text to S3, and enqueue normalized URLs at depth + 1 only if the insert won.
- Every URL ends fetched, skipped, dead after 5 real failures or still pending at day 5, and no domain gets two requests in one crawl delay.
- That is the invariant to close on. Skipped by rule covers
blocked,skippedandduplicate.
- That is the invariant to close on. Skipped by rule covers
What is Expected at Each Level
- Mid (E4):
- Understands the data flow and builds a simple working system, like the high-level design.
- Knows the basics of politeness and robots.txt.
- Has some idea of scaling.
- Not expected to go deep on queues or rate limiting.
- Senior (E5):
- Requirements, entities and API: states the purpose (text for LLM training), the two functional requirements and the targets with their numbers, sizes bandwidth apart from storage (20 PB transferred, 300 TB stored), and writes the six-step data flow with the URL and Domain records and a queue message that holds the url id.
- The high-level design, then detail on politeness and robots.txt.
- Detail on scaling to finish within 5 days.
- One answer per deep dive:
- Fault tolerance: splits the Crawler into a fetch stage and a parse stage joined by queues, so each retries and scales alone, retries failed fetches with growing backoff that survives the URL Fetcher, and recovers a mid-fetch crash by storing before deleting.
- Politeness: keeps each domain's parsed robots.txt rules in the Domain table, and holds the crawl delay across every URL Fetcher with one shared key per domain in Redis (
SET NX PX, never deleted). - Scale: walks the bandwidth math to 8 machines in 3.9 days, auto-scales Parser Workers on queue depth, caches DNS, checks the URL table before enqueueing a link, skips content it has already seen (a Bloom filter is acceptable), and stops past a depth of 15 to 20.
- Staff+:
- Deep on 3 or more areas, with insight the interviewer did not have.
- Covering every deep dive above, some at less than full depth, is "a good spot".
- Requirements, entities and API: names every URL status (
discovered,pending,fetched,duplicate,blocked,skipped,dead), which is what lets the closing invariant be stated. - Fault tolerance: counts real failures in
attemptsrather than SQS receives, so a politeness deferral never kills a URL, and raises the Frontier Queue's retention to 14 days. - Politeness: replaces the per-domain lock with the Domain Scheduler, and refreshes robots.txt every 24 h with errors that fail safe.
- Scale: makes URL and content dedup a conditional insert on a unique key, normalizes URLs first, and caps wide traps with a length cap, a repeated-segment check and a per-domain budget.
Say the opener out loud without notes: a data pipeline from seed URLs to text, stages joined by queues, polite per domain, sized by bandwidth.
Then write the math from memory: 200 Gbps / 8 / 2 MB = 12,500 pages/s, × 30% = 3,750, 10B / 3,750 = 30.9 days, / 8 machines = 3.9 days.
Then say politeness in one line: a URL Fetcher claims a due domain from the Redis ready set, sends one request, and the Domain Scheduler rescores it to now + crawl delay + jitter.
Then say the invariant: every URL ends fetched, skipped, dead after 5 real failures or still pending at day 5, and no domain gets two requests in one crawl delay.