Robots.txt and politeness constraints should absolutely be in your opening framing, not a footnote you add when prompted. When I prepped for a similar round, I made the same mistake of treating the URL frontier as the core problem and politeness as a detail. But crawlers that ignore rate limits and crawl directives are genuinely broken systems, not just impolite ones, and interviewers at companies like Meta who run actual large-scale crawl infrastructure know this viscerally. Leading with 'here are the constraints the real world imposes before we even talk about scale' reframes the whole answer and shows you've thought past the textbook BFS expansion.

For the prioritization piece, the honest answer is that PageRank-ish signals are expensive to compute in real time, so in practice you're working with proxy signals: inbound link count from your already-crawled graph, domain authority heuristics, and freshness decay on a per-URL basis. The frontier doesn't have to be a single priority queue either. A tiered structure where you have buckets for high-priority domains, freshness-driven recrawls, and new-discovery URLs lets you tune behavior without recomputing global scores constantly. The tricky part is keeping those tiers from starving each other, which is worth mentioning even if you don't fully solve it in the interview.

Inbound link count and Last-Modified headers are solid anchors, nothing wrong with that answer. The thing that would've elevated it is talking about how the scoring function degrades over time without feedback. If you never revisit a page to see whether your freshness estimate was accurate, you're flying blind. The more interesting design is a lightweight feedback loop where actual observed change rate per domain or URL pattern informs how aggressively you schedule recrawls. Sites that update hourly get treated differently from sites that haven't changed in six months, and you learn that from crawl history rather than assuming it upfront.

Acknowledgment-based delivery from a persistent queue is the right foundation. The exactly-once angle is genuinely hard and most real systems settle for at-least-once with idempotent workers, meaning if a URL gets processed twice the second write just overwrites the first with the same content hash and nothing breaks. Being explicit that you're choosing at-least-once deliberately rather than just not knowing about exactly-once is the difference between sounding like you understand the tradeoff versus sounding like you missed it. The checkpointing piece is mostly about the URL frontier state since the queue handles in-flight work, so clarifying what you're actually checkpointing (frontier snapshots, crawl stats, domain rate limit state) would tighten that part of the answer.

Content hashing for dedup at the body level is the move and you got that right. One thing worth layering in: store the hash alongside the URL mapping so you can answer 'has this content appeared before under a different URL' separately from 'have we visited this URL before.' Those are different questions with different implications for your index downstream.

The TTL refresh for robots.txt is completely defensible, you just have to own it. Something like a 24-hour TTL with a jitter component so you're not stampeding the same domains at the same time is pretty standard. If they push on 'what if it changes mid-crawl and you keep hitting pages that are now disallowed,' the honest answer is you'll have a window of non-compliance and that's an accepted tradeoff in practice. Saying that clearly is better than hedging.

On the distributed rate limiting side, the consistency question is real. A Redis-based token bucket works well when you accept that you might occasionally exceed the limit slightly due to race conditions across workers. For most crawl use cases that's fine, you're not doing financial transactions. The thing worth being crisp on is the key structure: you want per-domain buckets, not per-IP or per-worker, because the target server experiences load at the domain level. One thing I fumbled on a similar question was not distinguishing between rate limiting from the crawler's perspective versus courtesy delays from the target's perspective. They're related but not identical, and showing that distinction tends to land well.

The canonical normalization piece is actually where a lot of real dedup work lives. A bloom filter catches exact string matches, but two URLs pointing to the same page content are a much harder problem. Query param ordering, session tokens, tracking parameters like utm_source, trailing slashes, HTTP vs HTTPS variants, these all produce distinct strings that resolve to identical content. What I'd add to your answer: before any fingerprint hits the bloom filter, run the URL through a normalization pipeline that strips known noise params, lowercases the scheme and host, resolves relative paths, and sorts remaining query params alphabetically. That alone collapses a huge fraction of near-duplicates before you even touch the dedup store. The secondary exact-match store you mentioned for bloom filter false positives is the right call, just be clear it's also your source of truth for 'have we seen this normalized URL before' since the bloom filter can't answer that definitively.