Anthropic·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

Anthropic system design round, one big question about building a web crawler from scratch. Took longer than expected to get into the concurrency stuff and I think I lost points there.

Questions Asked (1)

Design a web crawler that starts from a seed URL, fetches pages, discovers and enqueues new links, avoids revisiting URLs, stays within a configurable domain allow-list, and handles concurrency, rate limiting, robots.txt, redirects, and failures.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

I started with the visited set and queue pretty quickly, that part felt natural.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale (e.g., pages per second, domain count, storage). Then design a modular pipeline: URL frontier with deduplication, fetcher with concurrency and rate limiting, parser for link extraction, and storage for visited URLs and content. Discuss trade-offs for each component, such as politeness vs. throughput, and how to handle robots.txt, redirects, and failures.

Pro tip: Emphasize politeness and robustness: implement per-domain rate limiting and respect robots.txt to avoid being blocked, and use exponential backoff with jitter for retries. Also, consider using a distributed queue and consistent hashing for scalability.

1. Clarify Requirements and Scope

Ask about scale (pages, domains, frequency), content types, storage needs, and freshness. Define success metrics like coverage, freshness, and politeness.

2. High-Level Architecture

Outline components: URL frontier (queue), fetcher (HTTP client), parser (link extractor), deduplication (visited set), and storage (content and metadata). Explain data flow from seed to enqueue.

3. Deep Dive into Key Components

Detail URL frontier design (priority, politeness), deduplication (Bloom filter or distributed set), concurrency model (thread pool or async), rate limiting (token bucket per domain), robots.txt handling, redirects, and failure retries.

4. Scalability and Trade-offs

Discuss scaling horizontally with distributed queues and workers, partitioning by domain, and trade-offs between consistency and availability for deduplication. Address bottlenecks and monitoring.

5. Wrap Up and Edge Cases

Summarize design, mention edge cases (e.g., infinite loops, dynamic content, traps), and suggest extensions like prioritization or recrawl scheduling.

Key Points to Mention

Deduplication using Bloom filters or distributed sets to avoid revisiting URLs, with trade-offs on false positives.
Per-domain rate limiting and robots.txt compliance to ensure politeness and avoid bans.
Concurrency model: thread pools vs. async I/O, and how to manage backpressure.
Handling redirects (301/302) and failures with retries and exponential backoff.
Domain allow-list enforcement at the frontier level to restrict crawling scope.
Scalability via distributed architecture: message queues, consistent hashing, and partitioning.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.