← Anthropic Interview Insights

Anthropic·Software Engineer·Technical Phone Screen·Senior

Senior
May 2026

Summary

Anthropic SWE interview with a web crawler problem, starting single-threaded and then getting pushed into a multi-threaded follow-up. Pretty classic problem but the hostname and fragment-stripping constraints added enough wrinkles to keep me on my toes.

Questions Asked (2)

Q1

Implement a single-threaded web crawler that starts from a given URL, uses a provided HTML parser to fetch links, and returns all unique reachable URLs restricted to the same hostname. URLs must have their fragment stripped before deduplication and before being included in the result.

Algorithms & Data StructuresSystem Design
Author's notes

BFS with a visited set, pretty mechanical once you figure out the hostname extraction.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Use BFS with a queue and a visited set to explore all reachable URLs, ensuring each URL is normalized (fragment stripped) before checking or adding to the set. Only enqueue URLs that share the same hostname as the starting URL, and return the visited set as the result.

Pro tip: Clarify upfront that you'll strip fragments before deduplication and hostname checks to avoid redundant work and ensure correctness. Mention that BFS naturally handles cycles and avoids deep recursion, which is important for robustness.

1. Normalize and initialize

Parse the start URL, strip its fragment, and initialize a queue with it and a visited set containing it. Extract the target hostname for later comparisons.

2. Process queue iteratively

While the queue is not empty, dequeue a URL, fetch its HTML using the provided parser, and extract all links.

3. Filter and normalize links

For each extracted link, strip the fragment, resolve it to an absolute URL, and check if its hostname matches the target hostname. If not, skip it.

4. Deduplicate and enqueue

If the normalized URL is not in the visited set, add it to the set and enqueue it for further crawling.

5. Return results

Once the queue is empty, return the visited set as the list of unique reachable URLs.

Key Points to Mention

  • Use BFS with a queue to ensure all reachable URLs are found and to avoid recursion depth issues.
  • Strip fragments before any deduplication or hostname comparison to treat URLs like 'page#section' and 'page' as identical.
  • Maintain a visited set to avoid infinite loops and redundant processing.
  • Resolve relative URLs to absolute form using the base URL of the current page.
  • Compare hostnames exactly (case-insensitive) to restrict crawling to the same hostname.
  • Handle edge cases: invalid URLs, non-HTML content, and parser errors gracefully.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Now make your web crawler multi-threaded. Multiple worker threads should be able to call the HTML parser concurrently, and the same URL must never be crawled more than once across all threads.

System DesignTechnical Trade-offs
Author's notes

This is where I fumbled a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then propose a thread-safe architecture using a shared concurrent data structure to track visited URLs and a thread pool to manage workers. Explain how you would ensure the HTML parser is thread-safe, either by making it stateless or using synchronization, and discuss trade-offs between locking granularity and throughput.

Pro tip: Mention that you would use a concurrent set with atomic check-and-insert (e.g., ConcurrentHashMap.newKeySet() in Java) to avoid race conditions, and consider using a work-stealing queue for better load balancing. Also, highlight the importance of idempotent crawling and handling redirects to prevent duplicate work.

1. Clarify Requirements and Constraints

Ask about expected scale, latency requirements, and whether the crawler must respect robots.txt or politeness policies. Confirm that the HTML parser is stateless or can be made thread-safe.

2. Design Thread-Safe URL Frontier

Propose a shared concurrent data structure (e.g., a concurrent set or a database with unique constraints) to track visited URLs. Ensure atomic check-and-insert to prevent duplicate crawling.

3. Implement Worker Thread Pool

Use a fixed-size thread pool to manage worker threads. Each worker fetches a URL from a thread-safe queue, checks if it's already visited, and if not, marks it as visited and proceeds to fetch and parse.

4. Ensure Thread-Safe HTML Parsing

If the parser is not thread-safe, either synchronize access (with minimal lock scope) or create a parser instance per thread. Prefer stateless parsers or thread-local instances to avoid contention.

5. Discuss Trade-offs and Optimizations

Compare coarse-grained vs. fine-grained locking, and consider using lock-free data structures. Address potential bottlenecks like DNS resolution and suggest caching or asynchronous I/O.

Key Points to Mention

  • Use of ConcurrentHashMap or similar concurrent set for visited URLs with atomic operations.
  • Thread pool configuration and work queue management (e.g., BlockingQueue, work-stealing).
  • Thread safety of HTML parser: stateless design, thread-local instances, or synchronization.
  • Avoiding duplicate crawling via atomic check-and-insert before fetching.
  • Handling redirects and canonical URLs to prevent duplicates.
  • Trade-offs between locking granularity, throughput, and complexity.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.