← Anthropic Interview Insights

Anthropic·Software Engineer·Technical Phone Screen·Intermediate

Intermediate
May 2026

Summary

Anthropic coding interview, got a web crawler problem that looked deceptively clean on the surface. The BFS structure wasn't the hard part, keeping the constraints straight in your head while coding live was.

Questions Asked (1)

Q1

Given an API that returns all URLs linked from a given URL, build a web crawler starting from a root URL that visits all reachable pages using BFS. Constrain it to the same domain only and avoid revisiting URLs.

Algorithms & Data StructuresSystem DesignAPI & Integrations
Author's notes

I jumped straight into BFS and got the basic loop working pretty fast, which felt good.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem constraints and API behavior, then outline a BFS traversal using a queue and a visited set to avoid cycles. Emphasize domain filtering and discuss edge cases like malformed URLs, non-HTML content, and rate limiting.

Pro tip: Mention that you would normalize URLs (e.g., resolve relative paths, strip fragments) before adding to the visited set to avoid duplicates. Also, discuss politeness policies like respecting robots.txt and adding delays to avoid overwhelming the server.

1. Clarify requirements and assumptions

Ask about API rate limits, expected scale, handling of non-HTML links, and whether the root URL's domain defines the scope. Confirm that BFS is required and that we only visit pages within the same domain.

2. Design data structures and algorithm

Use a queue for BFS and a set for visited URLs. Initialize with the root URL, then while the queue is not empty, dequeue a URL, fetch its links via the API, filter to same domain, and enqueue unvisited URLs.

3. Handle URL normalization and filtering

Normalize URLs by resolving relative paths, removing fragments, and standardizing case. Filter out URLs not matching the root domain (including subdomains if specified) and skip non-HTTP(S) schemes.

4. Address edge cases and robustness

Discuss handling API errors, timeouts, and retries. Consider rate limiting, concurrent requests (if allowed), and memory constraints for large sites. Mention robots.txt and politeness.

5. Analyze complexity and potential optimizations

State time complexity O(V+E) where V is pages and E is links, and space O(V). Suggest optimizations like using a Bloom filter for visited URLs if memory is tight, or parallelizing with a thread pool while respecting rate limits.

Key Points to Mention

  • BFS ensures shortest path and level-order traversal, which is suitable for crawling.
  • Visited set prevents infinite loops and redundant work.
  • Domain filtering: compare hostnames, handle subdomains carefully.
  • URL normalization: resolve relative URLs, strip fragments, lowercase scheme/host.
  • Rate limiting and politeness: respect robots.txt, add delays, handle 429 responses.
  • Scalability: consider distributed crawling, but for this problem, a single-threaded BFS is sufficient.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.