Start by clarifying requirements and constraints, then outline a BFS-based crawler using a thread-safe queue and a shared visited set to ensure each URL is visited once. Explain how a fixed-size worker pool processes URLs concurrently, and discuss testing strategies including unit tests with mocked get_links and integration tests for concurrency.
Pro tip: Mention that you would normalize URLs (e.g., remove fragments, handle trailing slashes) before deduplication to avoid visiting the same page under different forms. Also, discuss how you'd handle edge cases like redirects, non-HTML content, and rate limiting to show production awareness.
Ask about expected scale, politeness policies, error handling, and whether the crawler should respect robots.txt. Confirm that only same-hostname URLs are crawled and that get_links is the only way to discover links.
Use a thread-safe queue (e.g., LinkedBlockingQueue) for URLs to visit and a concurrent set (e.g., ConcurrentHashMap.newKeySet) for visited URLs. A fixed-size worker pool (e.g., ExecutorService with N threads) pulls URLs from the queue, calls get_links, and enqueues new same-host URLs not yet visited.
Use atomic operations or locks to check-and-add to the visited set before enqueueing. Track active tasks (e.g., with a counter or phaser) to know when all workers are idle and the queue is empty, then signal completion.
Normalize URLs (lowercase host, remove fragments, resolve relative paths) before deduplication. Handle exceptions from get_links gracefully, and consider backpressure or rate limiting to avoid overwhelming the target server.
Write unit tests with a mock get_links that returns a controlled graph, verifying all URLs are visited exactly once. Add concurrency tests to ensure no race conditions, and integration tests with a local web server to validate real-world behavior.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.