I got the BFS part down pretty fast, queue with a visited set, standard stuff.
Start by clarifying requirements and edge cases, then outline a BFS crawler using a queue for URLs to visit and a set for visited URLs. Explain how you normalize URLs, handle retries with a retry count per URL, and ensure domain restriction. Walk through the crawl loop, emphasizing duplicate detection and error handling.
Pro tip: Mention that you would use a separate retry queue or track retry counts to avoid blocking the main queue, and discuss how to handle redirects and URL fragments to demonstrate thoroughness.
Ask about domain definition (subdomains?), URL normalization rules, retry semantics (exponential backoff?), and whether concurrency is needed. Confirm that the crawler should be polite and avoid overloading the server.
Use a queue (FIFO) for BFS, a set for visited URLs to prevent revisits, and a map or separate queue for retry counts. Explain that the visited set stores normalized URLs to ensure duplicates are caught.
Normalize URLs by lowercasing scheme and host, removing default ports, resolving dot segments, and stripping fragments. Check that the host matches the starting domain (or subdomain if allowed) before enqueueing.
While the queue is not empty, dequeue a URL, attempt fetch. On failure, increment retry count and re-enqueue if under limit; on success, extract links, normalize and filter, then enqueue unseen URLs. Mark URLs as visited when first enqueued to avoid duplicates.
Talk about using a priority queue for importance, handling robots.txt, adding delays, and scaling with multiple workers. Mention that retries could be handled with exponential backoff and that permanent failures are skipped after max retries.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.