Google·Machine Learning Engineer·Technical Phone Screen
- Implement a depth-first search web crawler in Python, given a starting URL, a get_links function, and a max_pages limit. The crawler should visit each page at most once and handle cycles.
- Implement a breadth-first search version of the same web crawler using a queue, with the same constraints.
- What bugs can appear if you implement the DFS crawler using recursion instead of an explicit stack?
- How does Python's default recursion stack limit affect a recursive crawler implementation, and how would you work around it?
- Why should the visited set store URL strings rather than some mutable object representing a URL?
- How would you prevent bugs caused by shared mutable state if the same crawler instance is reused across multiple runs?
- How would you extend the crawler to handle timeouts, failed HTTP requests, and concurrent crawling?
“I went straight to recursion and got partway through before they asked about stack depth.”