Google·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jul 2026

Summary

System design round at Google for a senior engineering role. One question, big scope: design a web crawler at Google scale. The kind of problem where you can talk for an hour and still feel like you barely scratched the surface.

Questions Asked (1)

Design a distributed web crawler that continuously discovers, fetches, and indexes billions of web pages while respecting per-domain rate limits and avoiding duplicate work.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

I started with the URL frontier and felt okay about that part.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then outline a high-level architecture with key components: URL frontier, fetcher, parser, deduplication, and storage. Dive into critical challenges like per-domain rate limiting, duplicate detection, and politeness, discussing trade-offs and potential bottlenecks.

Pro tip: Emphasize the importance of a distributed URL frontier with per-domain queues and a global deduplication mechanism using Bloom filters or similar probabilistic data structures to handle billions of URLs efficiently. Also, mention the need for adaptive rate limiting based on server responses (e.g., 429, 503) to be a good web citizen.

1. Clarify Requirements and Scale

Ask questions to understand the scope: number of pages (billions), update frequency, politeness requirements, and indexing needs. Confirm assumptions about distributed environment and fault tolerance.

2. High-Level Architecture

Outline main components: URL frontier (scheduler), fetcher (downloader), parser, deduplication service, storage (for content and metadata), and indexing pipeline. Explain how they interact in a distributed setup.

3. Deep Dive into Key Components

Focus on URL frontier design: distributed queues, per-domain rate limiting, prioritization. Discuss deduplication: exact and near-duplicate detection using hashing (e.g., SHA-256) and SimHash. Explain fetching: distributed workers, politeness, retries, and handling dynamic content.

4. Address Scalability and Fault Tolerance

Discuss partitioning (e.g., by domain), replication, and consistency trade-offs. Explain how to handle failures, backpressure, and monitoring. Mention use of cloud services or custom distributed systems (e.g., Kafka, Redis, Cassandra).

5. Trade-offs and Optimizations

Summarize key trade-offs: consistency vs. availability, latency vs. throughput, and cost. Suggest optimizations like caching DNS, using connection pooling, and adaptive crawling strategies.

Key Points to Mention

Distributed URL frontier with per-domain queues and rate limiting
Deduplication using Bloom filters for URL seen and content hashing (e.g., SimHash) for near-duplicate detection
Politeness: respecting robots.txt, crawl-delay, and adaptive rate limiting based on server responses
Scalability: partitioning by domain, consistent hashing, and distributed storage (e.g., Bigtable, Cassandra)
Fault tolerance: replication, checkpointing, and handling failures gracefully
Indexing pipeline: extracting content, storing in inverted index, and updating incrementally

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.