This is one of those questions where you feel good for the first ten minutes and then realize you've only covered crawling and haven't touched ranking at all.
Start by clarifying requirements (scale, latency, freshness, etc.), then sketch the two main pipelines: offline (crawling, indexing) and online (query serving). Walk through each component, explaining data flow and key design choices, and highlight trade-offs and how they affect performance and relevance.
Pro tip: Emphasize the separation of concerns between offline and online systems, and discuss how you would measure and monitor end-to-end latency and relevance. Mention that you'd start with a simple design and iterate based on metrics.
Ask about scale (billions of pages, QPS), latency targets, freshness, and relevance expectations. Define functional and non-functional requirements to guide design decisions.
Outline crawling (URL frontier, politeness, deduplication), document processing (parsing, extraction), indexing (inverted index, forward index), and storage (distributed file system, sharding).
Describe query processing (parsing, spell correction, query rewriting), retrieval (using inverted index, ranking candidates), and ranking (learning-to-rank, blending signals).
Explain how to partition and replicate data, use caching (query results, posting lists), and employ distributed systems (e.g., sharded index servers, load balancing).
Highlight trade-offs like index size vs. query speed, freshness vs. cost, and relevance vs. latency. Mention techniques like tiered indexing, early termination, and approximate algorithms.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.