I started with URL ingestion and spent probably too long on canonicalization before the interviewer nudged me toward the crawling layer.
Start by clarifying requirements and scale, then design a high-level architecture covering data ingestion, storage, and notification. Dive into key components like scheduling crawls, handling price updates, and ensuring scalability and reliability.
Pro tip: Emphasize trade-offs between freshness and cost: for example, using adaptive polling intervals based on product popularity or price volatility. Also, discuss how to handle failures gracefully with retries and dead-letter queues.
Ask questions to understand expected number of users, products tracked, frequency of price checks, and notification latency requirements. Estimate scale to inform design decisions.
Outline main components: API for URL submission and data retrieval, scheduler for periodic crawling, crawler service, price storage, and notification service. Sketch data flow.
Design schemas for products, price history, user subscriptions, and alerts. Choose appropriate databases (e.g., time-series for prices, relational for users) and discuss indexing and partitioning.
Address how to scale crawling (distributed workers, queues), handle failures (retries, circuit breakers), and ensure timely notifications (push vs. pull, batching).
Discuss trade-offs like polling frequency vs. cost, consistency vs. availability, and potential optimizations (caching, CDN, adaptive polling).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.