I started with requirements which felt right, but I spent way too long on the ingestion side and barely got to the serving path before they nudged me along.
Start by clarifying requirements (number of companies, feed update frequency, query latency, consistency) and then design a scalable ingestion pipeline that fetches and parses RSS feeds, stores articles in a database optimized for per-company recency queries, and exposes a simple API. Focus on trade-offs between push vs pull ingestion, storage choices, and caching to meet low-latency reads.
Pro tip: Emphasize idempotency and deduplication in the ingestion pipeline—RSS feeds often republish or update items, so using a unique article ID (e.g., GUID or hash of link+title) prevents duplicates and ensures accurate 'top X' results.
Ask about number of companies, articles per day, query QPS, latency SLA, and whether 'top X' means strictly most recent or ranked by relevance. This shapes the entire design.
Decide on a pull-based scheduler (e.g., cron or distributed queue) that periodically fetches RSS feeds, parses them, and normalizes articles. Use a message queue to decouple fetching from processing and handle failures with retries.
Select a database that supports efficient per-company recency queries. For example, a wide-column store like Cassandra with clustering by company and descending timestamp, or a relational DB with proper indexing. Consider caching hot companies in Redis.
Define a REST endpoint like GET /articles?company_id=123&limit=X that returns the X most recent articles. Implement caching (e.g., Redis) for frequent queries and consider precomputing top X per company if X is small and fixed.
Discuss partitioning by company, handling feed failures, ensuring eventual consistency, and monitoring lag and error rates. Also consider rate limiting and backpressure.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.