I started with the data model and event schema which felt safe, but I spent too long there and had to rush through the streaming pipeline.
Start by clarifying requirements and scale, then design a pipeline that separates ingestion, processing, and serving layers. Use a streaming path for near-real-time metrics and a batch path for historical accuracy, with idempotent writes and windowing to handle duplicates and late data.
Pro tip: Explicitly discuss trade-offs between latency, cost, and accuracy, and propose a lambda or kappa architecture with a unified data model to avoid divergence between real-time and batch views.
Ask about event volume, required latency for real-time metrics, historical retention, and query patterns. Establish consistency and accuracy expectations.
Propose a scalable, durable message queue (e.g., Kafka) to buffer events. Ensure events have unique IDs and timestamps for deduplication and ordering.
Use stream processing (e.g., Flink) for near-real-time aggregations with windowing and watermarks to handle late data. Use batch processing (e.g., Spark) for historical aggregations and reprocessing.
Store real-time aggregates in a low-latency store (e.g., Redis, Druid) and historical aggregates in a scalable OLAP store (e.g., ClickHouse, BigQuery). Support queries that merge both.
Implement idempotent writes using unique event IDs, and use event-time processing with allowed lateness. Reconcile real-time and batch views periodically.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.