Expedia·Machine Learning Engineer·Onsite - System Design / Architecture
- Design a data pipeline that ingests events from multiple upstream producers, transforms them, and delivers them to analytics-ready storage. Walk through your full architecture.
- How would you handle schema evolution in a high-throughput streaming pipeline without breaking downstream consumers?
- Walk me through the tradeoffs between stream processing (e.g. Flink) and micro-batch approaches (e.g. Spark Streaming) for this kind of pipeline.
- How would you design for backfills when your pipeline has already been running in production for months?
- How do you ensure data quality and handle bad or malformed events at scale?
- How would you approach observability and lineage for a pipeline like this, and what does a meaningful SLA look like?
- What's your strategy for managing cost and scaling as event volume grows by an order of magnitude?
“This was the main question and it basically ate the whole session.”