Amazon·Machine Learning Engineer·Onsite - System Design / Architecture·Staff

StaffPrefer not to say

Jul 2026

Summary

System design round at Amazon for an ML Engineer role, focused entirely on building a distributed data-preprocessing pipeline for LLM training. The question was deep and principal-engineer-level, with follow-ups that kept pushing on every trade-off I made.

Questions Asked (1)

Design a distributed data-preprocessing pipeline for LLM training. The reference architecture is S3 as the source, feeding into Kafka, then a tokenizer stage, near-duplicate deduplication, a quality filter, and finally an output sink. For each stage, justify your technology choices, explain what breaks at scale (throughput bottlenecks, data skew, fault tolerance, exactly-once semantics, schema evolution), and walk through how you'd monitor and recover from failures.

System DesignTechnical Trade-offsData Modeling

Author's notes

This one ran the full hour.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements (data volume, latency, fault tolerance, exactly-once needs) and then walk through each stage of the pipeline, justifying technology choices and discussing failure modes and monitoring. Emphasize trade-offs and how you would handle scale, skew, and schema evolution. Conclude with a holistic view of monitoring, recovery, and iteration.

Pro tip: Anchor your design around idempotency and replayability—use deterministic processing and checkpointing so you can reprocess data safely after failures. Also, proactively discuss cost and operational complexity, as Amazon values frugality and simplicity.

1. Clarify Requirements and Constraints

Ask about data volume (TB/PB per day), latency requirements (batch vs streaming), fault tolerance, exactly-once semantics, and schema evolution needs. This shapes technology choices and trade-offs.

2. Design Each Stage with Justification

For S3, Kafka, tokenizer, dedup, quality filter, and sink, explain technology choices (e.g., Kafka for buffering and replay, Spark/Flink for processing) and how they handle scale, skew, and failures.

3. Address Scale and Failure Modes

Discuss bottlenecks (e.g., tokenizer CPU-bound, dedup memory-intensive), data skew (partitioning strategies), fault tolerance (checkpointing, replication), exactly-once (idempotent writes, transactions), and schema evolution (schema registry, backward compatibility).

4. Monitoring and Recovery

Outline monitoring metrics (throughput, latency, error rates, lag), alerting, and recovery strategies (replay from Kafka, checkpoint restore, dead-letter queues). Emphasize observability and automated recovery.

5. Summarize Trade-offs and Iterate

Conclude with key trade-offs (cost vs performance, complexity vs reliability) and how you would iterate based on monitoring and feedback.

Key Points to Mention

Kafka as a durable buffer enabling replay and decoupling, with partitioning for parallelism and exactly-once semantics via transactions or idempotent producers.
Tokenizer stage: CPU-bound, use horizontal scaling with load balancing; consider GPU acceleration for large models; handle skew via dynamic partitioning.
Deduplication: use MinHash/LSH for near-duplicate detection; memory-intensive, so use distributed frameworks (Spark) with partitioning and caching; handle skew by salting keys.
Quality filter: rule-based or ML-based; ensure low latency; use side outputs for rejected data; monitor filter rates for drift.
Fault tolerance: checkpointing in stream processors (Flink/Spark), Kafka replication, S3 durability; exactly-once via idempotent writes and transactional sinks.
Schema evolution: use schema registry (e.g., AWS Glue Schema Registry) with backward-compatible changes; handle malformed records via dead-letter queues.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.