Waymo·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

Waymo ML engineer system design round, one big open-ended question about inference serving at massive scale. Felt like a gauntlet that kept expanding the more I answered.

Questions Asked (1)

Design a production inference serving system for an ML model that needs to support 100 million daily active users. Walk through traffic and QPS estimates, memory requirements for weights and caches, network and accelerator bandwidth, hardware selection, latency optimization, and OOM failure handling.

System DesignTechnical Trade-offs

Author's notes

This question sprawls in every direction and I underestimated how much ground it actually covers.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying assumptions about the model (e.g., size, architecture) and usage patterns (e.g., requests per user per day, peak-to-average ratio). Then systematically walk through each layer of the system: traffic estimation, resource requirements, hardware selection, latency optimization, and failure handling, using concrete numbers and trade-offs. Emphasize that this is a back-of-the-envelope design, and iterate on numbers as needed.

Pro tip: Always state your assumptions explicitly and show your math—interviewers care more about your reasoning than exact numbers. Also, proactively discuss trade-offs between latency, cost, and reliability, as Waymo values safety-critical system design.

1. Clarify Requirements and Assumptions

Ask about model size, architecture, input/output, latency SLA, and usage patterns (e.g., requests per user per day, peak traffic multiplier). State assumptions clearly to ground your calculations.

2. Estimate Traffic and QPS

Calculate average and peak QPS based on 100M DAU and assumed requests per user per day. Consider diurnal patterns and peak-to-average ratio (e.g., 2-5x).

3. Compute Memory and Bandwidth Requirements

Estimate model weight memory (e.g., number of parameters × bytes per parameter), activation memory, and cache sizes (e.g., KV cache for transformers). Calculate network bandwidth for input/output and accelerator memory bandwidth needed for inference.

4. Select Hardware and Optimize Latency

Choose accelerators (e.g., GPUs, TPUs) based on compute and memory needs. Discuss batching, quantization, model parallelism, and caching to meet latency SLA. Consider edge vs. cloud deployment.

5. Design for Failures and OOM Handling

Propose strategies to handle out-of-memory (OOM) errors: graceful degradation, request shedding, dynamic batching, memory pooling, and monitoring. Discuss redundancy and failover for high availability.

Key Points to Mention

Traffic estimation: 100M DAU × requests per user per day → average QPS; apply peak multiplier.
Model memory: parameters × bytes (e.g., FP16), plus activations and KV cache for transformers.
Bandwidth: network ingress/egress and accelerator memory bandwidth (e.g., HBM) requirements.
Hardware selection: trade-offs between GPUs (e.g., A100, H100) and TPUs; consider cost and availability.
Latency optimization: batching, quantization, model pruning, caching, and speculative decoding.
OOM handling: dynamic batching, memory limits, request prioritization, and graceful degradation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.