Waymo·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

Waymo ML engineer system design round, single question but it had a lot of moving parts. The focus was on designing an inference system at massive scale and the math-heavy estimation portion took up more time than I expected.

Questions Asked (1)

Design an ML inference system for 100 million daily active users. Include back-of-envelope estimates for memory and bandwidth, and address latency optimization and out-of-memory risks.

System DesignTechnical Trade-offs

Author's notes

The estimation part tripped me up more than the architecture itself.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints (e.g., model size, latency SLA, hardware). Then walk through a high-level architecture, perform back-of-envelope calculations for memory and bandwidth, and discuss latency optimization and OOM mitigation strategies. Emphasize trade-offs and justify your choices.

Pro tip: At Waymo, safety and reliability are paramount, so highlight redundancy, failover, and graceful degradation in your design. Also, consider edge vs. cloud inference trade-offs, as latency and bandwidth are critical for autonomous driving.

1. Clarify Requirements

Ask about model size, latency SLA, throughput, hardware (GPU/TPU/CPU), and deployment environment (cloud/edge). Confirm assumptions like peak load vs. average load.

2. High-Level Architecture

Propose a scalable architecture: load balancers, inference servers (with model replicas), caching, and possibly a message queue for async requests. Consider edge inference for low latency.

3. Back-of-Envelope Estimates

Calculate memory per model instance, total memory for replicas, and bandwidth for input/output data. Use QPS = 100M DAU * average requests per user per day / 86400 seconds.

4. Latency Optimization

Discuss techniques: model quantization, pruning, batching, caching, hardware acceleration (GPU/TPU), and edge deployment. Mention tail latency and how to mitigate it.

5. Out-of-Memory (OOM) Mitigation

Address OOM risks: memory pooling, gradient checkpointing (if training), model sharding, swapping, and monitoring. Suggest autoscaling and graceful degradation.

Key Points to Mention

Back-of-envelope: QPS calculation, memory per model (e.g., 1GB per model), total memory for replicas, bandwidth per request (e.g., 1MB per request).
Latency optimization: model quantization (FP16/INT8), batching, caching, hardware acceleration, edge inference.
OOM mitigation: memory pooling, model sharding, swapping, autoscaling, monitoring, and graceful degradation.
Trade-offs: accuracy vs. latency, cost vs. performance, edge vs. cloud.
Reliability: redundancy, failover, health checks, and canary deployments.
Waymo context: safety-critical, real-time inference, sensor data processing, and regulatory constraints.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.