← Waymo Interview Insights

Waymo·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say
Jun 2026

Summary

Waymo ML engineer interview that went deep on model efficiency and training techniques. Four meaty topics back to back, each with a 'how would you evaluate this in production' follow-up that I was not fully ready for.

Questions Asked (4)

Q1

Explain quantization-aware training: what it is, why you'd use it, how it works, common pitfalls, and how you'd measure its impact on a production model.

Technical Trade-offsSystem DesignA/B Testing & Experimentation
Author's notes

I knew the basics but fumbled on the pitfalls section.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining quantization-aware training (QAT) and contrasting it with post-training quantization, then explain the motivation (e.g., latency, memory, power) in the context of autonomous driving. Walk through the mechanics of simulating quantization during training, highlight common pitfalls like gradient mismatch and calibration issues, and finish with a rigorous measurement plan that includes offline metrics, online A/B tests, and safety-critical validation.

Pro tip: Emphasize that for safety-critical systems like Waymo, quantization must be validated not just on average accuracy but on worst-case and rare-scenario performance, and that you'd use techniques like per-channel quantization and quantization-aware fine-tuning to preserve robustness.

1. Define QAT and its purpose

Explain that QAT simulates quantization during training so the model learns to be robust to low-precision arithmetic, enabling efficient inference on hardware like TPUs or edge devices.

2. Motivate with production constraints

Discuss why QAT is used: reduced model size, faster inference, lower power consumption, and meeting latency requirements for real-time perception in autonomous vehicles.

3. Describe the mechanics

Outline how QAT inserts fake quantization nodes to simulate rounding and clamping, uses straight-through estimators for gradients, and often requires fine-tuning with a lower learning rate.

4. Address pitfalls and mitigations

Cover issues like gradient mismatch, overfitting to quantization noise, calibration of activation ranges, and handling outliers; mention solutions like per-channel quantization and careful hyperparameter tuning.

5. Measure impact rigorously

Propose a measurement plan: offline evaluation on held-out data with metrics like mAP, IoU, and latency; online A/B testing with canary deployments; and safety validation on rare scenarios to ensure no regression in critical performance.

Key Points to Mention

  • Difference between QAT and post-training quantization (PTQ): QAT typically yields higher accuracy at lower bit widths.
  • Straight-through estimator (STE) and how gradients are approximated during backpropagation.
  • Quantization granularity: per-tensor vs. per-channel, and symmetric vs. asymmetric quantization.
  • Calibration of activation ranges and handling of outliers to minimize accuracy loss.
  • Impact on model size, inference latency, and power consumption, especially on target hardware.
  • A/B testing methodology: offline metrics, online canary tests, and safety-critical validation for autonomous driving.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Walk through knowledge distillation: the purpose, the mechanics, when it's the right tool, what can go wrong, and how you'd verify it actually helped in a real deployment.

Technical Trade-offsA/B Testing & ExperimentationSystem Design
Author's notes

This one I felt okay about.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the purpose, mechanics, trade-offs, and validation of knowledge distillation, using a concrete example from autonomous driving (e.g., compressing a perception model for on-vehicle deployment). Emphasize the end-to-end process: why you'd distill, how you'd implement it, potential pitfalls, and how you'd measure success in production.

Pro tip: Frame distillation as a system-level optimization: the goal isn't just a smaller model, but maintaining safety-critical accuracy under latency and compute constraints. Always tie back to measurable business impact, like reduced inference time or improved recall of rare objects.

1. Clarify Purpose and Context

Explain why knowledge distillation is used: to transfer knowledge from a large, accurate teacher model to a smaller, efficient student model, enabling deployment on resource-constrained hardware without significant accuracy loss.

2. Describe Mechanics

Outline the training process: the student learns from both hard labels and soft targets (teacher's logits or probabilities), often using a temperature parameter to soften distributions. Mention variants like response-based, feature-based, or relation-based distillation.

3. Identify When to Use It

Discuss scenarios where distillation is appropriate: when you have a high-performing but slow/expensive teacher, need a compact model for edge deployment, or want to ensemble multiple models into one. Contrast with alternatives like pruning or quantization.

4. Highlight Risks and Failure Modes

Cover potential issues: teacher bias or errors propagating to student, capacity gap making it hard for student to mimic teacher, overfitting to teacher's quirks, and degradation on rare classes. Mention the need for careful hyperparameter tuning (temperature, loss weighting).

5. Validate in Deployment

Explain how to verify impact: offline evaluation on held-out data (accuracy, latency, size), then online A/B testing with safety-critical metrics (e.g., precision/recall on edge cases, false positive rate). Monitor for distribution shift and compare against baseline.

Key Points to Mention

  • Soft targets provide richer information than hard labels, including inter-class similarities.
  • Temperature scaling in the softmax controls the softness of teacher probabilities.
  • Distillation can be combined with other compression techniques (pruning, quantization) for greater efficiency.
  • The capacity gap between teacher and student can limit effectiveness; sometimes an intermediate teacher helps.
  • Validation must include both offline metrics and online A/B tests with safety-critical KPIs.
  • In autonomous driving, distillation is often used for perception models to meet real-time latency constraints on vehicle hardware.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What is evaluation mode in deep learning frameworks, why does it matter, when do people get it wrong, and how would you catch bugs related to it in a production pipeline?

Root Cause AnalysisTechnical Trade-offs
Author's notes

Honestly the question I felt most confident on and probably gave the most concise answer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining evaluation mode and contrasting it with training mode, emphasizing its role in deterministic inference. Then discuss common pitfalls and how they manifest in production, and finally outline a systematic approach to detect and prevent such bugs, using concrete examples from deep learning frameworks.

Pro tip: Emphasize that evaluation mode is not just about disabling dropout and batch norm updates—it's about ensuring reproducibility and correctness in production. Mention that subtle bugs often arise from inconsistent mode settings across distributed components or during model export.

1. Define Evaluation Mode

Explain that evaluation mode (e.g., model.eval() in PyTorch, training=False in Keras) sets layers like dropout and batch normalization to inference behavior, ensuring deterministic outputs.

2. Explain Why It Matters

Highlight that without evaluation mode, predictions become stochastic and batch statistics leak, leading to inconsistent and incorrect results in production.

3. Identify Common Mistakes

Discuss scenarios where people forget to set evaluation mode, such as during validation, model export, or in distributed inference, causing silent performance degradation.

4. Detect Bugs in Production

Propose methods like unit tests comparing training vs. evaluation outputs, monitoring prediction variance, and logging mode flags to catch inconsistencies.

5. Prevent and Mitigate

Suggest best practices: explicit mode setting in inference code, using framework-specific export tools, and integrating mode checks into CI/CD pipelines.

Key Points to Mention

  • Dropout and batch normalization behavior differs between training and evaluation modes.
  • Forgetting to set evaluation mode leads to non-deterministic predictions and incorrect batch statistics.
  • Common pitfalls: validation loops, model export (e.g., ONNX, TorchScript), and distributed inference.
  • Detection: unit tests with fixed seeds, comparing outputs in train vs. eval mode, and monitoring prediction drift.
  • Prevention: explicit mode setting, using context managers, and automated checks in deployment pipelines.
  • Impact on production: silent failures, degraded model performance, and difficulty in debugging.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Describe contrastive learning: the core idea, how it's trained, what scenarios it's useful for, what tends to go wrong, and how you'd evaluate whether it improved a production model.

Technical Trade-offsA/B Testing & ExperimentationAlgorithms & Data Structures
Author's notes

This is where I ran out of steam a little.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining contrastive learning and its core idea of pulling positive pairs together and pushing negative pairs apart. Then walk through the training process, including loss functions and data augmentation, and discuss scenarios where it excels. Finally, cover common pitfalls and how to rigorously evaluate its impact on a production model, emphasizing offline metrics and online A/B testing.

Pro tip: Emphasize that contrastive learning is not a silver bullet—its success hinges on the quality of positive/negative pairs and careful tuning of temperature. In production, always validate with a well-designed A/B test that measures both model performance and business metrics, as offline gains may not translate.

1. Core Idea

Explain that contrastive learning learns representations by comparing similar (positive) and dissimilar (negative) pairs, aiming to bring positives closer and push negatives apart in the embedding space.

2. Training Process

Describe how it's trained: typically using a contrastive loss like InfoNCE or triplet loss, with data augmentation to create positive pairs and in-batch negatives or memory banks for negatives. Mention the role of temperature scaling.

3. Useful Scenarios

Highlight scenarios where labeled data is scarce or expensive, such as self-supervised pre-training, retrieval, clustering, and few-shot learning. Also mention its use in multimodal tasks like image-text alignment.

4. Common Pitfalls

Discuss what tends to go wrong: false negatives, large batch size requirements, sensitivity to augmentation, representation collapse, and difficulty in tuning temperature and loss margins.

5. Evaluation in Production

Explain how to evaluate improvement: offline metrics like linear probe accuracy, kNN, or retrieval recall; online A/B testing with guardrail metrics; and monitoring for distribution shift and embedding drift.

Key Points to Mention

  • InfoNCE loss and its relation to mutual information
  • Importance of large batch sizes and memory banks for negatives
  • Data augmentation strategies for positive pair generation
  • False negatives and debiasing techniques
  • Linear probe evaluation for representation quality
  • A/B testing with business metrics and guardrails

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.