DoorDash·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

May 2026

Summary

DoorDash MLE interview that went deep on experimentation theory. One long question that covered basically everything you'd ever need to know about A/B testing, and I mean everything. Felt like a stats oral exam more than a typical ML screen.

Questions Asked (1)

Walk me through A/B testing from start to finish: setting up hypotheses, picking metrics, calculating sample size and power, choosing the right statistical test, interpreting results, and then talk through the major pitfalls like peeking, multiple comparisons, novelty effects, network interference, Simpson's paradox, and sample ratio mismatch. Also briefly compare frequentist and Bayesian approaches.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was basically the whole interview in one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer as a chronological walkthrough of the experimentation lifecycle, from hypothesis to decision, explicitly covering each requested component. Use concrete examples from a marketplace or ML context (e.g., ranking model, delivery time prediction) to illustrate statistical concepts and pitfalls. Conclude with a balanced comparison of frequentist and Bayesian methods, emphasizing when each is appropriate.

Pro tip: Emphasize that experimentation is a decision-making tool, not just a statistical exercise—always tie results back to business impact and guardrail metrics. Mention that at DoorDash, network interference and sample ratio mismatch are particularly critical due to the two-sided marketplace and dynamic pricing.

1. Define Hypothesis and Metrics

Start with a clear, falsifiable hypothesis (e.g., 'Adding a new ranking feature will increase conversion rate by 2%'). Choose a primary metric (OEC) and guardrail metrics (e.g., delivery time, cancellation rate) that align with business goals.

2. Design Experiment and Calculate Sample Size

Determine unit of randomization (user, session, store), calculate required sample size using power analysis (α=0.05, power=0.8, MDE). Consider variance reduction techniques like CUPED if applicable.

3. Run Experiment and Monitor

Launch the test, but avoid peeking at results prematurely. Monitor for sample ratio mismatch (SRM) and other data quality issues. Use sequential testing or Bayesian methods if early stopping is necessary.

4. Analyze Results and Interpret

Apply the appropriate statistical test (e.g., t-test, Mann-Whitney, bootstrap) based on metric distribution. Check for pitfalls: multiple comparisons (use Bonferroni or FDR), novelty effects (run longer), network interference (cluster randomization), Simpson's paradox (segment analysis).

5. Make Decision and Compare Approaches

Decide whether to ship, iterate, or abandon based on statistical and practical significance. Briefly compare frequentist (p-values, confidence intervals) vs. Bayesian (posterior probabilities, credible intervals) approaches, noting trade-offs in interpretability and flexibility.

Key Points to Mention

Power analysis: relationship between sample size, effect size, alpha, and power; use of MDE to determine practical significance.
Common pitfalls: peeking (inflates Type I error), multiple comparisons (false positives), novelty effects (temporary lift), network interference (spillover), Simpson's paradox (aggregation bias), and sample ratio mismatch (SRM) as a validity check.
Statistical tests: choose based on metric type (continuous vs. binary) and distribution (parametric vs. non-parametric); consider bootstrap for complex metrics.
Frequentist vs. Bayesian: frequentist focuses on long-run error rates, Bayesian on posterior probability of hypotheses; Bayesian allows early stopping and incorporates priors.
Variance reduction: techniques like CUPED or stratification to increase sensitivity without increasing sample size.
Guardrail metrics: ensure experiment doesn't harm key business metrics (e.g., delivery time, customer satisfaction).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.