Uber·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

Uber DS interview that went deep on experiment design, like really deep. It was basically one long question about A/B testing assumptions and I was not fully prepared for how systematic they wanted me to be about it.

Questions Asked (1)

Walk through the core assumptions required for a valid online A/B test. For each assumption, explain what it means formally, give a realistic product scenario where it breaks down, describe how you'd detect the violation, and explain how you'd fix or redesign around it. They specifically wanted coverage of: randomization integrity and sample ratio mismatch, independence and interference like network effects or shared inventory, stable unit exposure and crossover or noncompliance, stationarity and novelty effects, metric logging bias and missingness, sequential peeking and error inflation, and heterogeneous treatment effects.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was the whole interview basically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the seven assumptions the interviewer listed, treating each as a potential failure mode. For each, briefly define it formally, then illustrate with a concrete Uber-relevant scenario (e.g., rider-driver interference, surge pricing, sequential testing), and finally describe detection and remediation strategies. Emphasize that these assumptions are often violated in practice and that robust experimentation requires proactive design and monitoring.

Pro tip: Show maturity by acknowledging that perfect randomization is rare in marketplace settings and that the best experiments combine design safeguards (e.g., switchback, cluster randomization) with post-hoc diagnostics (e.g., SRM checks, CUPED). Also, mention that heterogeneous treatment effects are not a violation but an opportunity to learn about segment-level impact.

1. Randomization Integrity & Sample Ratio Mismatch (SRM)

Define SRM as a statistically significant deviation from the intended assignment ratio. Use a chi-squared test to detect it. In Uber, a bug in the assignment service could cause SRM; fix by auditing the assignment logic and adding real-time monitoring.

2. Independence & Interference

Explain that SUTVA (Stable Unit Treatment Value Assumption) requires no interference between units. In Uber, driver-rider networks and shared inventory cause interference. Detect via cluster-level analysis or spatial correlation. Fix with cluster randomization (e.g., by city or driver cohort) or switchback experiments.

3. Stable Unit Exposure & Noncompliance

Formally, units must receive the treatment as assigned. In Uber, a rider assigned to a discount might not take a ride (noncompliance). Detect by comparing intent-to-treat (ITT) vs. treatment-on-treated (TOT) or using instrumental variables. Fix by using encouragement designs or analyzing compliers.

4. Stationarity & Novelty Effects

Assume treatment effect is constant over time. Novelty effects cause early spikes that decay. Detect by plotting treatment effect over time or using change-point detection. Fix by running longer experiments, using holdout groups, or modeling time-varying effects.

5. Metric Logging Bias, Missingness, Sequential Peeking, & Heterogeneous Effects

Address logging bias (e.g., missing events due to app crashes) by auditing data pipelines and using imputation or weighting. For sequential peeking, use alpha-spending or always-valid inference. For heterogeneous effects, pre-specify subgroups and use interaction tests, but avoid overfitting.

Key Points to Mention

Sample Ratio Mismatch (SRM) detection via chi-squared test and its causes (e.g., bot traffic, logging bugs).
Interference in two-sided marketplaces and solutions like cluster randomization or switchback experiments.
Noncompliance and the difference between ITT and TOT, with instrumental variables as a remedy.
Novelty effects and the importance of running experiments for sufficient duration to reach steady state.
Sequential testing pitfalls and methods to control false positive rate (e.g., alpha spending, sequential probability ratio test).
Heterogeneous treatment effects: pre-register subgroups, use interaction terms, and beware of multiple comparisons.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.