← Uber Interview Insights

Uber·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say
May 2026Remote

Summary

Uber DS interview with a single massive experiment design question covering basically every dimension of A/B testing you can think of. The depth expected was pretty intense for what felt like a single prompt.

Questions Asked (1)

Q1

Your company is running a promotion that gives users a discount when they open the app. Design a full randomized controlled experiment to evaluate whether this promotion works, covering: primary success metric and guardrails, unit of randomization and justification, eligibility and assignment rules, network effect detection and spillover quantification, sample size and test duration with stopping rules, and the instrumentation needed to compute intent-to-treat and treatment-on-treated estimates.

A/B Testing & ExperimentationProduct Analytics & MetricsPricing & Monetization
Author's notes

This is basically six questions stapled together and presented as one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business goal and defining a primary success metric with guardrails, then design the experiment with user-level randomization and justify it. Address network effects by detecting and quantifying spillover, and plan sample size, duration, and stopping rules. Finally, outline instrumentation for ITT and TOT estimates.

Pro tip: At Uber, network effects are critical due to the marketplace; consider using cluster randomization or switchback tests to mitigate spillover, and always pre-register your analysis plan to avoid p-hacking.

1. Define Metrics and Guardrails

Choose a primary success metric (e.g., incremental rides or revenue per user) and guardrail metrics (e.g., cancellation rate, driver utilization, customer satisfaction) to ensure the promotion doesn't harm the ecosystem.

2. Design Randomization and Eligibility

Randomize at the user level to avoid contamination, but justify if cluster randomization (e.g., by city) is needed due to network effects. Define eligibility criteria (e.g., new users, specific geographies) and ensure balanced assignment.

3. Detect and Quantify Network Effects

Use techniques like cluster randomization, switchback tests, or spatial separation to detect spillover. Quantify spillover by comparing treatment effects across different exposure levels or using instrumental variables.

4. Determine Sample Size, Duration, and Stopping Rules

Calculate sample size based on minimum detectable effect, power, and significance level. Set duration to capture full behavior cycle, and define stopping rules (e.g., sequential testing, futility) to avoid peeking.

5. Instrument for ITT and TOT

Log user-level exposure, treatment assignment, and outcomes. Use intent-to-treat (ITT) as the primary analysis, and treatment-on-treated (TOT) via instrumental variables or compliance-adjusted analysis to estimate the effect on those who actually used the promotion.

Key Points to Mention

  • Primary metric: incremental revenue or rides per user; guardrails: cancellation rate, driver earnings, customer satisfaction.
  • Randomization unit: user-level to avoid interference, but consider cluster randomization if spillover is likely.
  • Network effects: use switchback or cluster designs to detect and quantify spillover; measure via difference in outcomes between treated and control in overlapping areas.
  • Sample size: power analysis with MDE, alpha, beta; duration: at least one full business cycle; stopping rules: sequential testing or group sequential design.
  • Instrumentation: log assignment, exposure, and outcomes; ITT includes all randomized users; TOT uses instrumental variable (assignment as instrument for exposure).
  • Pre-register analysis plan and consider heterogeneous treatment effects by user segments.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.