Uber·Data Scientist·Technical Phone Screen
- You're running a weekly email campaign to 10 million users. How would you set up the randomization unit and any stratification blocks, and how do you prevent contamination from resends or overlap with other campaigns that same week?
- Walk through the sample size calculation for detecting a +0.18 percentage point absolute lift on a 3% baseline CTR, using 80% power and a two-sided alpha of 0.05. State your assumptions and show the formula.
- Define a primary success metric and at least two guardrail metrics for this email experiment. Be precise about numerators, denominators, and time windows. Why CTR over open rate?
- Leadership wants to peek at results daily and stop the experiment early if there's harm. How do you design a valid sequential monitoring plan that still controls Type I error?
- After 48 hours into the experiment, what diagnostics would you run to catch randomization failures, instrumentation delays, or traffic mix shifts like weekend effects? How do you fix issues without biasing the estimates?
- An interim look shows negative CTR lift but higher open rates. Give at least three plausible explanations and explain what you'd do next. Also cover how you'd handle intention-to-treat versus per-protocol analysis and what you'd tell stakeholders.
“I went with user-level randomization pretty quickly but fumbled a bit on the contamination angle.”