Uber·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Uber data scientist interview that was essentially a statistics gauntlet disguised as a single question. The problem packed in about six different concepts at once and I was not fully prepared for how deep they wanted me to go on each layer.

Questions Asked (1)

Walk through the full sample size calculation for a two-arm signup A/B test: baseline conversion 6%, relative uplift target of 5%, two-sided alpha 0.05, 80% power, unequal traffic split of 2:1 control to treatment, 15% bot sessions removed after the fact, clustered sessions with mean 1.4 sessions per user and ICC of 0.03, and 8% attrition from data quality filters. Include the design effect for clustering, attrition adjustment, allocation ratio correction, and convert to days given 120k sessions per day. Also explain what breaks if variance is mis-specified or if someone peeks at results mid-test.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This question wrecked me a little.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by computing the base sample size per arm using the standard two-proportion z-test formula, then sequentially apply corrections for unequal allocation, clustering design effect, attrition, and bot removal. Convert the final required sample to days using daily traffic, and clearly explain the consequences of variance mis-specification and peeking.

Pro tip: Always state your assumptions explicitly (e.g., independence, constant variance) and note that in practice you'd validate the ICC and attrition estimates from historical data; this shows rigor and awareness of real-world messiness.

1. Compute base sample size per arm

Use the two-proportion z-test formula for equal allocation: n = (Z_{α/2} + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2-p1)^2, with p1=0.06, p2=0.063 (5% relative uplift), α=0.05, power=0.80.

2. Adjust for unequal allocation (2:1)

Multiply the base per-arm sample by the allocation ratio correction factor: (1 + r)^2 / (4r) where r = n_treatment / n_control = 1/2. Then compute required control and treatment sizes.

3. Apply clustering design effect

Compute design effect DE = 1 + (m - 1) * ICC, where m = 1.4 sessions per user, ICC = 0.03. Multiply the sample sizes by DE to account for within-user correlation.

4. Adjust for attrition and bot removal

Inflate sample sizes by dividing by (1 - attrition_rate) and (1 - bot_rate): final_n = n_clustered / ((1 - 0.08) * (1 - 0.15)).

5. Convert to days and discuss risks

Divide total required sessions (control + treatment) by 120,000 sessions/day to get duration. Then explain how variance mis-specification (e.g., ignoring clustering) inflates false positives/negatives, and how peeking invalidates p-values and increases Type I error.

Key Points to Mention

Use of two-proportion z-test formula and correct Z-values (1.96 for α=0.05 two-sided, 0.84 for 80% power).
Allocation ratio correction factor for unequal splits: (1+r)^2/(4r) where r is treatment/control ratio.
Design effect formula for clustering: DE = 1 + (m-1)*ICC, and its impact on sample size.
Sequential adjustments: base → allocation → clustering → attrition/bots, and the multiplicative nature of these corrections.
Consequences of variance mis-specification: underpowered tests, inflated false positive/negative rates, and misleading confidence intervals.
Peeking problem: alpha inflation, need for sequential testing or alpha-spending to maintain error control.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.