Uber·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Uber data scientist interview with a meaty experiment design question that goes pretty deep into stats and product thinking. One question but it has about seven sub-questions packed inside it, so budget your time accordingly.

Questions Asked (1)

Design an experiment to test whether offering a bonus (cash or queue priority) increases the rate at which paperwork-complete candidates take their first action within 14 days. Cover eligibility rules, randomization unit, treatment arms (control, bonus upfront, bonus after first action, staged), stratification by motivation type, primary and secondary metrics including guardrails for quality and fraud, how you'd handle noncompliance and spillover, stopping rules, and when to roll out broadly vs. target narrowly based on heterogeneous effects. Also compute the minimum per-arm sample size to detect a 3 percentage point lift from a 20% baseline at 90% power and alpha 0.05 using a two-proportion z-test, showing the formula and a numeric answer.

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Strategy

Author's notes

This one is basically five interviews in a single prompt.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer by first defining the experiment's scope (eligibility, randomization, arms) and then detailing metrics, analysis, and decision rules. Emphasize practical considerations like noncompliance, spillover, and heterogeneity, and finish with the sample size calculation. Show a balance between statistical rigor and business context.

Pro tip: When discussing heterogeneous effects, propose a pre-registered subgroup analysis plan to avoid false positives, and suggest using a holdout group for long-term quality monitoring. Also, mention that the sample size calculation assumes no clustering or stratification effects, so you'd inflate the sample size to account for design effects.

1. Define eligibility and randomization

Specify who is eligible (e.g., paperwork-complete candidates who haven't taken first action) and the randomization unit (e.g., candidate-level). Discuss stratification by motivation type (e.g., high vs. low intent) to improve power and enable subgroup analysis.

2. Design treatment arms

Outline arms: control (no bonus), bonus upfront (cash or queue priority given immediately), bonus after first action (given upon completion), and staged (e.g., partial upfront, partial after). Ensure arms are mutually exclusive and cover the key hypotheses.

3. Select metrics and guardrails

Primary metric: first action within 14 days. Secondary metrics: time-to-action, completion rate, candidate satisfaction. Guardrails: quality of actions (e.g., fraud rate, error rate), long-term retention, and cost. Define how to measure and monitor them.

4. Address noncompliance, spillover, and stopping rules

Plan for noncompliance (e.g., intent-to-treat vs. per-protocol analysis) and spillover (e.g., cluster randomization if interference likely). Set stopping rules for futility and early success using sequential testing or alpha spending.

5. Compute sample size and plan rollout

Calculate minimum per-arm sample size using two-proportion z-test formula. Then, based on heterogeneous effects, decide whether to roll out broadly or target narrowly (e.g., only high-motivation candidates if effect is concentrated there).

Key Points to Mention

Eligibility: paperwork-complete candidates who have not taken first action within a defined window (e.g., 7 days).
Randomization unit: candidate-level, with stratification by motivation type (e.g., based on prior engagement or survey).
Treatment arms: control, bonus upfront, bonus after first action, staged bonus (e.g., 50% upfront, 50% after).
Primary metric: proportion taking first action within 14 days. Secondary: time-to-action, completion rate, cost per action.
Guardrails: fraud detection (e.g., anomalous action patterns), quality checks (e.g., error rates), and long-term retention.
Sample size formula: n = (Zα/2 + Zβ)^2 * (p1(1-p1) + p2(1-p2)) / (p2-p1)^2, with p1=0.20, p2=0.23, Zα/2=1.96, Zβ=0.84, yielding n ≈ 3,764 per arm.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.