DoorDash·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

DoorDash data science interview with a pretty intense stats-heavy question on experiment design. One question, lots of moving parts, felt like a take-home problem crammed into a live setting.

Questions Asked (1)

You have a baseline cold-food complaint rate of 3.50%, and all such orders currently get full refunds. A thermal-bag pilot is expected to cut that rate by 10% relatively (down to 3.15%). Using a two-sided z-test for two proportions at 95% confidence and 80% power: calculate the minimum sample size per arm for equal allocation, estimate the minimum test duration if only 60% of 200k daily orders are eligible and traffic is split evenly, and then redo the sample size calculation for a 6% relative reduction target instead. State your assumptions and round to the nearest 100 orders and 0.5 day.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This one had three sub-parts and I think I handled (a) okay but fumbled the transition to (b).

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly stating the assumptions (e.g., baseline rate, significance level, power, allocation ratio) and the formulas for sample size and duration. Then perform the calculations step-by-step, showing the plug-in values and rounding as requested. Finally, interpret the results and discuss practical implications such as feasibility and trade-offs.

Pro tip: Always sanity-check your sample size with a quick rule of thumb (e.g., for 80% power and 95% confidence, you need about 16 * p(1-p) / delta^2 per arm) to catch gross errors. Also, mention that in practice you'd use sequential testing or adjust for multiple comparisons if peeking, but for this calculation we assume a fixed horizon.

1. State assumptions and formulas

List all given parameters: baseline rate p1=0.035, relative reduction 10% => p2=0.0315, alpha=0.05 two-sided, power=0.80, equal allocation. Use the standard formula for two-proportion z-test: n per arm = ( (z_{1-alpha/2} * sqrt(2*p_bar*(1-p_bar)) + z_{1-beta} * sqrt(p1*(1-p1)+p2*(1-p2)) )^2 ) / (p1-p2)^2, where p_bar = (p1+p2)/2.

2. Calculate sample size for 10% relative reduction

Plug in values: z_{0.975}=1.96, z_{0.80}=0.84. Compute p_bar = (0.035+0.0315)/2 = 0.03325. Calculate the numerator and denominator, then round up to nearest 100 orders per arm.

3. Estimate test duration

Determine eligible daily orders: 60% of 200,000 = 120,000. With equal split, each arm gets 60,000 orders per day. Required days = n_per_arm / 60,000. Round to nearest 0.5 day.

4. Recalculate sample size for 6% relative reduction

New p2 = 0.035 * (1 - 0.06) = 0.0329. Recompute p_bar and sample size using same formula. Round to nearest 100 orders per arm.

5. Summarize and interpret

Present final numbers: sample size per arm for 10% and 6% reductions, and duration for 10% reduction. Discuss implications: smaller effect requires larger sample and longer duration, feasibility given traffic, and potential need for sequential testing or variance reduction.

Key Points to Mention

Assumptions: baseline rate 3.5%, relative reduction 10% (to 3.15%) and 6% (to 3.29%), two-sided alpha=0.05, power=80%, equal allocation, no continuity correction.
Formula for sample size per arm for two-proportion z-test, including pooled variance under null and unpooled under alternative.
Calculation steps: compute p_bar, plug in z-scores, solve for n, round up to nearest 100.
Duration calculation: eligible orders per day = 120,000, per arm = 60,000, days = n_per_arm / 60,000, round to nearest 0.5 day.
For 6% relative reduction, sample size increases because effect size is smaller; show recalculated n.
Practical considerations: feasibility of running test for required duration, potential for sequential testing or CUPED to reduce sample size, and business impact of detecting smaller effects.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.