Microsoft·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Microsoft DS interview with a brutal stats/experimentation question that covers basically everything at once. No fluff, just a wall of math and decision-making under uncertainty.

Questions Asked (1)

Baseline conversion is 5% and you want 90% power at a 0.05 significance level to detect a 6% relative lift. Calculate the per-variant sample size for a Bernoulli outcome using a normal approximation, then adjust for 10% bot traffic and a 7-day ramp assuming daily independence. After the experiment runs, A has 50k users and 2,650 conversions while B has 49,500 users and 2,820 conversions, with one interim look at day 3. Compute the p-value and 95% CI for the difference in proportions, apply a multiple-testing correction for the interim look, check for sample ratio mismatch, and decide whether to ship. Also discuss Type S/M error risks and what happens if your baseline was wrong.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was a single question that kept going.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly stating the formulas and assumptions for the sample size calculation, then walk through each adjustment step-by-step. For the analysis, compute the test statistic and confidence interval, apply the alpha-spending correction, check SRM, and interpret results in context of practical significance and error risks.

Pro tip: Always sanity-check your calculations with a quick simulation or known benchmarks, and remember that statistical significance does not guarantee practical significance—consider the business impact and potential errors.

1. Sample Size Calculation

Use the normal approximation formula for two proportions: n = 2*(z_{α/2}+z_β)^2 * p(1-p) / δ^2, where p is baseline conversion, δ is absolute lift. Plug in p=0.05, δ=0.003 (6% relative lift), z_{0.025}=1.96, z_{0.10}=1.28, and compute n per variant.

2. Adjustments for Bot Traffic and Ramp

Divide the required sample size by (1 - bot rate) to account for 10% bot traffic. Then, since the experiment ramps over 7 days with daily independence, multiply by 7 to get the total users needed per variant over the ramp period.

3. Analysis of Results

Compute the difference in conversion rates (p_B - p_A), the pooled standard error, and the z-statistic. Calculate the two-sided p-value and the 95% confidence interval for the difference. Apply a multiple-testing correction (e.g., Bonferroni or alpha-spending) for the interim look at day 3.

4. Diagnostics and Decision

Check for sample ratio mismatch (SRM) using a chi-square test on the user counts (50k vs 49.5k). If no SRM and the corrected p-value < 0.05, consider shipping, but also assess practical significance and error risks.

5. Error Risks and Sensitivity

Discuss Type S (sign) and Type M (magnitude) errors: the risk of getting the direction wrong or overestimating the effect. Also, analyze how a wrong baseline assumption would affect power and sample size, potentially leading to underpowered or overpowered experiments.

Key Points to Mention

Use of normal approximation for sample size and its assumptions (large sample, independence).
Adjustment for bot traffic by inflating sample size, and ramp-up period by multiplying by days if daily independence holds.
Calculation of p-value and confidence interval using pooled vs unpooled standard error, and interpretation.
Multiple testing correction for interim analysis: alpha-spending or Bonferroni, and its impact on significance threshold.
Sample ratio mismatch check: chi-square test on user counts, and implications if SRM is detected.
Type S and Type M errors: definition, risks, and how they relate to power and effect size estimation; sensitivity of results to baseline misestimation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.