← Flatiron Health Interview Insights
I've done this derivation before but blanked on the pooled variance step mid-explanation.
Start by clearly stating the formula for sample size per variant in a two-proportion z-test, then plug in the given values step by step, computing the z-scores for alpha and power, the baseline and treatment proportions, and the pooled variance. Finally, calculate the required sample size and interpret it in the context of the experiment.
Pro tip: Always clarify whether the sample size is per variant or total, and mention that this calculation assumes a simple random sample and no peeking, which can inflate Type I error if not accounted for.
Present the standard formula for sample size per variant: n = ( (z_{1-α/2} * sqrt(2 * p̄ * (1-p̄)) + z_{1-β} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / (p2 - p1)^2, where p̄ is the pooled proportion. Mention assumptions like independence and normality.
For two-sided alpha=0.05, z_{1-α/2} = 1.96. For 80% power, z_{1-β} = 0.84. Explain that these correspond to the standard normal quantiles.
Baseline conversion p1 = 0.045. A 10% relative lift means p2 = p1 * 1.10 = 0.0495. Compute the pooled proportion p̄ = (p1 + p2)/2 = 0.04725.
Compute the pooled variance term: 2 * p̄ * (1-p̄) = 2 * 0.04725 * 0.95275 ≈ 0.0900. Compute the unpooled variance term: p1*(1-p1) + p2*(1-p2) = 0.045*0.955 + 0.0495*0.9505 ≈ 0.0900. Note they are similar but use the correct formula.
Substitute values: n = ( (1.96 * sqrt(0.0900) + 0.84 * sqrt(0.0900))^2 ) / (0.0045)^2 = ( (1.96+0.84) * 0.3 )^2 / 0.00002025 = (2.8 * 0.3)^2 / 0.00002025 = (0.84)^2 / 0.00002025 = 0.7056 / 0.00002025 ≈ 34,844 per variant. Round up to 34,845.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by recalling that sample size is proportional to variance, so a 30% variance reduction means the required sample size is multiplied by 0.70. Then compute the absolute and relative reduction in sample size, and explain the practical implications for experiment duration and cost.
Pro tip: Mention that CUPED reduces variance without biasing the treatment effect, so you can achieve the same power with fewer users—but always validate the variance reduction on historical data before relying on it.
Explain that for a fixed power and effect size, required sample size is directly proportional to the variance of the metric. Therefore, reducing variance by 30% reduces the required sample size by 30%.
If the unadjusted sample size is N, the adjusted sample size becomes 0.70 * N. For example, if N = 100,000 per variant, the new size is 70,000 per variant.
The absolute reduction is N - 0.70N = 0.30N (e.g., 30,000 fewer users per variant). The relative reduction is (N - 0.70N)/N = 30%.
Highlight that this reduction can shorten experiment duration, lower costs, or allow detection of smaller effects with the same sample size. Also note that the variance reduction must be estimated accurately.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The math is straightforward but I fumbled the practical interpretation at the end.
First, compute the pooled proportion and standard error, then calculate the z-statistic and p-value for the two-sided test. Next, construct the 95% confidence interval for the difference in proportions. Finally, interpret the results in terms of statistical significance and practical impact for the product.
Pro tip: Always consider the practical significance alongside statistical significance; with large samples, even tiny differences can be statistically significant but may not be worth implementing. Also, mention that the confidence interval provides a range of plausible effect sizes, which is more informative than the p-value alone.
Define null hypothesis (no difference) and alternative (two-sided difference). Compute the pooled proportion: (x1 + x2) / (n1 + n2), where x1 and x2 are the number of successes in each group.
Calculate the standard error using the pooled proportion: sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2)). Then compute z = (p1 - p2) / SE.
For a two-sided test, p-value = 2 * (1 - Φ(|z|)). Compare to alpha = 0.05 to determine statistical significance.
Use the unpooled standard error for the CI: SE_unpooled = sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2). The 95% CI is (p1 - p2) ± 1.96 * SE_unpooled. Convert to percentage points by multiplying by 100.
Discuss whether the difference is statistically significant and what the effect size means in context. Consider the width of the CI and whether the lower bound is practically meaningful.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.