I knew the formula but fumbled the pooled variance piece.
Start by stating the standard formula for sample size per arm for a two-proportion z-test, then plug in the given values: baseline rate p1 = 0.12, target rate p2 = 0.125, alpha = 0.05 (two-sided), power = 0.80. Compute the required sample size step-by-step, showing intermediate calculations and rounding up to the nearest integer.
Pro tip: Mention that this calculation assumes no continuity correction and equal allocation; in practice, you might inflate the sample size by 5-10% to account for data loss or multiple comparisons, and consider using exact methods for small rates.
Write the formula for sample size per arm: n = ( (z_{1-α/2} * sqrt(2 * p̄ * (1-p̄)) + z_{1-β} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / (p2 - p1)^2, where p̄ = (p1+p2)/2.
Assign values: p1 = 0.12, p2 = 0.125, Δ = 0.005, α = 0.05 (two-sided) so z_{1-α/2} = 1.96, β = 0.20 so z_{1-β} = 0.84.
Calculate p̄ = (0.12+0.125)/2 = 0.1225. Then compute the two variance terms: 2*p̄*(1-p̄) = 2*0.1225*0.8775 ≈ 0.21495, and p1*(1-p1)+p2*(1-p2) = 0.12*0.88 + 0.125*0.875 = 0.1056 + 0.109375 = 0.214975.
Compute numerator: (1.96*sqrt(0.21495) + 0.84*sqrt(0.214975))^2. sqrt(0.21495) ≈ 0.4636, sqrt(0.214975) ≈ 0.4636. So (1.96*0.4636 + 0.84*0.4636) = (0.9087 + 0.3894) = 1.2981. Square: 1.685. Divide by Δ^2 = 0.000025: n ≈ 67,400 per arm.
Round up to the next integer: 67,400 per arm. Mention that this is a large sample due to the small effect size relative to baseline.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The stat sig vs business significance part is where I spent most of my time.
Start by calculating the conversion rates for control and treatment, then compute the absolute and relative lift. Use the two-proportion z-test to derive the standard error, confidence interval, and p-value, and interpret both statistical and practical significance in the context of Stripe's business metrics.
Pro tip: Always relate the lift to a business metric like revenue or conversion value, and consider the cost of implementation—statistical significance alone doesn't justify a rollout.
Compute control rate = 18,000/150,000 = 12%, treatment rate = 18,900/150,000 = 12.6%. Absolute lift = 0.6 percentage points, relative lift = 5%.
Use pooled proportion for the z-test: p_pool = (18,000+18,900)/(300,000) = 0.123. Standard error = sqrt(p_pool*(1-p_pool)*(2/150,000)) ≈ 0.0012. Z = (0.126-0.12)/0.0012 ≈ 5.0.
For 95% CI, use unpooled SE: sqrt(0.12*0.88/150000 + 0.126*0.874/150000) ≈ 0.0012. CI = (0.006 - 1.96*0.0012, 0.006 + 1.96*0.0012) ≈ (0.0036, 0.0084) or (0.36%, 0.84%). P-value for Z=5.0 is < 0.0001 (two-tailed).
Since p-value < 0.05 and CI excludes zero, the result is statistically significant. However, check if the CI includes the minimum detectable effect (MDE) or business threshold.
Evaluate if a 5% relative lift (0.6 pp) is practically significant. Consider factors like revenue impact, implementation cost, and whether it aligns with strategic goals. For Stripe, even small lifts can be valuable at scale, but ensure it's not due to novelty or segment-specific effects.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the CUPED variance reduction formula: the adjusted variance is (1 - ρ²) times the original variance, where ρ² is the R-squared between the pre-experiment covariate and the outcome. Then, since MDE is proportional to the standard error, which is proportional to the square root of the variance, the revised MDE is the original MDE multiplied by sqrt(1 - ρ²). Plug in ρ² = 0.30 to get the numerical reduction.
Pro tip: Emphasize that CUPED reduces variance without biasing the treatment effect estimate, and that the 30% variance reduction translates to a 16.3% reduction in MDE (or equivalently, a ~30% reduction in required sample size for the same power). This shows you understand the practical impact on experiment efficiency.
Recall that CUPED adjusts the outcome by subtracting a multiple of the pre-experiment covariate, and the variance of the adjusted outcome is (1 - ρ²) times the original variance, where ρ² is the squared correlation between covariate and outcome.
Explain that MDE is proportional to the standard error, which is the square root of the variance divided by sample size. Therefore, MDE scales with the square root of the variance reduction factor.
Substitute ρ² = 0.30 into the formula: adjusted variance = 0.70 * original variance. Then compute the MDE multiplier: sqrt(0.70) ≈ 0.8367.
Conclude that the effective variance is reduced by 30%, and the revised MDE is about 83.67% of the original MDE, i.e., a 16.33% reduction. Optionally, mention that this corresponds to a 30% reduction in required sample size for the same power.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on the ordering direction.
First, sort the p-values in ascending order and assign ranks. Then, for each rank k (from 1 to m), compare the p-value to the Holm-Bonferroni threshold α/(m - k + 1). Stop at the first non-significant p-value; all subsequent hypotheses are also non-significant. Finally, state which guardrails remain significant.
Pro tip: When applying Holm-Bonferroni, remember that it is a step-down procedure: once you fail to reject a null hypothesis, you must stop and declare all remaining hypotheses non-significant, even if their p-values are below their individual thresholds. This controls the familywise error rate without being overly conservative.
List the four p-values in ascending order: 0.01, 0.03, 0.04, 0.20. Assign ranks 1 to 4 respectively.
For each rank k, compute the Holm-Bonferroni threshold α/(m - k + 1), where m = 4 and α = 0.05. Thresholds: rank 1: 0.05/4 = 0.0125; rank 2: 0.05/3 ≈ 0.0167; rank 3: 0.05/2 = 0.025; rank 4: 0.05/1 = 0.05.
Compare each p-value to its threshold in order. Rank 1: 0.01 < 0.0125 → significant. Rank 2: 0.03 > 0.0167 → not significant. Stop here; ranks 3 and 4 are automatically not significant.
Only the guardrail with p-value 0.01 remains significant after Holm-Bonferroni correction. The others (0.03, 0.04, 0.20) are not significant.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining how to detect overdispersion in clustered conversion data using statistical tests and diagnostic plots, then discuss methods to correct for it such as mixed-effects models or cluster-robust standard errors. Emphasize the importance of accounting for geographic clustering to ensure valid inference and reliable conversion rate estimates.
Pro tip: When dealing with clustered data, always consider using Bayesian hierarchical models to naturally account for overdispersion and provide more stable estimates, especially when some regions have small sample sizes.
Compute the dispersion statistic (e.g., Pearson chi-square divided by degrees of freedom) and inspect residuals. If the value is significantly greater than 1, overdispersion is present.
Use intraclass correlation coefficient (ICC) or design effect to quantify how much geographic clustering inflates variance. This helps determine the necessary correction.
Select an appropriate model: quasi-Poisson/binomial for simple adjustments, mixed-effects models with random intercepts for regions, or cluster-robust standard errors to account for within-cluster correlation.
Compare corrected estimates with uncorrected ones and check model diagnostics (e.g., residual plots, overdispersion tests) to ensure the correction adequately addresses the issue.
Discuss the trade-offs between model complexity, interpretability, and computational cost. Recommend the method that best balances accuracy and practicality for the business context.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.