I knew the formula well enough but fumbled the plugging-in part under pressure.
Start by clearly stating the formula for sample size per arm in a two-proportion z-test, then plug in the given values: baseline conversion rate (p1 = 0.04), minimum detectable effect (0.4 percentage points, so p2 = 0.044), significance level (α = 0.05, two-sided), and power (1-β = 0.80). Compute the required sample size using the standard formula and round up to the nearest integer.
Pro tip: Mention that this calculation assumes no continuity correction and equal allocation; in practice, you might adjust for expected attrition or use sequential testing, but for a standard A/B test this is the right starting point.
Identify p1 = 0.04, p2 = 0.044, α = 0.05 (two-sided), power = 0.80, and the corresponding z-values: z_{α/2} = 1.96 and z_β = 0.84.
Use the standard formula for sample size per arm: n = ( (z_{α/2} * sqrt(2 * p̄ * (1-p̄)) + z_β * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / (p2 - p1)^2, where p̄ = (p1 + p2)/2.
Calculate p̄ = 0.042, then compute the terms: sqrt(2 * 0.042 * 0.958) ≈ 0.283, sqrt(0.04*0.96 + 0.044*0.956) ≈ 0.289, and the difference (p2-p1) = 0.004.
Substitute into the formula: n = ( (1.96 * 0.283 + 0.84 * 0.289)^2 ) / (0.004)^2. Compute numerator: (0.5547 + 0.2428)^2 = (0.7975)^2 ≈ 0.636, then divide by 0.000016 to get n ≈ 39,750 per arm.
Round up to the next whole number: approximately 39,751 per arm. Mention that this is the required sample size to detect a 0.4 percentage point absolute lift with 80% power at 5% significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by computing the pooled and unpooled standard errors for the difference in proportions, then calculate the z-statistics and corresponding p-values. Next, construct the 95% confidence interval using the unpooled standard error. Finally, interpret the results in practical terms, emphasizing the small effect size and business implications.
Pro tip: At Amazon, always tie statistical significance to practical significance—mention that with large samples, tiny differences can be significant but may not justify action. Also, note that unpooled variance is preferred for confidence intervals, while pooled is for hypothesis testing under the null.
Calculate control conversion rate (p_c = 0.0430) and treatment conversion rate (p_t = 0.0422). The observed difference is p_t - p_c = -0.0008 (or 0.08 percentage points).
For pooled: p_pool = (x_c + x_t) / (n_c + n_t) = (2150 + 2089) / 99500 ≈ 0.0426. SE_pooled = sqrt(p_pool * (1-p_pool) * (1/n_c + 1/n_t)). For unpooled: SE_unpooled = sqrt(p_c*(1-p_c)/n_c + p_t*(1-p_t)/n_t).
For pooled: z = (p_t - p_c) / SE_pooled. For unpooled: z = (p_t - p_c) / SE_unpooled. Then find two-tailed p-values from standard normal distribution.
Use unpooled SE: CI = (p_t - p_c) ± 1.96 * SE_unpooled. This gives a range for the true difference in proportions.
Discuss whether the difference is statistically significant (p < 0.05) and the width of the CI. Emphasize that the effect size is tiny (0.08 pp) and likely not business-relevant, even if significant.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the primary metric as the key success measure tied to the experiment's hypothesis, and guardrail metrics as those that ensure no harm to user experience or business health. Then discuss multiple comparison correction, emphasizing that guardrails are typically monitored for directional harm rather than statistical significance, so corrections like Bonferroni or Benjamini-Hochberg may be applied but with caution to avoid over-penalizing. Finally, recommend a practical approach such as using a lower alpha for guardrails or focusing on effect sizes and confidence intervals.
Pro tip: At Amazon, guardrails are often treated as one-sided tests to detect harm, and multiple comparison corrections are applied selectively—over-correcting can mask real issues. Emphasize that guardrails are about risk management, not proving superiority.
Identify the single metric that directly measures the experiment's goal (e.g., conversion rate, click-through rate) and is used to decide success.
Select metrics that monitor potential negative side effects (e.g., cancellation rate, latency) and ensure they do not degrade significantly.
Decide whether to apply multiple comparison correction. For guardrails, consider one-sided tests and less stringent corrections to avoid missing harm.
If correction is needed, use Bonferroni for strict family-wise error control or Benjamini-Hochberg for false discovery rate control, balancing power and risk.
Explain the rationale behind the chosen correction and how it aligns with the experiment's risk tolerance and decision-making framework.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Said CUPED regresses out pre-experiment covariate variance from the outcome, which shrinks the residual variance and effectively lets you reach the same power with fewer users.
Start by explaining the core idea of using pre-experiment data to explain post-experiment variance, then detail how CUPED and stratification implement this via regression adjustment or blocking. Finally, connect the variance reduction to the sample size formula, emphasizing the squared relationship and practical implications for experiment speed and cost.
Pro tip: Quantify the impact: mention that a 50% variance reduction cuts required sample size in half, and always check that pre-experiment covariates are balanced and truly predictive to avoid bias.
Describe how high variance in the outcome metric increases required sample size and reduces sensitivity. Mention that pre-experiment differences or noise can obscure true treatment effects.
Define CUPED as using pre-experiment covariates (e.g., user's prior behavior) to adjust post-experiment outcomes via regression, and stratification as blocking on pre-experiment variables to create homogeneous groups.
For CUPED, explain that the adjusted estimator subtracts the portion of variance explained by the covariate, reducing variance by a factor of (1 - ρ²). For stratification, explain that within-stratum variance is lower, and the overall variance is a weighted average of stratum variances.
Show that required sample size is proportional to variance (n ∝ σ²/Δ²). Therefore, reducing variance by a factor k reduces sample size by the same factor, assuming power and effect size are fixed.
Highlight that CUPED can achieve large variance reductions (e.g., 50%) if covariates are predictive, leading to faster experiments. Mention that stratification requires careful implementation and may have limited gains if strata are not predictive.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Two-parter and I think I handled the first better than the second.
Start by explaining the statistical risks of peeking (inflated type-I error) and methods to control it, such as sequential testing or alpha spending. Then, address the segment-level paradox by discussing heterogeneity, multiple comparisons, and the need for pre-registered subgroups or post-hoc validation. Emphasize a structured, data-driven approach that balances statistical rigor with business impact.
Pro tip: Mention that at Amazon, decisions are often made with incomplete data, so it's crucial to quantify the risk of false positives and use techniques like sequential testing to allow early stopping without inflating error rates. Also, highlight that segment-level insights should be validated with holdout data or a follow-up experiment to avoid overfitting to noise.
Describe how repeatedly checking results during an experiment increases the chance of false positives (type-I error) because each look is a new hypothesis test. This can lead to premature and incorrect decisions.
Discuss methods like sequential testing (e.g., group sequential designs, alpha spending functions), Bonferroni correction, or Bayesian approaches to maintain the overall false positive rate at the desired level.
Acknowledge that this is Simpson's paradox: the overall effect can mask heterogeneous treatment effects. Investigate segment-level results to understand the drivers, but be cautious of multiple comparisons.
Use pre-registered subgroups or apply corrections for multiple testing (e.g., Benjamini-Hochberg). If segments are post-hoc, validate with a follow-up experiment or holdout data to confirm the effect.
Consider the practical implications: if segments show opposite effects, decide whether to target specific segments or iterate on the experiment. Communicate uncertainty and recommend next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.