I knew the Wald formula cold but fumbled the explanation of why it breaks down near the boundaries.
First, clarify the baseline conversion rate and sample size (100,000 users). Then, compute the Wald interval using the standard formula and the Wilson or Agresti-Coull interval using their respective formulas. Finally, explain why the latter two are preferable, focusing on coverage probability and behavior with extreme proportions.
Pro tip: Mention that in practice, Amazon often uses Bayesian methods for A/B testing, but understanding frequentist intervals like Wilson is crucial for interpreting results and communicating uncertainty.
Confirm the baseline conversion rate (p-hat) and sample size (n=100,000). Assume the conversion rate is estimated from the sample.
Use the formula: p-hat ± z * sqrt(p-hat*(1-p-hat)/n), where z=1.96 for 95% confidence.
For Wilson, use the formula: (p-hat + z^2/(2n) ± z * sqrt(p-hat*(1-p-hat)/n + z^2/(4n^2))) / (1 + z^2/n). For Agresti-Coull, add z^2/2 successes and failures, then compute Wald on adjusted values.
Highlight that Wald can produce intervals outside [0,1] and has poor coverage for extreme p or small n. Wilson and Agresti-Coull have better coverage and are bounded within [0,1].
Explain why these intervals matter for decision-making in experiments, especially when conversion rates are low or sample sizes are not large.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Went fine until I forgot to account for the fact that p1 is derived as 1.05 times p0, not an absolute difference.
Start by clearly stating all assumptions: baseline conversion rate, significance level, power, and the relative lift to detect. Then derive the sample size formula for comparing two proportions using the normal approximation, and plug in the numbers to compute the required sample size per variant. Finally, interpret the result and discuss practical considerations.
Pro tip: Always mention that the sample size depends on the baseline conversion rate; if not given, you can express the formula in terms of it or assume a conservative 50% baseline. Also, clarify that the formula assumes equal allocation and independence.
Explicitly list the baseline conversion rate (p), significance level (α=0.05 two-sided), power (1-β=0.80), and the relative lift (δ=5%). Also assume equal allocation and independence.
Set up null hypothesis H0: p1 = p2 and alternative H1: p1 ≠ p2. Use the normal approximation for the difference in proportions.
Using the formula for comparing two proportions: n = ( (z_{α/2} + z_β)^2 * (p1(1-p1) + p2(1-p2)) ) / (p1 - p2)^2, where p2 = p1 * (1 + δ).
Substitute z_{α/2}=1.96, z_β=0.84, p1 (if known), and p2 = p1*1.05. If p1 is not given, express n as a function of p1 or assume a conservative p1=0.5.
Report the required sample size per variant, round up to the nearest integer, and mention that this is an approximation. Discuss sensitivity to assumptions and practical implications.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the multiple comparisons problem and why correction is needed when testing three metrics. Then walk through the Bonferroni and Holm procedures step-by-step, calculating the adjusted thresholds for three tests, and contrast them with Benjamini-Hochberg's FDR control, highlighting the trade-offs between Type I error control and power.
Pro tip: Mention that in practice, especially at Amazon, you'd often pre-register a primary metric and treat others as secondary or guardrail metrics, applying corrections only to the secondary family. This shows you understand experimental design beyond just the math.
Explain that testing three metrics inflates the family-wise error rate (FWER) if no correction is applied. For three independent tests at α=0.05, FWER ≈ 1 - (0.95)^3 ≈ 14.3%, which is unacceptably high.
Divide the significance level by the number of tests: α_adj = α / m = 0.05 / 3 ≈ 0.0167. Each metric's p-value must be below this threshold to be considered significant. This controls FWER but is conservative, reducing power.
Order the three p-values from smallest to largest. Compare the smallest to α/3, the next to α/2, and the largest to α. Stop at the first non-significant result. This is uniformly more powerful than Bonferroni while still controlling FWER.
Order p-values ascending. Find the largest k such that p_(k) ≤ (k/m) * α. Reject all hypotheses up to that k. For m=3 and α=0.05, thresholds are 0.0167, 0.0333, and 0.05. BH controls the expected proportion of false discoveries, not FWER, offering more power.
Bonferroni/Holm control FWER (probability of any false positive), while BH controls FDR (expected proportion of false positives among rejections). For three metrics, BH is less conservative and may be preferable if some false positives are acceptable, but FWER control is stricter for confirmatory analyses.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Honestly the hardest part of the whole thing.
Explain that user-level clustering violates independence because multiple observations from the same user are correlated, leading to underestimated variance and inflated false positive rates. Then describe correction methods like cluster-robust standard errors or mixed-effects models, and note that power estimates decrease because the effective sample size is smaller than the number of observations.
Pro tip: Mention that ignoring clustering can lead to a false sense of precision; always check for clustering in your data and adjust your analysis accordingly. Also, consider using the intraclass correlation coefficient (ICC) to quantify the degree of clustering and adjust power calculations.
Clarify that user-level clustering means each user contributes multiple data points, so observations are not independent. This violates the independence assumption of standard statistical tests.
State that ignoring clustering leads to underestimated standard errors, inflated Type I error rates, and incorrect confidence intervals. This can result in false positives.
Discuss approaches such as cluster-robust standard errors (e.g., CRSE), mixed-effects models, or aggregating data to the user level. Mention that the choice depends on the analysis goals and data structure.
Explain that clustering reduces effective sample size, so power decreases. The design effect (1 + (m-1)*ICC) quantifies the variance inflation, where m is cluster size and ICC is intraclass correlation.
Suggest planning for clustering in the design phase by increasing sample size or using cluster-level randomization. Also, recommend sensitivity analyses to check robustness.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.